Service Interruption [ ALL Services ]
Postmortem
AUTODESK EVENT ANALYSIS Incident Number: INC96462
Incident Date: November 17, 2022
Summary
On November 17, 2022, Autodesk Identity Authorization service experienced a service disruption that may have impacted customers’ ability to sign in to Autodesk cloud products and use cloud-connected workflows from within our desktop products between 5:14 AM PST to 12:01 PM PST.
Impacted Services
- Autodesk cloud products and services, as well as desktop applications with cloud-based features were impacted.
- Customers experienced intermittent issues where they were unable to sign in or could not stay signed in to impacted products and services.
Root Cause
- As part of a planned upgrade for the Autodesk Identity Authorization service, we updated a third-party vendor database component and added a new replication target to an existing replication of the authorization service database cluster. Unfortunately, this led to an unexpected database contention and caused latency spikes for database queries.
- The database latencies resulted in sign in and authorization timeouts in impacted Autodesk products and services. The timeouts triggered the impacted products and services to execute “retry” behavior, resulting in a significant increase in traffic to the system, which caused a service disruption.
- To resolve this issue and support the increased load, we introduced multiple new server clusters and server traffic-handling. With the new, scaled infrastructure in place, we started restoring service at 8:17 AM PST. We restored service gradually, reaching 100% restoration at 12:01 PM PST.
Autodesk Actions
Autodesk conducted a post-incident analysis of the event and identified actions we plan to take to prevent a recurrence of this issue. Some of these actions include:
- Engaging with our third-party vendor on remediating the database latency issue.
- Introducing improved high-availability and disaster recovery infrastructure, volume scaling, and policies for the supporting sign-in and authorization services. These changes will enhance sign-in and authorization services’ overall resiliency profile, with higher confidence fail-over and recovery.
- Expanding our service monitoring and observability capabilities to improve our ability for early detection, as well as support faster triage and recovery.
- Improving application traffic routing for managing infrastructure and server load. This will improve our services’ overall scale and availability when an exponential increase in traffic occurs.
- Introducing new load and production traffic simulation practices that will further validate and strengthen our resiliency and recovery measures.
Autodesk recognizes our responsibility to ensure maximum reliability and redundancy of our products and services, and we remain committed to consistently delivering reliable and world- class experiences for our customers. We thank you for your patience and understanding as we work to resolve this issue.
Posted Nov 18, 2022 - 14:18 PST
Resolved
Current status: Operational. Service is now working as expected and all features are operating normally.
Posted Nov 18, 2022 - 13:07 PST
Update
An issue with Autodesk’s Identity Authorization Service this morning impacted customers’ ability to login to Autodesk Products & Services. The issue has now been resolved and all Autodesk Products & Services have returned to full service. We understand the frustration this caused customers and apologize. We are still investigating the root cause, and will be identifying actions to be taken to prevent the issue from taking place in the future. As we learn more details, we will continue to communicate those to impacted customers.
Posted Nov 17, 2022 - 16:11 PST
Update
An issue with Autodesk’s Identity Authorization Service this morning impacted customers’ ability to login to Autodesk Products & Services. The issue has now been resolved and all Autodesk Products & Services have returned to full service. We understand the frustration this caused customers and apologize. We are still investigating the root cause, and will be identifying actions to be taken to prevent the issue from taking place in the future. As we learn more details, we will continue to communicate those to impacted customers.
Posted Nov 17, 2022 - 16:07 PST
Update
Service has been restored. The team will continue to actively monitor the status over the next 24 hours.
Posted Nov 17, 2022 - 13:54 PST
Update
Service has been restored. The team will continue to actively monitor the status over the next 24 hours.
Posted Nov 17, 2022 - 13:24 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 13:01 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 12:47 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 12:46 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 12:43 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 12:38 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 12:32 PST
Monitoring
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 12:29 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 12:21 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 12:16 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 12:12 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 12:09 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 11:58 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 11:53 PST
Update
Recovery continues to our Identity Authorization Service. All services have recovered and are operational except for Cloud-based features for Desktop applications which are now in recovery mode. Updates will continue every 30 minutes until these services are restored.
Posted Nov 17, 2022 - 11:51 PST
Update
Recovery continues to our Identity Authorization Service, which has impacted users' ability to login to Autodesk Products & Services. The team continues to triage the issue until all services have recovered. Cloud-based features for Desktop applications are now in recovery mode. Updates will continue every 30 minutes until all services are restored.
Posted Nov 17, 2022 - 11:48 PST
Update
We are continuing to investigate this issue.
Posted Nov 17, 2022 - 11:37 PST
Update
Recovery continues to our Identity Authorization Service, which has impacted users' ability to login to Autodesk Products & Services. The team continues to triage the issue until all services have recovered. Presently cloud-based features for Desktop applications are significantly degraded. Updates will continue every 30 minutes until all services are restored.
Posted Nov 17, 2022 - 11:14 PST
Update
We are continuing to investigate this issue.
Posted Nov 17, 2022 - 11:12 PST
Update
We are continuing to investigate this issue.
Posted Nov 17, 2022 - 11:03 PST
Update
We are continuing to investigate this issue.
Posted Nov 17, 2022 - 11:01 PST
Update
We are continuing to investigate this issue.
Posted Nov 17, 2022 - 11:00 PST
Update
Recovery continues to our Identity Authorization Service which has impacted users ability to login to Autodesk Products & Services. The team continues to monitor the issue until all services have recovered. At this time only Desktop clients are still impacted by this incident. Updates will continue 30 minutes until all services are restored.
Posted Nov 17, 2022 - 10:53 PST
Update
We are continuing to investigate this issue.
Posted Nov 17, 2022 - 10:41 PST
Update
Recovery continues to our Identity Authorization Service which has impacted users ability to login to Autodesk Products & Services. The team continues to monitor the issue until all services have recovered. Desktop clients are still impacted by this incident. Updates will continue 30 minutes until all services are restored.
Posted Nov 17, 2022 - 10:19 PST
Update
We continue to see recovery to our Identity Authorization Service which has been impacting users ability to login to Autodesk Products & Services. The team continues to triage the issue until all services have recovered. Updates every 30 minutes until all services are restored.
Posted Nov 17, 2022 - 09:36 PST
Update
We are experiencing a degradation of our Identity Authorization Service at this time. This is impacting users ability to login to Autodesk Products & Services. We are seeing a gradual restoration of services at this time and continue to triage the issue until all services have recovered. We will continue to post updates every 30 minutes until all services are restored.
Posted Nov 17, 2022 - 09:08 PST
Update
We are still experiencing an issue with our Identity Authorization Service. This is impacting users ability to login to Autodesk Products & Services. The team continues to work on the restoration of service, however we do not have an ETA for resolution at this time. We will continue to post updates every 30 minutes until the service is restored.
Posted Nov 17, 2022 - 08:46 PST
Update
We are still experiencing an issue with our Identity Authorization Service. This is impacting users ability to login to Autodesk Products & Services. The team continues to work on the restoration of service, and will continue to post updates every 30 minutes until the service is restored.
Posted Nov 17, 2022 - 08:18 PST
Update
We are experiencing an issue with our Identity Authorization Service. This is impacting all users ability to login to Autodesk Products & Services. The team continues to troubleshoot and work on restoration of service, the next update will be posted in 30 minutes.
Posted Nov 17, 2022 - 07:56 PST
Update
We are experiencing an issue with our Identity Authorization Service. This is impacting users ability to login to Autodesk Products & Services. The team continues to work on restoration of service, and will post the next update in 30 minutes.
Posted Nov 17, 2022 - 07:14 PST
Investigating
We are experiencing an issue with our Identity Authorization Service. This is impacting users ability to login to Autodesk Products & Services. The team is working on the restoration of service, and will post an update every 30 minutes until service is restored.
Posted Nov 17, 2022 - 05:57 PST
This incident affected: A360, A360 Mobile, ACC Admin Console, ACC Admin Console (Europe), APS - Data Exchange, APS - Design Automation, APS - Developer Portal, AutoCAD Online Services, AutoCAD Web, Autodesk App Store, Autodesk BIM Collaborate, Autodesk BIM Collaborate (Europe), Autodesk Build, Autodesk Build (Europe), Autodesk Construction Cloud Data Connector, Autodesk Construction Cloud Data Connector (Europe), Autodesk Docs, Autodesk Docs (Europe), Autodesk Drive, Autodesk Takeoff, Autodesk Takeoff (Europe), Autodesk Tandem™, AVA, BIM 360 Account Administration, BIM 360 Account Administration (Europe), BIM 360 Cost Management, BIM 360 Cost Management (Europe), BIM 360 Design Collaboration, BIM 360 Design Collaboration (Europe), BIM 360 Docs, BIM 360 Docs (Europe), BIM 360 Field, BIM 360 Field (Europe), BIM 360 Field Management, BIM 360 Field Management (Europe), BIM 360 Glue, BIM 360 Glue (Europe), BIM 360 Insight, BIM 360 Insight (Europe), BIM 360 Model Coordination, BIM 360 Model Coordination (Europe), BIM 360 Ops, BIM 360 Plan, BIM 360 Project Home, BIM 360 Project Home (Europe), BIM 360 Project Management, BIM 360 Project Management (Europe), BIM 360 Reports, BIM 360 Reports (Europe), BIM 360 Team Mobile, BuildingConnected, CER v2 services, Character Generator, Civil 3D Online Services, Cloud Rendering, Collaboration for AutoCAD Plant 3D, Collaboration for AutoCAD Plant 3D (Europe), Digital Workplace, Dynamo Machine Learning, Dynamo Package Manager, FormIt, Fusion, Fusion HQ, Fusion Manage, Fusion Mobile, Fusion Online, Fusion Team / BIM 360 Team, Generative Design, InfraWorks, Instructables, Licensing & Entitlement, My Profile and Settings, Plant Collaboration Services (based on BIM 360 Team), ReCap Services, Revit Cloud Model Upgrade, Revit Cloud Worksharing / Cloud Models, Revit Cloud Worksharing / Cloud Models (Europe), Tinkercad, and Vault Gateway.