AUTODESK EVENT ANALYSIS
Incident Number: #COE-INC102826
Incident Date: August 3, 2023
Summary
On August 3, 2023, Autodesk experienced a service disruption that may have impacted customers' ability to use Autodesk cloud products and to use cloud-connected workflows from within our desktop products on August 3, 2023 between 4:29 AM PDT and 5:13 AM PDT, and again on August 3, 2023 between 8:38 AM PDT and 12:01 PM PDT.
Impacted Services
- Autodesk cloud products and services, as well as desktop applications with cloud-based features were impacted (details in the following bullets).
- All cloud document management workflows were impacted.
- Desktop products' cloud management workflows were unavailable.
- Customers using impacted products and services experienced intermittent access or performance issues with impacted products and services.
- Customers using the Autodesk Construction Cloud mobile (iOS and Android) apps retained offline access to files and sheets downloaded prior to the incident.
- Fusion 360 customers experienced issues at the beginning of the service disruption at which time Fusion 360 was put into offline mode. On August 3, 2023 at 12:01 PM PDT, the Fusion 360 service was returned to an online state and fully operational.
- Note BIM 360 and Autodesk Construction Cloud European Data Center customers were not impacted by this service disruption.
Root Cause
- A database component detected an increased time for transactions to complete and attempted to mitigate the issue by restarting the database engine. The automated process to restart the engine failed on initial and subsequent attempts.
- The secondary automated process to automatically fail-over if the restart of the engine experienced a failure did not execute properly.
- Timeout parameters were adjusted, and the restart of the database engine was able to complete successfully. While this resolved the first occurrence of the issue and enabled all services to resume normal operations, the issue recurred later in the day.
- Our third-party vendor made additional updates to the database components which mitigated the second occurrence of the issue and all services resumed normal operation.
Autodesk Actions
Autodesk has completed a post-incident analysis of the event and identified actions to be taken. These include the following:
- Investigating the cause of the issue and why implemented fail-over and recovery processes did not execute properly.
- Investigating how the database engine is handling a parallel query setting to determine if that is contributing to the database restarts. Testing is underway to determine if disabling the setting will mitigate future occurrences.
- Expanding our service monitoring and observability to improve detection, as well assupport faster triage and recovery.
- Other improvements to our resiliency and recovery measures.
- Introducing process improvements to service availability and recovery verification.
Autodesk recognizes our responsibility to provide maximum reliability and redundancy of our products and services, and we remain committed to consistently delivering reliable and world-class experiences for our customers. Thank you for your patience and understanding.