AUTODESK EVENT ANALYSIS
Incident Number: #COE-INC100866
Incident Date: May 9, 2023
Summary
On May 9, 2023 between 6:03 AM PDT and 8:46 AM PDT, Autodesk experienced a service disruption that impacted North American Data Center customers using BIM 360 HQ, Autodesk Build, Autodesk Docs, BIM 360 Docs, BIM 360 Field Management, BIM 360 Insight, BIM 360 Design Collaboration, BIM 360 Model Coordination, Autodesk BIM Collaborate, Autodesk Takeoff, ACC AutoSpecs, and ACC Admin Console.
Impacted Services
- BIM 360 and Autodesk Construction Cloud (ACC) customers experienced degraded performance and the inability to view features within either product.
- Some BIM 360 and Autodesk Construction Cloud (ACC) customers were unable to login.
Root Cause
- A recent code change introduced a caching update that inadvertently caused the clearing of cache for a wider range of transactions than initially required, which resulted in a decrease in service performance. However, upon the routine collection of our app usage data, which typically causes spikes in memory and CPU usage, the combined volume of traffic was greater than our caching engine could handle. This resulted in degraded service and performance for BIM 360 and Autodesk Construction Cloud (ACC) customers.
Autodesk Actions
Autodesk has completed a post-incident analysis of the event and identified actions to be taken. These include the following:
- We have rolled back the recent code change which contained the code related to this performance regression.
- A new code change will be rolled out to only target the transactions requiring cache clearing updates.
- We are working to implement improvements to the monitoring and alerting for high memory and CPU usage to proactively identify and mitigate performance issues in the future.
- We are exploring new ways to collect routine app usage data without creating spikes in memory and CPU usage that could negatively affect service and performance for customers.
Thank you for your patience and understanding.