On Tuesday, October 13 at 17:38 UTC we began a cohort rollout of an update to a subset of Astro Data Planes. The update had already been successfully deployed to previous cohorts without any issue. At 17:55 UTC, we received automated alerts indicating an increase in errors within the Astro API. In response, we immediately began our incident management process. Within 2 minutes we had established that this was resulting in a full outage of our control plane.

At 18:03, we established that this was due to connection pool saturation in one of the essential databases that powers Astro's control plane. At 18:07, we took several steps that we believed might alleviate pressure on the database, including enabling a secondary connection pool for backend services, which we can now confidently state is the step that resolved the issue. The Astro Control Plane then recovered to its previous, healthy state. We continued to monitor the situation to verify that no further steps were required, and none were ultimately needed.

During the incident, we stated that there was no impact on tasks running. In our postmortem analysis, we determined that this was not completely true. We determined that there is a bug in the Airflow API server's Astro authorization plugin. In Astro, the Airflow API server reaches out to the Astro API to handle a subset of authorization requests. During this incident, the Astro API was responding slowly to these requests, rather than failing promptly. The bug we found led to the Airflow API server waiting on authorization requests for long enough that task heartbeats were not processed promptly. This led to tasks being interrupted and marked as "task heartbeat timeout" by Airflow, which erroneously concluded that they had already failed. We have developed a mitigation for this by ensuring that long authorization requests do not block task heartbeats, and this will be rolled out to all customers within the next 10 days.