Quay.io outage causing new pods to be stuck in Pending waiting to download container images

Incident Report for Astro

Resolved

This incident has been resolved.

Posted Nov 15, 2023 - 19:09 UTC

Monitoring

Quay.io has indicated that they have completed the fix and is operating correctly for pushes and pulls.

New Airflow services that come up are operational. We are continuing to monitor the situation as Quay.io has not marked their incident as resolved.

Posted Nov 15, 2023 - 01:41 UTC

Update

This issue is ongoing. We have observed some instances where images are able to be pulled, but we're continuing to observe widespread image pull issues.

We will update as more information becomes available.

Posted Nov 14, 2023 - 22:19 UTC

Update

Quay has indicated that they are continuing to experience instability and are moving their image repo to read-only mode, which will affect image push operations.

Posted Nov 14, 2023 - 21:01 UTC

Identified

Quay.io, the container image repository used by Astronomer is experiencing issues with image pull failures.

Quay.io incident: https://status.quay.io/incidents/z7sbjqmb34p1

We will continue monitoring the situation and update this incident as more information becomes available.

Existing pods should be unaffected and will continue executing tasks.

Posted Nov 14, 2023 - 20:53 UTC

This incident affected: Astro Hybrid (Cloud Image Repository) and Astro Hosted (Cloud Image Repository).