We are currently seeing a capacity shortage in the Google Cloud (GCP) us-central1 region. As a result, some Deployments running in this region may experience delays when scaling up worker nodes and pods.
Some tasks may remain in a queued state for longer than usual while waiting for compute capacity. In some cases, tasks that stay queued beyond their configured timeout may fail.
Recommended workaround: Configuring automated task retries (for example, setting retries and a retry_delay on your tasks or in your DAG's default_args) will help tasks recover automatically once capacity becomes available. https://www.astronomer.io/docs/learn/rerunning-dags#automatically-retry-tasks
We are working to address the issue and keep this page posted as we have more updates.
Investigating
We are currently seeing a capacity shortage in the Google Cloud (GCP) us-central1 region. As a result, some Deployments running in this region may experience delays when scaling up worker nodes and pods.
Some tasks may remain in a queued state for longer than usual while waiting for compute capacity. In some cases, tasks that stay queued beyond their configured timeout may fail.
Recommended workaround: Configuring automated task retries (for example, setting retries and a retry_delay on your tasks or in your DAG's default_args) will help tasks recover automatically once capacity becomes available. https://www.astronomer.io/docs/learn/rerunning-dags#automatically-retry-tasks
We are working to address the issue and keep this page posted as we have more updates.