Worker Nodes Not Spinning Up in GCP Dataplane Clusters

Resolved·Partial outage

This incident has been resolved.

Thu, Apr 4, 2024, 04:52 AM

(2 years ago)

Affected components

Apr 4, 2024, 03:27 AM

04:52 AM

Scheduling and Running DAGs and Tasks

Updates

Resolved

This incident has been resolved.

Thu, Apr 4, 2024, 04:52 AM

Monitoring

We are continuing to monitor for any further issues.

Thu, Apr 4, 2024, 04:51 AM

Monitoring

A fix has been implemented and we are monitoring the results.

Thu, Apr 4, 2024, 03:56 AM(54 minutes earlier)

Identified

The issue has been identified and the fix is being implemented.

Thu, Apr 4, 2024, 03:32 AM(24 minutes earlier)

Investigating

Incident Description: Some worker nodes within several GCP dataplane clusters are failing to spin up as expected. This issue is causing delays in task execution and may lead to DAGs/tasks getting stuck in the queued state or failing.

Current Status: We have pinpointed the issue and confirmed its existence. Our engineering team is actively collaborating to resolve the problem.

Impact: Delays in task execution within affected clusters. There is a risk of DAGs/tasks getting stuck in the queued state or failing due to the inability to spin up worker nodes.

Resolution: Our engineering team is working diligently to implement a fix for this issue.

Communication: Regular updates will be provided to keep you informed of any developments.

We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve this issue promptly. Please stay tuned for further updates.

Thu, Apr 4, 2024, 03:27 AM