We are declaring an end to this incident as the impact has been determined to be quite narrow.
This issue will only impact a deployment that at one point was on an Astro Runtime version below 12.3.0, then upgraded to 12.3.0 - 12.7.1, upgraded again to a version between 12.3.0 - 12.7.1, then used the downgraded back to the initial 12.3.0 - 12.7.1 version. For example, if a Deployment were to go from 11.0.0 -> 12.3.0 -> 12.7.1 and attempt to rollback to 11.0.0, the rollback would be performed as expected, but if it was attempted to rollback to 12.3.0, it would fail.
We are working on an update to the Astro Data Plane to fix this issue which we expect to release this month. Our support team will assist if any deployments do encounter this bug in the meantime.
Posted Mar 04, 2025 - 19:50 UTC
Monitoring
The issue has been verified as a missed minor database update that does not impact deployment behaviour unless a deployment rollback is applied. A deployment is affected if it was upgraded to a Runtime version greater than 12.3.0 from a Runtime version lesser than 12.3.0.
For example:
- Runtime 12.2.0 upgraded to Runtime 12.7.0 (Affected) - Runtime 12.0.0 upgraded to Runtime 12.3.0 (Affected) - Runtime 12.4.0 upgraded to Runtime 12.7.0 (Unaffected)
The underlying issue has been fixed and any upgrades going forward is unaffected. We are currently applying a permanent fix for affected deployments.
Posted Feb 28, 2025 - 17:40 UTC
Investigating
Astronomer has identified an issue with the upgrade to Runtime 12.3.0+ where a database migration is not applied. While the upgrade will complete without errors, the deployment will become unhealthy when attempting to downgrade from Runtime 12.3.0+.
We do not recommend upgrading to Runtime 12.3.0+ at this time as there may be other incompatible interactions between the database.
Posted Feb 28, 2025 - 16:21 UTC
This incident affected: Astro Hosted (Scheduling and Running DAGs and Tasks) and Astro Hybrid (Scheduling and Running DAGs and Tasks).