Modifying environment variables from the Astro UI may delete the values for other environment variables marked as "Secret"
Incident Report for Astro
Postmortem

Astronomer is undergoing a migration from a legacy, internal, API to the recently released Astro API, which is more resilient and performant. The two APIs handle falsy values differently, and this introduced a bug in Astro API’s updating of environment variables, resulting in the values for environment variables marked “Secret” being functionally deleted when environment variables were modified. When environment variables on Astro are saved, they are saved together, so updating any environment variable resulted in the deletion of values for all environment variables marked “Secret”.Beyond implementing the fix to resolve this issue, we are implementing the following remediations to prevent something similar from happening again:

  • Review functional testing for secret environment variables
  • Implement feature flags to enable staged rollouts for future API migration tasks
Posted Dec 14, 2023 - 16:50 UTC

Resolved
This incident has been resolved.
Posted Dec 07, 2023 - 01:52 UTC
Investigating
We are currently investigating this issue.

If you rely on setting environment variables in the Astro UI, please refrain from updating environment variables at this time.
Posted Dec 06, 2023 - 19:30 UTC
This incident affected: Astro Hybrid (Deployment Management, Cloud UI) and Astro Hosted (Deployment Management, Cloud UI).