Local Airflow stack with docker-compose

You should never write a DAG and ship it straight to a managed Airflow. Run it locally first. The companion repo ships a docker-compose stack with Postgres, Redis, the scheduler, and a webserver. Same DAG. Same operators. Real Airflow runtime, no AWS calls.

The compose file uses CeleryExecutor with one worker. Postgres is the metadata store. Redis is the broker. Webserver lives at localhost:8080. The whole stack starts in about 60 seconds and shuts down cleanly with docker-compose down.

Local rehearsal pattern: stub AWS connections (Conn id aws_conn pointing at a moto server or a dry-run client), trigger the DAG by hand, watch the task graph, fix what breaks, repeat. Once the DAG is green locally, the cloud version is mostly mechanical.

The DAG file does not change. Connection IDs change. Network policies change. Auth changes. Operator versions are pinned by the managed service. The DAG you wrote and tested locally ports unchanged; only the surrounding config differs.