Airflow DAGs as contracts

A DAG is a directed acyclic graph of tasks. In Airflow, each task is an operator that runs a unit of work. Dependencies encode "this must succeed before that starts". The scheduler keeps state about every run of every task, which is why you can tell what happened three weeks ago without reading logs.

A batch pipeline DAG

Extract, validate, transform, publish. Each is a task with its own retry and SLA.

Four stages, explicit dependencies, room for targeted retries.
airflow/dags/batch_ingestion_dag.py
python
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator

default_args = {
    'owner': 'data-eng',
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
    'sla': timedelta(minutes=30),
}

with DAG(
    dag_id='batch_ingestion_dag',
    start_date=datetime(2025, 1, 1),
    schedule='@hourly',
    catchup=False,
    default_args=default_args,
) as dag:
    extract = PythonOperator(task_id='extract_mysql', python_callable=run_extract)
    validate = PythonOperator(task_id='validate_raw', python_callable=run_validate)
    transform = PythonOperator(task_id='spark_transform', python_callable=submit_spark_job)
    publish = PythonOperator(task_id='publish_marts', python_callable=refresh_marts)

    extract >> validate >> transform >> publish

A minimal DAG with Python tasks. The decorators register tasks with the scheduler. The chain call makes the dependency graph explicit. Retries and SLA live on each task so a flaky upstream does not wake the whole team.

Airflow uses a metadata database, typically Postgres, that stores every task instance for every DAG run. That is how the UI shows you a grid of green and red squares across weeks. It also means a lost scheduler is not a lost history. You can stop and start Airflow and the next run picks up where it left off.

Matching exercise: Match the Airflow concept to its responsibility

Loading practice…

Quiz: Quiz

Loading practice…