Sync vs async triggers

A sync trigger holds the Airflow worker until the dbt build finishes. Simple. Reliable. Limited to one hour by Cloud Run timeout. An async trigger returns immediately, then polls for completion. More moving parts. Scales to long-running builds.

Sync wins when build is under 30 minutes. Async wins when build is 30 minutes plus, or when many builds run in parallel and you want to free Airflow workers. Sync is also easier to reason about during incident response.

airflow-async-pattern.py (illustrative)

python

fire_build = SimpleHttpOperator(
    task_id='fire_dbt_build',
    endpoint='/run_transformation_async',
    method='POST',
)

wait_build = HttpSensor(
    task_id='wait_dbt_build',
    endpoint='/build_status/{{ ti.xcom_pull(task_ids="fire_dbt_build")["job_id"] }}',
    poke_interval=60,
    timeout=7200,
    response_check=lambda r: r.json().get('status') in ('success', 'failed'),
)

Async pattern: fire the build, get a job_id, poll for completion in a sensor. The dbt runner endpoint returns the job_id immediately and processes the build in a background thread.

Sync. The Olist dataset builds in under a minute. Sync is simpler, debuggable, and fits 80% of teams. The course covers async so you have the pattern ready when build times grow past Cloud Run timeout.

Quiz: Quiz

Loading practice…