Sensor patterns and waiting on AWS

Sensors block until a condition is met. S3KeySensor waits for a file. GlueJobSensor waits for a job to finish. Each sensor takes mode (poke or reschedule) and timeout. Reschedule mode releases the worker between checks; poke holds it. Use reschedule for long waits.

airflow-sensor.py (illustrative)

python

wait_for_glue = GlueJobSensor(
    task_id='wait_for_glue',
    job_name='transform-weather-data',
    run_id="{{ ti.xcom_pull(task_ids='transform_actual_weather_data') }}",
    aws_conn_id='aws_conn',
    mode='reschedule',
    poke_interval=30,
    timeout=3600,
)

Wait for the Glue job, then for Snowflake to reflect the new snapshot. Both use reschedule mode so they do not pin a worker for an hour.

Poke mode is for short waits (under a minute) where holding the worker is cheaper than reschedule overhead. Reschedule mode is for waits longer than poke_interval where releasing the worker matters. Default to reschedule unless you measured otherwise.

Quiz: Quiz

Loading practice…