On-failure handling and retries
Three failure tiers. Transient (retry with backoff). Persistent (page on-call). Cosmetic (log and continue). Airflow lets you set per-task retries, retry_delay, and on_failure_callback. The right policy depends on which tier the failure is in.
GlueJobOperator(
task_id='transform_weather_data_glue',
job_name='transform-weather-data',
aws_conn_id='aws_conn',
retries=3,
retry_delay=timedelta(minutes=5),
retry_exponential_backoff=True,
on_failure_callback=page_on_call,
sla=timedelta(hours=2),
)Per-task retry policy. retry_exponential_backoff doubles the delay each attempt. on_failure_callback fires after retries exhaust.
airflow-callback.py (illustrative)
python
def page_on_call(context):
task_instance = context['task_instance']
msg = (
f"DAG {context['dag'].dag_id} task {task_instance.task_id} "
f"failed after retries. Run id: {context['run_id']}"
)
pagerduty_send(msg)
slack_send(msg)on_failure_callback gets the task context. Use it to page on-call, post to Slack, or open a ticket. Keep it idempotent.
Three is a good default. Two does not cover transient AWS hiccups. Five hides real bugs by burning hours of retries on a permanent failure. Add exponential backoff so the retries spread out: 5 min, 10 min, 20 min.