Failure handling and retries

Cloud Run failures fall into three buckets. Transient (network blip, retry). Build error (real dbt failure, page someone). Quota error (slot reservation hit, escalate). Each bucket gets a different on_failure response.

airflow/dags/pipeline-ecommerce-bq-dbt.py
python
trigger_dbt = SimpleHttpOperator(
    task_id='trigger_dbt_build',
    endpoint='/run_transformation',
    retries=2,
    retry_delay=timedelta(minutes=10),
    retry_exponential_backoff=True,
    on_failure_callback=alert_dbt_failure,
    sla=timedelta(hours=1),
)

Retry policy with exponential backoff. on_failure_callback differentiates between transient and persistent failures based on the response code.

airflow-callback-gcp.py (illustrative)
python
def alert_dbt_failure(context):
    ti = context['task_instance']
    last_response_code = context.get('last_response_code')
    if last_response_code in (500, 503):
        slack_send(f"dbt build failed at {ti.task_id}; investigating")
    else:
        pagerduty_send(f"dbt build hard-failed at {ti.task_id}; review the dbt run log")

on_failure_callback inspects the response. Transient (5xx from infra) gets a Slack note. Persistent (build failed) gets PagerDuty.

Twice the typical run time, plus retry budget. If a build takes 15 minutes and you allow 2 retries with 10-minute backoff, the SLA should be at least 60 minutes. Tighter than that produces noisy SLA misses on every retry.