DAG structure for the lakehouse

A lakehouse DAG has a recognizable shape. Refresh the dimension tables once. Generate the batch list. Fan out to many Lambda invokes. Wait for all extracts to finish. Run the Glue transform. Refresh Snowflake. Each step is a real task with retries and SLAs.

Daily lakehouse DAG

Linear flow with one fan-out for Lambda extracts. Glue runs sequentially after extracts; Snowflake refresh is the last step.

airflow/dags/pipeline-weather-data.py

python

with DAG(
    dag_id='pipeline-wheater-data',
    default_args=default_args,
    description='DAG to run AWS services for the ETL weather Data',
    schedule_interval=None,
    catchup=False,
    concurrency=2
) as dag:
    create_or_update_dim_locations_table_task = PythonOperator(
            task_id='create_or_update_dim_locations_table',
            python_callable=create_or_update_dim_locations_table,
            op_kwargs={'aws_conn_id': 'aws_conn', 'batch_size':  8000},
        )

The DAG instantiates with reasonable defaults: no past dependencies, zero retries (we test failures explicitly), and a concurrency cap.

During development, manual triggering keeps you in control. Once the DAG is stable, swap in a cron expression like @daily. The reason for the explicit None is that an unfinished DAG with a real schedule can fire at midnight and surprise everyone.