From manual to scheduled

A Lambda you invoke by hand is a script. A Lambda triggered on a schedule with idempotent payloads is a pipeline component. The course pairs the extractor with Airflow so the orchestrator owns scheduling and retries.

When the location set grows, batch it. The DAG splits the work into chunks sized to fit comfortably in one Lambda invocation, calls the Lambda once per batch, and lets Lambda concurrency limits absorb spikes. Smaller batches give faster recovery; bigger ones save Lambda invocations.

airflow/dags/pipeline-weather-data.py
python
s3_object_key = f"{project_pipeline_name}/staging/list_locations_dates_to_process/list_locations_dates_to_process_{key}_data_batch_{i}.json"
s3_client.put_object(
    Body=batch.encode('utf-8'),
    Bucket=s3_bucket_name,
    Key=s3_object_key
)

The DAG persists batch metadata to S3 with a deterministic key. If the run repeats, the same key gets overwritten cleanly. No duplicate-detection logic needed downstream.

EventBridge fires the Lambda. Airflow handles dependencies between Lambda, Glue, and Snowflake. A multi-step pipeline needs an orchestrator that can wait, retry, and branch on outcomes. EventBridge handles single triggers; Airflow handles workflows.

Quiz: Quiz

Loading practice…

Checkpoint: Checkpoint: the extract layer is real

Loading practice…