Capstone: wire the whole pipeline end to end

You have written every piece of this pipeline. The capstone is putting them together: a single push to main triggers CodeBuild, CodeBuild updates the Glue job, you start the run from the CLI, the transform produces curated Parquet, and pyarrow validates it. End to end. Reproducible. Boring.

The capstone loop

A single push runs the entire CI/CD chain: source upload, job refresh, manual run, validation, observability.

Validation checklist: Capstone checklist

Loading practice…

Checkpoint: Course-wide checkpoint

Loading practice…

You now have a Glue + PySpark pattern that ships through CodeBuild, runs locally for free, and lands curated Parquet you can validate. Next stop is the AWS lakehouse course, which keeps this Glue + PySpark base and layers Iceberg, Snowflake, Lambda, and Airflow on top.