Capstone: wire the whole pipeline end to end
You have written every piece of this pipeline. The capstone is putting them together: a single push to main triggers CodeBuild, CodeBuild updates the Glue job, you start the run from the CLI, the transform produces curated Parquet, and pyarrow validates it. End to end. Reproducible. Boring.
The capstone loop
Validation checklist: Capstone checklist
Loading practice…
Checkpoint: Course-wide checkpoint
Loading practice…
You now have a Glue + PySpark pattern that ships through CodeBuild, runs locally for free, and lands curated Parquet you can validate. Next stop is the AWS lakehouse course, which keeps this Glue + PySpark base and layers Iceberg, Snowflake, Lambda, and Airflow on top.