Iterating locally with main_local.py

The local runner is what makes this codebase iterable. It reads the bundled datasets/flights.csv, runs the same transform, and writes Parquet to ./out/. Total runtime is about thirty seconds for the full 336K-row file. You can iterate as fast as your editor saves.

main_local.py
python
parser = argparse.ArgumentParser(description="Run the flight ETL transform locally.")
parser.add_argument(
    "--input",
    default="datasets/flights.csv",
    help="Path to the input flights CSV (default: datasets/flights.csv)",
)
parser.add_argument(
    "--out",
    default="out",
    help="Output directory for the Parquet results (default: out/)",
)
parser.add_argument(
    "--limit",
    type=int,
    default=0,
    help="If > 0, sample this many rows after reading (useful for fast smoke runs).",
)
args = parser.parse_args()

Argparse handles the three knobs you reach for during dev: input path, output path, and a row limit so smoke tests run on a sample.

main_local.py
python
spark = (
    SparkSession.builder.appName("aws-glue-spark-etl-local")
    .master("local[2]")
    .config("spark.sql.shuffle.partitions", "4")
    .getOrCreate()
)
spark.sparkContext.setLogLevel("WARN")

A local-mode SparkSession with a small shuffle partition count. Two cores is plenty for the bundled file and keeps memory pressure low on a laptop.

main_local.py
python
carrier_df.write.mode("overwrite").parquet(str(carrier_out))
route_df.write.mode("overwrite").parquet(str(route_out))

carrier_count = carrier_df.count()
route_count = route_df.count()
print(f"carrier_analysis rows: {carrier_count} -> {carrier_out}")
print(f"route_analysis rows:   {route_count}   -> {route_out}")

spark.stop()

if carrier_count == 0 or route_count == 0:
    raise SystemExit("transform produced empty output")

Write Parquet, then count. A non-zero count assertion at the end is a cheap guard that the transform produced real output.

Default is 200 because Spark assumes a real cluster. On a laptop with two cores, 200 partitions adds task scheduling overhead that dwarfs the work itself. For the bundled dataset, 4 is a sweet spot. In Glue, leave the default (or tune for your DPU count).

Quiz: Quiz

Loading practice…

Code playground: Add a row-count assertion

Loading practice…