The bundled flights dataset and what it stands in for

The course ships with a real flight dataset: 336K rows of departures from New York airports in 2013. It is small enough to run on a laptop, big enough to expose every bug an inferred schema would hide, and dirty enough that you will write null handling, type coercion, and derived columns before the curated layer is queryable.

main_local.py
python
df = spark.read.option("header", True).csv(str(in_path), inferSchema=True)
if args.limit > 0:
    df = df.limit(args.limit)
carrier_df, route_df = transform(df)

The local runner reads the CSV with header inference. Notice we are not specifying the schema yet. The explicit-schema work comes right after, and you will see why autodetect is a trap.

Bundling the dataset with the workshop matters. You can run the full transform on a laptop in about 30 seconds and iterate without paying Glue. The same data shape exists in production datasets at most companies, so the patterns transfer directly.

transform.py
python
FLIGHT_COLUMN_TYPES: list[tuple[str, str]] = [
    ("id", "int"),
    ("year", "int"),
    ("month", "int"),
    ("day", "int"),
    ("dep_time", "double"),
    ("sched_dep_time", "int"),
    ("dep_delay", "double"),
    ("arr_time", "double"),
    ("sched_arr_time", "int"),
    ("arr_delay", "double"),
    ("carrier", "string"),
    ("flight", "int"),
    ("tailnum", "string"),
    ("origin", "string"),
    ("dest", "string"),
    ("air_time", "double"),
    ("distance", "int"),
    ("hour", "int"),
    ("minute", "int"),
    ("time_hour", "timestamp"),
    ("name", "string"),
]

The column type spec we will enforce once the PySpark transform comes online. Notice timestamp is the only non-primitive, and `carrier` is a string we will partition on later.

Quiz: Quiz

Loading practice…

AI prompt: Try it: profile a dataset you know

Loading practice…