Reading CSV with an explicit schema

Spark will happily guess column types for you with inferSchema=True. The guess depends on which rows it samples, which means two runs on the same file can produce two different schemas. That is fine for an ad-hoc notebook. It is a bug factory in production.

main_local.py
python
df = spark.read.option("header", True).csv(str(in_path), inferSchema=True)
if args.limit > 0:
    df = df.limit(args.limit)
carrier_df, route_df = transform(df)

The local runner does inferSchema for the smoke run. In Glue we will replace this with an explicit cast pass right after the read.

A column type spec is just a list of (name, dtype) tuples. Storing it as Python data, not as a Spark schema, lets you reuse it for casts, validation, and documentation. The transform module owns one of these and applies it explicitly.

transform.py
python
def coerce_types(df: DataFrame) -> DataFrame:
    """Cast every column to the FLIGHT_COLUMN_TYPES spec, preserving column order."""
    for column, dtype in FLIGHT_COLUMN_TYPES:
        if column in df.columns:
            df = df.withColumn(column, col(column).cast(dtype))
    return df

The cast pass walks the spec and casts each column it finds. Missing columns are skipped silently, so a schema change upstream does not crash the job.

No. Spark is lazy. Each withColumn adds a node to the logical plan. The cast pass produces one big plan that the Catalyst optimizer fuses before any executor runs. The cost is at plan time, not at execution time.

Quiz: Quiz

Loading practice…

AI prompt: Try it: write a column type spec

Loading practice…