GlueContext, DynamicFrame, and what changes inside Glue

The Glue runtime ships a few wrappers around regular Spark concepts. GlueContext is a SparkContext with extras. DynamicFrame is a DataFrame with optional schema flexibility. The catalog is where Glue stores table metadata so jobs can read by name instead of by S3 path.

main.py
python
glue_context.write_dynamic_frame.from_options(
    frame=DynamicFrame.fromDF(carrier_df, glue_context, "carrier_analysis"),
    connection_type="s3",
    connection_options={
        "path": f"s3://{BUCKET_NAME}/{PROJECT_NAME}/curated/carrier_analysis/",
        "partitionKeys": ["year", "carrier"],
    },
    format="parquet",
)

The Glue write pattern: convert the DataFrame back to DynamicFrame, then call `write_dynamic_frame.from_options` with S3 path and partition keys. Glue handles the rest.

Why DynamicFrame? It can carry rows with mixed schemas, useful when reading dirty source data. For our case (clean cast types in the transform), DataFrame is enough. We convert to DynamicFrame only at write time, which is the cheapest part of the lifecycle.

You can. The catalog gives you table metadata, schema versioning, and a name your DAG can reference. Reading raw S3 paths works for one-off jobs. Reading through the catalog scales to a team where multiple jobs share the same source. The course uses the catalog because that is the production pattern.

Matching exercise: Match Glue concept to Spark equivalent

Loading practice…

Quiz: Quiz

Loading practice…