Splitting the transform from the entry point

The single biggest improvement to a Glue codebase is moving the transform out of the entry point. Once transform.py has zero awsglue imports, you can run it anywhere PySpark is installed. Glue becomes one of many places the transform lives, not the only place.

Two entry points, one transform

Both runtimes import the same `transform.py`. Only the surrounding plumbing differs.
main.py
python
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame

from transform import transform

BUCKET_NAME = os.environ.get("DATALAKE_BUCKET", "learnwithparam-aws-flight-etl")
PROJECT_NAME = os.environ.get("PIPELINE_NAME", "lwp-aws-flight-etl")
GLUE_DATABASE = os.environ.get("GLUE_DATABASE", "learnwithparam_glue_db")
RAW_TABLE = os.environ.get("GLUE_RAW_TABLE", "raw-flight-data")


def main() -> None:
    spark = SparkSession.builder.appName(PROJECT_NAME).getOrCreate()
    glue_context = GlueContext(spark)
    raw_frame = glue_context.create_dynamic_frame.from_catalog(
        database=GLUE_DATABASE,
        table_name=RAW_TABLE,
    )
    df = raw_frame.toDF()
    carrier_df, route_df = transform(df)

The Glue entry point is intentionally thin: read from the Glue catalog, call `transform`, write Parquet to S3 with partitioning. Every line is plumbing.

Notice every external dependency comes through os.environ. Bucket names, project names, catalog database, raw table. Glue passes these as job arguments and the entry point reads them defensively. Hardcoding any of these into the transform would tie it to one environment.

Pass it as a function argument to transform. The transform stays pure (same input, same output) and the caller is the one that knows the runtime context. If you find yourself reaching for os.environ inside the transform, that is the signal you have not separated concerns yet.

Quiz: Quiz

Loading practice…

AI prompt: Try it: refactor a tangled Glue script

Loading practice…