The Glue job descriptor

Click-ops in the Glue Console works for a single demo. The moment you have a teammate, a second job, or a need to reproduce yesterday's setup, every config has to live in version control. The Glue job descriptor is a JSON file you commit and apply mechanically.

glue_job_config.json
json
{
    "Role": "arn:aws:iam::864826018661:role/GlueETLRole",
    "Command": {
        "Name": "glueetl",
        "ScriptLocation": "s3://learnwithparam-aws-flight-etl-scripts/aws_glue/lwp-aws-flight-etl/main.py"
    },
    "DefaultArguments": {
        "--TempDir": "s3://learnwithparam-aws-flight-etl-scripts/aws_glue/temp/",
        "--job-bookmark-option": "job-bookmark-enable",
        "--job-language": "python",
        "--enable-metrics": "",
        "--enable-continuous-cloudwatch-log": "true"
    },
    "MaxRetries": 0,
    "Timeout": 2880,
    "WorkerType": "Standard",
    "NumberOfWorkers": 1,
    "GlueVersion": "4.0"
}

Every Glue job has a Role, a Command (script location and language), DefaultArguments, retry/timeout policy, and worker config. The CLI consumes this exact shape.

Two arguments deserve attention. job-bookmark-enable makes Glue track which input rows it has already processed, so reruns skip them. That is what makes a Glue job idempotent. enable-continuous-cloudwatch-log ships logs to CloudWatch as the job runs, so you can watch failures in real time instead of after the run.

WorkerType + NumberOfWorkers control DPU allocation. Standard with 1 worker is the cheapest config and fits the bundled dataset. Scale up to G.1X or G.2X for production-sized data. Always leave the maxConcurrentRuns at 1 unless you know your job is concurrency-safe.

A failed Glue job that auto-retries hides bugs. You want the failure to surface, the team to look, and the fix to be deliberate. Set retries to 0 during development and let your orchestrator (Airflow, Step Functions) handle retries with proper backoff.

Quiz: Quiz

Loading practice…

AI prompt: Try it: write a Glue job config from scratch

Loading practice…