IAM, S3 paths, and the first successful run

The most common reason a first Glue run fails is IAM. Two roles matter: the role Glue assumes to run your job, and the CodeBuild role that needs iam:PassRole to hand the Glue role over. Get those right and the rest is mechanical.

Two roles, one PassRole bridge

CodeBuild assumes its own role, then PassRoles the Glue role into the create-job call. Glue assumes its role at runtime to read S3 and write S3.
iam/GlueETLRole-policy.json (excerpt)
json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::learnwithparam-aws-flight-etl/*",
        "arn:aws:s3:::learnwithparam-aws-flight-etl"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::learnwithparam-aws-flight-etl/lwp-aws-flight-etl/curated/*"
    },
    {
      "Effect": "Allow",
      "Action": ["glue:GetTable", "glue:GetDatabase", "glue:CreateTable", "glue:UpdateTable"],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:*:*:log-group:/aws-glue/jobs/*"
    }
  ]
}

Minimum policy for the Glue role: read raw, write curated, talk to the catalog, and CloudWatch. Avoid `s3:*` on a wildcard bucket. Scope to your specific buckets.

first-run.sh
bash
aws glue start-job-run \
  --job-name "glue-job-lwp-aws-flight-etl" \
  --arguments '--JOB_NAME="lwp-aws-flight-etl"'

Trigger the first job run from the AWS CLI. The job name matches what CodeBuild created.

CloudWatch log group /aws-glue/jobs/output for stdout, /aws-glue/jobs/error for stderr, and /aws-glue/jobs/logs-v2 for the structured runtime logs. Most first-run failures are AccessDenied messages from S3 or the catalog. Read those literally and check the IAM policy.

Quiz: Quiz

Loading practice…