CodeBuild ships the script on every push

Glue runs the script from S3, not from your laptop. Every change to the script means re-uploading. Doing that by hand is a recipe for stale jobs running yesterday's code. CodeBuild watches the repo, uploads on push, and refreshes the job descriptor.

Push to deploy

A commit to main triggers CodeBuild, which uploads the script to S3 and tells Glue to refresh.
buildspec.yaml
yaml
version: 0.2

env:
  variables:
    PROJECT_NAME: "etl-flight-data"
    BUCKET_NAME: "learnwithparam-aws-flight-etl-scripts"

phases:
  pre_build:
    commands:
      - echo Uploading the script to S3...
      - aws s3 cp --recursive . s3://$BUCKET_NAME/aws_glue/$PROJECT_NAME --exclude "src/*"

  build:
    commands:
      - export GLUE_JOB_NAME="glue-job-$PROJECT_NAME"
      - export GLUE_JOB_CONFIG="glue_job_config.json"
      - JOB_EXISTS=$(aws glue get-job --job-name $GLUE_JOB_NAME 2>/dev/null || echo "NOT_FOUND")
      - if [ "$JOB_EXISTS" = "NOT_FOUND" ]; then
          aws glue create-job --name $GLUE_JOB_NAME --cli-input-json file://$GLUE_JOB_CONFIG;
        else
          aws glue update-job --job-name $GLUE_JOB_NAME --job-update file://$GLUE_JOB_CONFIG;
        fi

Two phases: pre_build uploads the script to S3, build creates or updates the Glue job using the `glue_job_config.json` descriptor.

The conditional handling is what makes the pipeline idempotent. The first run creates the job. Every subsequent run updates it. No manual intervention. No drift between repo state and AWS state.

Two layers. The script in S3 is overwritten on each push, so revert the commit and let CodeBuild re-deploy the previous version. The Glue job descriptor is also overwritten, so the descriptor follows the same revert. Keep git revert as the rollback verb and never reach into the AWS Console to fix things.

Quiz: Quiz

Loading practice…