CloudWatch logs, metrics, and the audit trail

Glue ships three log streams: stdout, stderr, and structured logs-v2. With enable-continuous-cloudwatch-log set, those stream in near real time. The dashboard you actually want is simpler: did the job succeed, how many rows, and what did the runtime look like.

tail-logs.sh

bash

aws logs tail \
  --follow \
  --since 10m \
  /aws-glue/jobs/output \
  --filter-pattern "ERROR"

Tail the Glue log group while a job runs. The output stream is what you want during dev. The error stream is what you want when things break.

main.py (extension)

python

import boto3

cw = boto3.client("cloudwatch")
cw.put_metric_data(
    Namespace=PROJECT_NAME,
    MetricData=[
        {
            "MetricName": "carrier_analysis_rows",
            "Value": carrier_df.count(),
            "Unit": "Count",
        },
        {
            "MetricName": "route_analysis_rows",
            "Value": route_df.count(),
            "Unit": "Count",
        },
    ],
)

Emit a custom CloudWatch metric at the end of the job: row counts per output. Then build a CloudWatch alarm on the metric.

Yes. Glue passes JOB_RUN_ID as a job argument. Embed it in the output S3 path under a _metadata/ directory or include it in a separate Glue catalog column. That gives you a join key from output rows to the run that produced them.

Quiz: Quiz

Loading practice…