Glue Data Catalog provisioning

The Glue Data Catalog is just an AWS resource Terraform can provision. Create the database, register the job, and the catalog appears in the AWS Console exactly as planned. No ClickOps. No drift.

terraform/glue.tf
hcl
resource "aws_glue_catalog_database" "warehouse" {
  name = "warehouse_${var.pipeline_name}"
}

resource "aws_glue_job" "transform" {
  name     = "transform-${var.pipeline_name}"
  role_arn = aws_iam_role.glue_etl.arn
  command {
    script_location = "s3://${var.scripts_bucket}/${var.pipeline_name}/glue/transform-weather-data/main.py"
    name            = "glueetl"
    python_version  = "3"
  }
  glue_version = "4.0"
  worker_type  = "G.1X"
  number_of_workers = 2
  default_arguments = {
    "--enable-continuous-cloudwatch-log" = "true"
    "--job-bookmark-option"              = "job-bookmark-enable"
  }
}

The Glue database holds Iceberg metadata. The job resource registers the script in S3 and lets CodeBuild update it on every push.

G.1X with 2 workers is the smallest meaningful production size. It runs the bundled dataset cheaply. Production workloads scale by bumping number_of_workers. Worker types map to DPU sizes: G.1X (1 DPU), G.2X (2 DPU), G.4X (4 DPU).

No. Tables are managed by the Glue job that writes them. Terraform owns infrastructure (catalogs, IAM, jobs). The Glue job owns table DDL. Mixing the two creates drift between Terraform-state tables and runtime-state tables.