Creating Iceberg tables in Glue

Iceberg tables are created with SQL inside the Spark session. Three knobs matter on day one: format version, partition spec, and where the catalog lives. Get them right and Snowflake reads work the first time.

glue/transform-weather-data/main.py

python

query = f"""
    CREATE TABLE IF NOT EXISTS {full_path_table_to_create} (
        {', '.join([f'{field.name} {field.dataType.simpleString()}' for field in schema])}
    )
    USING iceberg
    PARTITIONED BY ({', '.join(partition_keys)})
    TBLPROPERTIES ("format-version"="2")
"""
spark.sql(query)

The job builds CREATE TABLE IF NOT EXISTS for each Iceberg table dynamically from a JSON schema. Notice format-version=2, USING iceberg, and the explicit PARTITIONED BY clause.

format-version=2 enables row-level deletes, equality deletes, and the merge-on-read pattern. Format-version=1 is read-only most engines; v2 is what modern engines (Snowflake, Athena, Spark) all support. Pick v2 unless an engine you depend on cannot read it.

Partition by what queries filter on, plus what writes naturally cluster. Time-series data partitions on year and month almost always. High-cardinality dimensions (user_id, event_id) partition badly and create thousands of tiny files. Start coarse; you can transform partition spec later.

Matching exercise: Match the Iceberg table property

Loading practice…