Configuring SparkSession for Iceberg

The Iceberg-on-Glue config is the single most failure-prone line in this codebase. Get the catalog name and impl right and writes commit cleanly. Get them wrong and you get cryptic "table not found in catalog" errors with no clear cause.

glue/transform-weather-data/main.py
python
spark = SparkSession.builder \
    .appName("pipeline-weather-data") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.glue_catalog", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.glue_catalog.warehouse", f"s3://{bucket_name}/{project_name}/warehouse") \
    .config("spark.sql.catalog.glue_catalog.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \
    .config("spark.sql.catalog.glue_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \
    .getOrCreate()

Five SparkSession config calls turn vanilla Spark into an Iceberg-on-Glue writer. The catalog name "glue_catalog" is what your DDL references in `glue_catalog.<db>.<table>` paths.

extensions adds the Iceberg SQL grammar (USING iceberg). spark.sql.catalog.glue_catalog declares a catalog by that name. The warehouse points at the S3 prefix. catalog-impl tells Iceberg to register tables in the Glue Data Catalog. io-impl uses the AWS S3 file IO instead of the default Hadoop one.

You can name it anything. The string here is just the namespace for table references in SQL. "glue_catalog.warehouse_weather_data.weather_actual" means "the catalog I configured under the name glue_catalog, database warehouse_weather_data, table weather_actual". Convention is to match it to the implementation, which makes the DDL self-documenting.

Quiz: Quiz

Loading practice…