Schema evolution in practice

Schema evolution is the feature that pays back the lakehouse setup cost. Adding a column to an Iceberg table is a metadata operation: zero data rewrite, zero downtime, zero coordination with downstream readers.

glue/transform-weather-data/iceberg_evolution.sql
sql
-- Add a nullable column. Old rows show NULL. No rewrite.
ALTER TABLE glue_catalog.warehouse_weather_data.weather_actual_data_timeseries
  ADD COLUMN wind_chill DOUBLE;

-- Rename a column. Pure metadata.
ALTER TABLE glue_catalog.warehouse_weather_data.weather_actual_data_timeseries
  RENAME COLUMN temperature_c TO temp_celsius;

-- Promote int to long. Safe widening.
ALTER TABLE glue_catalog.warehouse_weather_data.weather_actual_data_timeseries
  ALTER COLUMN flight_id TYPE BIGINT;

Three evolution operations Iceberg makes safe: add a column, rename a column, promote a type. Each is a metadata commit, not a data rewrite.

Safe evolutions: add a column, rename a column, widen a type (int to long, float to double). Unsafe evolutions: drop a column (data loss), narrow a type (data loss), reorder columns in source files. The boundary is whether old data is still readable under the new schema.

Snowflake reads the latest Iceberg snapshot on a refresh interval (we set 600 seconds in the Catalog Integration). After that interval, the new column is visible. To force an immediate refresh, run ALTER ICEBERG TABLE ... REFRESH on the Snowflake side.

Ordering exercise: Order the safe evolution path for adding wind_chill

Loading practice…