Compaction and small-file mitigation

Iceberg writes one or more files per partition per commit. Daily writes over a year produce hundreds of small files per partition. Reads slow down. Compaction merges them into a few large files.

glue/transform-weather-data/iceberg_maintenance.sql
sql
CALL glue_catalog.system.rewrite_data_files(
  table => 'warehouse_weather_data.weather_actual_data_timeseries',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-input-files', '5'
  )
);

Compaction is one Spark command. Run it weekly on Iceberg tables where you commit frequently. The target file size of 512 MB is a sane default.

glue/transform-weather-data/iceberg_maintenance.sql
sql
CALL glue_catalog.system.expire_snapshots(
  table => 'warehouse_weather_data.weather_actual_data_timeseries',
  older_than => TIMESTAMP '2026-04-01 00:00:00',
  retain_last => 5
);

Old snapshots accumulate manifest data and reference files marked for deletion. Expire snapshots older than 30 days to release storage.

Daily commits, weekly compaction. Hourly commits, daily compaction. The principle: compact often enough that "small files per partition" stays under 50 most of the time. Add an Airflow task that runs the compaction CALL on a schedule.

Checkpoint: Iceberg writer checkpoint

Loading practice…