Partitioning and sort orders for query patterns

Partitioning is how Iceberg makes queries fast. A query that filters on partitioned columns prunes whole files before reading. A query that does not, scans everything. Picking the right partitions means knowing the dominant query patterns.

Too few partitions and queries scan too much. Too many and you create the small-file problem: thousands of tiny Parquet files that hurt every read. The sweet spot is partitions that contain hundreds of MB to a few GB each.

Iceberg lets you change partition specs. New writes use the new spec. Old data keeps its old spec. Reads merge transparently. This is a feature plain Parquet does not have. The course shows the SQL: ALTER TABLE ... ADD PARTITION FIELD.