Partitioning and clustering on raw

BigQuery has two layers of physical organization. Partitions divide a table into chunks pruned by filter predicates. Clustering sorts rows within each chunk. Both speed up queries that filter on the right columns. Both cost money to get wrong.

Three partition types: ingestion-time (free, automatic), date or timestamp column, integer range. Use ingestion-time for streaming. Use date column when you query by purchase_date. Use integer range only when you have natural numeric grouping.

scripts/partition_orders.sql
sql
CREATE OR REPLACE TABLE `${GCP_PROJECT_ID}.${BQ_RAW_DATASET}.orders`
PARTITION BY DATE(order_purchase_timestamp)
CLUSTER BY customer_id, order_status
AS SELECT * FROM `${GCP_PROJECT_ID}.${BQ_RAW_DATASET}.orders_external`;

A partitioned + clustered table. Partition by purchase date for time-range filters. Cluster by customer_id for customer-centric queries.

Partitioning splits a table into separate physical chunks, one per partition value. Clustering sorts the rows within each chunk by the clustering keys. A query that filters on the partition column reads fewer chunks. A query that filters on a clustering column reads fewer blocks within the chunks it does scan.

Quiz: Quiz

Loading practice…

Checkpoint: Checkpoint: GCS and BigQuery raw are wired

Loading practice…