GCS bucket layout for raw CSVs

GCS layout drives every downstream step. The path you pick becomes the contract that bq load reads, that BigQuery external tables reference, and that retention rules apply to. Get it right once and forget about it.

A working layout: gs://learnwithparam-ecommerce-raw/raw/<table_name>/year=YYYY/month=MM/day=DD/<file>. Hive-style partition keys make BigQuery external tables happy. Embedding the date in the path makes retention rules clean.

GCS path drives every downstream surface

One Hive-partitioned path feeds bq load, external tables, and retention rules.

scripts/gcs_upload.sh
bash
TODAY=$(date +%Y-%m-%d)
YEAR=$(date +%Y)
MONTH=$(date +%m)
DAY=$(date +%d)

for file in datasets/olist_ecommerce_dataset/*.csv; do
  table=$(basename "$file" .csv)
  gsutil cp "$file" "gs://${GCS_RAW_BUCKET}/raw/${table}/year=${YEAR}/month=${MONTH}/day=${DAY}/${TODAY}.csv"
done

Upload the bundled Olist CSVs to GCS. The path embeds the date so the same command run tomorrow lands in a different prefix.

Long enough to rebuild from raw if BigQuery has to be wiped. 30 to 90 days is a common window. Apply a Lifecycle rule that transitions to Coldline after 30 days and deletes after 90. The point of raw is recovery, not permanent storage.

Quiz: Quiz

Loading practice…