Bronze: register the dataset, touch nothing

Time to put the theory in a warehouse. We will pull a real HackerOne bug bounty dataset from Hugging Face, land it in a DuckDB table called bronze, and write zero cleaning logic while we do it. Every byte from upstream lands untouched.

data_engineering.ipynb
python
from datasets import load_dataset

ds = load_dataset('cyberagent/hackerone', split='train')
raw = ds.to_pandas()
print(raw.shape)
print(raw.columns.tolist()[:6])

Pull the public dataset using the datasets library. It returns a pandas dataframe, which DuckDB can read directly. No schema inference on our side, no cleaning, no renaming.

Notice I print the shape and the first column names before writing anything. That tells me what I am dealing with without inventing a story. If upstream renames a column next month, this same print will show the difference and my bronze write will still succeed.

data_engineering.ipynb
python
import duckdb

con = duckdb.connect('hackerone.duckdb')
con.register('raw', raw)
con.sql('CREATE OR REPLACE TABLE bronze AS SELECT * FROM raw')

count = con.sql('SELECT COUNT(*) AS n FROM bronze').df()
print(count)

Open the persistent DuckDB file and register the raw dataframe as a table named bronze. DuckDB copies the rows into its own storage here, so the warehouse file is self-contained once this runs.

Never. Bronze is verbatim. If the source has duplicates, bronze has duplicates. The reason is: you might one day want to know how many duplicates you received, or debug a bug that only shows up with duplicates. Dedup lives in silver, where it is visible, testable, and reversible by rereading bronze.

Bronze is a one-way mirror

Source rows land unchanged. Downstream layers read from bronze and never write back.

Bronze is written once per load. Every other consumer reads only.

Validation checklist: Verify your bronze load

Loading practice…

Quiz: Quiz

Loading practice…