Why raw data is not queryable

Welcome! I'm Param, and this is the foundations course for the learnwithparam data engineering track. We are going to take a public HackerOne bug bounty dataset from raw dump to BI-ready marts using only DuckDB and pandas. No Spark, no cloud warehouse, no orchestrator. Everything we build grounds in the companion workshop at github.com/learnwithparam/data-engineering-medallion.

Before we talk about bronze, silver, or gold, let us agree on why the pattern exists at all. A pattern that solves nothing is a pattern nobody remembers. So we start with the problem: what happens when a business question lands on a raw dataset with no model.

Because every piece of that question hides a decision. What counts as critical. What counts as last quarter. Which timestamp. Which state of the report. Whether duplicates matter. In a raw CSV those decisions live in your head. Every time someone asks the question, the answer changes depending on who runs the notebook.

Raw vs modeled

A question hitting raw data spawns ad hoc logic. A question hitting a modeled warehouse reuses known joins.

Unmodeled data forces ad hoc scripts. Modeled data reuses known contracts.

notebooks/ad_hoc.ipynb

python

import pandas as pd

df = pd.read_csv('reports.csv')

# product question 1: critical bugs last quarter
crit = df[df['severity'] == 'critical']
crit_q = crit[crit['submitted_at'] >= '2025-10-01']
len(crit_q)

# product question 2: critical bugs this month, open state
crit2 = df[df['Severity_Level'] == 'CRITICAL']
open_only = crit2[crit2['state'].isin(['new', 'triaged'])]
open_only = open_only.drop_duplicates(subset=['id'])
len(open_only)

This is what every growing analytics team writes within the first month. Each question adds a few lines. After a quarter nobody knows which filter chain belongs to which report.

Because raw data lies. Upstream renamed the column mid-month. The first analyst caught it and rewrote their block. The second analyst copied from a blog post and now their number silently uses a non-existent column, which pandas returns as an empty filter. Neither analyst knows the other exists. Both reports ship. Both are wrong.

Quiz: Quiz

Loading practice…