Quality gate: catch drift before it hits gold

Silver is trusted by convention. A quality gate turns that convention into a contract. We are going to write five assertions that run after every silver build. If any one of them fails, gold does not load. The dashboard stays on the last known good numbers instead of showing you wrong ones.

The checks you will always want: row count is within an expected band, required columns have zero nulls, every enum value is in the known domain, and the freshest record is younger than your SLA. One more belongs on the list, that every foreign key in a fact table resolves to a dimension, but it needs the fact table to exist first, so it joins the runner when we build gold.

data_engineering.ipynb
python
CHECKS = [
    ('row-count-lower-bound', 'SELECT 1 WHERE (SELECT COUNT(*) FROM silver) < 1000'),
    ('no-null-ids',           'SELECT * FROM silver WHERE id IS NULL LIMIT 5'),
    ('known-severity',        "SELECT DISTINCT severity FROM silver WHERE severity NOT IN ('low','medium','high','critical','unknown')"),
    ('known-state',           "SELECT DISTINCT state FROM silver WHERE state NOT IN ('new','triaged','resolved','informative','duplicate')"),
    ('freshness-24h',         "SELECT 1 WHERE (SELECT MAX(submitted_at) FROM silver) < CURRENT_TIMESTAMP - INTERVAL 24 HOUR"),
]

def run_checks(con):
    failures = []
    for name, sql in CHECKS:
        rows = con.sql(sql).df()
        if len(rows) > 0:
            failures.append((name, rows))
    return failures

failures = run_checks(con)
for name, rows in failures:
    print(f'FAIL {name}')
    print(rows)
assert not failures, f'{len(failures)} data-quality checks failed'

A tiny assertion runner. Each entry is a SQL query that must return zero rows. If any returns rows, the build fails loudly and the offending rows get printed so a human can investigate.

Yes. A new state value is a schema change, and you want to know before the dashboard shows it as "unknown" and someone files a bug. Strict checks push you to update silver on purpose when upstream changes. That keeps the contract explicit instead of silently absorbing drift.

Quality gate position

The gate sits between silver and gold. Gold never loads on red.

Gate runs after silver, before gold. Dashboards read gold only when gate is green.

Quiz: Quiz

Loading practice…

Checkpoint: Checkpoint: bronze, silver, and quality

Loading practice…