Quality gates with Great Expectations

Great Expectations is a Python library that lets you declare what you expect from a dataset and runs those expectations as a checkpoint. Your Airflow DAG runs the checkpoint between extract and transform. Green goes forward. Red halts the run.

Where Great Expectations sits

Between raw and transform. A red checkpoint halts the DAG and notifies oncall.

Validation is a first-class task, not a log warning.
great_expectations/expectations/orders_raw.py
python
from great_expectations.core.expectation_suite import ExpectationSuite
from great_expectations.core.expectation_configuration import ExpectationConfiguration

suite = ExpectationSuite('orders_raw')
suite.add_expectation(ExpectationConfiguration(
    expectation_type='expect_table_row_count_to_be_between',
    kwargs={'min_value': 100},
))
suite.add_expectation(ExpectationConfiguration(
    expectation_type='expect_column_values_to_not_be_null',
    kwargs={'column': 'order_id'},
))
suite.add_expectation(ExpectationConfiguration(
    expectation_type='expect_column_values_to_be_in_set',
    kwargs={'column': 'status', 'value_set': ['new', 'paid', 'shipped', 'refunded', 'cancelled']},
))
suite.add_expectation(ExpectationConfiguration(
    expectation_type='expect_column_max_to_be_between',
    kwargs={'column': 'updated_at', 'min_value': 'now-36h'},
))

A suite with four typical expectations: row count minimum, not-null on business keys, allowed severity values, and timestamp recency. Each one turns into a pass/fail the DAG can act on.

Start with expectations tied to business-critical invariants: no null order_id, status in a known set, row count within a sane band. Those failing are real problems. Avoid expectations like column variance in a narrow range on every run. The rule: each expectation should be something a human could defend as an actual contract with upstream.

Ordering exercise: Order the GE integration steps

Loading practice…

Quiz: Quiz

Loading practice…