Why a notebook stops scaling
Welcome back! I'm Param, and this is the intermediate tier of the learnwithparam data engineering track. You already know the medallion pattern from a notebook. Now we put it on a schedule, gate it with quality checks, and front it with a real API. Everything we build grounds in the companion workshop at github.com/learnwithparam/data-engineering-pipeline.
Before we touch Airflow, let us be precise about what actually breaks when a notebook enters production. Most teams only realize what broke after on-call gets paged. We are going to walk through the failures on purpose so you can design around them before they happen to you.
What breaks when a notebook goes to production
Each failure mode maps to a piece of the platform we are about to build.
0 * * * * cd /opt/pipeline && python run_all.py >> pipeline.log 2>&1This is where most teams start. One line, done. It runs. Nobody owns it. Nobody notices when it fails. Nobody can explain why yesterday looked fine and today looks broken.
Three failures hide in that one line. No retry. No dependency graph, so the transform runs whether or not ingestion succeeded. No way to trigger a backfill other than writing a new script. You will replace it with an Airflow DAG that makes every one of those a first-class concept.
Because retries are the easy part. You also need a UI that shows which runs failed and why, a way to backfill a date range without rewriting the script, and a dependency graph so a failed ingestion halts the transform automatically. By the time you build that yourself you have built a small Airflow. Using Airflow means the team knows the vocabulary.
Quiz: Quiz
Loading practice…