Loading...
Loading...
Stop running ad-hoc SQL queries in the Cloud Console. Ship a versioned dbt project to BigQuery, host the runner on Cloud Run, schedule it with Airflow, deploy on every push.
Message a mentor about fit, prerequisites, or where to start. Replies come on WhatsApp, usually within a day.
Engineers are learning here from
Build a GCP analytics stack end to end: GCS landing zone, BigQuery raw, dbt star schema, Cloud Run-hosted dbt runner exposed via Flask, Airflow scheduling, Cloud Build for push-to-deploy, and Terraform owning the infrastructure. The GCP track of the learnwithparam data engineering catalogue.
GCS to BigQuery to dbt star schema, with Cloud Run hosting the dbt runner and Cloud Build deploying on push. Airflow schedules the run. Terraform owns the infra.
What you'll ship
What you'll learn
Curriculum
Choosing the GCP analytics stack
Pick BigQuery and dbt deliberately, understand the pricing model, and tour the Olist e-commerce dataset before writing any SQL.
GCS landing and BigQuery raw
Land CSVs in GCS, load them into BigQuery raw datasets, and partition for the dominant query patterns.
dbt project anatomy
Author the dbt project: profiles, sources with freshness, stg cleaning, marts star schema, BI aggregates. Run it locally with dbt-duckdb shim before pointing at BigQuery.
Hosting dbt on Cloud Run
Wrap dbt in a Flask app, ship to Cloud Run, secure with identity tokens, and add the health probes Cloud Run needs.
Airflow triggers Cloud Run
Schedule the dbt run from Airflow with identity-token auth, decide between sync and async patterns, and wire failure handling.
Cloud Build and Terraform
Wire Cloud Build for push-to-deploy, Terraform for everything else, and run the whole pipeline end to end as a capstone.
Who it's for
maintaining a pile of one-off SQL queries in BigQuery and wanting a real dbt project with version control and tests
asked to build a GCP analytics stack and looking for a reference end-to-end pattern instead of a tutorial-of-the-week
inheriting a Cloud Run dbt runner and needing to understand how Flask, dbt, identity tokens, and Cloud Build cooperate
wiring Cloud Build, Cloud Run, Artifact Registry, and IAM for the first time and discovering each has its own gotchas
FAQ
No. The smoke test runs locally with a dbt-duckdb shim profile, the Flask app boots without GCP credentials, and Terraform validate works without a real backend. Deployment chapters show the GCP Console and gcloud commands for when you do want to deploy.
Cloud Run runs containers, which means you can ship the exact dbt image you tested locally. Cloud Functions is too restrictive for the dbt runtime. Composer is overkill (and expensive) for a single dbt job. Cloud Run is the goldilocks middle.
Yes. dbt is engine-agnostic. The course uses BigQuery-specific features (partitioning syntax, scheduled queries) but the project structure, source freshness, and star schema patterns are identical across adapters. The CI profile uses dbt-duckdb to prove the point.
For a single dbt run on a schedule, yes. Cloud Scheduler plus Cloud Run is the lightest setup. The course keeps Airflow because real pipelines have multi-step dependencies (raw load, dbt build, downstream BI refresh) that benefit from a real orchestrator.
Pricing
One subscription unlocks every paid course and workshop replay. Pick yearly or monthly.
Unlock with Pro
You save 47% with regional pricing
Billed annually. Cancel anytime.
Still deciding? Ask Param a question
GCP analytics with BigQuery, dbt, and Cloud Run
From $16/mo with Pro