Spark for batch, Postgres for the warehouse

Compute and storage do different jobs. Spark reads parquet at scale, runs transforms in parallel, and writes derived tables. Postgres is a durable, queryable warehouse that serves dashboards, APIs, and the control plane. MinIO is the lake between the two, where raw files land before Spark ever sees them.

Where each tool sits

Airflow schedules, Spark computes, MinIO stores raw, Postgres serves modeled.

Each tool owns one thing. Swap tools later, shape stays.
spark/spark_batch_job.py
python
from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName('warehouse-transform')
    .config('spark.hadoop.fs.s3a.endpoint', MINIO_ENDPOINT)
    .config('spark.hadoop.fs.s3a.access.key', MINIO_ACCESS_KEY)
    .config('spark.hadoop.fs.s3a.secret.key', MINIO_SECRET_KEY)
    .getOrCreate()
)

raw = spark.read.parquet('s3a://raw-data/orders/')
silver = raw.dropDuplicates(['order_id']).dropna(subset=['order_id', 'amount'])

fact_orders = silver.selectExpr(
    'order_id AS natural_key',
    'customer_id', 'product_id', 'order_ts',
    'amount'
)

(fact_orders.write
    .format('jdbc')
    .option('url', PG_URL)
    .option('dbtable', 'fact_orders')
    .option('user', PG_USER)
    .option('password', PG_PASSWORD)
    .mode('overwrite')
    .save())

A minimal Spark batch job. It reads raw parquet from MinIO via S3 protocol, transforms into dimensional shape, and writes back to Postgres through JDBC. This is the transform stage the Airflow DAG calls.

Parquet is columnar, compressed, and carries a schema. Spark can read only the columns it needs, which cuts IO by an order of magnitude on wide tables. It also means the raw layer has self-describing types, so a downstream reader does not guess. CSVs would work but force every consumer to re-infer types and read the whole row.

Validation checklist: Walk the stack for your own pipeline

Loading practice…

Quiz: Quiz

Loading practice…

Checkpoint: Checkpoint: you can defend the architecture

Loading practice…