Why twenty services, not a monolith

Twenty services sounds heavy. It is not. Each service owns one well-scoped job. The point of the full shape is to see the contracts between them clearly so when the managed platform equivalents (Databricks, Snowflake, Confluent) show up, you know what they are replacing.

Not every service carries equal weight. Airflow, Kafka, Spark, Postgres, and FastAPI are the platform spine. MLflow, Atlas, Prometheus, and Grafana are the governance and observability layer and they matter at enterprise scale. MongoDB, Redis, InfluxDB, and Elasticsearch are optional stores for specific workloads. Never add a service because it is fashionable. Add it because a concern is unowned.

You can, up to a point. Minute-level Airflow runs work for near-real-time that tolerates ten to sixty second latency. Below that, the scheduler overhead dominates and you pay for a lot of container starts. Kafka is the right answer when you need event-driven consumers, sub-second p99, or exactly-once semantics. Use the simpler option first and move only when the SLA demands it.