FastAPI endpoints and health stubs

Welcome! I am Param. Over the next few hours we are going to take a RAG service from a laptop demo to a horizontally scaled Kubernetes deploy. You will add Ray for parallel embedding, persist ChromaDB, ship a hardened container, and watch an HPA move replicas under a real load test.

We always start with the API shape. If the contracts are wrong, no amount of clever embedding will save you. Ingest, query, health, and stats need clean Pydantic models and a routable prefix so every downstream layer can plug in without guessing.

terminal
bash
# Clone the workshop repository
git clone https://github.com/learnwithparam/enterprise-rag-kubernetes-ray.git
cd enterprise-rag-kubernetes-ray

# Create env and install deps
make setup

# Edit .env with your OPENROUTER_API_KEY (or Fireworks, Gemini, OpenAI)
# Then run the dev server
make dev

The workshop ships a Makefile that wraps uv, docker, and kubectl. make dev sets up the venv, installs dependencies, and runs uvicorn with reload.

main.py
python
from dotenv import load_dotenv
load_dotenv()

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from router import router

app = FastAPI(
    title='Enterprise RAG on Kubernetes with Ray',
    description='Distributed RAG pipeline using Ray for parallel embedding and Kubernetes for deployment',
    version='1.0.0',
)
app.add_middleware(CORSMiddleware, allow_origins=['*'], allow_credentials=True, allow_methods=['*'], allow_headers=['*'])
app.include_router(router)


@app.get('/')
async def root():
    return {'service': 'enterprise-rag-kubernetes-ray', 'docs': '/docs'}

The entry point stays tiny. load_dotenv runs before any imports that read env. The router carries every real endpoint. CORS is wide open for the workshop and locked down later per environment.

router.py
python
from fastapi import APIRouter, HTTPException

from models import (
    IngestRequest,
    IngestResponse,
    QueryRequest,
    QueryResponse,
    StatsResponse,
)
from service import rag_service

router = APIRouter(prefix='/enterprise-rag', tags=['enterprise-rag'])


@router.get('/health')
async def health():
    return {
        'status': 'healthy',
        'service': 'enterprise-rag-kubernetes-ray',
        'embedder': rag_service.embedder.backend,
        'documents': rag_service.index.count(),
    }

Health is intentionally more than a liveness ping. It reports the embedder backend (ray or sequential) and the current document count so Kubernetes probes, and you, can see at a glance whether the pod is really ready.

service.py
python
from typing import List

from models import Document
from pipeline.embed import RayEmbedder
from pipeline.index import ChromaIndex
from pipeline.query import answer_question
from utils.llm_provider import get_llm_provider


class RAGService:
    def __init__(self):
        self.embedder = RayEmbedder()
        self.index = ChromaIndex()
        self.llm = get_llm_provider()

    def ingest(self, docs: List[Document]) -> dict:
        ...

    async def query(self, question: str, top_k: int = 4) -> dict:
        ...

    def stats(self) -> dict:
        ...


# Module-level singleton: models and Ray actors load once at pod start
rag_service = RAGService()

This is the object the router just imported. RAGService composes the embedder, the vector index, and the LLM provider into one glue layer, and you build each of those pieces as the course progresses. Constructing the singleton at import time means models and Ray actors load once when the pod starts, never per request.

A plain ok lies. A pod can be listening on port 8000 before the embedding model finishes loading or before Ray connects its actors. By reporting the embedder backend and the document count, the same endpoint doubles as a real readiness signal for Kubernetes and a diagnostic you can curl during an incident.

Quiz: Quiz

Loading practice…