The lifespan that loads the model once

So far every request has paid the cost of whatever work the handler does. That is fine for a health check. It is disastrous when the handler instantiates a client that reads a config file, opens a connection, or loads a model. The fix is the lifespan: an async context manager that runs once at startup and once at shutdown.

Request cost with and without lifespan

Without a lifespan, every request reloads the client. With one, the load happens once.

main.py
python
from contextlib import asynccontextmanager
from dotenv import load_dotenv
load_dotenv()

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from router import router
from utils.llm_provider import get_llm_provider

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Startup: load expensive clients once.
    app.state.llm_provider = get_llm_provider()
    yield
    # Shutdown: release resources.
    app.state.llm_provider = None

app = FastAPI(
    title="Streaming LLM Applications with FastAPI",
    version="1.0.0",
    lifespan=lifespan,
)
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
app.include_router(router)

Wrap startup and shutdown in a single async context manager, then pass it to FastAPI as lifespan=. Attach resources to app.state so routes can read them.

Two reasons. First, app.state is tied to the FastAPI app instance, which means tests can create a fresh app with a different client without tearing down process globals. Second, it makes the dependency explicit: anything attached to app.state is visible in the route and shareable through Depends, which we will wire up soon.