The real-time story

This is the most important lesson in the course. Without streaming, your users stare at a loading spinner for seconds while the LLM generates the full response. With streaming, they see text appear word by word, just like ChatGPT. The difference in perceived speed is dramatic.

Streaming vs buffered responses

Without streaming, the user waits for the entire response. With streaming, chunks arrive in real-time.

service.py
python
async def generate_story_stream(request: StoryRequest) -> AsyncGenerator[str, None]:
    # Step 1: Notify client that connection is established
    yield f"data: {json.dumps({'status': 'connected'})}\n\n"

    try:
        prompt = build_story_prompt(request)

        # Step 2: Stream chunks as they arrive from the LLM
        async for chunk in llm_provider.generate_stream(
            prompt,
            temperature=0.8,
            max_tokens=800
        ):
            # Each chunk is sent immediately to the client
            yield f"data: {json.dumps({'content': chunk})}\n\n"

        # Step 3: Signal completion
        yield f"data: {json.dumps({'done': True})}\n\n"
    except Exception as e:
        yield f"data: {json.dumps({'error': str(e)})}\n\n"

This is an async generator. Each yield sends a Server-Sent Event to the client immediately. The format is "data: {json}\n\n" which is the SSE standard.

router.py
python
@router.post("/stream")
async def stream_story(request: StoryRequest):
    return StreamingResponse(
        generate_story_stream(request),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
        }
    )

FastAPI's StreamingResponse wraps the async generator. The media_type "text/event-stream" tells the browser this is an SSE connection.

SSE is simpler and sufficient for this use case. LLM streaming is one-directional: the server sends chunks to the client. WebSockets add bidirectional complexity you do not need. SSE also works over standard HTTP, reconnects automatically, and requires no special client library.

Quiz: Quiz

Loading practice…