Env-driven model swaps in CI

The final move is to make model selection a pure runtime decision. Your tests run against a cheap local model. Staging runs against a free OpenRouter tier. Production runs against your premium choice. Nothing about the code changes, only the environment.

.github/workflows/test.yml
yaml
name: tests
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      ollama:
        image: ollama/ollama:latest
        ports:
          - 11434:11434
    env:
      PRIMARY_MODEL: ollama/qwen2.5:0.5b
      OLLAMA_BASE_URL: http://localhost:11434/v1
      OLLAMA_API_KEY: unused
    steps:
      - uses: actions/checkout@v4
      - run: ollama pull qwen2.5:0.5b
      - run: pytest

CI runs the whole integration suite against a tiny local model. No cloud spend, no flaky shared quota, and the same code path as production.

.env
bash
# Local dev (laptop)
PRIMARY_MODEL=openrouter/google/gemini-2.5-flash
FALLBACK_MODELS=gpt-4o-mini,ollama/llama3.1:8b

# Staging
# PRIMARY_MODEL=openrouter/meta-llama/llama-3.1-70b
# FALLBACK_MODELS=gpt-4o-mini

# Production
# PRIMARY_MODEL=gpt-4o
# FALLBACK_MODELS=openrouter/anthropic/claude-3.5-sonnet,openrouter/google/gemini-2.5-flash

Three environments, three policies, zero code changes. Each layer picks the model that fits its cost and quality bar.

Three habits. First, a small golden test suite runs in CI against any candidate PRIMARY_MODEL so prompt regressions show up as a red build. Second, canary a new model on a percentage of traffic before full rollout. Third, log the effective model alongside every response so you can rollback cleanly if something looks off in production.

AI prompt: Try it: design a golden test

Loading practice…