Ollama as an OpenAI-compatible endpoint

Ollama is a tiny local server that runs open-weight models like Llama, Mistral, and Qwen on your laptop. The best part for us: Ollama exposes an OpenAI-compatible endpoint, which means your existing adapter works against it with zero new code.

Local models, same seam

The same chat() function talks to a cloud API or your laptop, depending on the model prefix.

terminal
bash
# Install once, then pull a model
brew install ollama
ollama pull llama3.1:8b

# The server is running on http://localhost:11434
ollama serve

Ollama runs as a background service. Any process on your machine can reach it on port 11434, just like a cloud API.

router.py
python
def resolve(model: str) -> tuple[OpenAICompatibleProvider, str]:
    if model.startswith('ollama/'):
        return _get_or_create(
            name='ollama',
            api_key_env='OLLAMA_API_KEY',  # any string works; the server ignores it
            base_url=os.environ.get('OLLAMA_BASE_URL', 'http://localhost:11434/v1'),
        ), model.removeprefix('ollama/')
    if model.startswith('openrouter/'):
        return _get_or_create('openrouter', 'OPENROUTER_API_KEY', 'https://openrouter.ai/api/v1'), model.removeprefix('openrouter/')
    if model.startswith('fireworks/'):
        return _get_or_create('fireworks', 'FIREWORKS_API_KEY', 'https://api.fireworks.ai/inference/v1'), model.removeprefix('fireworks/')
    return _get_or_create('openai', 'OPENAI_API_KEY', None), model

Ollama drops right in. Notice the api_key is still required by the SDK but ignored by the server, so any string works. The base_url is configurable so CI can point at a remote Ollama host.

For latency-sensitive, privacy-sensitive, or offline use cases, yes. Teams run Ollama on a GPU box inside their VPC and route internal traffic to it for redaction, embedding, or simple classification. The provider abstraction is what makes it cheap: you can start on OpenAI, add Ollama for a single feature, then expand without rewriting.

Quiz: Quiz

Loading practice…