LM Studio and vLLM with the same pattern

LM Studio is a desktop app that runs local models with a friendly UI. vLLM is a high-performance inference server used in production for self-hosted open-weight deployments. Both expose OpenAI-compatible endpoints, which means the adapter you already wrote is the adapter you use.

.env
bash
# LM Studio on your laptop
LMSTUDIO_BASE_URL=http://localhost:1234/v1

# vLLM running on a GPU host inside your VPC
VLLM_BASE_URL=https://inference.internal.acme.corp/v1

# Which one is primary this week? Change one line.
PRIMARY_MODEL=vllm/meta-llama/Llama-3.1-70B-Instruct

The only thing that changes between LM Studio and vLLM is the base URL and the model name. The rest of your stack does not know or care.

router.py
python
SELF_HOSTED = {
    'lmstudio/': ('LMSTUDIO_BASE_URL', 'LMSTUDIO_API_KEY'),
    'vllm/': ('VLLM_BASE_URL', 'VLLM_API_KEY'),
    'ollama/': ('OLLAMA_BASE_URL', 'OLLAMA_API_KEY'),
}

def resolve(model: str) -> tuple[OpenAICompatibleProvider, str]:
    for prefix, (base_url_env, key_env) in SELF_HOSTED.items():
        if model.startswith(prefix):
            return _get_or_create(
                name=prefix.rstrip('/'),
                api_key_env=key_env,
                base_url=os.environ.get(base_url_env),
            ), model.removeprefix(prefix)
    # cloud providers fall through...
    return _get_or_create('openai', 'OPENAI_API_KEY', None), model

The registry now scales cleanly. Every new OpenAI-compatible backend is one row in a dict. The pattern carries you very far before you need anything more complicated.

Matching exercise: Pick the right local backend

Loading practice…

Checkpoint: Checkpoint: foundations

Loading practice…