One client, multiple LLM providers

Nobody puts a database hostname in source code any more. We learned that one properly: the connection string lives in one place, the code names a service, and moving the database is an edit in the place that owns the fact — not in every repository that reads a row.

We have not learned it for models. A project that calls an LLM almost always carries three constants inline: a base URL, a key, and a model id. All three are correct on the day you write them and wrong a few weeks later, because the model you are using gets replaced, re-quantised, moved to another engine, or turns out to cost more than it is worth. Then the swap is an edit in every project that names it, and every project names it, because that is how every SDK example is written.

And it is worse than an edit, because the three constants are really four: the wire format is in there too, unwritten. Move the same request from OpenAI to Anthropic and it is not a new value in a config file — it is a different message shape, a different place for the system prompt, a different way to attach an image, and a different type for the arguments of a tool call.

ai-gateway is my answer to that, and it is deliberately unexciting: one OpenAI-compatible endpoint on localhost:24000, in front of every model on the machine. Projects ask for a name.

curl http://localhost:24000/v1/chat/completions \
  -H "Authorization: Bearer sk-litellm-master" \
  -H 'Content-Type: application/json' \
  -d '{"model":"lms-4b","messages":[{"role":"user","content":"hi"}]}'

lms-4b is not a model. It is a name that one config file points at a model, and that file is the only place the real id appears. This post is about what that one level of indirection buys, because “you can swap models faster” is the smallest of the five things, and I did not predict the other four.

One: one client, whatever is behind it

This is the one that costs the most to get wrong, so it goes first.

Every project here imports the OpenAI client, builds an OpenAI-shaped message, and points base_url at the gateway. That is the whole integration, and it does not change when the model behind the name changes — not when it moves from LMStudio to Ollama, not when it moves from a local Gemma to a hosted one, not when it stops being an open-weights model at all.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:24000/v1", api_key=KEY)
r = client.chat.completions.create(model="lms-4b", messages=[...])

Without that boundary, “use a different model” is not a config change. It is a rewrite of the request, because every vendor spells the same request differently. Here is one request — a PNG and a question about it — in four APIs:

Start with the image, because it is the shortest thing to compare. Same PNG, same four bytes of intent, four different spellings:

// OpenAI — a block inside messages[].content[]
{ "type": "image_url",
  "image_url": { "url": "data:image/png;base64,iVBORw0…" } }

// Anthropic — a block inside messages[].content[]
{ "type": "image",
  "source": { "type": "base64", "media_type": "image/png", "data": "iVBORw0…" } }

// Google Gemini — a part inside contents[].parts[]
{ "inline_data": { "mime_type": "image/png", "data": "iVBORw0…" } }

// Ollama native — a field on the message itself
{ "role": "user", "content": "…", "images": ["iVBORw0…"] }

Read down the first key of each: type, type, inline_data, images. One wraps the bytes in a data: URL, three want them bare. One calls the format media_type, one calls it mime_type, two do not carry it at all. Nothing about this is hard — it is just four spellings of one idea, and your code has to pick one.

The rest of the request is the same story:

API	System prompt	Image lives at
OpenAI chat completions	a `"role": "system"` message	`content[].image_url.url`
Anthropic messages	a top-level `system` field	`content[].source.data`
Gemini `generateContent`	a top-level `systemInstruction`	`parts[].inline_data.data`
Ollama native `/api/chat`	a `"role": "system"` message	`message.images[]`

The knobs move as well. OpenAI and Anthropic both take a top-level max_tokens — except that Anthropic requires it, and the gpt-5 family renamed it to max_completion_tokens. Gemini keeps its knobs in a generationConfig object, Ollama in an options object where the same idea is called num_predict.

Whole requests, then. The same question about the same image, sent to a hosted OpenAI model and to a Claude model:

{
  "model": "gpt-5.4-mini",
  "max_completion_tokens": 512,
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": [
        { "type": "text", "text": "What shape and colour is in this image?" },
        { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0…" } }
    ] }
  ]
}

{
  "model": "claude-opus-5",
  "max_tokens": 512,
  "system": "You are a helpful assistant.",
  "messages": [
    { "role": "user", "content": [
        { "type": "text", "text": "What shape and colour is in this image?" },
        { "type": "image", "source": {
            "type": "base64", "media_type": "image/png", "data": "iVBORw0…" } }
    ] }
  ]
}

The system prompt moved out of the array. The image grew a source wrapper. And the cap on the answer is not even called the same thing, which is the detail that catches people: the gpt-5 family rejects max_tokens outright.

Now the replies, because that is the half that runs in your code every single turn:

{
  "id": "chatcmpl-…",
  "model": "gpt-5.4-mini",
  "choices": [
    { "index": 0,
      "message": { "role": "assistant", "content": "A red circle on white." },
      "finish_reason": "stop" }
  ],
  "usage": { "prompt_tokens": 812, "completion_tokens": 7, "total_tokens": 819 }
}

{
  "id": "msg_…",
  "type": "message",
  "role": "assistant",
  "model": "claude-opus-5",
  "content": [
    { "type": "text", "text": "A red circle on white." }
  ],
  "stop_reason": "end_turn",
  "usage": { "input_tokens": 812, "output_tokens": 7 }
}

Four differences in seven lines of answer. The text is at choices[0].message.content on one side and content[0].text on the other — a string versus an array of typed blocks you have to walk. The reason it stopped is finish_reason: "stop" or stop_reason: "end_turn". And the token counts are prompt_tokens / completion_tokens against input_tokens / output_tokens, which is the one that quietly corrupts a cost dashboard rather than throwing.

Tool calling — the thing an agent lives on — diverges hardest:

	OpenAI	Anthropic
Declaring a tool	`{"type": "function", "function": {…}}`, schema under `parameters`	`{"name", "description", "input_schema"}` — no wrapper
The model asks	`message.tool_calls[]`, `finish_reason: "tool_calls"`	a `tool_use` block in `content[]`, `stop_reason: "tool_use"`
The arguments	a JSON string you must parse	already a parsed object
Sending the result back	a message with `"role": "tool"` and `tool_call_id`	a `user` message carrying a `tool_result` block

Row three is the one that gets you at runtime rather than at review, and it is worth seeing literally. Same tool call, both APIs:

// OpenAI — arguments is a STRING containing JSON
"tool_calls": [
  { "id": "call_abc123", "type": "function",
    "function": { "name": "get_stock_price",
                  "arguments": "{\"ticker\": \"MSFT\"}" } }
]

// Anthropic — input is already an OBJECT
"content": [
  { "type": "tool_use", "id": "toolu_abc123", "name": "get_stock_price",
    "input": { "ticker": "MSFT" } }
]

One hands you a string and the other hands you a value. So json.loads is mandatory in one branch and a TypeError in the other, and no schema, type hint or linter tells you which branch you are in — the model id does, three layers away.

None of that divergence disappears — it moves. The gateway holds it, and it is the only thing that has to know: LiteLLM speaks OpenAI to my code, and whatever the upstream wants on the other side. The translation goes both ways, too. LiteLLM also exposes /v1/messages, the Anthropic route, so Claude Code — which speaks Messages and nothing else — drives a local Gemma and never learns it is not talking to Anthropic. Two protocols in the front, five engines at the back, one vocabulary in the middle.

Two: the swap becomes an edit in one place

This is the obvious one, so it gets one paragraph.

The five engines here are LMStudio, Unsloth Studio and Ollama running natively on the machine, plus OpenRouter and OpenAI in the cloud. Each serves two or three aliases: a small chat model, a large one, an embedder. Changing which weights lms-26b means is one line in litellm/lms.yaml. Nothing that called it has to know, and nothing that called it has to be redeployed.

Three: comparison stops being a refactor

This is the one I would put first now.

The alias names are laid out as a grid on purpose. lms-26b, unsloth-26b, ollama-26b and openrouter-26b are the same weights on four engines — the row is the model, the column is the engine. So the question “is Ollama actually slower than LMStudio for my workload” becomes: change one word, up -d, run the same test suite again.

uv run run_all.py --model lms-4b        # 3 call kinds x 2 gateways
uv run run_all.py --model ollama-4b     # same scripts, same bodies

Without the indirection this is not an experiment, it is a refactor — you edit the client, and now you are comparing two engines and whatever you changed in the call while you were in there. With it, the only variable that moved is the one you meant to move.

That is what turned vague impressions into things I can write down with a date on them. The same weights behave differently per engine, and not subtly: unsloth-26b emits a reasoning block while lms-26b on identical weights does not (2026-08-27). Ollama’s Gemma pulls are Q4_K_M where LMStudio’s are QAT, so “the same model” is not the same file. None of that is discoverable if switching engines costs you an afternoon.

Four: it is where the money boundary goes

A local-only setup does not need this. A setup with any hosted route in it needs it badly, because the failure mode is silent.

Four rules do the work, and three of them are things the gateway refuses to do:

The prefix names the payer. lms-*, unsloth-* and ollama-* are this machine and free; openrouter-* and openai-* bill a real account. There is deliberately no engine-neutral name and no capability name. local existed once and hid which engine answered. cheap, standard and frontier existed and hid who was billed. Both were deleted for the same reason: a name should answer the question you would otherwise have to go and look up.
No alias falls back to another. A fallback chain is the industry default and it breaks both things this repo is for. A comparison stops being a comparison the moment a request can quietly run somewhere else, and a free session can become a paid one while every log still looks normal.
Keys carry a ceiling and an expiry. The master key mints other keys and has no budget of its own, so it is not what a project should hold. A project gets a key scoped to two aliases, capped at $0.50, expiring in 24 hours.
Local routes are shadow-priced. They are free, but they carry a cloud twin’s rate, so spend accrues and a ceiling still trips on a model that bills nobody. That number is “what this workload would cost in the cloud”, not money anyone paid — which is a distinction anything reporting it has to make out loud.

curl -X POST http://localhost:24000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"models":["lms-4b","lms-embed"],"max_budget":0.50,"duration":"24h"}'

The point is not that a laptop needs spend control. It is that an agent loop is a program that decides how many calls to make, and a cap you set once at the boundary is worth more than a discipline you have to keep remembering at the call site.

Five: one place where every request is written down

Every call through the gateway lands in the admin UI’s Logs tab with its prompt and its response. Not “a metric” — the actual text, the alias, the spend, the upstream it went to.

This is the part that has changed how I debug agents. A tool call that came back as prose instead of a structured tool_calls reply is invisible from the agent’s side: it executes nothing and exits cleanly, looking like a model that decided not to bother. In the gateway log it is one line, and the cause is legible — which is how the provider pin in litellm/openrouter.yaml came to exist, because OpenRouter load-balances its free tier and one provider returns tool calls as raw text.

It is also how I know the translation in point one actually survives contact with a small local model. Tool calling is the part that decides whether a model is usable from an agent at all, and structured tool_calls replies — not the raw-text imitation — are verified on lms-4b, unsloth-4b and ollama-4b (2026-08-27, re-verified on ollama-4b 2026-08-31).

What it does not fix

A post that lists five wins and stops is an advertisement, so here is the bill.

An alias is two edits. There are two gateways here — LiteLLM on 24000 and MLflow on 25000 — and each owns its own alias list, one in YAML and one in Python, neither reading the other. That is deliberate, so either can be deleted and the other still serves, and the price is exact: add a name to one and it answers on 24000 and 404s on 25000, with nothing in either log to say why.

A shared vocabulary is not interchangeability. The gateway makes two models callable the same way. It does not make them the same:

Looks like one name	Is not one thing
Any chat alias	Thinking models spend the reply’s budget on thinking. A `max_tokens` set too low returns empty content, `finish_reason: "length"`, and no error at all — `lms-4b` once spent 65 of 70 completion tokens reasoning
The three `*-embed` aliases	All nomic v1.5 at 768 dims in three different builds — Q4_K_M, Q8_0, F16. A query embedded with one and matched against an index built with another returns quietly worse neighbours and never errors
A local alias that worked this morning	LMStudio JIT-loads a model that is not resident, and a JIT load does not inherit your hand-load flags: a model you loaded at 262144 context comes back at 8192

It is another hop, and hops have timeouts. Prompt processing here measures around 100 tokens/second, and Claude Code re-sends its system prompt and every tool schema each turn, so a real agent turn can need 5–15 minutes before the first token. Both the route timeout and the client timeout have to be raised or the patience of one is wasted by the impatience of the other.

Why this is worth a repository rather than a note

Because the half-life of a model id is now shorter than the life of any project that names one.

Everything above is a consequence of a single decision — that projects name a capability, speak one protocol, and let something else decide what serves it — and that is not a new idea. It is the same reason we stopped putting hostnames in source. The only thing that changed is that the fast-moving dependency is now a model, and models change faster than databases ever did.

The repository is on GitHub, MIT, four stock containers and no build step — cp .env.example .env and docker compose up -d is the whole install. Everything in it is measured on one Apple-Silicon MacBook with 128 GB of RAM, which is the honest scope of every number in this post.