llmbroker
Turn a crowd of free, rate-limited LLMs into one reliable model — no premium subscription, no single point of failure. No heavy dependencies like LangChain.
Quick start
Install llmbroker, then:
llmbroker preset freetier > llms.toml # ready-made pool of free models
llmbroker env llms.toml > .env # which keys you need, and where to get them
llms = llmbroker.Broker("llms.toml")
print(llms.ask("Hello, how are you?").text)
Fill in whichever keys are easy to get: a model without a key simply stays inactive. When a model hits its rate limit, the broker cools it down and switches to the next one — you get an answer, not an error, as long as any model is up.
Where to go next
| Your scenario | Read |
|---|---|
| A simple script | Usage: the pool, timeouts, quality rating |
| FastAPI, agents, workers | Async: the same API with await |
| Function calling | Tools & agents: the whole tool loop in one call |
| Secrets already in AWS or Vault | API keys: the broker reads them right from there |
| Multiple instances, a shared DB | Servers & clusters: sqlite / Postgres / MongoDB, per-user keys |
Features
- Automatic failover — an error only when no one is left at all
(
NoLLMAvailableError). - Chat, tools & agents —
ask, multi-turnchat, tool calling. - Async-first —
AsyncBroker;Brokeris a blocking wrapper around the same engine. - Self-learning pool — rate the replies, weak models sink to the back of the queue.
- Keys anywhere — environment,
.env, DB, AWS, Vault or your own backend. - Scale out without code changes — a shared DB across instances, a per-user key.
- Disabling models manually, plus a pool state snapshot.
Full API reference — Reference.