What it does
A routing policy is versioned data: a catalogue of models with their context window, cost, capabilities and privacy tags, task routes with an author preference order, an allow-list and a deny-list, health thresholds and a circuit-breaker setting. POST /v1/model-routing/decide takes a request and returns a winner, an ordered fallback chain and an explanation, or an explicit denial. The caller then executes the call itself.
How it works
A request names a task and its constraints: a privacy requirement, a latency budget, a cost ceiling, a minimum context window and required capabilities. The decision applies eight inputs. Task type selects the candidate set and its preference order. Privacy tags, the latency budget (against observed latency), the cost ceiling, the context window and required capabilities filter candidates out. An approved list acts as a hard allow-list and a blocked list as a hard deny-list. Observed health demotes models with a high error rate.
Survivors are ranked by (health rank, preference index, model id), so the same inputs always give the same winner and the same bounded fallback chain (three models by default). If every candidate is filtered out, the decision is a denial with the full explanation, not an arbitrary pick. The explanation lists each model considered, why it was removed and why the winner won, and carries no prompt or answer content.
POST /v1/model-routing/outcome records the success or failure of a routed call, which drives a per-model circuit breaker: repeated failures open it and the model is skipped until a cooldown allows a trial. Health comes from model observability, and a model with no observed calls ranks as neutral, not as unhealthy.
Policies are immutable numbered versions with a content hash, published with a compare-and-swap, and the catalogue is part of the hashed content, so a decision is reproducible from the policy version and the health snapshot. Routing is opt-in and additive: it does not change what /v1/llm/complete does.
Where you reach it
- HTTP
POST and GET /v1/model-routing; GET /v1/model-routing/{name}/versions; POST /v1/model-routing/decide; POST /v1/model-routing/outcome
Limits
- Routing decides; it does not call the model. Your application sends the request to the chosen provider and reports the outcome.
- Health is built from model observability, which covers /v1/llm/complete calls alone, so it is a partial picture.
- Circuit-breaker state is held in memory per project and model, and every breaker reads closed after a restart.
- Model cost and context figures in the catalogue are authored by the policy writer, not looked up live.
- Publishing needs an admin or service credential. Decide and outcome are open to any authenticated caller for their own project policies.
Evidence
- On this siteAI features catalogue
- Tests23 policy tests and 5 route tests cover each filter, the ranking and tie-break, denial, the breaker state machine, the fallback bound and versioning
Test coverage is listed as present in the engine repository at this edition. This page does not report test results.
KynetraDB is in private preview
Request a preview to talk through the limits above against your own workload.