AI & agents

LLM provider layer

Puts local and hosted language models behind one completion interface with ordered fallback and token accounting.

BetaEngine edition 5f8ea3af

What it does

POST /v1/llm/complete sends a chat request to a named provider, or to the first provider in the fallback chain that answers. Three backends are built in: a local Ollama server, the Anthropic Messages API and any OpenAI-compatible chat endpoint. The response carries the provider, model, stop reason and, when the provider reports it, token usage. GET /v1/llm/providers lists what is configured.

How it works

The provider registry is built at boot from the environment. An Ollama provider is always registered (OLLAMA_BASE_URL, OLLAMA_MODEL), an Anthropic provider when ANTHROPIC_API_KEY is set (ANTHROPIC_BASE_URL, ANTHROPIC_MODEL) and an OpenAI-compatible provider when OPENAI_API_KEY is set (OPENAI_BASE_URL, OPENAI_MODEL), which covers any server that speaks the chat-completions protocol. The registration order is the fallback order: Ollama, then Anthropic, then the OpenAI-compatible endpoint.

A request carries messages with a role and content, and optionally provider, model, max_tokens (default 1024), temperature (default 0.2) and an action label used for model-usage attribution. With provider set, that provider alone is called and there is no fallback. Without it, providers are tried in order until one succeeds, and the number of attempts and whether a fallback happened are recorded. The response body is a completion object with text, tool calls, stop reason, model, provider and usage.

The adapters translate a common message and tool model into each provider wire format, including tool definitions with JSON-schema inputs. Token usage is read from each provider response when present; a provider that does not report it yields no usage, and downstream cost stays unknown rather than being estimated.

Calls through this route are recorded as metadata (project, action, provider, model, tokens, latency, retries, error class), never prompt or completion text; see the model observability entry. The same registry backs the agent runtime and agent memory consolidation.

Where you reach it

  • HTTPPOST /v1/llm/complete, GET /v1/llm/providers, GET /v1/llm/observability
  • ConfigOLLAMA_BASE_URL, OLLAMA_MODEL, ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, ANTHROPIC_MODEL, OPENAI_API_KEY, OPENAI_BASE_URL, OPENAI_MODEL

Example

Ask a named provider, with an attribution labelbash
curl -s "$KYNETRA_URL/v1/llm/complete" \
  -H "Authorization: Bearer $EDITOR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "ollama",
    "messages": [
      { "role": "system", "content": "Answer in one sentence." },
      { "role": "user", "content": "What is a write-ahead log?" }
    ],
    "max_tokens": 200,
    "action": "docs.explain"
  }'

# response: { "completion": { "text", "tool_calls", "stop_reason", "model", "provider", "usage" } }

Limits

  • The route needs a platform credential with the editor role. Project keys and end-user tokens are refused.
  • The adapters talk to the providers over plain HTTP clients with their default settings. There is no per-request timeout, retry with backoff or circuit breaker inside the provider layer; the fallback chain is a simple ordered try.
  • The local Ollama provider is always first when the registry is built from the environment, so an unreachable local server costs an attempt before a hosted provider is tried. Name a provider to skip the chain.
  • Tool use is supported by the adapters, but the completion route sends no tools. Tool loops are the agent runtime, whose multi-turn behaviour has been tested with scripted providers and with nothing else.
  • Provider errors are returned to the caller as bad-request responses, including the provider text.
  • The tests in the engine repository use scripted providers; no automated test calls a live hosted provider.

Evidence

  • On this siteAI features catalogue
  • Tests13 test functions cover fallback ordering with attempt and retry attribution, token pricing for each provider family, and the embedding providers; the observability hook has 4 route-level tests

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

Request a preview to talk through the limits above against your own workload.