AI & agents

Model usage observability

Shows what language-model calls cost and how they behave, by project, action, provider, model and run, without storing prompts or answers.

BetaEngine edition 5f8ea3af

What it does

Every completion made through POST /v1/llm/complete is recorded as metadata: who called, which action label, which provider and model, tokens when the provider reported them, latency, retries, fallback and error class. GET /v1/llm/observability returns a tenant view and GET /admin/db/model-metrics returns the fleet view, grouped by model, action, provider, project or run.

How it works

The record type has no field for a prompt or a completion, so content cannot reach the store or the API by construction. A test drives a completion with a marker string in both prompt and answer and checks that the marker appears in no serialised field.

Tokens are present when the provider reports them and absent otherwise, and a missing count is a typed not-available state, never zero. Cost is derived from tokens and the built-in price table and is also not-available when tokens are. Latency is always measured at the call site. Cache, region and redaction have no source on this path and are reported as explicit not-available states. Retries count the providers that failed before the one that answered. A failed call is classified as timeout, rate_limit, provider_error, invalid_request, network, configuration or other, and the aggregates count each class.

History is an in-process ring of at most 4,096 calls per project, dropping the oldest. The tenant route takes its scope from the credential and ignores any project query parameter; the admin route may filter by project. The ?by= parameter accepts model, action, provider, project and run.

The model routing and incident response capabilities read the same aggregates.

Where you reach it

  • HTTPGET /v1/llm/observability?by=model|action|provider|project|run
  • HTTPGET /admin/db/model-metrics?by=...&project=

Limits

  • The /v1/llm/complete call site is the one instrumented. Completions made inside the agent loop, streaming runs, memory consolidation and evaluation runs are not recorded, so these metrics cover HTTP-originated calls.
  • History is held in memory, is bounded, and starts empty after a restart.
  • Cost is an estimate from a built-in price table; use token totals for budgeting.
  • The tenant route needs a platform credential with the editor role, and the fleet route an admin role.

Evidence

  • On this siteAI features catalogue
  • Tests9 aggregation tests cover per-dimension sums, honest not-available states, bounded dimension strings and project isolation; 4 hook tests cover cost derivation from tokens, failed calls and the no-content property

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

Request a preview to talk through the limits above against your own workload.