Deployment & operations

Retrieval objectives, burn rate and alert webhooks

Turns retrieval quality and freshness signals into error budgets and burn rates, and sends a webhook when a budget is being used up too fast.

BetaEngine edition 5f8ea3af

What it does

GET /admin/db/slo/retrieval reports ten default objectives, such as availability, latency, recall, index and source freshness, saturation and policy denial, each with an error budget and a burn rate. A background task can post an incident to a webhook when a burn rate crosses a threshold in two windows. A simpler error-rate alert posts when the share of 4xx and 5xx responses exceeds a threshold. Three objectives have no data source, and recall needs an external feed.

How it works

Every objective reduces to good and bad observations: a request that is not a server error, an evaluation run whose recall met the target, a gauge sample inside its bound. The registry turns that into an error budget, the share of the budget consumed over a long window, and a burn rate over a short recent window. History is a ring of up to 4,096 observations per objective and scope, and the server writes it to a file so burn state survives a restart.

Six objectives are fed automatically from the server: retrieval latency, retrieval availability, index freshness, source freshness, saturation and policy denial. Recall at 10 is push-fed: the evaluation harness, or any producer, posts to /admin/db/slo/retrieval/eval. Embedding backlog, stream lag and cache hit rate have no source yet and report not applicable. The report is available to platform super administrators.

The burn-alert task polls on an interval set by KYNETRA_SLO_BURN_CHECK_INTERVAL_SECS. An objective trips when its burn rate is above the threshold in the short window and in a longer confirmation window, so a blip that has already stopped does not page. It posts a JSON incident to KYNETRA_SLO_BURN_WEBHOOK_URL, or to KYNETRA_ERROR_WEBHOOK_URL when that is unset, and repeated firing for one ongoing incident is held back by a cooldown. GET /admin/db/slo/retrieval/alerts shows whether a webhook is configured, the threshold and the recent deliveries, including attempts the receiver did not accept.

The error-rate monitor is separate. It polls the request counters and posts an alert to KYNETRA_ERROR_WEBHOOK_URL when the error share exceeds KYNETRA_ERROR_THRESHOLD_PCT.

Where you reach it

  • HTTPGET /admin/db/slo/retrieval
  • HTTPPOST /admin/db/slo/retrieval/eval
  • HTTPGET /admin/db/slo/retrieval/alerts
  • ConfigKYNETRA_ERROR_WEBHOOK_URL, KYNETRA_ERROR_THRESHOLD_PCT, KYNETRA_ERROR_CHECK_INTERVAL_SECS
  • ConfigKYNETRA_SLO_BURN_WEBHOOK_URL, KYNETRA_SLO_BURN_THRESHOLD, KYNETRA_SLO_BURN_CHECK_INTERVAL_SECS, KYNETRA_SLO_BURN_DEDUP_SECS

Limits

  • Three of the ten default objectives (embedding backlog, stream lag and cache hit rate) have no data source in this edition. Recall appears once something posts evaluation results to the eval route.
  • The error-rate alert is labelled as a five-minute rate in its payload, but it compares lifetime request counters and not a rolling window. A long-running instance dilutes a fresh error burst.
  • Alert delivery is one POST per firing. A failed delivery is recorded and the cooldown still advances, so this is not durable retry scheduling. The payload is not signed.
  • History is bounded per objective, and the number of distinct project scopes is capped.
  • The reports and the alert state need platform super administrator authority.

Evidence

  • On this siteDocumentation hub: operations
  • TestsCovered in the engine repository by 22 objective and burn-rate model tests, 11 webhook and burn-alert tests, 21 server-side objective tests, and HTTP tests for incident firing, a known-incident replay and burn state surviving a restart

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

Request a preview to talk through the limits above against your own workload.