AI & agents

Model incident response

Gives operators a governed workflow to detect a failing model, apply a bounded and reversible mitigation, and undo it on resolution.

BetaEngine edition 5f8ea3af

What it does

An incident moves through detected, acknowledged, mitigating and resolved or closed. It can be opened by hand or from an error-rate spike seen in model observability. Mitigations come from a closed set (open a model circuit breaker, add the model to the policy deny-list, or advise exclusion), impactful ones need an explicit approval, and resolving the incident lifts every override it applied.

How it works

POST /v1/model-incident-response opens an incident for a model, either manually or with a detect block that names an error class, a threshold in basis points (default 5,000) and a minimum call count (default 20); the incident opens solely when the observed rate over at least that many calls meets the threshold. The state machine allows detected to acknowledged to mitigating to resolved, with closed available from any non-terminal state. Both end states are terminal.

A mitigation is one of three actions: force a model circuit breaker open, publish a routing-policy version that blocks the model, or add an advisory exclusion that a routing caller may honour. The first two change where traffic goes and are impactful, so they execute solely after the approve call, where the approving administrator is the decision. The advisory one is safe and may apply directly. Deny never executes.

Every step appends to a content-free timeline of ids, event kinds and the affected model. Resolving lifts all overrides the incident applied, and ephemeral overrides are re-applied after a restart while the incident is open. Notifications go to a pluggable sink whose default writes a log line; nothing is sent externally.

Where you reach it

  • HTTPPOST and GET /v1/model-incident-response; GET .../{id}; POST .../{id}/acknowledge, /mitigations, /mitigations/{mid}/approve, /mitigations/{mid}/deny, /resolve

Limits

  • Automatic detection reads model observability, which covers /v1/llm/complete calls alone. A failure spike inside agent runs is not seen, though an incident can always be opened by hand.
  • The default notifier writes to the server log. There is no email, pager or webhook adapter.
  • Circuit-breaker overrides live in memory and are re-driven after a restart from the stored incident, so a mitigation lasts as long as its open incident.
  • All routes need an admin or service credential.

Evidence

  • On this siteAI features catalogue
  • Tests13 domain tests and 14 route tests cover the state machine, spike detection, impactful and safe mitigations, approval and denial, rollback on resolve and restart re-drive

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

Request a preview to talk through the limits above against your own workload.