AI & agents

Agent evaluation suites

Checks that an agent definition, its tool allow-list and its approval behaviour still work, as a gate in a build.

BetaEngine edition 5f8ea3af

What it does

agentevals runs a versioned suite of agent tasks against a deterministic scripted model in a throwaway engine, scores task outcomes and tool-permission violations, and exits non-zero when a promotion threshold fails. It tests the harness around an agent: allow-lists, approvals, assertions and traces. It does not evaluate a real model.

How it works

The command is agentevals run followed by a scenario file and an optional output path. A scenario holds an optional project, a suite id, the suite specification, a mock configuration and an approval policy. The suite is published as an immutable numbered version using the same compare-and-swap and content-hash pattern as prompt versions, and the run pins that version and the hash of the mock configuration.

Each task declares the tools it may use and the assertions it is scored on. The mock model plays a script per task: a turn is either a set of tool calls or a final answer. Tool calls execute against deterministic stand-in tools in the throwaway engine, so no side effect reaches real data. The approval policy is always approve, always deny, always time out, or a per-action script, which lets a suite prove that a gated action does not run without approval.

A call outside a task allowed tools is recorded as a policy violation and fails the task. The result lists how many tasks met their assertions out of the total, violations, the captured step trace, and honest cost fields: the number of mock calls, with no invented token counts. The process exit code follows the promotion thresholds, so a build can gate on it.

Where you reach it

  • CLIagentevals run <scenario.json> [out.json]

Limits

  • Scripts replace the model, so a suite shows how your allow-lists, approvals and assertions behave, not how a real model behaves.
  • Tools are deterministic stand-ins. A suite does not exercise your real tools or data.
  • The engine is ephemeral: suites and runs live for one invocation.
  • There is no HTTP route; the command line is the surface.

Evidence

  • On this siteAI features catalogue
  • Tests12 harness tests and 2 approval-decider tests cover versioned suites, tool permissions, sandboxing, trace capture, violations, thresholds and the approval simulation

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

Request a preview to talk through the limits above against your own workload.