What it does
evalharness is a command-line tool and library that runs a scenario against an engine, scores retrieval against a published relevance dataset with recall@k, NDCG@k and MRR, and exits non-zero when a threshold fails. A run pins the dataset digest, the prompt version and the corpus lineage state, so the same scenario run twice gives the same result.
How it works
The CLI has five commands: run scores a scenario, compare reports metric deltas between a baseline and a candidate scenario, demo runs a self-contained example, regression compares two prompt versions with everything else fixed and exits non-zero on a regression, and regression-demo shows a comparison that holds and one that regresses. run can also write a benchmark artifact that conforms to the capability-ledger schema.
A scenario names an engine source, either an inline fixture (a small corpus and judgements built into a temporary engine) or the path of an existing engine directory, plus a run configuration: project, retrieval mode (vector, BM25 text or hybrid fusion), queries with text or vectors, metric keys and thresholds, and pins. Retrieval calls the engine search, vector search or hybrid search directly.
Metrics follow the standard definitions over graded judgements. Recall@k divides relevant results in the top k by the relevant results in the judgement set, and queries with no relevant candidate are excluded rather than counted as zero. NDCG@k uses a gain of 2 to the grade minus 1 and a log2 discount. MRR is the reciprocal rank of the first relevant result. All three are pure functions with no randomness or clock.
Reproducibility comes from pinning. The run pins the relevance dataset by id, version and digest (a wrong digest is refused), the prompt by family, version and content hash, and the model configuration; the run id is a canonical hash of that configuration, and each retrieved candidate carries its lineage pin. Text queries are embedded with a deterministic local hash embedder, so the default path makes no network call.
The regression command derives a baseline run and a candidate run from one configuration that differs solely in the prompt pin, applies a per-metric tolerance (the candidate must stay at or above the baseline minus an allowed drop) and optional budgets, and reports a verdict. Latency, tokens and cost are reported as declared and not measured when the evaluation makes no model call, never as zeros.
Where you reach it
- CLI
evalharness run --scenario eval.json [--out result.json] [--benchmark-out bench.json] - CLI
evalharness compare --baseline base.json --candidate cand.json - CLI
evalharness regression --scenario regression.json
Limits
- The harness calls the engine in process, so it measures what the engine returns, not the HTTP layer.
- The local hash embedder is deterministic but not semantic. For a meaningful semantic evaluation, supply query vectors produced by your real model.
- Recall, NDCG and MRR are the implemented metrics.
- The evaluation makes no model call. The prompt pin in a regression run is recorded but does not change retrieval, so on this path the regression command reports a zero delta by construction, and a test asserts it. It becomes a real prompt comparison once an evaluation path calls a model, which none does today.
- Judgements must already exist as a published relevance dataset.
Evidence
- On this siteAI features catalogue
- Tests30 integration test functions cover hand-computed metrics, stable run ids, bit-identical repeat runs, pin refusal, threshold exit codes, project isolation, the regression verdict and tolerance rules; 5 in-module tests cover the metrics
Test coverage is listed as present in the engine repository at this edition. This page does not report test results.
KynetraDB is in private preview
Request a preview to try this against your own workload.