Search & retrieval

Labelled relevance datasets

Keeps the judgements that retrieval quality is measured against as versioned, content-hashed data next to the corpus.

Available in previewEngine edition 5f8ea3af

What it does

A relevance dataset holds graded judgements linking a query id to a candidate id, split assignments, provenance on every judgement and adjudication records where assessors disagree. A draft is edited freely; publishing freezes it into an immutable numbered version with a deterministic digest that an evaluation run can pin.

How it works

A dataset fixes its grade scale at creation: binary (0 or 1), or graded from 0 to a maximum between 1 and 100. Each judgement records the query id, candidate id, grade, assessor id, how it was produced (human, model or heuristic), a caller-supplied timestamp and an optional note. Grades outside the scale are refused. Several assessors may judge the same pair. A pair with more than one distinct grade is a derived conflict, and an adjudication resolves it to a single grade, the latest adjudication for a pair winning. An unresolved conflict leaves the pair out of the effective set, but the conflict stays visible through the conflicts route.

A query belongs to at most one split (for example train, dev or test); reassigning moves it. Editing happens on one mutable draft per dataset through judgement appends, a bulk import that replaces the draft, split assignment and adjudication. The import body has the same shape as an export, so an export round-trips into a fresh dataset. Publishing takes a snapshot of the draft and appends an immutable version whose number is assigned by the server with a compare-and-swap, so two concurrent publishes cannot mint the same number.

The digest is a canonical hash of the content alone, with objects key-sorted and judgements put in canonical order, so identical judgements give identical digests regardless of insertion order or dataset id. Export of a version is deterministic and can be limited to one split with ?split=. Everything is stored under project-prefixed kinds. Management needs an admin or service credential; reads need any authenticated project credential.

Where you reach it

  • HTTPPOST /v1/relevance, GET /v1/relevance/{dataset}, POST .../judgements, POST .../import, PUT .../splits, POST .../adjudications, GET .../conflicts, POST .../publish, GET .../versions, GET .../versions/{version}/export

Example

Create a graded dataset and add a judgementbash
curl -s "$KYNETRA_URL/v1/relevance" \
  -H "Authorization: Bearer $KYNETRA_API_KEY" -H "Content-Type: application/json" \
  -d '{ "id": "support-faq", "name": "Support FAQ", "scale": { "kind": "graded", "max": 3 } }'

curl -s "$KYNETRA_URL/v1/relevance/support-faq/judgements" \
  -H "Authorization: Bearer $KYNETRA_API_KEY" -H "Content-Type: application/json" \
  -d '{ "judgements": [
        { "query_id": "q-refund", "candidate_id": "doc-12", "grade": 3,
          "assessor_id": "assessor-1", "method": "human" }
      ] }'

Limits

  • Provenance is recorded, not verified. The platform stores who or what produced a judgement but cannot attest it.
  • There is no judging interface; judgements arrive through the API.
  • Published dataset size limits are not documented, and a dataset is stored as one draft entity and one entity per version, so very large judgement sets should be sized before use.
  • Read routes accept any authenticated project credential, so treat the judgements as readable by every holder of a project key.

Evidence

  • On this siteHTTP API reference
  • Tests16 store tests and 2 route tests cover digest determinism, immutable versions, concurrent publish, grade-range validation, conflict derivation, adjudication, splits and export

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

Request a preview to try this against your own workload.