AI & agents

Vector schemas and embedding freshness

Refuses to mix incompatible embeddings in one index and refuses to serve vectors that are older than the data they describe.

RoadmapEngine edition 5f8ea3af

What it does

Two registries are implemented: a model-aware vector schema per index (dimension, metric, embedding family, normalisation, tokenizer, version) with a controlled reindex state machine, and an embedding freshness watermark per project and model that reports fresh, stale or unknown. Memory recall consults both. No route declares a schema or advances a watermark, so users cannot configure them yet.

How it works

A declared schema validates every vector write and query before any index change: a vector of the wrong width or a query for another metric is refused with a typed error. A redeclaration that changes any of the version, dimension, metric, family, normalisation or tokenizer is refused; the sanctioned path is begin reindex, then complete or roll back. While a reindex is in flight, both writes and queries for that index are refused, so mixed-schema access cannot happen.

The freshness registry tracks, per project and model, the committed sequence through which source data has been observed and the sequence through which the index was rebuilt. It reports fresh when the index covers every observed source revision, stale when source changes are past the watermark and unknown when no watermark exists. A fail-closed check refuses retrieval unless the watermark is at least the required one, and a store read error is its own typed answer, never read as unknown or fresh.

In the stock server the memory write path marks source changes in the same commit as the write, and memory recall performs the freshness and schema checks. Nothing sets a watermark or declares a schema, so a recall without allow_stale reports an unknown freshness state and is refused.

Where you reach it

  • LibraryVectorSchemaRegistry and FreshnessRegistry (Rust crate kynetra-server; exercised through /v1/memory/recall)

Limits

  • Not configurable through any route, command or setting of the private-preview server.
  • The registries are enforced for the memory kinds and for nothing else. The generic vector routes do not consult them.
  • Reindex completion is a declaration by the caller; it is not tied to the embedding jobs capability.

Evidence

  • Tests18 schema tests and 16 freshness tests cover the redeclare fence, the reindex state machine, rows planted in storage, fail-closed reads, bounded retries and watermark validation against the log

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

This is not available in the preview. Request a preview to discuss the current scope.