Search & retrieval

BM25 full-text search

Gives an application relevance-ranked keyword search over its own records without running a separate search cluster.

Available in previewEngine edition 5f8ea3af

What it does

Every entity written to KynetraDB is indexed for keyword search as part of the same write path. POST /v1/search returns the highest-scoring hits ranked by BM25, optionally restricted to one entity kind. The index is a segmented inverted index held in the engine process and snapshotted at checkpoints, so it recovers together with the rest of the database.

How it works

On every write the engine tokenises the entity: the kind name, every string value and object key found anywhere in its attributes, and numbers rendered as text. Tokens are split on Unicode word boundaries, lower-cased and folded to ASCII for common Latin accents, so a search for cafe matches a record that says café. Embeddings and anything outside the attributes are not indexed, and there are no per-field weights.

New documents land in an in-memory table that is sealed into an immutable segment every 65,536 documents. An update tombstones the old document and appends a new one, so postings are never rewritten. Each query is scored with BM25 (k1 1.2, b 0.75) and pruned with MaxScore: per-term upper bounds let the engine skip documents that cannot reach the current top-k, so work follows k and the strongest posting lists rather than the whole corpus. An exhaustive reference scorer with the same arithmetic ships in the engine, and a test checks that the pruned and exhaustive results are identical across segment boundaries.

Persistence follows the log. The index snapshot is written with a checksum and the log sequence it reflects, and it is never stamped ahead of the durable log. At boot the engine loads the snapshot and replays the tail of the log; a missing or damaged snapshot is rebuilt from the entities, not treated as data loss. While the engine is fenced for recovery, the route answers with the typed 503 refusal instead of an empty result list.

The request carries q, top_k (default 24) and an optional kind. Each hit is returned as id, kind and score, after the same read-visibility check that entity reads use. A project service key is confined to its own namespace. When no kind is given the server fetches up to four times top_k, never more than 1,000, and filters to the namespace afterwards, so a crowded corpus can return fewer than top_k hits; give kind for an exact fill.

Measured, local scope, one host (Apple M5, 32 GB RAM): BM25 top-24 query p99 of 2.305 ms over 5,000,000 documents, against a declared gate of 8 ms. The corpus is synthetic, with 12 common, 12 medium and 4 rare tokens per document and a 26.1k vocabulary, and the artifact states that relevance against human judgements remains open. A second run, also local scope on one host, measured top-24 query p50 of 0.025 ms over 100,000 documents at 19,103 queries/second on a single thread; agreement with the exhaustive scorer (NDCG@10) was 0.9983 at that size, which the artifact attributes to tie reordering, and reached 1.0 at 1M and 5M documents.

Where you reach it

  • HTTPPOST /v1/search { q, top_k, kind }
  • SDKSearchClient.bm25 in the TypeScript, Python, Go and Rust client sources
  • MCPkynetra_search (HTTP /mcp and the stdio kynetra-mcp server)

Example

Top five product hits for a phrasebash
curl -s "$KYNETRA_URL/v1/search" \
  -H "Authorization: Bearer $KYNETRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "q": "single origin espresso", "top_k": 5, "kind": "product" }'

# response shape
# { "hits": [ { "id": "...", "kind": "product", "score": <float> } ] }

Limits

  • Keyword matching on exact tokens after case and accent folding. There is no stemming, no phrase or proximity query, no per-field boost and no highlighting.
  • Fuzzy matching and synonym expansion exist as engine library functions, but no route, command or client call reaches them, so they are not part of this capability.
  • Document frequency still counts tombstoned postings until a future merge, so inverse document frequency drifts slightly on corpora with heavy update and delete churn. Scores of live documents stay exact for the postings that exist.
  • The index is held in memory and the server indexes every kind; there is no server setting to narrow it. Plan memory for the whole text of the corpus.
  • Relevance against human-labelled judgements has not been measured on a public corpus. The published artifacts measure latency and agreement with the exhaustive scorer on synthetic data.
  • The route accepts service credentials: the instance key or a project service key. Anonymous project keys and end-user tokens are refused.

Evidence

  • ArtifactBM25 scale run (local scope, Apple M5, 32 GB RAM)
  • On this siteHTTP API reference
  • Tests15 engine test functions cover ranking, case and accent folding, kind isolation, deletion, restart, snapshot damage, pruned-versus-exhaustive agreement and a concurrent write storm; HTTP-level tests cover the mounted route under recovery refusal and tenant scoping

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

Every figure on this site is local-scope: one host, one operator, no third-party reproduction. Artifacts are linked next to each number, and gates that were missed are published beside the gates that were met.

KynetraDB is in private preview

Request a preview to try this against your own workload.