Search & retrieval

Durable streaming retrieval sessions

Lets a producer push retrieval output to a client as it is produced, and lets the client reconnect and resume without losing or repeating events.

BetaEngine edition 5f8ea3af

What it does

A retrieval session is a durable, ordered event log with a numeric cursor. Producers append events, clients read them over server-sent events, and a reconnecting client resumes from the last cursor it saw. Events are ordinary database entities, so sessions survive a restart, and retention is bounded by dropping the oldest events.

How it works

POST /v1/retrieval/sessions opens a session with an optional retention_max (default 1,024 events) and an optional policy expression, and returns the session id, state and cursor. A producer with an editor role or a service key appends events with POST /v1/retrieval/sessions/{id}/events, giving a kind and a JSON payload. Each append is one compare-and-swap transaction that assigns the next cursor, writes the event and prunes the oldest events beyond the retention bound, so the cursor is monotonic even with concurrent producers. A cancelled session is terminal and rejects appends. The store also has completed and failed end states, but no route sets them today.

GET /v1/retrieval/sessions/{id}/events is a server-sent event stream. The event id field is the durable cursor. A client resumes with ?after= or the standard Last-Event-ID header and receives every retained event after that cursor, then the live tail. The live tail is an in-memory broadcast of 256 events; a subscriber that falls behind has its stream ended so that the browser reconnects and the durable log fills the gap. A cursor older than the retained window returns a 409 conflict with an explicit body instead of a silent gap.

Sessions and events are stored under project-prefixed kinds and ids, so a reader in one project cannot name another project session. The session itself does not run a search; whatever produces retrieval results writes them as events.

Where you reach it

  • HTTPPOST /v1/retrieval/sessions, GET /v1/retrieval/sessions/{id}, GET and POST /v1/retrieval/sessions/{id}/events, POST /v1/retrieval/sessions/{id}/cancel

Example

Open a session and append an eventbash
# open
curl -s -X POST "$KYNETRA_URL/v1/retrieval/sessions" \
  -H "Authorization: Bearer $KYNETRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "retention_max": 512 }'

# append (producer side)
curl -s -X POST "$KYNETRA_URL/v1/retrieval/sessions/$SESSION_ID/events" \
  -H "Authorization: Bearer $KYNETRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "kind": "passage", "payload": { "doc": "faq-12", "text": "..." } }'

# read and resume from cursor 40
curl -N "$KYNETRA_URL/v1/retrieval/sessions/$SESSION_ID/events?after=40" \
  -H "Authorization: Bearer $KYNETRA_API_KEY"

Limits

  • Retention is bounded: a consumer that is further behind than the retained window must restart without a cursor.
  • Treat the optional session policy as a filter on replay, not as an access control. Decide who may read a session with project credentials and roles.
  • Payloads are opaque JSON; the platform does not validate their shape or check them against a search.
  • The live tail is an in-process broadcast. Resume after a disconnect relies on the durable log, not on the broadcast.

Evidence

  • On this siteHTTP API reference
  • Tests8 library test functions cover reconnect without duplicates or loss, resume across a restart, an explicit expired-cursor response, bounded retention for a slow consumer, durable cancellation, cross-project isolation and a policy change mid-session; a mounted-route test drives open, append, get and read while healthy and every route under a recovery fence

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

Request a preview to talk through the limits above against your own workload.