KynetraDB architecture

One log, many views.

KynetraDB writes every change to one append-only log and derives its record store, its full-text index and its vector search from that log. This page describes the engine as it stands in its main branch: what each layer does, how you reach it, and where it stops.

Source of truth
One append-only log of puts, deletes and transactions.
Writer
One thread. Many callers share each fsync.
Default storage
Local segment files. No external database.

Overview

A write goes to the log first, and the views follow

A write is validated, appended to the log, applied to the engine’s in-memory state and then published as a new snapshot. Reads, search and vector queries run against that state. The record store and the search indexes are derived from the log, and are rebuilt from it when a snapshot is missing or damaged.

KynetraDB architectureClients call an HTTP server. The server calls the engine, which has a single writer thread and snapshot reads. The engine appends to the Universal Log, which is kept in a storage backend: local files, Postgres, an object layout limited to local files, or a roadmap Cloudflare D1 backend. Projections on the right are derived from the engine state. Read replicas stream records from the log.Reachable in this buildRoadmap or library codeClientsHTTP and REST · server-sent events · WebSocket · command line · MCPHTTP serverPrincipal · rate limits · role overlayProject scope and rules, on the routes that apply themEngineOne writer thread: validate, append, apply, publishGroup commit: many callers share one fsyncSnapshot roots: readers pin a root and never wait for writesUniversal LogAppend-only segments · CRC-32 frames · checkpoints beside the logStorage backends: where the log is keptFiledefaultPostgresbetaObjectslice 1, local filesCloudflare D1roadmapProjectionsDerived from the engine stateRecords · filter · KynQLFull text (BM25)Vector (exact) · hybridANN · aggregates · viewsapplyRead replicasPull records over HTTP, own logstream, asynchronous

Solid outlines are reachable through the server in this build. Dashed outlines are roadmap or engine library code. The same labels as the capability catalogue are used throughout this page.

Text version of the diagram
  1. Clients reach the HTTP server over HTTP and REST, server-sent events, WebSocket, the command line and MCP.
  2. The server resolves the caller, applies rate limits and role checks, then project scoping and collection rules on the routes that apply them, and calls the engine.
  3. The engine has one writer thread. It validates a batch, appends it to the Universal Log, applies it to state and publishes a new snapshot. Readers use the published snapshot.
  4. The Universal Log is a set of append-only segment files with checkpoints beside it.
  5. The log is kept in a storage backend: local files by default, Postgres (beta), an object layout limited to a local file store in its first slice (roadmap), or Cloudflare D1 (roadmap).
  6. Projections are derived from the engine state: records, full-text search, exact vector search and hybrid fusion. Approximate vector indexes, aggregates and views exist as library code and have no route.
  7. Read replicas pull records from the log over HTTP and keep their own copy.

The Universal Log

A segmented, checksummed, append-only log

Every write becomes a record with a sequence number, a timestamp and one operation: put an entity, delete an entity, or commit a transaction. Nothing else changes the data. The glossary names the guarantees that hang off this design, such as Tideline.

The log is a directory of segment files. Each segment has a header and a run of frames, and each frame carries its length, a CRC-32 checksum, a flags byte and the payload. Payloads above a size threshold are compressed with zstd. When the active segment reaches its size limit (128 MiB by default) it is sealed and a new one begins. Within a segment sequence numbers are contiguous.

The engine can encrypt segment payloads with AES-256-GCM and rotate the key. That is an engine option: the server has no setting that turns it on, so a stock server does not encrypt the log. Use disk or volume encryption, and see Security.

What the log does when something goes wrong

SituationBehaviour
A crash leaves a torn write at the end of the last segment.Open truncates the log at the last frame whose checksum verifies, and carries on.
A checksum fails in an earlier, sealed segment.Open fails with a corruption error. The engine does not skip the segment, because skipping it would lose data without telling you.
A write fails before anything was made durable, for example a full disk.The torn tail is cut back to the last complete frame and the log stays writable.
An fsync fails.The log is poisoned. It refuses later appends and syncs until it is reopened, because the state of the page cache is unknown and a retried fsync can report success falsely. Reads keep working.
A checkpoint file is missing or fails its checksum.The engine replays the log from the last good point. A checkpoint speeds up boot; it is never the sole copy of the data.

Covered by the WAL-format, crash-recovery, write-failure and checkpoint tests in the engine repository. TheReliability page describes what review caught that those tests missed.

Durability modes

Strict (the default)
The write is acknowledged after the segment is flushed to stable storage. Writes that arrive together share one flush.
Interval
The write is acknowledged once handed to the operating system and flushed on a timer. A crash can lose the last interval. This is an engine option.
None
No explicit flush. A crash can lose anything the operating system had buffered. The server’s KYNETRA_SYNC=false selects this.

Checkpoints and boot

Every so many writes (10,000 by default) the engine writes a checkpoint: a full base file, then a chain of small delta files that hold just what changed. On boot it loads the chain, replays the log after it, and is ready. If a delta is missing or fails its checksum, the engine falls back to replaying the log; the deltas it cannot link are ignored, never half-applied.

History can be rewound. A point-in-time restore or a delete-all records a durable intent before it touches the log, and keeps an undo copy. If one is interrupted, the engine refuses authoritative reads and writes with a typed recovery_uncertain 503 until the intent is resolved, rather than serve a state it cannot vouch for.

The engine

One writer, group commit, snapshot reads

One thread owns the write path. It drains the queue into batches and takes each batch through four steps, so two concurrent writes to the same key can never apply in the opposite order of their log positions.

  1. Validate and stage. Conditions, idempotency keys and guards are checked against the published state plus the writes already staged in this batch.
  2. Make durable. One append, and in strict mode one flush, for the whole batch.
  3. Apply. The records are applied to state in log order and a new snapshot is published with one atomic swap.
  4. Resolve. Callers are answered after publication, so a caller that re-reads after a conflict sees the write it lost to.

State is a persistent ordered map behind an atomic pointer. A reader pins the current snapshot with one atomic load and sees a consistent point in time for as long as it holds it, without coordinating with the writer. Reads therefore do not wait for writes.

Transactions

Over HTTP, POST /v1/transactions commits up to 256 mutations (puts and deletes) guarded by up to 256 conditions (exists, missing, attribute equals) as one atomic record in the log. An optional idempotency key makes a retry return the original receipt instead of applying twice, and reusing a key for a different request is refused.

The engine library also offers interactive transactions with snapshot isolation and first-committer-wins conflict detection. The server uses them internally; no public route exposes them in this build.

Projections

Query shapes are views over the same state

Projections, which the engine calls personalities, are read paths. They never have a write path of their own: a write goes to the log and the views follow. Some are reachable through the server; some exist as engine library code and nothing more.

ProjectionWhat it servesReach it throughState in this build
Record storeGet by id, filter on attributes, typed collections with rules.The default entity profile keeps every entity in memory, in a persistent ordered map.POST /v1/entities and /v1/filter (service credential); /rest/v1 (rules and policies apply); /v1/collections records (role lists apply)Available in preview
KynQLA bounded query language over one collection: where, select, sort, limit, offset.It cannot write, join or aggregate. It reads the collection in memory, so cost grows with collection size.POST /v1/queryAvailable in preview
Full text (BM25)Ranked keyword search over the text values of every entity.A segmented inverted index with MaxScore pruning. Its snapshot is stamped with a log position; a missing or damaged snapshot is rebuilt, not trusted.POST /v1/searchAvailable in preview
Vector search (exact)Cosine similarity over the embeddings you store with entities.On the server this is an exact scan of every entity that has an embedding, so recall is complete and cost grows with the number of embedded entities. The route needs a service credential, so call it from your server.POST /v1/vectorBeta
Hybrid searchOne ranked list from the keyword and vector results, with optional metadata filter.Weighted reciprocal-rank fusion, so BM25 scores and cosine scores never have to be compared directly.POST /v1/hybridBeta
Approximate vector indexesHNSW graphs in memory, SPANN on disk, int8, binary and RaBitQ-style compression.Built and benchmarked in the engine library. The server never builds one and has no setting that does, so the stock server does not use them.Engine library codeRoadmap
Aggregates, existence index, materialised viewsCount, sum, min, max and average; a per-collection Bloom filter; refreshed group-by views.No server route, CLI command or query language reaches them in this build.Engine library codeRoadmap

The /v1/views routes are a different feature: versioned, typed views over collections. Measured results for keyword and vector search are on Benchmarks; the vector figures there were taken on the engine library, not on the stock server’s exact-scan path.

Storage backends

Where the log is kept

The engine talks to its log through one storage interface. The default needs nothing but a directory. The others change where the log is durable, not what the engine does with it.

File

Available in preview
Where the log lives
Segment files in a directory on a persistent volume, with checkpoints and search snapshots beside them.
How it is selected
The default. KYNETRA_STORAGE_BACKEND=file.
Know before you use it
Needs a persistent volume and a backup plan. The tiny-vps runtime profile requires this backend.
Where the log lives
One row per log record in a table you name. A database-held writer lease admits one primary; replicas open the same table without write access.
How it is selected
KYNETRA_STORAGE_BACKEND=postgres with KYNETRA_PG_URL. Optional pgvector offload of vector search.
Know before you use it
The product’s own deployment-fit check lists this mode as configuration-ready preview with no measured durability claim. The connection uses TLS when the URL asks for it or the host is a known managed provider, and otherwise does not. Its integration tests run when a test database URL is provided and are skipped otherwise.
Where the log lives
The local log is a write-through cache. Each sealed segment is uploaded as one object, and a cold boot restores from the store.
How it is selected
An engine option. No environment setting of the server selects it.
Know before you use it
Slice 1 supports a local file:// store alone. There are no cloud object stores, no checkpoints in the store and no store-side cleanup yet.

Cloudflare D1

Roadmap
Where the log lives
Not available.
How it is selected
Not selectable.
Know before you use it
A D1 backend is in development and is not in the engine’s main branch. It is roadmap work, with no date.

File blobs, such as uploads, are separate from the log: they go to a local directory or an S3-compatible store, and just their metadata is in the log. The server accepts file or postgres for the storage setting. Any other value falls back to the file backend, so check the start-up log line that names the backend. Capability pages:PostgreSQL backend ·Object-storage backend ·Encrypted log ·Disk-resident entity store

Replication and failover

One primary, read replicas, manual promotion

There is one writer per log. Replicas add read capacity and a warm copy. They do not add write capacity, and in the supported path nothing promotes a replica for you.

MechanismHow it worksLimitState
Streaming read replicasA replica started with KYNETRA_ROLE=replica and KYNETRA_PRIMARY_URL pulls log records from the primary over HTTP, writes them unchanged into its own log, then applies them. A restarted replica resumes where it stopped. Writes sent to a replica are forwarded to the primary.Asynchronous, so a replica can lag. A replica compares its tail with the primary’s before it continues, and a node whose log differs takes itself out of service until it is reseeded by hand. Nothing promotes a replica automatically.Beta
Shared-log replicas (Postgres)With the Postgres backend, replicas poll the same log table and apply what the primary wrote.Carries every limit of the Postgres backend above.Beta
Semi-synchronous quorum commitThe primary pushes each batch to configured peers and waits for a set number of acknowledgements. If the quorum is not reached in time, the client receives a 503 marked indeterminate.An indeterminate write is durable on the primary and may survive a failover, so treat it as unknown, not as failed. Promotion is an operator action: restart the follower with the highest applied position as the primary.Beta
Raft transportAn opt-in Raft message transport and consensus-gated acknowledgement, enabled with KYNETRA_RAFT_ENABLED.The code describes itself as a prototype that is independent of the primary and replica path. It cannot run on the Postgres backend. Do not plan around it.Research

Automatic failover is not available

An earlier automatic leader election was retired after proof testing found safety defects in it, and the server now refuses to start with its configuration. Semi-synchronous replication has no automatic failover by design. Replication between nodes can use mutual TLS when you configure it; without that configuration it sends the instance service key as a bearer token over whatever transport the primary URL names, so use https or a private network.

Request path

What happens between the socket and the engine

The stages run in this order, outermost first. The server listens on plain HTTP. Put TLS termination in front of it, on a reverse proxy or your platform’s load balancer.

  1. Correlation and slow-query logging

    Every request gets a correlation id. Slow queries are logged against it.

  2. Principal resolution

    The credential is verified once and attached to the request: the global API key, a project service or publishable key, a project user token, or a platform token.

  3. Metrics

    Per-route counters feed the /metrics endpoint and the per-route SLO tracker.

  4. Admission

    Fixed-window rate limits by class (authentication, administration, writes, everything else) and by project, plus the security response headers.

  5. Role overlay

    If a user has any custom role binding, mapped routes need an explicit grant.

  6. Body limit, CORS and compression

    Request bodies are capped at 64 MiB. Responses are compressed.

  7. Mount layers

    Usage metering; on a replica, forwarding of mutating requests to the primary and the freshness gate; and the read-consistency budget on /v1.

  8. Handler

    Project scoping of kinds and ids on the routes that carry a project, and a check that refuses with a typed 503 while the engine is recovery-uncertain. The raw /v1 routes (entities, filter, search, vector, hybrid, transactions) need a service credential and apply no row-level policy. /rest/v1, KynQL and typed views apply collection rules and row-level policies to end-user callers, the record routes under /v1/collections apply role lists alone, and the WebSocket applies row-level policies per subscriber. The engine call then runs off the async threads.

  9. After the write

    The change is published to realtime subscribers and, when mirror sinks are enabled, fanned out to them without waiting.

The routes themselves are listed on API; the machine-readable description is served by the instance at /openapi.json.

Multi-tenancy

Tenants are projects, separated where the surface enforces it

Organisations own projects, and a project is a tenant. On the project-facing surfaces, tenant data is addressed under a per-project prefix and each credential carries its project, so a project’s calls act on that project’s data. A project credential also cannot act as a platform administrator, whatever role it carries.

Projects have a lifecycle of six states (provisioning, active, paused, suspended, deprovisioning and purged), a per-project request budget, a per-tenant export with credential material removed, and a purge. Per-project entity quotas exist in the data model, but no API in this build sets them. The Multi-tenancy page covers lifecycle, export and purge, including exactly what a purge does not do.

That separation is not universal. Platform accounts and the global key are operator credentials with deployment-wide authority; keep them and operator endpoints away from tenants.

Tenants share one process, one log and one set of CPU and memory. A per-project request budget bounds a noisy tenant; it does not give each tenant its own machine, its own encryption key or its own failure domain.

Limits

What this architecture does not do

  • It is not a distributed database. One writer thread owns each log. There is no sharding in the server’s request path, no cross-node serialisability and no multi-region active-active mode.
  • Memory and disk scale with data. The server runs the in-memory entity profile. A disk-resident entity store exists in the engine library and is not selectable on the server. In this profile the server does not trim the log, so disk use grows with write history.
  • No SQL wire protocol. Clients speak HTTP. KynQL is a bounded query language that cannot write.
  • No encryption at rest from the server. Use volume encryption. Backup bundles can be encrypted and signed by the backup command.
  • Failover is yours to run. See the replication table above.

Roadmap

In development, or in the library but not wired

None of the following is reachable through the server today, and none carries a date.

  • A Cloudflare D1 storage backend is in development and is not in the engine’s main branch.
  • Approximate vector indexes (HNSW, SPANN and the compressed codecs) exist as engine library code with published benchmark artifacts. The server does not switch them on.
  • Aggregates, the existence index and materialised views exist as engine library code with no route.
  • Encrypted log segments with key rotation exist as engine library code with no server setting.
  • The object-store layout is limited to a local file store in its first slice.

Scope an evaluation against your workload

KynetraDB is in private preview. Tell us the shape of your data and your recovery needs, and we will scope what the current build covers.