Durability & recovery

Group commit and write durability

Lets an application choose whether a write is acknowledged after it is on stable storage, while concurrent writers share the cost of each flush.

Available in previewEngine edition 5f8ea3af

What it does

Writes from many clients are queued to one writer, folded into batches and flushed once per batch. In the default strict mode a caller is answered after the flush, so an acknowledged write has reached stable storage. One server setting turns flushing off for workloads that accept losing recent writes in a crash.

How it works

Callers send writes to one writer thread through a bounded queue of 4,096 pending requests; beyond that, callers wait. The writer drains up to 512 requests into a batch and runs four steps. It validates each request against the current state plus the requests staged earlier in the same batch. It appends the whole batch to the log in one write. It applies the batch to memory in sequence order. Then it replies to every caller. Because staging, appending and applying all happen on that one thread, two concurrent writes to the same key cannot reach memory in the opposite order to their log positions.

In strict durability the flush happens inside the append step, once per batch, and replies are sent afterwards. The server exposes this through a single setting, KYNETRA_SYNC. When it is true, which is the default, the engine runs strict. When it is false the engine never flushes explicitly: acknowledged writes can be lost in a crash, up to whatever the operating system had buffered.

The engine library also has an interval mode that flushes at least every configured number of milliseconds, and a choice of flush primitive. On macOS the primitives range from fsync(2) to F_BARRIERFSYNC to F_FULLFSYNC, and the default is the strongest. On other operating systems every setting maps to sync_data. The server does not expose either setting; it runs strict with the default primitive, or no explicit flush.

The engine tracks the highest sequence number it has actually flushed, separately from the highest it has appended. Checkpoints and index snapshots are never stamped above the flushed position, so after a crash a derived index cannot claim to contain writes that the log no longer holds.

Measured: durable writes at 189,065 operations/second with 64 concurrent clients, p99 of 1.299 ms, in strict group-commit mode acknowledging after fsync(2) — against a declared gate of p99 under 1.5 ms. This is local scope, one host (Apple M5, 32 GB RAM), one operator, no third-party run. A durability number means nothing without its sync primitive. This figure is os_fsync — fsync(2), the primitive Postgres and SQLite default to on macOS. The stronger F_BARRIERFSYNC (110,464/s) and F_FULLFSYNC (14,005/s) modes are in the same artifact and are slower.

Readers do not wait for the writer. Each read pins an immutable root of the in-memory state. Measured in local scope on a single host: point-read p99 under a 32-writer storm stays within 2x of the quiescent read p99.

Where you reach it

  • ConfigKYNETRA_SYNC=true|false (strict acknowledgement, or no explicit flush)
  • LibraryEngineConfig durability: Strict | IntervalMs(n) | None, and wal_sync_mode (engine library, not wired to the server)

Limits

  • With KYNETRA_SYNC set to false, a crash can lose acknowledged writes.
  • Interval durability and the flush-primitive setting exist in the engine library but cannot be configured through the server.
  • The published throughput figure used fsync(2) on macOS. The server uses a stronger default flush on macOS, which the same artifact shows is slower, and no figure for Linux hosts has been published.
  • One writer thread serialises all writes on an instance, so write throughput on one instance is bounded by that thread.

Evidence

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

Every figure on this site is local-scope: one host, one operator, no third-party reproduction. Artifacts are linked next to each number, and gates that were missed are published beside the gates that were met.

KynetraDB is in private preview

Request a preview to try this against your own workload.