Data model & realtime

Universal Log

Gives every write one durable, checksummed home, so a crash or a damaged disk block is detected and handled instead of silently corrupting the database.

Available in previewEngine edition 5f8ea3af

What it does

Every write to KynetraDB is appended to a single log made of checksummed segment files. The key-value, document, search and vector views are in-memory state rebuilt from that log, so recovery after a crash is a replay of the log rather than a repair of many files. Damage in the newest segment is treated as a torn write and cut back; damage in an older, sealed segment stops the engine from opening.

How it works

The log is a directory of segment files, each named for the first sequence number it holds. A segment starts with a 48-byte header: a magic value, a format version, the first sequence number, a random 128-bit salt and a key identifier. Records follow as frames made of a length, a CRC-32 checksum, a flags byte and the payload. The checksum covers the flags and the payload, so a torn or bit-flipped frame is detected before its payload is ever decoded. Payloads at or above a configurable size are compressed with zstd; the default threshold is 4,096 bytes.

There is one writer. Callers queue writes to a single writer thread, which groups waiting requests into a batch, appends the batch to the log, applies it to the in-memory state in sequence order and then answers each caller. A segment rotates to a new file once it passes a size limit, 128 MB by default. Sequence numbers are contiguous inside a segment and may have gaps between segments.

On start-up the engine loads the latest checkpoint and replays the log above it. If the final segment ends in a damaged frame, the engine treats it as a torn write and truncates the segment back to the last intact frame. If any earlier, sealed segment is damaged, the engine refuses to open: truncating there would silently discard data that was acknowledged. Adversarial tests flip bytes across segments, swap intact frames between positions and truncate files, and check that the engine either recovers an acknowledged prefix or refuses to open.

A write that fails part-way, for example on a full disk, is rolled back to the last complete frame when that is safe. A failed fsync is handled differently: after it the state of the operating system cache is unknown and a retried fsync can falsely report success, so the log is marked poisoned and refuses further appends, flushes and key rotations until the process reopens it. Reads keep working while a log is poisoned.

Format version 4 is the single version the engine writes. Segments written in version 3 stay readable, and the older single-file log format is migrated into segments the first time such a directory is opened. The file log takes an advisory lock on its data directory when it opens, and the engine library can request that lock exclusively so that offline maintenance is refused while a server holds the directory.

The kynetra command line can read records back from a local data directory by sequence range, either to the terminal, into a fresh engine, or through a simple filter.

Where you reach it

  • ConfigKYNETRA_DATA (data directory that holds the log)
  • ConfigKYNETRA_COMPRESS_WRITES_OVER_BYTES (compression threshold, 4096 by default)
  • CLIkynetra replay --from-seq <n> [--to-seq <m>] [--into stdout|engine|filter]

Example

Read the first twenty records of a local data directorybash
kynetra --data ./.kynetra replay --from-seq 0 --to-seq 20

Limits

  • The log is written by appending. A delete appends a tombstone, and earlier versions of a record stay in retained segments, in replicas and in restore backups.
  • In the default in-memory entity profile the server never trims the log, because the log is the sole durable copy of history. Disk use grows with write history.
  • At-rest encryption of log segments is implemented in the engine library but is not switched on by the server; see the encrypted log entry.
  • The server can place the log on local files or in PostgreSQL. Other locations are described in the storage backend entries.

Evidence

  • On this siteReliability: how recovery findings are tested
  • TestsCovered in the engine repository by 7 WAL format tests (rotation and replay, torn-tail truncation, corruption in a sealed segment, legacy migration), 5 adversarial tests (byte-flip sweep, frame swap, wrong key), write-failure rollback tests, and a crash harness that kills a writing process with SIGKILL and checks every acknowledged write after reopen

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

Request a preview to try this against your own workload.