Durability & recovery

Disk-resident entity store

Would keep entity values on disk in an LSM projection, with a bounded hot overlay in memory, so capacity is not limited by RAM.

RoadmapEngine edition 5f8ea3af

What it does

The engine library has a disk-backed entity profile: a fjall LSM projection of the log holds the latest version of every entity, and boot replays the log above its watermark and nothing below it. The server always uses the in-memory profile, so this profile is not available to users yet.

How it works

In the disk profile the roots keep a bounded hot overlay and the LSM holds the latest version of every entity, keyed by id, with the applied log position stored in each value. It is a derived, rebuildable projection of the log, never the source of truth. Batches commit with an OS-buffered persist mode because the log already made them durable, and the watermark rides in the same atomic batch, so the projection can lag but cannot misstate what it holds.

Readers pin an LSM snapshot per state root. Equality indexes and a per-kind posting list live in the same keyspace and are written in the same atomic batch as the entity. While the projection lags, the overlay is capped, and exceeding the cap fails new writes closed with a storage-degraded error; reads are unaffected. An explicit administrator operation can trim log segments that the projection fully covers, which the in-memory profile refuses.

Measured: disk-resident entity store serving at 100,000,000 entities against gates of RSS under 14 GB, boot under 30 s, and get p99 under 1 ms. This is local scope, one host (Apple M5, 32 GB RAM) and a benchmark of the engine library, not of the server. Single host. WAL is trimmed above the projection watermark every 10M writes.

Where you reach it

  • LibraryEngineConfig::entity_store(EntityStoreProfile::Disk { cache_entities }) (engine library, not wired to the server)

Limits

  • The server does not configure this profile, so a server holds all entity values in memory.
  • The benchmark ran the engine library on one host. It says nothing about a server deployment.
  • The disk profile requires the file log. Deleted values remain in the LSM until compaction, which the engine does not expose.

Evidence

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

Every figure on this site is local-scope: one host, one operator, no third-party reproduction. Artifacts are linked next to each number, and gates that were missed are published beside the gates that were met.

KynetraDB is in private preview

This is not available in the preview. Request a preview to discuss the current scope.