Partial backup
A backup scan failure could produce a partial bundle
Backup creation and the require-empty restore preflight both enumerate the store. When that scan failed, the result could be a partial bundle, or a preflight that passed although the store could not be read.
Both now enumerate the store through a fallible, pinned-snapshot scan, and a scan failure fails the backup or the preflight instead. Ordinary filter and get paths are unchanged. An independent review gave a bounded GO, and all 14 programme gates passed.
Functional only: no performance number was measured. Merged to KynetraDB main (310af5d), and not yet deployed to the hosted KynetraDB instance: the 2026-10-05 host inventory showed it still running an earlier build, and a host migration and a copied-data rehearsal must happen before any upgrade.
Recovery
A failed restore could leave the store serving unresolved history
restore_to_seq, delete_all and undo_restore change history. If one failed, panicked or was killed part-way, the store could go on serving reads and writes over history whose state was unresolved.
Each now records a durable intent before it touches history. If one fails, panics or is killed after that point, the engine refuses authoritative reads and writes with RecoveryUncertain (HTTP 503 recovery_uncertain) until the intent is resolved, and an open on an unresolved intent is refused. An independent review of the first closeout found blocking gaps; the bounded fixes in0bfd2f9 cover HNSW generation publication, WAL range and mmap-open admission, and organization authorization ordering, and an independent review gave them GO. All 14 programme gates passed, and four independent mutations were caught and restored byte-identically.
Disclosed limit: callers that must distinguish “absent” from “failed” use the fallible try_get / try_snapshotpaths. The legacy Option/Vec/usize reads keep their signatures and still answer None, empty or zero while the engine refuses. Functional only: no performance number was measured. Merged to KynetraDB main, and not yet deployed to the hosted KynetraDB instance: the 2026-10-05 host inventory showed it still running an earlier build, and a host migration and a copied-data rehearsal must happen before any upgrade.
Silent data loss
A stale checkpoint could resurrect restored-away entities and drop new writes
restore_to_seq / delete_all left stale on-disk checkpoint artifacts, so a later delta linked onto a stale base. Two failures at once: restored-away entities came back, and new post-restore writes that reused sequence numbers were silently dropped after the next boot.
Fixed structurally as the Tideline guarantee: delete checkpoint artifacts before truncating the WAL, fence detached writer threads with a monotonic checkpoint epoch, and delete any checkpoint the reject-guard discards on open. Two adversarial rounds; round 1 alone found six follow-on findings including a detached-writer TOCTOU.
Silent data loss
WAL-tail loss: entities written between checkpoint and restart vanished
Engine::open seeded last_checkpoint_seq but never seededdirty_ids from the replayed WAL tail, so the first post-reboot delta claimed a coverage range that omitted those records. Reproduced exactly: 5 puts, restart, 2 puts, restart, 5 puts, restart — the count was 10 when it should have been 12.
Fixed by seeding dirty tracking from the tail replay, with red-to-green regression tests covering both the row and columnar checkpoint formats.
Cross-tenant takeover
A project-plane credential could assert platform authority
One root cause, three routes. A tenant holding its own project'sjwt_secret (previously leaked in project-JSON responses) could mint a JWT whose sub was a victim org's owner uid and pass the owner check — rotating the victim's keys or deleting the victim org. Separately, cluster and settings routes did bare role=="admin" checks that a per-project service key satisfied. And namespace selection came from an unverified pid JWT claim.
Took three review rounds — each of the first two found a HIGH cross-tenant hole, including one that the previous round's own fix introduced. That is precisely why the final fix is structural (Bulkhead) rather than point-patched. Round 3 shipped with a full enumeration of every remaining role check, proving no admin route was missed.
Account takeover
The export path handed out brute-forceable OTP hashes
The t_{pid}___ prefix scan is exactly right for isolationand wrong for sensitivity: a tenant's own namespace includes its auth kinds.auth_otp stores an unsalted SHA-256 of a 6-digit code — all 106 hashes take milliseconds on a laptop. Anyone holding an export taken during a live 15-minute OTP window could recover the code and replay it to take over an end-user account. The same file carried session token hashes and password hashes for offline cracking.
Fixed by redacting at the export boundary (Ferry), keyed on the auth crate's own kind constants rather than string guesses. The lesson generalises:an export boundary needs a sensitivity filter distinct from its isolation filter.
Silent failure
A restore reported success while writing zero rows
HttpSink counted a non-2xx response as a successful write, so a restore that wrote zero rows reported full success. Separately,ScopedSink re-scoped only kind and not id, so restores minted fresh ids instead of preserving them.
Both fixed; the NDJSON restore path is now byte-faithful including embeddings, and the attrs-only HTTP path is documented as such rather than advertised as a full restore.