Reliability

restore_to_seq / delete_all left stale on-disk checkpoint artifacts, so a later delta linked onto a stale base. Two failures at once: restored-away entities came back, and new post-restore writes that reused sequence numbers were silently dropped after the next boot.

Fixed structurally as the Tideline guarantee: delete checkpoint artifacts before truncating the WAL, fence detached writer threads with a monotonic checkpoint epoch, and delete any checkpoint the reject-guard discards on open. Two adversarial rounds; round 1 alone found six follow-on findings including a detached-writer TOCTOU.

Engine::open seeded last_checkpoint_seq but never seeded dirty_ids from the replayed WAL tail, so the first post-reboot delta claimed a coverage range that omitted those records. Reproduced exactly: 5 puts, restart, 2 puts, restart, 5 puts, restart — the count was 10 when it should have been 12.

Fixed by seeding dirty tracking from the tail replay, with red-to-green regression tests covering both the row and columnar checkpoint formats.

One root cause, three routes. A tenant holding its own project's jwt_secret (previously leaked in project-JSON responses) could mint a JWT whose sub was a victim org's owner uid and pass the owner check — rotating the victim's keys or deleting the victim org. Separately, cluster and settings routes did bare role=="admin" checks that a per-project service key satisfied. And namespace selection came from an unverified pid JWT claim.

Took three review rounds — each of the first two found a HIGH cross-tenant hole, including one that the previous round's own fix introduced. That is precisely why the final fix is structural (Bulkhead) rather than point-patched. Round 3 shipped with a full enumeration of every remaining role check, proving no admin route was missed.

The t_{pid}___ prefix scan is exactly right for isolation and wrong for sensitivity: a tenant's own namespace includes its auth kinds. auth_otp stores an unsalted SHA-256 of a 6-digit code — all 10⁶ hashes take milliseconds on a laptop. Anyone holding an export taken during a live 15-minute OTP window could recover the code and replay it to take over an end-user account. The same file carried session token hashes and password hashes for offline cracking.

Fixed by redacting at the export boundary (Ferry), keyed on the auth crate's own kind constants rather than string guesses. The lesson generalises: an export boundary needs a sensitivity filter distinct from its isolation filter.

HttpSink counted a non-2xx response as a successful write, so a restore that wrote zero rows reported full success. Separately, ScopedSink re-scoped only kind and not id, so restores minted fresh ids instead of preserving them.

Both fixed; the NDJSON restore path is now byte-faithful including embeddings, and the attrs-only HTTP path is documented as such rather than advertised as a full restore.