# KynetraDB — Benchmark Ledger

Append a dated entry **with every build**. Each entry: what was measured, the
numbers, the method, and the commit. Newest first.

---

## 2026-07-27 (13:30 IST) — 9-slice engine DEPLOYED to the live DO demo (b23b93b)

**What:** deployed the full day's engine delta to `kynetradb-quantumos-mvp-blr1`
(blr1) via the kynetrapods OCI flow — Tideline, Bulkhead, Ferry, Tidewater, Ebb,
Meridian, Lodestar, Ledger, Watermark, i.e. `07643e4..b23b93b`. Built amd64-native
ON the droplet with buildah (`localhost/kynetradb:b23b93b`, 150 MB), pushed to an
OCI layout, `umoci unpack`ed to a new bundle, atomic base swap + restart with
health-gated auto-rollback.

**DATA SURVIVED — the headline result.** The boot log is the proof:

```
checkpoint loaded — replaying tail only  checkpoint_seq=30018  log_seq=38423
```

That is precisely the WAL-tail replay path the **Tideline** fix hardened, running
on 38k sequences of real production data. `log_seq` is *up* from the 38414
recorded on the previous deploy, so nothing was lost and writes continued
throughout. WAL 37M, checkpoint 19M, zero errors/panics/projection warnings.

**End-to-end health after the swap:** live site 12/12 × HTTP 200; homepage serves
**48 unique product links** (150 KB, live data — no `cf-cache-status`, so not a
stale edge cache); a real product page returns 200.

**Live confirmation of the new tenancy code:** the global service key sees
`{"collections":[]}` — correct, and exactly **Bulkhead** working as designed,
since tenant data lives under `t_<project>___coll:*` and an unscoped credential
must not see it. Studio mode (`X-Project-Id`) authorizes with 200.

**Latency — reported honestly, because the clean engine number was NOT
obtainable.** End-to-end p50 measured 0.251s pre-deploy and 0.286s post-deploy
(12 samples each, from India). **That difference is noise, not a regression:** both
runs contain a ~2.4s cold outlier, n is small, and the path is
my-machine → Cloudflare → Worker → droplet, so the engine is a small fraction of
it. Server-side numbers that *are* trustworthy: `/v1/collections` p50 **1.0ms**
(p95 1.4ms, 20 samples, no network).

What blocked a real engine-level catalog comparison: **Caddy on the droplet serves
only `168-144-85-189.sslip.io` → :8080; it does not serve camerademo**, which is a
Cloudflare Worker calling the API. The storefront's actual data route is
`/rest/v1/:table` (20 × 200 in the request log), but reaching it with data needs
the **project-scoped** key the Worker holds — the global key legitimately cannot,
by Bulkhead's design. Options to close this next time: provision a dedicated
benchmark project and measure against it, or capture the Worker's own timing.
**Not claiming a speed win I did not measure.**

**Rollback:** `base.pre-20260727` (a copy of the *previously running* base, created
before the swap). Note `base.pre-perf10x` was stale — it predates the previous
deploy, so rolling back to it would have reverted two deploys, not one. Data lives
outside the bundle (`DATA_DIR=.../kynetra`), so a bundle swap cannot touch it.

**Two host-side gotchas worth recording:** `/etc/containers/registries.conf` had
lost its `unqualified-search-registries = ["docker.io"]` line, so the build failed
instantly on short-name resolution until re-added — expect this again on any
droplet rebuild. And `/` has only 4.2G free while the **50G `/dev/sda` volume**
(47G free) is where the build must run; the build script already targets it.
Reclaimed the data volume 2.5G → 608M of orphaned buildah layers on the way.

**Commit:** `data/db` `b23b93b`.

---

## 2026-07-27 (12:45 IST) — Equality index projection-lag guard + leftover toggles removed (Watermark)

Two commits (`data/db@cbf3df0`, `@b23b93b`). Coined **Watermark**; Foundry 86 -> 87.

### 1. The equality index could silently lose rows

`kind_indexed_ids` falls back to a full scan when `store_watermark != seq`, because
on a frozen watermark (a store write failed) the overlay holds live entities the
projection has never seen — so the posting list **under-reports**. Lane B flagged
in `3755145` that the equality arm had no such guard and called it out-of-scope;
`42e161d` then hardened *over*-reporting only. Confirmed still open at that tip.

**Why under-reporting is the dangerous direction:** the `f.matches` re-check can
only REMOVE ids, never add back rows the index failed to return. Over-reports are
survivable by construction; under-reports silently lose a tenant's rows. So no
amount of re-checking downstream repairs this.

Fixed by factoring the invariant into ONE helper, `projection_is_current(state,
filter_shape)`, called by both the equality arm and the per-kind arm (which
previously inlined its own copy). Two copies of an invariant drifting apart is
exactly what produced the `vector_search` bug fixed the same day — so the guard is
now single-sourced, with the shape named in the warn! log.

**Test** (white-box, justified): there is no existing projection-lag test to model
on, and an integration test *cannot* express this — `StateRoot`, `store_watermark`,
`VersionedEntity` and `filter_root` are all crate-private and there is no seam for
injecting a real fjall write failure. So the test doctors a Disk root exactly as
`project_and_apply` leaves it after a failed store write (seq advanced, watermark
frozen, one entity live in the overlay that `store.get()` confirms the projection
never saw) and asserts on the **read's output** via `filter_root`, not on the
guard's return value. Mutation-verified twice: deleting the guard gives
"the equality arm must refuse a posting list that cannot describe the overlay",
and with that assertion also removed, the row loss itself fires —
`left: ["e0"..."e5"]` vs `right: ["e0"..."e5", "overlay-only"]`.

### 2. Three leftover benchmark toggles, one a real footgun

Three sites labelled `TEMP BENCHMARK TOGGLE (reverted before finishing)` came in
with `1b919a8` and were **never actually removed** — still at HEAD, gated on
`KYNETRA_BENCH_DISABLE_EQ_INDEX`:

- **read path** — returns `None`, falling back to a correct scan. Safe, just slow.
- **write path** (`project_records`, `rebuild_disk_index_if_absent`) — skips ALL
  `index_eq` maintenance. Set the var, write, unset it, restart: reads then trust a
  silently incomplete posting list.

**And `index_ready` masks that rather than catching it.** The early return fired
*before* `need_eq = !store.index_ready()?`, so on any store already marked ready the
flag stays set, `need_eq` is false forever, and no later boot ever backfills — rows
written during the off window are missing **permanently**. The per-kind index
handles its own off window by *clearing* `kind_index_ready` so the next enabled boot
rebuilds; the equality index has no counterpart.

Removed rather than cfg-gated, and that is the *complete* fix rather than a partial
one: there is no `disk_eq_index` config flag — only `disk_kind_index` is gated — so
the equality index is unconditional on Disk and this toggle was the **only** thing
that could ever create an off window for it. With it gone, the missing
clear-on-off-window counterpart is moot. Nothing under `crates/core/examples/` read
the var (the bench documents its "before" arm as a git stash), so no bench broke.
Behaviour with the var absent — the only shipped behaviour — is unchanged:
`is_err()` was true, so both guarded bodies always ran and the early return never
fired. 6 insertions, 30 deletions.

**Process note:** the leftover toggles sat in merged code through every review this
session. Design review and guard mutation-testing both missed them because neither
was *looking* for debug scaffolding. A grep for leftover toggles/TODO-reverts is now
part of the pre-merge sweep.

**Suite:** `cargo test -p kynetra-core` green — 315 tests, 25 binaries, 3
consecutive runs in the worktree plus a clean run after cherry-picking.

**Minor gap noted, not fixed:** `disk_equality_index_bench.rs`'s doc comment points
at `benchmarks/evidence/2026-07-24-linux-disk-equality-index.json`, which was never
committed. The bench still runs; only its cited artifact is missing.

---

## 2026-07-27 (11:45 IST) — Filter fidelity + test determinism (Ledger)

Two follow-ups filed during Lane B, now closed. Coined **Ledger**; Foundry 85 -> 86.

### 1. `vector_search` must honor EVERY conjunct (`data/db@42e161d`)

`extract_kind_eq` is a last-wins loop: it keeps only the LAST `Kind` and LAST `Eq`
of an `And(..)` and silently drops the rest. `filter_root` was safe because it
re-fetches each candidate and re-checks `f.matches` — **that re-check is what makes
an over-approximating index resolution sound.** `vector_search` consumed the same
lossy set with no re-check, so `And([Kind, Eq, Gt])` returned rows failing the
`Gt`, and `And([Eq(a), Eq(b)])` honored only the last `Eq`.

**Severity, stated precisely:** NOT a cross-tenant leak for Kynetra — tenancy lives
in the entity KIND, which is part of the index key and always honored — so the
in-product impact is **wrong results**. It *would* be a leak for a caller that put
tenancy in an attrs conjunct, which is why the invariant is load-bearing and now
has a tenancy-shaped guard test saying so. (I had initially overstated this as a
possible tenant leak; review corrected it and the correction is carried here.)

**The fix is structural, because the root cause was a duplicated invariant** — two
call sites resolved pre-filters and only one knew resolution may over-approximate.
Rather than a second copy of the re-check, a `FilterCandidates` type now *carries
the obligation*: `index_resolution_is_exact` marks a resolution exact only for a
bare `Kind` or a lone `Kind`+`Eq` pair, `retain_admitted` discharges the re-check
when owed and no-ops when not, and `resolve_filter_candidates` is the single way to
obtain a resolved set. The two paths cannot drift apart again.

**Hot path preserved:** on the exact arm entities are fetched anyway so the
re-check is ~free; on the approximate arm only ids that become candidates are
re-checked, with `widen_overfetch` compensating k so a filtered search still
returns a full page.

**Verification note worth recording:** my first mutation disabled `retain_admitted`
and 4 of 5 tests still passed — which looked like weak tests. It wasn't: for those
shapes the candidate set is already exact by another route, so skipping the
re-check is a legitimate no-op. Mutating `index_resolution_is_exact` to always
return true — reproducing the *actual* original defect — fails **all five**.
**A mutation must reconstruct the real bug's semantics, not merely disable code
near it.**

### 2. Two Disk-tier tests made deterministic (`data/db@535912e`)

Both built corpora with `Entity::new(..)`, i.e. a fresh random ULID per entity per
run. `StateRoot.index` is an `im::OrdMap`, so entities reached `SpannIndex::build`
sorted BY those random ids — a different corpus **order** every run — and `kmeans`
samples by index into that slice, so every run built a materially different index.
Measured failure rate in isolation, no contention, no code change: **2/8 at
`3755145`, 1/6 at `ea0e399`.** Inherent, not caused by recent work.

Fixed with stable `item-{i:06}` ids. Now reproducible — **8 isolated runs each,
bit-identical**: recall gate test **1.0000** (gate 0.99), boot-rebuild test
**0.873** (gate 0.70). No gate moved, no seed tuned, no corpus resized.

**Flagged rather than hidden:** the deterministic corpus yields 1.0000, above both
the old 0.972-0.990 spread and Caustic's representative ~0.995. `clustered_vec`
derives its cluster from the raw LCG state, so ordering ids by insertion correlates
corpus order with cluster identity and gives k-means unusually clean seeding — a
favourable draw from the same distribution, arrived at without tuning. Documented
in-code that 1.0000 is **this corpus's number, not a Caustic benchmark**, and that
a 0.99 bar against a 1.0000 baseline catches a >=1% codec regression but not a
smaller one. Re-basing the bar deliberately is the honest fix if tighter
sensitivity is wanted — never swapping in a different corpus.

**A mismatched assertion also fixed:** the boot-rebuild test's message claimed *"a
write covered by the checkpoint must survive the rebuild"* but tested top-3 ANN
membership — so a pure *ranking* miss would have been reported as *data loss*. Now
split into survival (`get` + byte-identical embedding) and indexing (the ANN check,
honestly labelled). Coverage grew: embedding integrity was never asserted before.

**Result:** `cargo test -p kynetra-core` is now reliably green — 3 consecutive full
runs plus 8/8 isolated runs of each previously-flaky test.

---

## 2026-07-27 (06:55 IST) — SPANN large-filter scan: covering the path the engine actually uses (Lodestar)

**What:** `data/db@ea0e399`, engine-1 follow-up — the final item of the 10x queue.
`SpannIndex::search`'s filter has two branches: a **small** exact branch (<= 4096
members) and the **large** predicate-during-scan branch. The Engine resolves
<= 4096 filters with its *own* exact scan and only ever calls `SpannIndex::search`
with LARGE filters — so **the >4096 posting scan is the only SPANN filter path
production reaches, and it was the one with no real coverage.** The committed
matrix test used N=10,000, making its 10%/1%/0.1% selectivities 1000/100/10
members — all small-path — and its `>=0.99` bar validated brute force, not the new
scan. Coined **Lodestar**; Foundry 84 -> 85.

**Measured (N=30,000, filters asserted >4096, both codecs, 12 queries):**

| codec | filter | native | post-filter equal-k | post-filter overfetch |
|---|---|---|---|---|
| RaBitq | 4,200 (14%) | **0.8000** | 0.1417 | 0.7833 |
| RaBitq | 6,000 (20%) | **0.8167** | 0.2250 | 0.7500 |
| Caustic | 4,200 (14%) | **0.7917** | 0.1667 | 0.7750 |
| Caustic | 6,000 (20%) | **0.7833** | 0.2167 | 0.6833 |

Bars derived from these numbers rather than assumed: `native >= 0.75`,
`native > equal-k + 0.30`, and `native > overfetch` strict and **unpadded** — the
+0.03 pad from the earlier shape was dropped because the new data doesn't support
it. A caller who already knows the selectivity and pays a 1/selectivity-wider
search genuinely is close; the robust win is the **order of magnitude** over
equal-budget post-filtering (0.80 vs 0.14).

**Calibration mattered more than the assertions.** With the sibling test's 60
tight clusters, native saturates at **1.0000** at every filter size — and even
`nprobe: 3` changes nothing, meaning the *generator* was being measured, not the
scan. Eight broad clusters make each query's true top-k straddle postings.
Dropping to N=20,000 saturates it back to 1.0000, so 30,000 is the floor that
keeps recall off the ceiling. **A test that cannot fail measures nothing.**

**Also:** a large-path (5,001-member filter) tombstone + memtable test — a
tombstoned member stays absent *before* the flush (memtable shadow over a live
posting row) *and after* it, with `memtable_len() == 0` asserted in between so the
post-flush case genuinely exercises the settled mmap'd posting scan. Both new
tests assert no ineligible id reaches the exact-rerank closure; I verified
non-vacuity myself by bypassing the scan's filter predicate, which fails both with
`"large-filter posting scan reranked an ineligible id"`.

**The small branch's O(corpus) walk is documented, not optimized** — deliberately.
Making it O(filter) needs a resident whole-corpus `id -> (centroid, row)` map, i.e.
putting the id strings back on the heap: exactly the cost Tidewater moved to mmap
(~16% -> ~0.13% resident). Paying that for a path the engine never reaches would
trade the subsystem's reason for existing for nothing.

**A latent test-infra bug this surfaced:** `wait_for_graph`'s ceiling was 600 x
50ms (~30s) and its own comment warned it "flakes under contention" — a wall-clock
timeout inside a parallel suite is a bug waiting for someone to add a heavy test.
The new large-filter test starved that background compaction thread and failed
`rabitq_persist_load_round_trip` **reproducibly (2 of 2)**, while the tip passed
129/129. Raised to 1800 polls (~90s): it is a **liveness bound — a hang detector —
not a performance assertion**, so a larger ceiling weakens no property. Now
**130/130 on three consecutive full-suite runs.**

**Process note worth keeping:** the coding agent's single green full-suite run did
not catch this, because contention-dependent failures need *repeated* runs. One
green run is not evidence. That rule is now part of the loop.

**Pre-existing flake, filed not fudged:** `fresh_disk_tier_engine_default_passes_caustic_recall_gate`
has a knife-edge `>= 0.99` bar with a nondeterministic corpus (random ULIDs ->
`im::OrdMap` iteration order -> different k-means sampling each run; measured
0.9720-0.9900). I independently reproduced **1 failure in 6 isolated runs at base
`3755145`** with none of this change present, confirming it predates it. The fix
is deterministic entity ids via `Entity::with_id` — and if a deterministic corpus
lands below 0.99, that is a real finding for the gate's owner, not something to
tune away. Filed separately.

**Suite:** 130 lib + vector_tiering 18 + all other suites, 0 failed.
**Commit:** `ea0e399`.

---

## 2026-07-27 (04:45 IST) — Per-kind index: tenant-scoped reads go O(tenant) (Meridian)

**What:** `data/db@3755145`, tenancy Lane B (tasks 6-8). A tenant-scoped list is
`filter(Filter::Kind("t_<pid>___..."))`, but a **bare** `Filter::Kind` was not
index-answerable (`extract_kind_eq` only matched `And([Kind, Eq, ..])`), so it fell
through to a full-corpus scan — a scoped read cost O(**all** tenants' data). Coined
**Meridian**; Foundry 83 -> 84.

**Measured (release, fixed 10k tenant-A corpus, background scaling 10k -> 1M):**

| arm | p50 @ 10k -> 1M | shape |
|---|---|---|
| ram, index **on** | **4.5 -> 3.0 ms** | flat (growth <= 1.00x) |
| ram, index off | 5.0 -> 82.5 ms | linear (14.5-17.5x) |
| disk, index **on** | **6.8 -> 13.6 ms** | monotone 1.90-2.37x |
| disk, index off | 9.1 -> 443 ms | linear (40-49x) |

At 1M background that is **~28x** (ram) and **~33x** (disk) faster than index-off.

**Ship-gates, reported honestly:**
- **Gate 2 (disk write regression < 10%): PASS at 0.1%** — 33153 vs 33112 puts/s,
  median of 3 interleaved rounds at 100k puts/arm. An early single-sample protocol
  read exactly 10.0%, and single 20k rounds ranged -23%..+12%; the **measurement**
  was hardened, not the bar. The extra posting per write is inside noise, so the
  `t_`-prefix-only fallback proved unnecessary.
- **Gate 1 (scoped-read p50 varies < 2x): NOT cleanly met, reported as-is.** Ram
  passes on substance (cost does not track background size on any run) but max/min
  exceeded 2.00x on 2 of 5 runs — with the worst cell at the *smallest* background,
  i.e. between-cell drift from separately seeded engines, not scaling. Disk shows
  **genuine** monotone growth: `filter_root` re-fetches each of the 10k ids
  individually and an fjall point-get costs more as the LSM deepens, so the read is
  O(tenant . log total). Verdict: **Ram default-on, Disk default-off** until that
  read path stops doing per-id point-gets. (The reviewer's note: the bar itself is
  the wrong statistic — it should be a growth factor, not max/min spread.)

**The engineering that mattered — making the trap unrepresentable.** `kind_ids` must
be rebuilt in every path that reconstructs state, and missing one diverges silently
*only after* that lifecycle op. Instead of vigilance, two structural defences:
`rebuild_json_indexes` became `rebuild_ram_indexes`, rebuilding `json_eq` and
`kind_ids` **together** so "rebuilt one, forgot the other" cannot be expressed; and
`StateRoot::fresh(config)` replaced `StateRoot::default()` at every from-scratch
site, where `default()` leaves the flag **false** — the fail-safe direction, since
reads then scan (correct, merely slow). A future missed site is a perf regression,
never a correctness bug. Adversarial review audited **all 12** `StateRoot`
construction sites and found **no missed path**.

**Two migration traps caught in design, not production:** (1) `kind_index_ready` is
deliberately *separate* from `index_ready` — any store written between ADR-014 and
this change has `index_ready` set and zero kind postings, so one shared flag would
make it look caught up and **silently serve empty tenant-scoped reads**; (2) the
disk read arm falls back to a scan when `store_watermark != seq`, because on a
frozen watermark the overlay holds entities the projection never saw, so postings
would **under**-report — and under-reports silently lose a tenant's rows, where
over-reports are caught by `filter_root`'s `f.matches` re-check.

**Exactness is load-bearing, not optional:** `Engine::vector_search` resolves its
pre-filter through `indexed_filter_ids` but, unlike `filter_root`, does **not**
re-check `f.matches` — so a stale posting would surface as a vector hit for another
entity. Pinned by a leak test per profile; I verified myself that no-op'ing the
retraction makes it fail with the exact cross-tenant message. (Filed separately:
the pre-existing equality index *does* over-approximate there for
`And([Kind, Eq, Gt])` — a filter-fidelity bug, **not** a tenant leak under
kind-based tenancy, since the kind is always honored as part of the index key.)

**Suite:** 129 lib + all integration suites, 0 failed. **Commit:** `3755145`.

---

## 2026-07-27 (03:45 IST) — Honest tenant erasure — Lane A complete (Ebb)

**What:** `POST /admin/v1/projects/:id/purge` (`data/db@a7d2d1d`) tombstones every
`t_{pid}___*` entity, destroys the tenant's keys, then compacts. This completes
**all of tenancy Lane A (tasks 1–5)**. Coined **Ebb**; Foundry 82 → 83.

**Idempotent and resumable by construction:** re-lists kinds after each sweep
until none remain, then does an honest live **recount** rather than trusting its
own loop counter. If residue remains or the chunk budget was hit it returns
`complete: false` *without* deleting the project row and *without* compacting — so
the tenant stays `Deprovisioning` and the purge can be re-driven. The project row
is deleted last, then the KeyIndex rebuilt. `force_checkpoint`/`trim_wal` failures
are recorded in `compaction_notes` and never fail the purge (tombstones are
already durable, so worst case is under-compaction, never data loss).

**The bug adversarial review found (state-machine trap):** `DELETE /projects/:id`
(→ Deprovisioning) followed by `POST /projects/:id/status {"purged"}` persisted a
`Purged` row while deleting **no data**. After that `purge_project` refused
forever ("is purged, not deprovisioning"), `Purged` was terminal, and the
tenant's data sat resident, unreachable and **unerasable through any API** — the
exact leak this task exists to close, reachable by an *org owner* rather than a
super-admin. Fixed by guarding `set_project_status` (the authoritative writer, so
no caller can bypass it) as well as the handler, plus making the purge interlock
accept `Purged` so an already-stranded tenant can still be erased. **Lesson: a
lifecycle edge that only one privileged path should traverse must not also be
reachable through a generic "set status" endpoint.**

**Cross-tenant safety (the highest-stakes property):** the cascade scans prefix
`t_{pid}___`, so a pid containing `_` could match another tenant (pid `X` vs
`X___Y`). `validate_purgeable_pid` rejects empty/`_`-bearing ids on **both** paths
including the orphan sweep where no row exists to cross-check. Ids are always
server-minted ULIDs, so it can't fire in production — a structural guard. I
verified by mutation that loosening the prefix turns the isolation test red.

**HONESTY — `erasure_scope = "live checkpoint + retained WAL"`,** returned in
every `PurgeReport` and documented in `deploy/tenancy/README.md`. Real grep output
from the residue test:

```
[honest-purge] purged tenant A sentinel still found in: ["wal/00000000000000000000.seg"]
[honest-purge] surviving tenant B sentinel found in:   ["checkpoint.bin", "wal/...seg"]
```

A is **gone from `checkpoint.bin`**, still in the WAL. The B-sentinel is a
deliberate control proving the grep can see checkpoint contents, so "A absent"
cannot pass vacuously. This is tombstone + checkpoint compaction, **not**
crypto-erasure: bytes survive in the retained WAL, `restore_to_seq` undo backups,
replica logs, and Disk-profile fjall segments (tombstoned pending compaction —
major compaction isn't exposed by the engine). A true-erasure contract needs
out-of-band per-tenant key destruction plus retention expiry.

**Suite:** 219 lib + 16 + 2, 0 failed. **Commit:** `data/db` `a7d2d1d`.

---

## 2026-07-27 (03:35 IST) — SPANN postings served from mmap at query time (Tidewater)

**What:** the 10x program's last big memory goal (`data/db@0b40e88`). SPANN's
premise is billion-scale ANN at ~1% DRAM, but `SpannState.postings` held every
payload in RAM (`HashMap<CentroidId, Posting>`) and `search` read straight from
it — so the disk tier existed on disk yet was *paid for* in RAM. Beacon slice 4's
mmap was load-time only. Coined **Tidewater**; Foundry 81 → 82.

**Numbers (measured, 50 000 × 64 corpus):**

| | resident | % of raw f32 |
|---|---|---|
| before (RAM HashMap) | ~2.1 MB | ~16.4% |
| after (mmap at query time) | **16 400 B** | **0.13%** |

≈**125× less resident memory**, search results **byte-identical** (same bits, same
corrections, same ordering). Only the navigator, centroid vectors, per-posting
headers and the memtable stay resident; the page cache *is* the tier now.

**Key enabler:** the `.post` file became a raw fixed-stride layout (`SPP1` magic,
32 B header, then packed_bits / corrections / residual_norms / id_offsets /
id_blob), replacing the `rmp_serde` blob. A serialized blob must be fully
deserialized to read one row, so **offset-addressable rows are what make
zero-copy possible at all**. `search` now reads sign bits zero-copy from mapped
pages and materializes an owned `EntityId` only for actual candidates.

**Why it's safe:** a reader `Arc`-clones posting handles under a short read lock
then scans lock-free — a concurrent flush/rebalance that unlinks the old epoch
file cannot tear the scan, because unlinked-but-mapped stays valid on Linux and
postings are immutable per epoch. Flush maps only *after* fsync, installs under
one state write lock, with the manifest as commit point. Rebalance installs
postings + epochs + centroid_vecs + navigator in one write section, so a query
sees fully-old or fully-new.

**Bug caught by adversarial review:** `MappedPosting::parse` validated only the
*first and last* `id_offsets` entries — a corrupt `.post` file could pass parse
and then panic inside `search`, or in a **release** build wrap the offset
arithmetic and *silently serve garbage bytes as an entity id*. Now the whole
table is walked for monotonicity and blob bounds, with `checked_add` +
`slice::get` in `id_str` as a second layer. I verified the three rejection tests
fail against a revert of the validation loop while the over-rejection guard still
passes.

**Suite:** full `cargo test -p kynetra-core` green — **265 tests, 24 binaries, 0
failed**, including the new flush-during-query and rebalance-during-query tear
tests, and the restore/checkpoint (Tideline) suites.

**Commit:** `data/db` `0b40e88`. Note: the Caustic-vs-RaBitq density test now sums
on-disk `.post` bytes, since `memory_bytes()` is codec-independent once payloads
leave the heap.

---

## 2026-07-27 (02:55 IST) — Per-tenant export + auth-material redaction (Ferry)

**What:** `GET /admin/v1/projects/:id/export` (`data/db@f9fe907`) streams one
tenant's namespace as integrity-hashed NDJSON from a single pinned
`Engine::snapshot()` — no writer quiesce, no torn output — plus an `ndjson`
migrator source that verifies before it writes. Coined **Ferry** and registered
it in Foundry (Engine Fabric group; platform-term count 80 → 81).

**The bug adversarial review caught (account-takeover vector, blocked merge):**
the `t_{pid}___` prefix scan is exactly right for *isolation* but wrong for
*sensitivity* — a tenant's own namespace includes its auth kinds. `auth_otp`
stores an **unsalted** SHA-256 of a **6-digit** code, so 10⁶ hashes ≈
milliseconds on a laptop: anyone holding an export taken during a live 15-min OTP
window could recover the code and replay `POST /auth/v1/verify` to take over an
end-user account. The same file carried `auth_refresh` session token hashes and
`user.password_hash` for offline cracking. Fixed by redacting at the export
boundary — `auth_otp`/`auth_refresh` excluded outright, `password_hash` emptied —
keyed on `kynetra_auth`'s own kind constants (not string guesses). **Lesson: an
export boundary needs a sensitivity filter distinct from its isolation filter.**

**Second bug (silent restore data-loss):** `HttpSink` counted a non-2xx as a
successful write, so a restore that wrote **zero rows** reported full success;
and `ScopedSink` re-scoped only `kind`, not `id`, so restores minted fresh ids.
Both fixed — `ndjson → ScopedSink → EngineSink` is now byte-faithful including
embeddings, and the attrs-only HTTP path is documented as such rather than
advertised as a full restore.

**Integrity:** trailer SHA-256 now covers the header line *and* every entity
line (the header carries `pid`, so leaving it unhashed meant a relabelled file
could restore the wrong tenant silently); `attrs` keys recursively sorted, since
`serde_json`'s `preserve_order` **is** enabled transitively via `bson` — the
original "BTreeMap sorts keys" assumption was wrong, so the hash was not
canonical. Importer takes `--expect-pid`. Audit: bulk egress records
`project.export` in the hash-chained admin log; Deprovisioning/Purged refused;
1 GiB cap returns 413 instead of OOMing the shared monolith.

**Verification:** I re-ran the suites myself and **empirically proved the
redaction test has teeth** — reverting the exclusion guard makes it fail with
`OTP code_hash leaked`, and it passes restored. Server **204+16+2**, migrators
**38+1**, 0 failed.

**Commit:** `data/db` `f9fe907`. Task 5 (erasure) in progress.

---

## 2026-07-27 (02:10 IST) — Tenant-isolation hardening, Lane A 1-3 (Bulkhead)

**What:** landed the first three tasks of the tenancy production-hardening plan
(`data/db@fc469ee`) — ProjectStatus lifecycle, per-project rate limits + entity
quotas, and a structural platform/project **plane fence**. Coined **Bulkhead**
and registered it in Foundry (Engine Fabric group; platform-term count 79 → 80).
Took **three KynetraResolve adversarial-review rounds** — each of the first two
found HIGH cross-tenant-takeover holes (including one the previous round's fix
*introduced*), which is why the fix ended up structural rather than point-patched.

**The class of bug closed (all one root cause):** a project-plane credential
asserting platform-plane authority. (1) a tenant holding its *own* project's
`jwt_secret` (previously leaked in project-JSON responses) could mint a JWT with
`sub` = a victim org's owner uid and pass `can_manage_org`'s owner check →
rotate the victim's keys / delete the victim org; (2) `/admin/v1/clusters/*` and
`/admin/v1/*/settings` did bare `role=="admin"` checks that a per-project service
key satisfied → global cluster/settings control; (3) `resolve_project_ctx` chose
which tenant namespace to write into from an *unverified* `pid` JWT claim →
signup/OTP/token into a victim's namespace.

**Fix (Bulkhead):** a `platform_identity: bool` on `AuthCtx`, set once at
credential resolution (true only for the global service key, an unscoped
platform JWT, and a genuinely-super_admin Studio session), checked by
`is_super_admin`/`require_platform_admin`/`can_manage_org` before any role/uid
comparison. `AuthCtx` derives only `Debug+Clone` (no `Deserialize`/`Default`, no
struct-update sites) so the flag can't be set from untrusted input. All 19 bare
role checks in clusters.rs/settings_admin.rs swept to `require_platform_admin`;
`resolve_project_ctx` verifies the JWT signature against the candidate project's
own `jwt_secret` before a `pid` can select a namespace; `Project.jwt_secret` is
now `#[serde(skip_serializing)]`. Paused tenants keep `grant_type=refresh`
(reads stay alive) while password/signup/OTP are blocked.

**Method:** independent verify (re-ran the suite in-worktree AND main tree) +
Fable review rounds 1→3 (round 3 verdict SHIP, with a full enumeration of every
remaining role check in `crates/server` proving no admin route was missed). Full
`cargo test -p kynetra-server` green: **215 tests (197 lib + 16 + 2), 0 failed.**

**Commit:** `data/db` `fc469ee`; Foundry term in `ml/foundry`. Lane A 4 (export)
in progress; Lane B (engine per-kind index) queued behind engine-4.

---

## 2026-07-27 (02:00 IST) — Durability hardening: two silent-data-loss bugs closed (Tideline)

**What:** closed two empirically-confirmed silent-data-loss bugs in the
checkpoint/restore path, both found by adversarial review (KynetraResolve), each
landed with red→green regression tests covering both the row and columnar
checkpoint formats. Coined the durability guarantee **Tideline** and registered
it in Foundry (Engine Fabric group; platform-term count 78 → 79).

**Bug 1 — WAL-tail loss (`data/db@a9e39ad`):** entities written after the last
checkpoint but before a restart vanished after the next automatic *delta*
checkpoint + a second restart. Cause: `Engine::open` seeded `last_checkpoint_seq`
but never seeded `dirty_ids` from the replayed WAL tail, so the first
post-reboot delta claimed a coverage range that omitted those records; the next
boot's `seq > cp.seq` tail filter then skipped them. Repro: `checkpoint_every(5)`,
5 puts → restart → 2 puts → restart → 5 puts → restart → count was **10, should
be 12**. Fixed by seeding dirty tracking from the tail replay (puts + deletes).

**Bug 2 — restore-path resurrection/drop (`data/db@16a8c5e`):** `restore_to_seq`
/ `delete_all` left stale on-disk checkpoint artifacts, so a subsequent delta
linked onto a stale base → restored-away entities **resurrected** and new
post-restore writes with reused seqs were **silently dropped** after the next
boot. Fixed with three mechanisms (Tideline): (1) delete every checkpoint
artifact *before* truncating the WAL (crash-between = full replay = "restore
didn't happen"); (2) a monotonic checkpoint epoch bumped under the
checkpoint-write lock fences detached base/delta writer threads — a writer whose
corpus predates the rewind aborts before its rename; (3) `Engine::open` deletes
any checkpoint its `cp.seq >= next_seq` reject-guard discards. Also re-persists
the search snapshot on restore and resets the vector index on `delete_all`.

**Method:** independent verify (re-ran the agent's tests in-worktree AND in the
main tree) + two Fable adversarial review rounds on bug 2 (round 1 found 6
follow-on findings incl. the detached-writer TOCTOU; round 2 verdict SHIP with a
per-attack safety argument). Full `cargo test -p kynetra-core` green (exit 0);
`restore_checkpoint_consistency` 6/6, `checkpoint` 13/13.

**Commit:** `data/db` `a9e39ad` (WAL-tail) + `16a8c5e` (restore-path);
Foundry term in `ml/foundry`.

---

## 2026-07-26 (22:35 IST) — Session engine DEPLOYED to the live DO demo

**What:** built the session's engine (`data/db@cd7f0e4`+ — Beacon 4-5, Caustic,
ACORN filtered search, columnar-checkpoint format, D1 migrator, Raft crate)
amd64-native ON the blr1 droplet via buildah (the Dockerfile from-source build),
and swapped it into the running `kynetrapods-vps` OCI flow
(`kynetradb-kynetrapods.service`, runc + shim) — replacing the pre-built binary
image `sha256:0c1c…` with the freshly-compiled `sha256:009b268f…`.

**Deploy method (kynetrapods, not the bootstrap.sh path):** buildah build →
`buildah push oci:` → `umoci unpack` into a fresh `base.new` bundle → atomic
base swap + `systemctl restart` with **auto-rollback on unhealthy** (old bundle
kept as `base.pre-perf10x`; DO boot-disk snapshot `238562267` as the outer net).

**Result:** clean cutover. `applied_seq` **38414 → 38414** (data intact — the new
engine loaded the existing WAL/checkpoints via serde back-compat, no migration
needed). Service `active`, `role: primary`.

| Path (new engine, through CF-SIN edge) | p50 | p90 | max |
|---|---|---|---|
| KynetraDB catalog build (`server-timing`) | **9 ms** | **12 ms** | 145 ms (first-touch) |
| Live site TTFB (warm) | ~215 ms | — | 2.28 s (cold, first hit post-restart) |

**Takeaway:** the session's engine is live on the demo with zero data loss and
the same low-latency profile (catalog ~9 ms p50). The 145 ms catalog max and the
2.28 s first TTFB are the expected post-restart cold-start (page cache + edge
keep-alive), settling to ~9 ms / ~215 ms within a few requests. Fixed a
pre-existing Dockerfile bug en route (`COPY benchmarks/` — the server
`include_str!`s a repo-root evidence file the build context omitted; `cd7f0e4`).

---

## 2026-07-26 (21:00 IST) — Live DO demo re-benchmark (`camerademo.qosx3.space`)

**What:** re-measured the live demo after a maintenance window (root
password-reset + two power-cycles while restoring admin access — the DB
auto-restarted on each boot, as designed). Client: India → Cloudflare-**SIN**
edge (`cf-ray …-SIN`; still Singapore, not Mumbai — Cloudflare edge PoP is not
repo-controllable, as documented). Read-only GETs, `curl` timing, `xargs -P`
for concurrency.

| Path | p50 | p90 | p99 |
|---|---|---|---|
| Edge-HIT (cached HTML) | 199 ms | 207 ms | 215 ms |
| Origin-MISS (Worker→KynetraDB→render) | 217 ms | 340 ms | 586 ms |
| **KynetraDB catalog build** (`server-timing: catalog;dur=…`) | **10 ms** | **13 ms** | **17 ms** (mean 10.8 ms, n=12) |

**Concurrency sweep** (origin MISS, 30 req/level, unique cache-busted URLs, **0 errors**):

| Concurrency | req/s | p50 | p90 | p99 |
|---|---|---|---|---|
| 1 | 2.5 | 308 ms | 534 ms | 2034 ms |
| 10 | 24.9 | 322 ms | 727 ms | 1073 ms |
| 20 | 48.5 | 345 ms | 762 ms | 1166 ms |
| 40 | 77.2 | 497 ms | 670 ms | 1100 ms |

**Takeaway:** unchanged from the prior run — **the DB is not the bottleneck**
(catalog reads 10 ms median / 17 ms max, 0 errors across ~150 requests). The
one cold spike (2.09 s on the very first request post-reboot, and the conc=1
p99 of 2.03 s) is the documented cold-start: OS page cache evicted + edge
keep-alive dropped during the maintenance window — successive requests settle to
~200 ms, exactly what the `kynetra-keepwarm.timer` exists to prevent once it's
had a few cycles to re-warm. Throughput ceils at ~77 req/s because it's one
client → the Singapore edge, not an origin limit. **Note:** this measures the
*currently-deployed* build — the session's new engine work (Beacon 4-5, Aether
slice 3, Caustic, Raft slices 4-6) is committed but **not yet deployed here**;
that redeploy is the pending `attach-existing.sh` step, blocked on shell access
(see session notes).

---

## 2026-07-26 — Caustic sub-bit residual quantization at 768-dim (Task 5)

**Build:** `data/db@507f204`. **Sonnet-coded.** **What:** Task 5 of the Caustic
sub-bit quantization plan — the ship gate (`tests/vector_tiering.rs::
caustic_shrink2_recall_gate`, recall@10=0.9940 at shrink=2) only proved the
codec at DIM=64 (a fast smoke width); this reruns the identical clustered-corpus
technique at DIM=768, a real embedding width, where a fixed `shrink` compresses
a bigger `proj_dim` and is a strictly harder test of the JL projection. 10k
vectors, 50 clusters, 100 queries, `examples/caustic_bench.rs`.

| Config | recall@10 | bytes/vector (amortized) | ×-vs-f32 | figure of merit (recall/byte) |
|---|---|---|---|---|
| f32 (brute-force truth) | 1.000 | 3072.0 | 1.0× | 0.000326 |
| RaBitq (default) | **0.9950** | 101.54 | 30.3× | 0.00980 |
| **Caustic shrink=2** | **0.9950** | **57.54** | **53.4×** | **0.01729** |
| Caustic shrink=4 | 0.9950 | 33.54 | 91.6× | 0.02967 |

**Takeaway:** shrink=2's recall held at 0.9950 at 768-dim — no measurable
degradation from the 64-dim gate result, and (honestly unexpectedly) shrink=4
matched it exactly on this corpus rather than showing the predicted falloff;
5 centroids (10k/2000 target_posting_len) meant every posting scan stayed
small relative to `rerank_budget`, so the exact-rerank tier likely absorbed
most of the estimator noise difference between shrink levels — a real result,
not a tuned one, and a reason to retest with more centroids/less rerank
headroom before calling shrink=4 "free." Caustic shrink=2 is the clear
per-byte winner among the three real configs. **Caveat: synthetic clustered
corpus (LCG-generated, not real embeddings); real-embedding-model validation
still pending.** Evidence: `benchmarks/evidence/2026-07-26-linux-caustic-subbit.json`.

---

## 2026-07-26 — Beacon/SPANN slice 2: incremental writes (Silt)

**Build:** `data/db@a3e9495`. **Sonnet-coded.** **What:** SPANN takes live
inserts/deletes — a memtable (**Silt**) makes writes searchable immediately, then
they settle into epoch-versioned Strata postings on a crash-safe flush. Unblocks
engine-tiering (slice 3).
**Results (15 tests, 4 new + 11 prior, all green):** insert searchable before AND
after flush; delete hidden (before/after flush); update relocates; flush→persist→
load round-trip; **crash-before-manifest recovers the pre-crash state** (no data
loss); **truncated manifest → treated absent, never silently-empty**; search merges
the mid-flush snapshot; **emptying a whole posting doesn't brick the next load**.
**Hardening (Fable found 4 crash-durability bugs → all fixed; re-review confirmed
closed + caught a 5th):** delete-before-durable-manifest (silent centroid loss),
non-atomic manifest, search-during-flush tombstone-resurrection, flush-error batch
drop, and (5th, caught on re-review) an emptied posting left a stale epoch entry
naming a deleted file → hard-error on reload. New order: write+fsync → install →
durable manifest (commit point) → delete old; load errors on a referenced-missing
posting; empty-posting drops its epoch entry.
**Takeaway:** this is the clearest proof of the **Sonnet-codes / Opus-verifies /
Fable-reviews** model — Sonnet built it fast, the review caught 4 real durability
bugs, all fixed before commit. Quality held by the gate, not the author.

---

## 2026-07-26 — Beacon/SPANN slice 1: disk-resident ANN (Sounding)

**Build:** `data/db@5f99890`. **What:** static SPANN — in-RAM HNSW centroid
navigator routes to RaBitQ postings on disk; a query does a **Sounding** (navigate
→ scan nprobe postings → estimate → exact rerank). 10k clustered vectors, dim 64,
100 queries, module-only (no engine tiering yet).

| Metric | Result |
|---|---|
| recall@10 (nprobe=10/20) | **1.000** (= HNSW 1.000) |
| **recall@10 (nprobe=2/20, ~10% of postings)** | **1.000** — routing carries recall, not coverage |
| memory | **16.43%** of raw f32 (postings-in-RAM regime) |

**Takeaway:** the navigator routes each query to the 1–2 centroids its true
neighbors live under, so ~10% of postings suffices for full recall — the
billion-scale-at-low-RAM premise holds. Slice-1 keeps postings in RAM (16.43%);
moving them to mmap'd disk drops the resident footprint to the ~2.5%
navigator-only regime. Fable-reviewed → fixed a posting-file invariant panic + a
query-dim guard (both regression-tested). Later slices: posting replication (lets
nprobe shrink further), on-disk mmap postings, SPFresh incremental rebalance,
engine auto-tiering by collection size.

---

## 2026-07-26 — Lens slice 1: exact filtered fallback (Sieve)

**Build:** `data/db@9752dd3` (+ de-flake `185e576`). **What:** high-selectivity
filtered vector search collapsed (graph traverses unfiltered then discards
non-members → almost nothing eligible survives). Fix: **Sieve** — when the filter
set is small (< 1%·N or < k·8), skip the graph and exactly score every filter
member. N=5000, k=10.

| Selectivity | \|filter\| | graph (forced) | **Lens** |
|---|---|---|---|
| 0.1% | 5 | — | **1.000** (exact) |
| 1% | 50 | — | **1.000** (exact) |
| 3% (collapse demo) | 150 | **0.265** | **1.000** |
| 10% | 500 | ~0.90 | ~0.90 (graph path, ≥0.80 floor) |

**Takeaway:** the exact fallback turns the 0.1–3% recall cliff (down to 0.265)
into 1.0, and becomes the recall oracle for slice 2 (ACORN predicate-subgraph
traversal via an instant-distance fork). Fable-reviewed → fixed a stale-filter
resurrection bug (membership-gate before exact_lookup). 3 Lens tests + 11 hnsw
green; the 10% assertion was de-flaked (graph noise) to a 0.80 floor.

---

## 2026-07-26 — Aether slice 2: replay_floor + incremental upload (Waterline)

**Build:** `data/db@2522a0e`. **What:** production-hardening the object-store
backend. Boot skips segments below the **Waterline** (replay_floor); `uploaded_floor`
is seeded from real store coverage and trusts an object only when its SIZE matches
the local file; sync/flush upload only the tail + unconfirmed seals (not the whole
WAL each time).
**Results (7 object_backend + 7 wal_v3, all green):** replay_floor skips
below-floor segments on download+replay; repeated syncs upload only the tail
(bounded PUTs, not O(log size)); uploaded_floor honest after crash-before-flush.
**Hardening (Fable-caught → fixed): a CONFIRMED WAL data-loss bug** — a truncated
store object (tail synced mid-growth, seal-upload lost on crash) was trusted as a
complete upload, then trim deleted the only good local copy → cold-boot corruption.
Fix: verify by size, not presence; download via tmp+atomic-rename. Regression test
truncates a store object and proves full recovery.
**Takeaway:** Aether is now efficient (incremental) and durable (content-verified) —
the object-storage-native log is production-shaped. Slice 3+: columnar checkpoints
on object storage, MinIO smoke, strict ObjectPut mode.

---

## 2026-07-26 — Aether slice 1: object-storage-native log (functional)

**Build:** `data/db@16a7537`. **What:** `ObjectBackend` — the WAL's source of truth
on an object store, local `UniversalLog` as an NVMe write-through cache; sealed
segments upload as atomic PUTs. Functional (durability) gates against a LOCAL
object store (`file://`, no cloud/network).
**Results (3 tests + 7 wal_v3 + 11 hnsw regression, all green):** round-trip —
uploaded segment objects are **byte-identical** to local sealed segments; **cold-boot**
— delete the entire local WAL, reopen from the store only → all records replay
identically (proves object storage is the source of truth); **trim-barrier** —
`trim_below` uploads all segments before local removal (the **Tidemark** guarantee),
then a cold-boot after trim still recovers the full set — no segment vanishes from
both tiers.
**Hardening (Fable-caught, fixed):** trim/upload race (fire-and-forget upload could
lose a segment) → barrier before trim; flush now fsyncs the local tail.
**Takeaway:** the platform bet works — a KynetraDB node's log can live on object
storage with RAM/NVMe as cache. Slice-2: replay_floor plumbing, incremental
uploads, MinIO smoke, columnar checkpoints.

---

## camerademo.qosx3.space — live-store speed tracking (ongoing)

Recurring speed/perf checks of the live KynetraDB-on-DO demo (India → CF-Singapore
edge). Newest first.

- **2026-07-26 14:34Z** — edge-HIT p50 **227ms** / p90 300 / p99 311; KynetraDB
  catalog build **8ms p50 / 17ms p90** (max 151 cold); concurrency-20 (60 req)
  **0 errors**, lat p50 354 / p90 525 / max 711ms. DB not the bottleneck (catalog
  8ms); latency is edge/network-bound. Consistent with the baseline below.
- **2026-07-26 (earlier)** — edge-HIT p50 252ms; catalog **9ms p50 / 18ms p99**;
  concurrency sweep to 52 req/s @ P40, 0 errors across ~340 req. (Baseline run.)

*(These measure the DEPLOYED engine; the session's new work — Prism-R, Beacon,
Lens, Aether, Mnemos — is not deployed to the droplet yet. The KynetraSearch 10x
demo, task #5, will run against this URL once the DB build + that upgrade land.)*

---

## 2026-07-26 — Mnemos slice 5: Reflect action + AppState hoist (functional)

**Build:** `data/db@a3ec39c`. **First Sonnet-coded slice** (Sonnet codes / Opus
verifies / Fable reviews). **What:** agents can now **Reflect** — recall their own
memory mid-run via a `recall_memory` action; MemoryService hoisted into AppState
(shared Arc) so Lore's graph adjacency persists across requests instead of
rebuilding per request. **Completes Mnemos (all 5 slices).**
**Results (40 lib + 6 integration tests, all green):** recall_memory returns
seeded episodes; missing-query errors; top_k/graph_hops clamped; builtin action
registry now 13 (registration verified).
**Review:** Fable found **no correctness bugs** — the now-shared graph_cache RwLock
is safe (no lock held across await, benign concurrent double-rebuild, watermark
re-checked per call). Confirms Sonnet + the verified loop holds the quality bar.
Follow-up flagged: thread agent_id through ActionContext to scope the action's
recall (currently cross-agent).
**Takeaway:** Mnemos is end-to-end complete — form (Distill) → store (Echo/Lore/
Craft) → recall (Confluence) → Reflect (agent self-recall), all over the one log.

---

## 2026-07-26 — Mnemos slice 4: consolidation / Distill (functional)

**Build:** `data/db@f1da5f9`. **What:** the memory-formation loop — an async,
watermarked LLM job (**Distill**) turns raw `agent_run` entities into durable
episodes/facts/procedures, written with content-hash ids so replay never
re-invokes the LLM. Route `POST /v1/memory/consolidate`. Injectable extractor →
hermetic tests.
**Results (25 memory tests, 3 new + 22 prior, all green, stable 5/5):**
extract→episode/facts/procedures + watermark advance; **replay-idempotent** (run
twice → byte-identical `mnm-*` state, no dups); **ms-collision** (two runs same
timestamp across a batch boundary → neither stranded); **supersession** (a newer
contradicting fact closes the prior — Palimpsest, even for consolidated facts);
**poison-pill** (a 16KB-capped oversized/failing run is skipped-and-logged and the
watermark advances past it — can't wedge consolidation).
**Hardening (Fable-caught → fixed):** watermark ms-collision skip, missing fact
supersession, poison-pill wedge, 128-bit hash.
**Takeaway:** Mnemos is now end-to-end — memory is *formed* (Distill) from runs,
stored across 3 tiers, and recalled (Confluence), all over the one log,
time-travelable + tamper-evident. Remaining Mnemos: slice 5 (agent recall action +
hoist MemoryService into AppState). Coding shifts to **Sonnet-codes /
Opus-verifies / Fable-reviews** starting with the next slice (Beacon 2).

---

## 2026-07-26 — Mnemos slice 3: Craft procedural tier (functional)

**Build:** `data/db@217a99d`. **What:** the Craft (procedural) tier — learned
skills as `memory_proc` entities, upsert-by-name (**Groove**), success reinforced
via OCC. Completes all 3 Mnemos tiers (Echo + Lore + Craft). Routes
`/v1/memory/procedures`.
**Results (19 memory tests, 6 Craft + 13 prior, all green):** roundtrip; **upsert
by name** (re-put same name → one record, not versioned history like Lore);
`record_success` bumps count under OCC retry; list orders by success_count.
**Hardening (Fable-reviewed → fixed):** re-put PRESERVES the proven `success_count`
(client can't forge/reset it) and carries the prior embedding forward on an embed
outage (no vector drop); input caps (name/trigger/steps) guard an oversized-id DoS.
**Takeaway:** episodic + semantic + procedural memory now all present. Remaining
Mnemos: slice 4 consolidation (LLM background job → durable memory, idempotent),
slice 5 agent-recall action + hoist MemoryService into AppState.

---

## 2026-07-26 — Mnemos slice 2: Lore semantic graph (functional)

**Build:** `data/db@6c5e7d9`. **What:** the Lore tier — facts/edges as bi-temporal
log entities, in-RAM adjacency (LoreGraph), directed BFS `neighbors(node,hops,as_of)`,
and 3-way recall (BM25+vector+**graph-hop**) fused via RRF. Functional gates.
**Results (13 tests, 5 new + 4 temporal-edge, all green):** graph traversal
respects the hop budget (C reachable at hops=2, not hops=1); `as_of` before an
edge's `valid_from` hides it and `as_of=None` == `Some(now)`; **Palimpsest** —
a contradicting fact closes the prior's window at `new.valid_from` (not delete),
so both are queryable as-of; backfilled (closed) facts don't disturb the live one;
`graph_hops=0` preserves slice-1 recall.
**Hardening (Fable-caught, fixed):** 4 bi-temporal bugs (as_of=None semantics,
backfill closing the live fact, boundary overlap, TOCTOU race) → the close+insert
is now one atomic OCC transaction under a global write lock.
**Takeaway:** episodic + semantic tiers now compose; Craft (procedural) +
consolidation are slices 3–5.

---

## 2026-07-26 — Mnemos slice 1: Echo + Confluence recall (functional)

**Build:** `data/db@024d7bc`. **What:** 3-tier agent memory (this slice: Echo
episodic + fused recall) as a view over the log; routes `/v1/memory/{recall,
episodes,stats}`. Functional (not perf) gates.
**Results (4 hermetic tests, no network):** planted-needle recall finds the target
(top-1, within top-5) among 100 noise episodes; **agent-scoping-without-starvation**
— target agent still returned amid 60 noisy-neighbour episodes from another agent
sharing query terms, no foreign leak; **degradation** — embeddings unreachable →
recall still serves BM25 hits with `degraded:true` (never fails); remember/recent
round-trip.
**Hardening (Fable-caught, fixed):** `top_k` clamped 1..100 (was a Viewer-triggerable
DoS via `top_k*8` overflow / `Vec::with_capacity(huge)`); agent_id scoped in
retrieval before fusion; vector-index errors surfaced as `degraded` not swallowed.
**Takeaway:** Confluence (BM25+vector RRF, per-agent, graceful) works end-to-end;
Lore (semantic graph) + Craft (procedural) + consolidation are later slices.

---

## 2026-07-26 — Prism-R slice 2: Seedprint persistence (boot-from-disk)

**Build:** `data/db@75f9d6e`. **What:** a RaBitQ HNSW index now persists as
`{ seed, dim, codes }` and reloads without re-encoding the corpus — the rotation
is reconstructed deterministically from the stored **seed** (Seedprint), not the
matrix. `SNAPSHOT_VERSION` 3→4 (old snapshots gracefully rebuild); seed mismatch
→ refuse+rebuild.
**Result:** round-trip recall@10 **1.000 → 1.000** (unchanged); reloaded codes
**byte-equal** to freshly-encoded (determinism tripwire); seed-B load correctly
refused. Boot now loads the index instead of re-encoding every vector.
**Takeaway:** completes Prism-R end-to-end (build → search → persist/reload). Next
Prism-R work is optional (mmap the snapshot vs read-into-RAM). 7 rabitq + 11 hnsw
+ 18 vector_hnsw tests green.

---

## 2026-07-26 — Prism-R in HNSW (graph-integrated) + Refract

**Build:** `data/db@850e7a0`. **What:** `QuantMode::RaBitQ` now drives HNSW graph
traversal (not just brute-force) — the "Refract" step (rotation + unbiased
estimate) navigates on 1-bit codes, Facet rerank corrects. DIM=32, N=2000, 40
clusters, 50 queries, both indices exact-reranked.

| Mode (in HNSW) | recall@10 | compression |
|---|---|---|
| Prism-8 (int8) | 1.000 | 8× |
| **Prism-R (RaBitQ, graph-integrated)** | **0.996** | 32× |

**Takeaway:** the codec that scored **0.10** as ad-hoc binary now **matches int8
(0.996 vs 1.000) inside the graph at 32× compression** — 4× denser than int8 at
near-identical recall. Dimension-mismatch guards match Int8/Binary sentinel
semantics (no panic; regression-tested). Fable-reviewed → dim-guard bug caught →
Opus-fixed. Next: v4 loadable RaBitQ snapshot (slice 2, so boot doesn't rebuild).

---

## 2026-07-26 — Prism-R (RaBitQ) codec + packed API

**Build:** `data/db@5bc251d` (codec), packed-bits API (this build).
**What:** vector quantization recall on clustered data (DIM=32, N=1000, 40 clusters).

| Codec | recall@10 | compression | estimator bias | source |
|---|---|---|---|---|
| Prism-8 (int8) | ~0.96 top-1 | 8× | — | `hnsw.rs` (existing) |
| Prism-1 (ad-hoc binary) | **0.10** | 32× | — | `hnsw.rs::prism1_binary_quant_via_config_recall` |
| **Prism-R (RaBitQ)** | **0.96** | 32× | **−0.019** | `rabitq.rs::prism_r_recall_beats_binary_baseline` |

**Method:** brute-force index + exact rerank, over-fetch 64, recall@10 vs exact
cosine. Estimator bias = mean(estimate − true IP) over 400 random pairs.
**Packed API:** `encode_packed`/`estimate_packed` verified byte-identical to the
reference `Vec<bool>` path (`packed_encode_and_estimate_match_bool_version`).
**Takeaway:** RaBitQ's unbiased estimator closes the binary recall gap (0.10 →
0.96) at the same 32× compression — the mandate for integrating it into the HNSW
graph (Prism-R→HNSW, roadmap #1). Next benchmark: recall@10 of the *graph-
integrated* RaBitQ vs int8 at equal memory.

---

## 2026-07-26 — Live DO demo store (`camerademo.qosx3.space`)

**Build:** the deployed KynetraDB on a DigitalOcean droplet behind Cloudflare
(session-new work NOT yet deployed there). **Client:** India → CF-Singapore edge.
**What:** end-to-end latency, KynetraDB catalog build time, concurrency, errors.

| Path | p50 | p90 | p99 |
|---|---|---|---|
| Edge-HIT (cached HTML) | 252 ms | 349 ms | 367 ms |
| Origin-MISS (Worker→KynetraDB→render) | 260 ms | 371 ms | 506 ms |
| **KynetraDB catalog build** (`server-timing`) | **9 ms** | **11 ms** | **18 ms** |

**Concurrency sweep** (origin MISS, 60 req/level, **0 errors across ~340 req**):

| Concurrency | req/s | p50 | p90 | p99 |
|---|---|---|---|---|
| 1 | 3.0 | 265 ms | 340 ms | 890 ms |
| 10 | 30.9 | 262 ms | 334 ms | 571 ms |
| 20 | 43.0 | 313 ms | 438 ms | 639 ms |
| 40 | 52.0 | 581 ms | 818 ms | 895 ms |

**Method:** `curl` timing; catalog time from the origin's `server-timing:
catalog;dur=…` header on cache-busted (MISS) requests; concurrency via
`xargs -P`. Read-only GETs.
**Takeaway:** the **DB is not the bottleneck** — catalog reads are 9 ms median /
18 ms p99, **0 errors**. Throughput ceils at ~52 req/s because it's one client →
the Singapore edge, not the origin. Only wart: a rare cold-start spike (→1.9 s on
the first request after idle), addressed by the keep-warm timer.

---

## 2026-07-26 — Lens slice 2 (ACORN) wired into the hot search path; Raft slice 4 (real HTTP transport)

**Lens/ACORN** (`crates/instant-distance-fork` + `vector_hnsw.rs`'s
`search_graph_filtered`) replaces unfiltered-graph-search-then-post-filter-
discard with a real predicate-aware traversal for filtered vector queries.
Recall@10, measured on a 6,000-vector clustered corpus (before = naive
post-filter, after = ACORN):

| Selectivity | Before | After |
|---|---|---|
| 10% | 0.9067 | 1.0000 |
| 5% | 0.5200 | 1.0000 |

Fable-reviewed clean (PointId indexing safety across all 3 quant modes
confirmed, param wiring confirmed complete, no persistence round-trip risk).
Two non-blocking perf follow-ups noted: no distance-based early termination
in the fork's traversal (~10k node-expansion fixed cost per filtered query
above the brute-force threshold), and `ef` can narrow the rerank pool below
`fetch_pool`'s intended sizing at large `k` — both flagged for a later pass,
neither a correctness issue.

**Raft slice 4** (`crates/server/src/raft_net.rs`) proves the sim-tested
election/replication protocol also holds over a REAL network stack (real
HTTP via `reqwest`/axum, real async, real serialization) — not just the
in-process `SimNet`. Opt-in via `KYNETRA_RAFT_ENABLED` (default off, zero
behavior change for existing deployments — verified: all 164 pre-existing
`kynetra-server` tests pass unmodified with the feature off). Test
`real_http_transport_elects_and_replicates` proves a real 2-node election +
replication round-trip over real sockets.

---

## 2026-07-26 — Beacon slices 4-5 close out; the original 6-subsystem 10x build is complete

**Beacon/SPANN** gains posting replication, mmap-based posting reads, and
LIRE split/merge/reassign rebalancing (`crates/core/src/personalities/
vector_spann.rs`). Recall@10 at nprobe=2: **0.829 → 0.953** from replication
alone. `posting_replication` defaults to 1 everywhere (RAM economy is the
whole point of the disk tier); the engine's Disk-tier construction sites opt
into 2 explicitly, so nothing changes silently for any other caller.

Two Fable review rounds — round 1 caught a real navigator-visibility race (a
rebalance could update postings before the in-memory centroid graph, so a
concurrent search briefly saw new rows with a navigator that didn't know
their new home existed — a committed row transiently unfindable) and a
replica-unaware merge/reassign path (could duplicate an id within one
posting, or collapse its replica coverage, completely untested at
replication ≥ 2). Round 2, targeted specifically at the fixes, verified the
navigator fix by reading the actual lock-acquisition sequence on both sides
(not just trusting the prose) — confirmed it genuinely serializes rather
than narrowing the race window.

**Raft is now fully done through slice 6** (membership change + learner
catch-up, per-tenant shard groups) — 1000-seed sweep green with partition,
kill, and membership-change fault injection all genuinely wired into the
randomized property loop, not left as fixed scenarios.

**All 6 subsystems of the original build — Prism-R, Aether, Lens, Beacon,
Mnemos, Raft — are now shipped and independently verified.**

---

## How to reproduce
- **Codec/engine benches:** `cd data/db && cargo test -p kynetra-core --lib rabitq -- --nocapture` (and `--test hnsw`).
- **Live store:** the scripted sweep lives in the session scratchpad; re-run any
  `curl … -w "%{time_total}"` loop against the origin and read `server-timing`
  for the KynetraDB slice.
- Follow the repo convention `data/db/benchmarks/evidence/*.json` for
  machine-readable per-build evidence when wiring CI gates (ties to the Foundry
  `VECTOR_SUBSTRATE_KPIS`).