Search & retrieval

Vector quantisation (int8, binary, RaBitQ, Caustic)

Shrinks the memory an approximate vector index needs, while exact reranking protects recall.

RoadmapEngine edition 5f8ea3af

What it does

The engine implements four ways to store index payloads in fewer bytes: int8 scalar quantisation, 1-bit sign quantisation, a 1-bit RaBitQ code with an unbiased estimator, and the Caustic sub-bit residual codec used by the disk tier. Each has tests and, for most, a published artifact that includes the trade-off and the misses. None of the modes is selectable on the server, so the capability is roadmap.

How it works

int8 is the default for the HNSW graph. Each vector is L2-normalised and stored as one signed byte per dimension plus a four-byte per-vector scale, with an exact integer accumulator for the dot product. The query is never quantised for the final score: graph hits are reranked as the cosine between the raw query and the dequantised candidate, and the engine can rerank against the exact f32 embedding in the entity store.

Binary mode keeps one sign bit per dimension and ranks by Hamming distance. It is a coarse routing signal, so exact rerank is effectively mandatory and the default over-fetch is four times. RaBitQ mode keeps a 1-bit code at the same footprint but ranks with an unbiased inner-product estimator, which routes closer to the true neighbourhood and needs a narrower net (default over-fetch of two). The disk-tier index stores RaBitQ or Caustic codes in its postings; Caustic encodes a residual against the centroid at sub-bit cost and is the default for a fresh disk tier.

Measured, local scope, one host (Cortex-X925, 121 GB RAM): RaBitQ quantization holds recall@10 at 0.995 on a 768-dimension corpus while storing 101.5 bytes per vector, 30.3x smaller than f32. The run is an A/B on one corpus reused across all four codecs, and bytes per vector come from the codec encoder, not a formula. The artifact also records that whole-search throughput is lower than f32 at this posting length: the win is memory, not latency.

Measured, local scope, one host (Cortex-X925, 121 GB RAM): binary quantization recovers recall@10 to 0.9855 at the default 4x over-fetch, but solely with exact rerank, which is mandatory in this mode. Raw 1-bit Hamming routing recalls about 0.20 before rerank. The artifact states that the memory and build-time gains are large and real, and that the speed half of the usual memory-and-speed framing does not carry over end to end once rerank and wider over-fetch are paid for; the raw distance primitive is the one part that is faster.

Measured, local scope, one host (Cortex-X925, 121 GB RAM): int8 quantization meets its memory and latency targets at both measured scales, and its recall gate at 100k, but misses that gate at 1M on the default search path. The artifact records the miss under recall_gate_at_scale and states that it is not fixed.

Where you reach it

  • LibraryHnswParams.mode = Int8 | Binary | RaBitQ via EngineConfig::vector_quant; SpannParams.codec = RaBitq | Caustic via EngineConfig::disk_vector_codec (Rust engine API; not exposed by the server)

Limits

  • Not selectable through any route, command or setting of the private-preview server.
  • Quantised payloads trade recall for memory. Binary and RaBitQ need exact rerank to be usable, and rerank reads the full-precision embedding.
  • The int8 recall miss at 1M vectors is open in the published artifact.
  • All published figures are local scope on one Linux host, from a single operator, with no third-party reproduction.
  • Caustic results are published in the same artifact as RaBitQ but are not quoted here, because they are not part of the registered evidence set for this site.

Evidence

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

Every figure on this site is local-scope: one host, one operator, no third-party reproduction. Artifacts are linked next to each number, and gates that were missed are published beside the gates that were met.

KynetraDB is in private preview

This is not available in the preview. Request a preview to discuss the current scope.