What it does
The bulk crate implements streamed NDJSON export with a manifest and an order-independent dataset digest, and chunked import with resume and quarantine. It is a library, and no route or command calls it.
How it works
Export pulls entities from any iterator without materialising the dataset and writes bounded chunk files, each with a header line, key-sorted entity lines and a trailer carrying the chunk's SHA-256 and record count. A manifest with the chunk list and an aggregate digest is committed atomically and last, so a source error leaves no manifest. A durable export state makes export resumable, and a resumed export ends byte-identical to an uninterrupted one.
Import validates the manifest and the exact chunk set before any record moves, verifies each chunk against the manifest in a pass that never touches the sink, and then applies the chunk in bounded batches. A durable checkpoint per chunk makes it resumable, malformed and sink-refused records go to an appended quarantine file, and the run reconciles applied, skipped and quarantined counts to the manifest total. A digest recorded at export is recomputed from what was applied and compared.
Where you reach it
- Library
kynetra_bulk: export_records, export_chunks, import_chunks, DatasetDigest (engine library, not wired to the server)
Limits
- Not reachable from the server or the command line.
- The crate states that a terabyte-scale run is external evidence: the harness is parameterised by scale, and the local run used the largest fixture the disk allowed. Nothing here asserts terabyte behaviour.
- CSV is a non-streaming conversion helper outside the streaming path.
Evidence
- TestsCovered in the engine repository by about 100 tests for export, import, lossless CSV conversion and export resume
Test coverage is listed as present in the engine repository at this edition. This page does not report test results.
KynetraDB is in private preview
This is not available in the preview. Request a preview to discuss the current scope.