What it does
Administrator routes rewind the log to a chosen sequence number, keep a backup of what was dropped so the rewind can be undone, and wipe everything on explicit confirmation. Each destructive step writes a durable intent first. If a step fails, panics or is killed, the engine answers 503 recovery_uncertain instead of serving state that may not match the log.
How it works
POST /admin/db/restore takes a target sequence number and a typed confirmation phrase, and rewinds the log: every record at or above the target is dropped, the in-memory state and the search and vector indexes are rebuilt, and the records that were dropped are first written to a restore backup file. GET /admin/db/restore-backups lists those backups, and POST /admin/db/restore-undo replays one back; the backup path must lie inside the data directory. POST /admin/db/delete-all wipes the database after its own confirmation phrase. GET /admin/db/time-travel reports the current position, the entity count and an estimated time window. All of these need an administrator, and restore is recorded in the compliance trail.
Rewinding re-issues sequence numbers, so stale derived state is the hazard. The engine deletes stale checkpoint artifacts before it truncates the log, fences detached checkpoint writers with an epoch, and rejects a checkpoint that claims to be ahead of the log. Concurrent writers are quiesced for the operation, and after a rewind the index snapshots are rewritten. A restore does not carry file object bytes; a request that asks for them is refused.
Before a destructive step touches durable history, the engine persists and fsyncs an intent record in the data directory, using a temporary file, an fsync, a rename and a directory fsync. A completion marker replaces it after the operation finishes durably, not before. An intent that is still present at open marks the engine uncertain: authoritative reads and writes fail closed with a 503 and the code recovery_uncertain, nothing is loaded as authoritative for mutation, and opening refuses until the intent is resolved. If a failure leaves a live process uncertain, an in-process latch closes the gate for queued writers as well.
While uncertain, the engine also refuses forced checkpoints, log trimming, vector-index builds, temporal snapshots, replication ingest and backups, so it cannot persist or copy state derived from an unresolved rewind. Real-process tests kill a child with SIGKILL at each durable boundary of the operation and check that the next open refuses with the typed error.
Where you reach it
- HTTP
POST /admin/db/restore; GET /admin/db/restore-backups; POST /admin/db/restore-undo - HTTP
POST /admin/db/delete-all; GET /admin/db/time-travel
Limits
- A rewind is destructive: records at or above the target are dropped. Take a backup first, and keep the restore backup until you are sure.
- Resolving an unresolved recovery intent is a manual operator action on the data directory followed by a restart. There is no command for it yet.
- File object bytes are outside the log and are not rewound.
- For library callers, the older read methods keep their signatures and still return none, empty or zero while the engine refuses; callers that must tell absence from failure use the fallible read methods.
- The time window reported by time-travel is estimated from entity timestamps and is approximate.
Evidence
- On this siteReliability: recovery and Tideline findings
- TestsCovered in the engine repository by 11 restore tests, 7 restore-and-checkpoint consistency tests, 21 recovery-fence tests across vector search, aggregates, snapshots, checkpoints and replication, a SIGKILL crash harness with 15 cases, and HTTP tests that check the typed 503 on mounted routes
Test coverage is listed as present in the engine repository at this edition. This page does not report test results.
KynetraDB is in private preview
Request a preview to try this against your own workload.