What it does
The ingestion pipeline sits on top of the connector fabric. A pipeline names a source, a sealed credential and a chunking policy; a job runs it through fetch, normalise, identity, deduplicate, chunk, redact, embed and index, and records lineage. Jobs persist their checkpoint, retry within a bound and dead-letter.
How it works
POST /v1/ingestion/pipelines stores a pipeline: a source inlet, a reference to a sealed credential (never the credential itself), a chunk policy and optional quotas. The embedding model is pinned when the pipeline is created. POST /v1/ingestion/jobs queues a job, POST /v1/ingestion/jobs/{id}/run drives it, and POST /v1/ingestion/jobs/{id}/cancel stops it.
A run pulls records through the chosen inlet and processes each document in order: it normalises the content, derives a stable document identity, and skips content it has already seen by hash. It splits the text with either a fixed character window with overlap or a paragraph-first splitter, passes each chunk through a redaction hook, embeds the chunks and indexes them. Quotas can cap documents per run, chunks per run, chunks per document and characters per document, which bounds embedding spend.
A job records its state (queued, running, succeeded, cancelled or dead-lettered), its checkpoint and any failures. A failure is classified as retryable or not; retryable failures back off up to a bound and then dead-letter, and a run that reaches a recorded verdict returns 200 even when that verdict is dead-lettered. Cancelling stops the run; work already committed stays committed. Each commit also records a document-lineage revision, so a chunk can be traced back to its source.
Everything is stored as entities in the log under project-scoped kinds and ids, so a pipeline or job in one project is invisible to another.
Where you reach it
- HTTP
POST, GET /v1/ingestion/pipelines; GET /v1/ingestion/pipelines/{id} - HTTP
POST /v1/ingestion/jobs; GET /v1/ingestion/jobs/{id}; POST /v1/ingestion/jobs/{id}/run; POST /v1/ingestion/jobs/{id}/cancel - Config
KYNETRA_ROOT_KEY (enables the routes); embedding provider settings read from the environment
Limits
- Running a job is synchronous: the request stays open until the job reaches a terminal state, so a large source holds the connection.
- The routes are switched off without KYNETRA_ROOT_KEY, because a pipeline cannot reach its source without the credential vault.
- Embeddings come from an Ollama-compatible provider configured in the environment.
- The default redaction hook passes content through. A deployment that needs redaction must supply a policy.
- Sources are the connector fabric's inlets, with the limits listed on that entry.
Evidence
- On this siteAI data plane
- TestsCovered in the engine repository by more than 80 ingestion tests (identity and deduplication, chunking, quotas, retry and dead-letter, lineage, project isolation) and 17 HTTP tests (authentication, pinned response shapes, deduplicated second run, cancelled and dead-lettered runs, project isolation)
Test coverage is listed as present in the engine repository at this edition. This page does not report test results.
KynetraDB is in private preview
Request a preview to talk through the limits above against your own workload.