Data model & realtime

Knowledge ingestion pipeline

Turns documents from a connected source into deduplicated, chunked, embedded and indexed records, with a resumable job and a recorded lineage for each chunk.

BetaEngine edition 5f8ea3af

What it does

The ingestion pipeline sits on top of the connector fabric. A pipeline names a source, a sealed credential and a chunking policy; a job runs it through fetch, normalise, identity, deduplicate, chunk, redact, embed and index, and records lineage. Jobs persist their checkpoint, retry within a bound and dead-letter.

How it works

POST /v1/ingestion/pipelines stores a pipeline: a source inlet, a reference to a sealed credential (never the credential itself), a chunk policy and optional quotas. The embedding model is pinned when the pipeline is created. POST /v1/ingestion/jobs queues a job, POST /v1/ingestion/jobs/{id}/run drives it, and POST /v1/ingestion/jobs/{id}/cancel stops it.

A run pulls records through the chosen inlet and processes each document in order: it normalises the content, derives a stable document identity, and skips content it has already seen by hash. It splits the text with either a fixed character window with overlap or a paragraph-first splitter, passes each chunk through a redaction hook, embeds the chunks and indexes them. Quotas can cap documents per run, chunks per run, chunks per document and characters per document, which bounds embedding spend.

A job records its state (queued, running, succeeded, cancelled or dead-lettered), its checkpoint and any failures. A failure is classified as retryable or not; retryable failures back off up to a bound and then dead-letter, and a run that reaches a recorded verdict returns 200 even when that verdict is dead-lettered. Cancelling stops the run; work already committed stays committed. Each commit also records a document-lineage revision, so a chunk can be traced back to its source.

Everything is stored as entities in the log under project-scoped kinds and ids, so a pipeline or job in one project is invisible to another.

Where you reach it

  • HTTPPOST, GET /v1/ingestion/pipelines; GET /v1/ingestion/pipelines/{id}
  • HTTPPOST /v1/ingestion/jobs; GET /v1/ingestion/jobs/{id}; POST /v1/ingestion/jobs/{id}/run; POST /v1/ingestion/jobs/{id}/cancel
  • ConfigKYNETRA_ROOT_KEY (enables the routes); embedding provider settings read from the environment

Limits

  • Running a job is synchronous: the request stays open until the job reaches a terminal state, so a large source holds the connection.
  • The routes are switched off without KYNETRA_ROOT_KEY, because a pipeline cannot reach its source without the credential vault.
  • Embeddings come from an Ollama-compatible provider configured in the environment.
  • The default redaction hook passes content through. A deployment that needs redaction must supply a policy.
  • Sources are the connector fabric's inlets, with the limits listed on that entry.

Evidence

  • On this siteAI data plane
  • TestsCovered in the engine repository by more than 80 ingestion tests (identity and deduplication, chunking, quotas, retry and dead-letter, lineage, project isolation) and 17 HTTP tests (authentication, pinned response shapes, deduplicated second run, cancelled and dead-lettered runs, project isolation)

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

Request a preview to talk through the limits above against your own workload.