Data model & realtime

Pipelines control plane

Stores the state of pipeline runs and jobs, and the operations that move them, with retries that replay the original answer instead of repeating a change.

BetaEngine edition 5f8ea3af

What it does

The control plane is a set of HTTP operations over the engine for repositories, runs, jobs, leases, approvals and reporting. Each mutating operation records an immutable outcome row in the same transaction as its effects, so a retry replays the original ids and decisions. It is served at /v1/pipelinecp to controller credentials.

How it works

Operations cover the lifecycle of a pipeline run: ensuring a repository record, atomic intake of a run with its jobs, job admission and skip, termination, approval, retry and cancellation, an executor family (begin a lease, attach a resource, open a session, heartbeat, complete, report cleanup, expire a lease), run finish, a reporter outbox with bounded retry and dead-lettering, and a queue family backed by the Pulse queue. GET /v1/pipelinecp/capabilities returns a document that describes what this node actually serves, including its live durability mode, writer role and envelope limits, and lists just the primitives that are implemented.

Every mutation first reads an immutable outcome row, then commits a new one in the same transaction as its effects. A retry therefore replays the original ids, times and decisions and never recomputes them; a changed request conflicts with no effects; and racing identical requests converge. The operation identity is minted by the server from the authenticated project, the resolved tenant, the operation and a bounded client hint, never taken from a request body. Every envelope is counted, and one above the engine's limit is refused before any effect.

Every mutation refuses before any effect unless the node is the primary writer, the engine runs strict durability, and the engine is not recovery-uncertain. Reads are fallible, paginated and report partial results honestly. A guarded migration import (begin, import, validate, activate) holds a project in a migration-in-progress state, in which every other operation answers 409, until the imported state is validated and activated; nothing imported carries authority and nothing is overwritten.

Control-plane rows use reserved kinds and ids that the generic read and write routes hide and refuse, and they carry no top-level user identifier, so a user erasure cascade cannot delete lifecycle state.

Authentication reuses the server's credential resolver. A project service key, the instance key with a project header, or an operator session with an administrator or owner role maps to the controller principal. End-user and anonymous credentials are refused.

Where you reach it

  • HTTPGET /v1/pipelinecp/capabilities
  • HTTPPOST /v1/pipelinecp/repositories:ensure, /runs:intake, /runs/{id}:finish, /runs/{id}:cancel
  • HTTPPOST /v1/pipelinecp/jobs/{id}:admit, :skip, :terminate, :approve, :retry; /jobs/{id}/leases:begin, /leases/{generation}:heartbeat, :complete, :expire
  • HTTPPOST /v1/pipelinecp/migration:begin, :import, :validate, :activate; GET /v1/pipelinecp/migration

Limits

  • The controller principal alone can call it. The tenant principal class exists in the control plane, but no existing credential maps to it yet, so tenant-scoped callers are refused today.
  • Delivery is at least once. Nothing here claims exactly-once execution of external work.
  • The control plane stores and orders state. The work itself runs in workers outside the database.
  • The expired-lease listing is a bounded scan and reports a partial result when it hits its bound. An intake above the engine envelope is refused with a 413 rather than staged.
  • A replica refuses control-plane mutations instead of forwarding them.

Evidence

  • On this siteHTTP API reference
  • TestsCovered in the engine repository by more than 250 control-plane tests (intake, admission, executor leases, approval and retry, reporter, queue operations, replay, serving gate, migration), 34 in-process HTTP tests and 9 server-level HTTP tests

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

Request a preview to talk through the limits above against your own workload.