What it does
Pulse is a durable job queue built on the engine's transactions. Every state change is one atomic commit, claims carry a generation that fences older holders, and retries back off before dead-lettering. Over HTTP it serves the Pipelines control plane's own queue; the general queue is a library API.
How it works
Queue state is a set of entities: a queue row, job rows, an event per terminal transition, a due marker per waiting job and a durable outcome row per operation. Every transition commits as one engine transaction, so the job row, its due marker, its event and, for a claim, its outcome row appear together or not at all. Identities are scoped by project, queue and job type.
A worker claims by discovering due markers in order of scope, job type, due time and job id. Discovery pages that ordered range, so it never scans the job corpus, and a claim is a conditional commit that binds both the job row and its marker as observed. A claim bumps the job's lease generation. Completing or failing a job presents the generation and, in the control plane, a claim token, so a holder whose lease expired and was reclaimed is refused. A scan budget and a bounded number of compare-and-set retries stop a claim call from spinning; a bounded call that cannot decide reports that more work may remain, never that the queue is empty.
Each claim writes an immutable outcome row in the same transaction. If a response is lost and the worker retries with the same operation identity, it receives the original selection, not a second job; a changed request under the same identity conflicts with no effects. Failed attempts back off with a capped exponential delay and dead-letter once their attempts are used. A bounded expiry sweep counts the attempt that an expired claim consumed, so a worker that crashes on every delivery cannot be redelivered forever.
Delivery is at least once. A claimed job whose worker dies is redelivered after its lease deadline, so work must tolerate repeats.
Over HTTP, queue operations live under /v1/pipelinecp: queue:claim_next, queue/items/{id}:complete, queue/items/{id}:release, queue:requeue_expired, queue:enqueue and GET queue/health. They act on one system queue with fixed job types for scheduling, provisioning, teardown, reporting and reconciliation, and they need a controller credential.
Where you reach it
- HTTP
POST /v1/pipelinecp/queue:claim_next, /queue/items/{id}:complete, /queue/items/{id}:release, /queue:requeue_expired, /queue:enqueue - HTTP
GET /v1/pipelinecp/queue/health - Library
ScopedQueue, QueuePlanner (kynetra-pulsequeue)
Limits
- The HTTP surface exposes a single system queue with fixed job types, for the Pipelines control plane. There is no public API to create arbitrary queues; the general scoped queue is a library API.
- Delivery is at least once, never exactly once.
- The crate authenticates nobody. The server authenticates callers and supplies the project and principal.
- A claim can return a continuation instead of work when the scan budget or the retry bound is reached; callers poll again.
Evidence
- On this siteReliability
- TestsCovered in the engine repository by more than 100 queue tests (claim contract, scoped contract, outcome replay, due-index and plan builders), including a harness that kills a claiming process with SIGKILL at durable boundaries and checks that retries replay the original claim
Test coverage is listed as present in the engine repository at this edition. This page does not report test results.
KynetraDB is in private preview
Request a preview to talk through the limits above against your own workload.