What it does
An embedding job selects records of a kind (or the embeddings of an earlier model), embeds them in batches through the configured embedding provider and stores the vectors as managed embeddings. A re-run skips whatever is current, a killed job resumes from its checkpoint, failing records go to a dead-letter list instead of stopping the job, and a re-embed job migrates a population to a new model.
How it works
POST /v1/embedding-jobs/jobs creates a job for a project. The selector is either records, which names a source kind, an optional text attribute path and an optional attribute-equals filter, or reembed, which selects existing embeddings made by an earlier model. Batch size defaults to 32, retries use a bounded exponential backoff, and an optional rate cap limits embeddings per second. The provider, model and vector width are pinned from the live provider at creation, not chosen by the caller, so the model cannot drift under a running job.
POST /v1/embedding-jobs/jobs/{id}/run drives the job to a terminal state inside the request. Each embedding is stored as its own entity under a kind that encodes the model and under an id derived from the model and the source record, so a re-run finds the existing embedding and skips it when the content hash still matches. A positional checkpoint is saved after every batch, advancing solely after the vectors are durably written, so a crash repeats at most one batch.
A record that cannot be embedded, for example because the provider returns the wrong width, is retried within the budget and then recorded as a dead letter with its source id, stage and reason, never its text. The job still finishes. A job-level failure, such as a quota breach or an unreachable provider after retries, ends in the dead-lettered state. Cancel is a durable flag re-read at each batch boundary.
Re-embed jobs read the embeddings of an old model and write a new population under a different kind, so a model migration, including a change of width, never overwrites the old vectors. The stale policy chooses whether the old embeddings are marked superseded (the default) or left live so both populations stay searchable while the new model is validated. Per-project quotas cap concurrent running jobs and records per job.
Where you reach it
- HTTP
POST /v1/embedding-jobs/jobs, GET /v1/embedding-jobs/jobs, POST /v1/embedding-jobs/jobs/{id}/run, POST .../cancel, GET .../dead-letters, POST /v1/embedding-jobs/reembed, POST and GET /v1/embedding-jobs/quotas - Config
KYNETRA_EMBED_URL, KYNETRA_EMBED_MODEL, KYNETRA_EMBED_DIM
Example
curl -s "$KYNETRA_URL/v1/embedding-jobs/jobs" \
-H "Authorization: Bearer $KYNETRA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"project": "acme",
"name": "embed articles",
"selector": { "type": "records", "source_kind": "article", "text_path": "body" },
"batch_size": 32
}'
curl -s -X POST "$KYNETRA_URL/v1/embedding-jobs/jobs/$JOB_ID/run" \
-H "Authorization: Bearer $KYNETRA_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "project": "acme" }'Limits
- run is synchronous: the request stays open until the job reaches a terminal state, so long jobs are exposed to client and proxy timeouts. Polling the job record works, but a background run is not offered.
- The provider is an Ollama-compatible embedding endpoint read from the environment. No other provider is wired.
- Token counts and cost are reported as null; the embedding providers do not report usage, and the job does not invent figures.
- The routes accept the instance API key, and the project is a field of the request body rather than bound to the credential.
- Batch concurrency is one; throughput is bounded by sequential provider calls and the optional rate cap.
- Managed embeddings live under model-encoded kinds, separate from the embedding field of the source record. Search them with a kind filter.
Evidence
- On this siteAI features catalogue
- Tests16 integration tests cover end-to-end embedding and search, skip-on-rerun, resume after a kill, wrong-width dead letters, retries, rate pacing, cancel, refresh, model migration, quotas and project isolation; 15 more cover refusal under a recovery fence and storage faults; mounted-route tests cover the quota routes
Test coverage is listed as present in the engine repository at this edition. This page does not report test results.
KynetraDB is in private preview
Request a preview to talk through the limits above against your own workload.