APIs, integrations & migration

Postgres migration job with progress stream

Would let an operator start a PostgreSQL copy from the admin console and watch per-table progress while it runs.

RoadmapEngine edition 5f8ea3af

What it does

The server has a route that starts a PostgreSQL copy as a background job and a second route meant to stream its progress as server-sent events. In this build the progress stream does not carry the events of the job, and the job writes through an HTTP destination that depends on routes the server does not expose. That makes this entry roadmap. Use kynetra-migrate on the command line instead.

How it works

POST /admin/migrate/postgres takes a connection URL, a schema, include and exclude table lists, a batch size, relation and email-and-URL options and a dry-run flag. It returns a job id and starts the copy on a background task, using the same PostgreSQL source as the command line, so it has the same limits: no TLS, and a one-time copy.

GET /admin/migrate/postgres/{job_id}/stream is meant to send plan, progress and done events. In this build each request opens a new, empty channel that is not connected to the running job, so the stream ends without events. The job result is also not stored anywhere a client can read.

For a real run the job builds an HTTP destination that points at the server itself on the loopback address. That destination asks for collections to be created, and for records to be sent in bulk, through routes this server version does not expose. It then falls back to sending records one at a time, which fails for collections that do not exist. A dry run prints the plan to the output of the server process.

Where you reach it

  • HTTPPOST /admin/migrate/postgres
  • HTTPGET /admin/migrate/postgres/{job_id}/stream

Limits

  • Treat the route as not working end to end. Starting a job produces no usable progress output, and the writes can fail for missing collections.
  • Use kynetra-migrate with --to stdout or --to engine, and read the per-source limits on the migration page.
  • No test in the engine repository starts a job or reads its stream. The one test that names the stream route checks that a replica refuses it.
  • Unrelated to the PostgreSQL storage backend, which runs the database itself on PostgreSQL.

Evidence

Test coverage is listed as present in the engine repository at this edition. This page does not report test results.

KynetraDB is in private preview

This is not available in the preview. Request a preview to discuss the current scope.