Documentation / Reliability & Operations

Run Lifecycle

The exact states a Nexus run moves through — queued, claimed, executing, terminal — how the durable Postgres run queue dispatches work, and how the heartbeat and orphan reaper recover runs whose worker died.

This page is for engineers who need to reason about what happens to a run between "I pressed go" and "it's done," including what happens if a process dies mid-run. Every state described here is persisted in your own per-tenant PostgreSQL, so the lifecycle is inspectable and recoverable rather than living in ephemeral process memory.

The states

A run moves through a small, explicit set of states. Every transition is written to the database before the next step proceeds, which is what makes the lifecycle durable across a replica restart.

State	Meaning	Where recorded
`queued`	Enqueued and waiting for a worker to claim it	`run_queue` (pending), `pipeline_runs` (created)
`claimed`	A worker has atomically taken ownership	`run_queue` (claimed by worker), heartbeat started
`executing`	The sandboxed `run(ctx)` is actually running	`pipeline_runs` status, heartbeat updated
`succeeded`	`run(ctx)` returned normally	`pipeline_runs` terminal
`failed`	`run(ctx)` raised; full traceback captured	`pipeline_runs` terminal + `exception_text`
`timeout`	The wall-clock deadline elapsed before completion	`pipeline_runs` terminal
`cancelled`	Cancellation was requested and honored	`pipeline_runs` terminal

succeeded, failed, timeout, and cancelled are the four terminal states. A run in a terminal state is never re-dispatched — there is no automatic requeue and no dead-letter queue (see Retries & Idempotency).

Enqueue → dispatch: the durable run queue

When any trigger starts a run — on demand, a schedule firing, an approval clearing, or a parent pipeline spawning a child — Nexus does not hand the work directly to a thread. It writes a row to the run_queue table in Postgres and records the run in pipeline_runs. The queue is the durable system of record for "work that should happen": if every replica restarted right now, the queued work would still be there when they came back.

Dispatch uses two Postgres primitives together:

LISTEN/NOTIFY for low-latency wakeups. When a run is enqueued, the enqueuing code issues a NOTIFY on a channel the workers LISTEN on, so an idle worker wakes immediately instead of polling on a timer. A periodic sweep also runs as a backstop, so a missed notification (for example, a worker that was mid-restart) never strands a queued run.
SELECT … FOR UPDATE SKIP LOCKED for safe claiming. When a worker wakes, it claims work with a FOR UPDATE SKIP LOCKED query. This is the standard Postgres pattern for a concurrent work queue: two workers can query simultaneously and each atomically takes a different row, because the lock on an already-claimed row causes the other worker to skip it rather than block. No two workers ever claim the same run.

enqueue                              claim (per worker)
────────                             ─────────────────────────────
INSERT INTO run_queue (...)          SELECT ... FROM run_queue
INSERT INTO pipeline_runs (...)        WHERE status = 'pending'
NOTIFY run_queue_channel               ORDER BY created_at
                                       FOR UPDATE SKIP LOCKED
                                       LIMIT 1;
                                     -- mark claimed, start heartbeat

At-least-once at the claim boundary

The claim is atomic, so a run is never claimed by two workers concurrently. But the queue's delivery contract is at-least-once, not exactly-once: if a worker claims a run and then dies before it can record a terminal state, the orphan reaper (below) will return that run to a claimable state so another worker can take it. That is the correct, safe behavior for a durable queue — and it is exactly why pipelines should be idempotent. There is no end-to-end exactly-once execution guarantee. See Scheduling Guarantees for how this contrasts with cron firing, which is exactly-once.

Executing

Once claimed, the worker moves the run to executing and invokes the sandboxed async def run(ctx) inside the timeout wrapper described in Timeouts & Limits. During execution:

The worker maintains a heartbeat — a periodically updated timestamp on the run — so the rest of the system can tell a live long-running run from an abandoned one.
Structured logs stream to the run record as the pipeline emits them via ctx.logger, so a run that later fails still has every log line it wrote before the failure. See Observability.
Row counters (rows_read, rows_created, rows_updated, rows_deleted) accumulate as adapter operations happen.

The executor is designed never to raise out of the run. A pipeline that throws produces a failed terminal record with the complete Python traceback attached in exception_text — not a silent process crash. You diagnose from the run record itself.

Reaching a terminal state

succeeded — run(ctx) returned. The return value is captured as the run result.
failed — run(ctx) raised. The traceback is persisted. The run is not requeued.
timeout — the wall-clock deadline elapsed. The worker abandons the in-flight work and records timeout.
cancelled — a cancellation request was honored. Because cancellation is best-effort, reaching this state means the request took effect; a run that cannot be cooperatively interrupted may complete before the request lands. See Cancellation.

Every terminal transition is a single committed write to pipeline_runs, and the corresponding run_queue row is cleared. After that, the run is immutable history.

When a worker dies: heartbeat + orphan reaper

The failure mode a durable queue must handle is a worker that claims a run and then vanishes — an OOM kill, a node eviction, an image roll. Nexus handles it with the heartbeat plus an orphan reaper:

While a run executes, its worker updates the run's heartbeat on an interval.
A reaper periodically scans for runs that are claimed/executing but whose heartbeat has gone stale past a threshold — the signature of a worker that died mid-run.
The reaper transitions those runs out of limbo: the run is returned to a claimable state so it can be retried by a healthy worker, or marked terminal, depending on policy — so a dead worker never leaves a run stuck "running forever."

This is what makes the lifecycle robust to infrastructure churn without any operator intervention. It is also the mechanism that makes the delivery contract at-least-once: a reaped run may execute again, which is safe precisely when your pipeline is idempotent.

Inspecting the lifecycle

Every state and every transition is queryable from the desktop app or the MCP surface — there is no hidden runtime state you can't see:

uql_list_pipeline_runs            → recent runs with status, counters, timing
uql_get_pipeline_run              → one run's full record and terminal state
uql_get_pipeline_run_logs         → the run's structured log stream
uql_get_pipeline_run_diagnostics  → traceback + sandbox diagnostics on failure

Limits & guarantees

Guaranteed: every run is durably recorded before dispatch; a run is claimed by at most one worker at a time; a worker that dies mid-run does not strand the run; every terminal state, including the full failure traceback, is persisted.
At-least-once, not exactly-once: a reaped run can execute more than once. Design for that (see idempotency).
No automatic requeue of a cleanly failed run. A run that reached failed on its own is terminal history, not queue backlog. Re-running it is an explicit action (yours, or an external scheduler's).