Documentation / Reliability & Operations

Run Cancellation

Cancellation in Nexus is best-effort — it requests cancellation rather than guaranteeing termination. How the three cancellation paths behave, why an isolated-worker run can be terminated where an in-memory task can only be cooperatively interrupted, and how cancellation is permission-gated.

This page is for operators who need to stop a run, and for reviewers who need the precise semantics of what "cancel" does. The single most important thing to internalize: cancellation requests cancellation — it does not guarantee termination. Anywhere this documentation, the desktop app, or an API says "cancel," read it as "ask the run to stop," not "force the run dead." The rest of this page explains exactly how far the request can go.

What cancellation is

Cancelling a run signals that the run should stop and reach the cancelled terminal state (see Run Lifecycle). It is initiated by an operator from the desktop app or the MCP surface (uql_cancel_pipeline) and is governed by RBAC — only a principal with the appropriate permission may cancel a run.

Cancellation is best-effort by design, because a general-purpose Python runtime cannot be forcibly and safely frozen at an arbitrary instruction without risking corrupt half-writes. So Nexus does the safe thing: it makes the request effective through whichever mechanism the run's execution mode allows.

The three paths

How forcefully a cancellation takes effect depends on how the run is executing at that moment.

Path	Mechanism	How forceful
In-memory async task exists	The task is cooperatively cancelled	Takes effect at the next `await` boundary
Isolated-worker execution	The worker process can be terminated	Can stop CPU-bound work the async path can't
No live task reachable	Run status is set to `cancelled` in the DB	Bookkeeping; a detached run may still finish

1. Cooperative cancellation of the in-memory task

When the run is executing as an async task in a worker that receives the request, Nexus cancels that task. Python async cancellation is cooperative: the cancellation lands the next time the task hits an await — a network call, an adapter read, a sleep. This is the common case and it works well for the typical I/O-bound pipeline, which spends most of its life awaiting adapters and HTTP. The caveat is the honest one: a section of pure CPU-bound Python with no await will run to its next await before the cancellation is observed. Cooperative means cooperative.

2. Process termination on the isolated-worker path

Nexus is rolling out a process-isolation execution model in which a run executes in a separate, env-scrubbed worker process (holding no signing or managed-identity secrets, receiving bridge secrets only over data-only IPC). On that path, cancellation has a stronger tool available: the worker process itself can be terminated, which stops even CPU-bound work that cooperative async cancellation could not interrupt. This is architecturally a firmer stop than path 1.

Two honest qualifications:

Process isolation is live on the stingraytest tenant and rolling out — describe it as a control being deployed, not a universal guarantee present on every tenant today. See Architecture and the security section for the isolation model.
Even process termination does not retroactively undo writes the run already committed to your systems before it was killed. Termination stops further work; it is not a transaction rollback. This is why idempotent design matters even here.

3. Status-only cancellation

If there is no live, reachable task for the run at the moment of the request — for example the run is executing on a replica the request can't signal, or the process has already detached — cancellation records the run as cancelled in the database. This makes the operator's intent durable and correct in the run history, but it is bookkeeping: a genuinely detached run could still run to completion in the background. The heartbeat and orphan reaper are the backstop that keeps such a run from being stuck "running" forever, but the reaper reconciles state — it is not itself a kill switch.

Timeout is the host-side relative of cancellation

Hitting the wall-clock timeout is functionally a host-initiated stop, subject to the same reality: the worker abandons the in-flight work and records timeout. On the process-isolation path the worker can be torn down; on the in-memory path the same cooperative constraints apply. In both cancellation and timeout, the guarantee is on the bookkeeping and the lane — the run will be marked terminal and its execution lane freed — not on instantaneously freezing arbitrary Python.

Permissions

Cancellation is an RBAC-gated action. Only principals granted the relevant pipeline-cancel permission can cancel a run, and — like every governed action — a cancellation is attributable to the identity that issued it. See Roles & RBAC.

Limits & guarantees

Guaranteed: a cancellation request is durably recorded and attributable; an in-memory async run is cooperatively cancelled at its next await; on the isolated-worker path the process can be terminated; the run reaches the cancelled terminal state and its lane is freed.
Not guaranteed: immediate termination of arbitrary CPU-bound code on the in-memory path; rollback of writes the run already committed before stopping; that a fully detached run stops before finishing (status is corrected regardless). Never treat cancellation as a transactional undo — pair it with idempotent, keyed writes.