Documentation / Pipelines

Scheduling Pipelines

Run Nexus pipelines automatically on a cron schedule that fires exactly once per tick, on demand with parameters, or from an external trigger — and gate any write operation behind a required human approval step.

Once you've authored a pipeline in Nexus, you decide when it runs: on a recurring cron schedule, on demand from the desktop app or the MCP surface, or in response to an external trigger. Cron ticks fire exactly once even across multiple server replicas, and write operations can be held behind an approval gate so a person signs off before any data is changed. This page is for analysts and operators who run pipelines, and for admins who set the rules around them.

For the runtime ctx surface referenced below, see the Pipeline SDK reference; for run history and diagnostics, see Monitoring & Auditing Pipelines.

How a pipeline runs

A pipeline in Nexus is a Python module that defines async def run(ctx). Every run — scheduled, manual, or externally triggered — goes through the same sandboxed executor inside your tenant, so behavior is identical regardless of how the run was started. Each run produces a result record that captures status (succeeded or failed), row counters, structured logs, and a full traceback if anything went wrong.

Each run receives a ctx object that exposes the parameters it was started with, your configured data adapters, secrets from Key Vault, an SSRF-safe HTTP client, the internal workspace, and a structured logger.

Runs do not execute inline on the request thread. A trigger enqueues a run onto a durable Postgres-backed run queue, and a worker claims and executes it. That decoupling is what keeps triggers cheap and lets scheduling, concurrency, and cancellation be governed centrally. The full queue and concurrency model lives in Execution & Concurrency.

Trigger types

Trigger	How it starts	Typical use
On demand (manual)	You invoke the pipeline directly with a parameter dict	Backfills, ad hoc reruns, operator-triggered jobs
Scheduled (cron)	Nexus fires the pipeline on a wall-clock-aligned cron tick	Incremental syncs, nightly refreshes, report generation
External	Another system starts a run via the REST API or the MCP surface	Event-driven kickoffs from an upstream workflow

All three land in the same executor and the same run history, so a daily sync and a manual rerun of that sync behave identically.

On-demand runs

The simplest way to run a pipeline is to invoke it directly. You pass parameters as a dictionary, and the run is enqueued immediately.

# Author-facing pipeline entry point
async def run(ctx):
    since = ctx.params.get("since", "2026-01-01")
    rows = await ctx.adapters.netsuite.read({"collection": "invoices", "updated_after": since})
    await ctx.logger.info("read rows", count=len(rows))
    return {"rows": len(rows)}

On-demand runs are useful for:

One-off backfills and ad hoc reporting.
Testing a pipeline before you put it on a schedule.
Operator-triggered jobs that don't follow a fixed cadence.
Reproducing a failed scheduled run with the same parameters to read its traceback.

Every on-demand run is recorded with its parameters, logs, and result, so you always have an audit trail of who ran what and when.

Scheduled runs

To run a pipeline automatically, attach a cron schedule to it — either in the desktop app or inline at the top of the file:

# nexus:trigger scheduled 0 9 * * *

async def run(ctx):
    ...

The five cron fields are the standard minute hour day-of-month month day-of-week. Nexus fires the pipeline on wall-clock-aligned ticks — 0 9 * * * runs at 09:00, not "roughly nine hours after the last run" — so schedules stay predictable and don't drift with execution time. Scheduled runs carry a defined set of parameters, the same way an on-demand run does, so a daily sync and a manual rerun of that sync behave identically.

Common scheduling patterns:

Pattern	Cron example	Use
Hourly	`0 * * * *`	Incremental syncs, near-real-time refreshes
Nightly	`0 2 * * *`	Full refreshes, report generation
Off-hours weekday	`0 3 * * 1-5`	Heavy backfills that should avoid peak load
Weekly	`0 6 * * 1`	Weekly rollups and exports

Exactly-once firing

The scheduler runs on every server replica, which raises an obvious question: if three replicas all notice that a 09:00 tick is due, does the pipeline fire three times? It does not. Nexus guarantees exactly-once firing of each cron tick:

Each due tick is claimed with a compare-and-swap against the schedule's last-fired marker — only one replica wins the swap for a given tick.
Cross-replica Postgres advisory locks serialize the scheduler's due-ness check so two replicas can't both evaluate and claim the same tick.
Because due-ness is anchored on the last scheduled fire (not the last completed run), a single failed or slow run does not silence the schedule — the next tick still fires on time.

Be precise about what this guarantees: exactly-once applies to the firing of a cron tick — the decision to enqueue a run at 09:00 happens once. It is not end-to-end exactly-once execution of the pipeline. Queue dispatch is at-least-once at the claim boundary, and Nexus does not deduplicate the side effects your pipeline performs. If a run is retried by an operator, or a claim is redelivered, your pipeline's writes can run again — which is why write-heavy pipelines should be idempotent. The complete guarantee model, including where at-least-once applies and how to build idempotent pipelines, is documented in Scheduling Guarantees and Retries & Idempotency.

Because scheduled runs and manual runs flow through one executor, you can debug a failing scheduled job by rerunning it on demand with the same parameters and reading the captured traceback — no special diagnostics tooling required.

The write-approval gate

Reading data is safe to automate. Writing data — creating, updating, or deleting records in a connected system — is where mistakes are expensive. Nexus lets you require that any pipeline performing write operations is approved by a person before it proceeds.

When the approval gate is in effect, a pipeline run that would write data does not execute its writes immediately. Instead it is held as a pending approval. An authorized reviewer sees the request, inspects what the run intends to do, and then approves or rejects it:

Approve — the run proceeds, is dispatched from where it paused, and performs its writes.
Reject — the run stops without changing any data.

This gives you a clean separation of duties: the person who authors or schedules a pipeline isn't necessarily the person who authorizes a production write. Pending approvals, approvals, and rejections are all recorded for audit, and the approval itself becomes part of the run's audit record.

Reviewing approvals

Authorized reviewers can list everything currently waiting and act on it:

List pending approvals to see queued write runs and what each intends to do.
Approve a specific run to let it proceed via resumable dispatch.
Reject a run to cancel it cleanly, changing no data.

Approval authority is governed by RBAC, so only principals you've granted the right permission can clear write runs. Read-only pipelines and report queries are unaffected — the gate applies specifically to runs that modify data.

Putting it together

A typical production setup looks like this:

Author the pipeline and validate it. Invalid source is rejected at save time, before it can ever be scheduled.
Test it on demand with sample parameters and confirm the result and logs.
Schedule it for its recurring cadence, or wire an external trigger via the API or MCP.
Gate writes so a reviewer approves any data-changing run before it commits.
Make writes idempotent so a manual rerun after a failure converges rather than double-writing (Nexus does not auto-retry runs).

Every run — however it started — lands in the same run history with status, counters, logs, and (on failure) a complete traceback, so scheduled automation stays observable and auditable end to end.

Limits & guarantees

Guaranteed: each cron tick fires exactly once across all replicas; scheduled and manual runs share one executor and one run history; a failed fire doesn't silence the schedule; every run is recorded with parameters, logs, counters, and (on failure) a full traceback.
Not guaranteed / not present: end-to-end exactly-once execution; automatic retry of a failed run; a dead-letter queue for failures. Retry and DLQ are roadmap — today you rerun manually and rely on idempotent pipeline design. See Reliability & Operations for the full, honest treatment.