Documentation / Reliability & Operations

Retries & Idempotency

The honest account — Nexus has no automatic run-level retry and no dead-letter queue. What it does have (adapter 429 backoff, child-run polling backoff, idempotent workspace writes) and how to build the safely re-runnable pipeline pattern buyers use today.

This is the page a skeptical platform engineer should read first, because it is the one where over-claiming would be most tempting and most damaging. So, stated plainly and up front:

Nexus does not automatically retry a failed run, and it has no dead-letter queue. When a run fails, it is recorded — with its full traceback — as terminal history. It is not silently requeued, and it is not moved to a DLQ for later reprocessing.

That is the truth of the shipped product. This page tells you exactly what retry-adjacent behavior does exist, teaches the idempotent-pipeline pattern that makes re-running safe, and marks the missing pieces as roadmap so you can plan around them honestly.

What is NOT built (do not design around it)

Not built	What it would be	Status
Automatic run-level retry	The platform re-runs a failed run for you, on a policy	Roadmap
Exponential backoff at the run level	Increasing delays between automatic run retries	Roadmap
Dead-letter queue	Failed runs park in a DLQ for later reprocessing/inspection	Roadmap
Durable step-level orchestration	A run resumes from the step it failed on, not from the top	Roadmap
End-to-end exactly-once execution	A run is guaranteed to take effect exactly once	Not built — see Scheduling Guarantees

If your architecture requires one of these as a platform primitive today, you build it at the edge (an external scheduler that re-invokes, plus an idempotent pipeline) — the rest of this page shows how. Do not assume the server will retry for you.

What DOES exist (precisely)

There are four real, narrow mechanisms. None of them is run-level retry; describing them as such would be wrong.

1. Adapter HTTP 429 backoff

Individual adapters that talk to rate-limited HTTP APIs handle throttling within a single adapter call. For example, the NetSuite adapter retries a throttled request with a bounded number of attempts and a backoff delay (on the order of retries=3, backoff≈2s). This is transparent to your pipeline: await ctx.adapters.netsuite.read(...) absorbs a transient 429 and returns the data. It is a per-call resilience feature, not a re-run of your pipeline.

2. Child-run polling backoff

When a parent pipeline spawns a child run and waits for it (via ctx.nexus), the wait loop polls the child's status with a backoff so it doesn't busy-spin. Again: this is polling backoff inside one running pipeline, not a retry of a failed run.

3. Narrow idempotency-key passthrough

Where an adapter's upstream API supports an idempotency key (so re-sending the same create is de-duplicated by the provider), Nexus can pass that key through. This is a narrow passthrough of a provider capability, not a platform-wide idempotency guarantee — it applies only where the underlying API implements it.

4. Idempotent writes via upsert / merge

The SQL adapters and the internal workspace support merge (upsert) semantics keyed on the adapter's configured key columns. Writing with op="upsert" is naturally idempotent: running it twice with the same input converges to the same state instead of creating duplicates. This is the building block you use to make your own pipeline safely re-runnable — covered next.

The pattern buyers use today: idempotent pipelines

Because the platform does not retry for you, the durable pattern is to make re-running safe and cheap, then let something re-invoke on failure. An idempotent pipeline is one where running it again — after a failure, a timeout, or an at-least-once reap — converges to the correct state rather than double-applying effects. Four techniques compose into it.

Technique 1 — Upsert / merge on a stable key

Never blind-insert records you might process again. Write with op="upsert" against a target keyed on a natural or synthetic business key, so a re-run overwrites rather than duplicates.

async def run(ctx):
    invoices = await ctx.adapters.netsuite.read(
        {"collection": "invoices", "updated_after": ctx.params["since"]}
    )
    rows = [{"invoice_id": i["id"], "amount": i["amount"], "status": i["status"]}
            for i in invoices]

    # Idempotent: upsert on the target's key column instead of inserting duplicates.
    await ctx.adapters.postgres.write(rows, op="upsert")
    await ctx.logger.info("upserted invoices", count=len(rows))
    return {"upserted": len(rows)}

Run this twice and you get one row per invoice_id, not two. That single property is what turns "the run failed halfway, do I dare re-run it?" into "just re-run it."

Technique 2 — Checkpoint progress in the workspace

For a long job, record what you've completed so a re-run skips finished work. The workspace is your durable scratch space for exactly this — it is structurally isolated from Nexus's operational tables and has its own managed identity.

async def run(ctx):
    done = {r["batch"] for r in await ctx.workspace.read("sync_progress")}
    for batch in ctx.params["batches"]:
        if batch in done:
            continue                                   # already completed — skip
        rows = await ctx.adapters.mssql.read({"batch": batch})
        await ctx.adapters.postgres.write(rows, op="upsert")
        await ctx.workspace.insert("sync_progress", [{"batch": batch}])
    return {"processed": len(ctx.params["batches"])}

Now a re-run after a timeout resumes near where it stopped instead of redoing everything — you have built step-level resume at the application layer, which the platform does not yet provide natively.

Technique 3 — Make writes to external systems keyed, not additive

When writing back to a system of record (NetSuite, SQL Server, MySQL), prefer upsert/merge on the target's key over append. The per-adapter capability matrix tells you which connectors support merge natively (Postgres, MySQL, SQL Server, NetSuite, and the internal workspace do); see Connectors Overview. Where the target lacks upsert, guard the write by reading the current state first, or use a provider idempotency key where available (mechanism 3 above).

Technique 4 — Re-invoke from an external scheduler for retry semantics

Because there is no automatic run-level retry, the supported way to get retry-on-failure today is to drive it from outside: a scheduler or orchestrator (Azure Logic Apps, a GitHub Actions cron, an existing enterprise scheduler, or even a parent pipeline) invokes the pipeline via the REST API or MCP, checks the returned terminal status, and re-invokes on failure with its own backoff policy. Paired with an idempotent pipeline (techniques 1–3), this gives you at-least-once-with-retry semantics that are safe, because a duplicate invocation converges rather than corrupts.

external scheduler                     Nexus
──────────────────                     ─────────────────────────
invoke pipeline (REST / MCP)  ───────► run executes → terminal status
if status == failed:                   (run recorded with full traceback)
    wait(backoff); invoke again ─────► idempotent re-run converges state

A checklist for a production-safe pipeline

Every write is a merge/upsert on a stable key — no blind inserts of reprocessable data.
Long jobs checkpoint completed units in the workspace and skip them on re-run.
Row-count constraints are set so a runaway re-run can't cause outsized damage.
Retry is driven by an external scheduler that reads the terminal status — not assumed from the platform.
Failure is expected to be diagnosable and re-runnable, because the full traceback is persisted (see Observability).

Limits & guarantees

Guaranteed: a failed run is durably recorded with its full traceback; adapter-level 429 backoff and child-run polling backoff exist as described; workspace/SQL merge gives you idempotent writes to build on.
Not provided: automatic run-level retry, exponential run-level backoff, a dead-letter queue, native step-level resume, or end-to-end exactly-once. These are roadmap; today you achieve equivalent safety with idempotent design plus external re-invocation.