NexusReliability & Operations Overview
How reliability works in Nexus's tenant-owned model — the durable run queue, wall-clock timeouts, concurrency controls, and observability that exist today, stated plainly alongside what is not yet built.
This section is for platform, SRE, and security teams evaluating how Nexus behaves under load, under failure, and over time. It documents the operational runtime honestly: the reliability primitives that exist, how they work, and — just as importantly — what Nexus does not do yet. Where a capability is on the roadmap rather than shipped, this section says so in plain language. You should be able to make an operational risk decision from these pages without talking to a salesperson.
The model in one paragraph
Nexus runs as an Azure Marketplace managed application inside your own Azure subscription. There is no Stingray-hosted data plane, no shared execution tier, and no multi-tenant scheduler you contend with. Every reliability primitive described here — the run queue, the scheduler, the timeout enforcer, the connection governor, the run history — executes on your Container App against your per-tenant PostgreSQL Flexible Server. That has a direct consequence for how you reason about reliability: the durability and availability of the platform's operational state is, to a large degree, the durability and availability of the Azure services you already run and already govern. See Architecture for the component map.
What this section covers
| Page | Question it answers |
|---|---|
| Run Lifecycle | What are the exact states a run moves through, and where is each state persisted? |
| Execution & Concurrency | How many things run at once, and what happens when I exceed that? |
| Timeouts & Limits | How long can a run take, and what caps bound its blast radius? |
| Retries & Idempotency | What retries automatically, what does not, and how do I build a safely re-runnable pipeline? |
| Cancellation | What exactly happens when I cancel a run? |
| Scheduling Guarantees | Does cron fire exactly once? Does a run execute exactly once? |
| Observability | How do I see what ran, what it changed, and why it failed? |
| Backup & Disaster Recovery | How is operational state backed up and recovered? |
What this does — and doesn't — do
Reliability claims are only useful if they're precise. This table is the honest summary; every row is expanded on its own page.
| Capability | Status | Notes |
|---|---|---|
| Durable run queue in Postgres | Shipped | run_queue table, FOR UPDATE SKIP LOCKED claim + LISTEN/NOTIFY dispatch. Survives a replica restart. |
| Persisted run state & history | Shipped | Every run recorded in pipeline_runs with status, counters, logs, and full traceback on failure. |
| Per-run wall-clock timeout | Shipped | Default 3600s, hard ceiling 6h, per-pipeline # nexus:timeout override, enforced by the worker wrapper. |
| Pipeline row-count constraints | Shipped | Caps on rows read / created / updated / deleted as a defense-in-depth blast-radius limit. |
| Concurrency queue (global + worker + per-tenant) | Shipped | A queue that delays work, not a limiter that rejects it. |
| Bridge connection governor | Shipped | Per-replica DB connection caps with orphan reaping. |
| Cron firing exactly once | Shipped | Compare-and-swap claim + cross-replica advisory locks. |
| Heartbeat + orphan reaper | Shipped | Detects runs whose worker died and marks them terminal. |
| Run cancellation | Shipped, best-effort | Requests cancellation; does not guarantee termination. See its page. |
| Structured logs, tracebacks, audit snapshots | Shipped | Full traceback persisted per failed run; before/after audit snapshots on writes. |
| Automatic run-level retry / exponential backoff | Not built | Backoff exists only for adapter HTTP 429s and child-run polling. Retry is on the roadmap. |
| Dead-letter queue | Not built | Roadmap. Failed runs are recorded, not requeued. |
| End-to-end exactly-once execution | Not built | Exactly-once applies to cron firing only; queue dispatch is at-least-once at the claim boundary. |
| Durable step-level orchestration / checkpointed resume | Not built | Roadmap. Today you checkpoint yourself via the workspace (see the idempotency page). |
| Published SLA / uptime / RPO / RTO numbers | Not published | Nexus inherits the posture of the Azure services in your subscription; you set and measure your own targets. |
How to read the "roadmap" rows
When a page marks something as roadmap, it means the primitive is not in the shipped server and you must not design around it existing. It does not mean the gap is unaddressed. In nearly every case there is a supported pattern that closes the gap today — most importantly, designing pipelines to be idempotent and safely re-runnable, and re-invoking them from an external scheduler when you need retry semantics. Those patterns are documented, not hand-waved. Read Retries & Idempotency before you conclude anything is missing that you actually need.
Where reliability state lives
Because everything runs in your tenant, so does every durable artifact of the runtime:
- Run queue and run history — your per-tenant PostgreSQL Flexible Server (
run_queue,pipeline_runs). - Authored assets (pipelines, reports, sites, skills) — a per-tenant git repository on Azure Files, with history/diff/restore. See Version Control.
- Secrets and signing keys — your per-tenant Key Vault.
- Audit and telemetry — your per-tenant Postgres. See Auditing & Cost Telemetry.
Nothing on this list transits Stingray. Your disaster-recovery story for Nexus is therefore an extension of your existing Azure DR story, which is exactly the point of the Backup & Disaster Recovery page.
Maturity context
Nexus is a versioned product on the 2.1.x server line with roughly 1,300+ automated server tests, shipping to two live tenants — a commercial tenant and an Azure Government (GCC-High) tenant — and carrying a real production workload (a nightly ERP sync). The reliability primitives here are the ones exercised by that workload. The honest framing throughout this section reflects a deliberate product stance: we would rather you deploy knowing exactly what the runtime guarantees than discover a missing guarantee in an incident. See Platform Maturity for the fuller maturity picture.