Documentation / Reliability & Operations

Observability

How you see what Nexus is doing — structured per-run logs, the full persisted traceback on failure, run history and detail, sandbox diagnostics, before/after audit snapshots, opt-in cost telemetry, and health checks — all inside your own Azure tenant.

This page is for operators and auditors who need to answer "what ran, what did it change, and why did it fail?" without leaving your tenant. Nexus is built so you diagnose from durable records rather than from log-spelunking a live process: every run leaves a queryable trail, and every failure leaves its complete traceback. None of it transits Stingray.

Structured, persisted run logs

A pipeline emits logs through ctx.logger, and what you see in the run record is exactly what it emitted. Logging structured fields — rather than string-formatting them into the message — keeps them filterable after the fact.

async def run(ctx):
    rows = await ctx.adapters.netsuite.read({"collection": "invoices", "limit": 500})
    await ctx.logger.info("read source rows", count=len(rows), source="netsuite")
    written = await ctx.adapters.postgres.write(rows, op="upsert")
    await ctx.logger.info("upserted", written=written, target="postgres")
    return {"written": written}

Logs are attached to the run and survive a later failure: if a pipeline logged three steps and raised on the fourth, all three log lines remain on the failed run alongside the traceback. Levels are debug, info, warning, error — all async, all awaited.

The full traceback on failure

The executor is designed never to raise out of a run. A pipeline that throws produces a failed run with the complete Python traceback persisted in the run record — the exception message and the offending line — not a silent crash. This is the single most useful observability property in the platform: you debug a production failure from the real stack trace, captured in place, the same way you would locally.

Run history and detail

Every run — scheduled, on demand, approval-cleared, or a child of another run — lands in the same history with a consistent record. Inspect it from the desktop app or the MCP surface:

uql_list_pipeline_runs            → recent runs: status, row counters, timing, identity
uql_get_pipeline_run              → one run's full record and terminal state
uql_get_pipeline_run_logs         → that run's structured log stream, in order
uql_get_pipeline_run_diagnostics  → traceback + sandbox diagnostics on failure

Each record carries the fields you need to reason operationally:

Field	Tells you
`status`	`succeeded`, `failed`, `timeout`, or `cancelled`
`rows_read` / `created` / `updated` / `deleted`	Exactly what the run touched, per operation
Structured logs	Every `ctx.logger` emission, ordered
`exception_text`	Full traceback on failure
Identity	Run ID, pipeline ID, and the principal that started the run
Timing	Enqueue, start, and terminal timestamps

See Monitoring & Auditing Pipelines for the operator workflow around these.

Sandbox diagnostics

For failures that stem from the sandbox itself — a disallowed import, a missing entrypoint, an UNCONFIGURED_* capability call — the diagnostics surface names the cause explicitly rather than surfacing a generic error. Invalid source is rejected at save time, before it can ever be scheduled, so a whole class of failure never reaches production in the first place.

Before/after audit snapshots

Writes are the operations that matter most for governance, so Nexus captures before/after snapshots around data-changing operations: the audit trail records not just that a write happened but the state on each side of it, attributable to the triggering identity. Combined with the per-run row counters, this answers the audit question directly — which run, started by which principal, changed how many rows in which system, from what to what. Approvals, when a write-approval gate is in effect, are themselves part of the record. See Auditing & Cost Telemetry and Scheduling Pipelines.

Cost telemetry (opt-in beyond the tenant)

Nexus rolls up the cost of each run as it completes and aggregates it into per-run, per-pipeline, and per-tenant totals, with recursive attribution across child runs and anomaly flagging when a pipeline's recent spend spikes above its baseline. All of this lives in your tenant by default. Sending aggregated cost/usage signals to the Stingray control plane is opt-in and administrator-controlled — when it's off, nothing leaves your instance; when it's on, only aggregated metrics (counts, totals, breakdowns) are sent, never your data, results, or credentials. Full detail is in Auditing & Cost Telemetry.

Health checks

Nexus exposes health endpoints so your Azure platform tooling (Container Apps probes, monitors, or an external uptime check) can observe liveness and readiness, and a deeper health check that reports on the server's dependencies (database reachability and the like). Use the shallow check for load-balancer/liveness probing and the deep check for operational dashboards. Health surfaces are also reachable through the MCP/admin tooling (uql_health, uql_health_deep) for scripted checks. Because Nexus runs on Azure Container Apps and Azure Database for PostgreSQL in your subscription, you also inherit Azure's own platform metrics and diagnostic logging for those resources — wire them into whatever monitoring you already run.

What stays in your tenant

Every observability artifact on this page — run history, structured logs, tracebacks, audit snapshots, in-tenant cost rollups — lives in your own per-tenant PostgreSQL and storage. Stingray does not receive your logs, query results, file contents, or business data. The only thing that can leave is opt-in aggregated cost telemetry, and only when an administrator turns it on.

Limits & guarantees

Guaranteed: every run is durably recorded with status, counters, and ordered logs; every failure persists its full traceback; writes carry before/after audit snapshots attributable to an identity; observability data stays in your tenant.
Note: Nexus does not ship an application metrics/tracing pipeline (Prometheus/OpenTelemetry-style) as a product feature today; observability is the durable run record plus Azure's native platform diagnostics for the underlying services. Export from the run records via the API/MCP if you need to feed an external system.