Documentation / Pipelines

Monitoring & Auditing Pipelines

Track every Nexus pipeline run, inspect structured logs and persisted full tracebacks, review before/after audit snapshots and per-write accounting, and watch per-pipeline cost — all inside your own Azure tenant.

Every pipeline run in Nexus produces a durable, queryable record. This page is for operators who need to know whether a run succeeded, analysts diagnosing a failure, and admins who need a defensible audit trail of every change written back to their systems. All of it lives inside your own Azure tenant — Nexus does not ship your run history or logs anywhere else.

This page covers the run record, structured logs, tracebacks and diagnostics, write auditing, and cost. For the deep runtime-guarantees treatment — the queue model, orphan detection, timeouts, and the observability contract as a reliability property — see Reliability & Operations.

The run record

Whether a pipeline is triggered on a schedule, on demand, or by an external event, it goes through the same sandboxed executor and produces the same result record. Each run captures:

Field	What it tells you
`status`	`succeeded` or `failed`
`rows_read` / `rows_created` / `rows_updated` / `rows_deleted`	Row counters per operation, so you can see exactly what a run touched
Structured logs	Every `ctx.logger` emission, with the message and any structured fields, in order
`exception_text`	The full Python traceback when a run fails — including the exception type, message, and offending line
Identity	Run ID, pipeline ID, tenant ID, and the principal that started the run
Timing	When the run was enqueued, started, and finished
Cost	Per-run cost units, plus the recursive cost of any child pipelines it invoked

Because the executor never raises out of a run, a failure is always recorded as a failed result with the traceback attached, not a silent crash. You diagnose from the run record itself — no log spelunking, no reproducing the failure blind.

Run history

List and inspect runs from the desktop app, the REST API, or the MCP surface. A typical operator flow:

List recent runs for a pipeline, filtered by status or time window.
Open a failed run to read its full traceback and the logs emitted before the failure.
Fix the cause, then re-run with the same parameters.

uql_list_pipeline_runs            → recent runs with status, counters, timing
uql_get_pipeline_run              → a single run's full record
uql_get_pipeline_run_logs         → that run's structured log stream
uql_get_pipeline_run_diagnostics  → traceback + sandbox diagnostics

Logs emitted before a failure are never lost — if a pipeline logged three steps and then raised on the fourth, all three log lines remain attached to the failed run alongside the traceback. That property is what makes tracebacks actionable: you see both what the run was doing and exactly where it broke.

Structured logging from a pipeline

What you see in the run record is exactly what the pipeline emits through ctx.logger. Logging structured fields (rather than string-formatting them into the message) keeps them filterable later.

async def run(ctx):
    rows = await ctx.adapters.netsuite.read({"collection": "invoices", "limit": 100})
    await ctx.logger.info("read source rows", count=len(rows))

    updated = await ctx.adapters.mssql.write(rows, op="upsert")
    await ctx.logger.info("wrote rows", updated=updated, target="mssql")
    return {"updated": updated}

The keyword arguments (count, updated, target) are preserved as discrete fields on each log entry, so you can scan a run for "how many rows did the write step report" without parsing prose. See the Pipeline SDK reference for the logger's full method set.

Diagnostics and persisted tracebacks

When a run fails, the diagnostics record is your starting point. It carries the complete exception_text — the same traceback you would see in a local Python session, including the exception class, its message, and the line that raised — plus sandbox diagnostics that distinguish a compile-time rejection (a disallowed import or syntax error caught before execution) from a runtime exception thrown by your logic. This lets you tell "the sandbox refused this code" apart from "the code ran and hit a real error" at a glance, without reproducing anything.

Because tracebacks and logs are persisted per run and never overwritten, a run from three weeks ago is as diagnosable today as it was the moment it failed.

Write-operation audit

Pipelines that write back to your systems are the ones that matter most for governance. The row counters (rows_created, rows_updated, rows_deleted) are recorded per run, giving you a per-write accounting of what changed and when. Combined with the run's identity fields, this answers the audit question directly: which run, started by which principal, changed how many rows in which system.

For data-changing operations, Nexus also captures before/after audit snapshots, so the audit trail records not just that rows were touched but the state on each side of the change. Write-heavy pipelines can additionally be held behind an approval gate so a person signs off before any data is changed — and that approval is itself part of the audit record. See Scheduling Pipelines for the approval workflow, and Audit & Telemetry for the tenant-wide audit surface.

Orphan detection

A run that starts but whose worker dies mid-execution — a pod restart, a lost lease — would otherwise sit "running" forever. Nexus detects these: workers heartbeat while a run is in progress, and a reaper marks a run whose heartbeat has gone stale as orphaned rather than leaving it hung. That keeps run history honest — a run you see as "running" really is, and a stuck run is surfaced as failed/orphaned instead of silently lingering. The heartbeat/reaper mechanics and their timing are documented in Reliability & Operations.

Cost and anomaly tracking

Nexus rolls up the cost of each run as it completes — data-adapter calls and any AI usage both contribute. Costs are aggregated automatically, accumulating into per-pipeline and per-tenant lifetime totals, and cost is attributed recursively: when one pipeline invokes another via ctx.nexus, the parent's record reflects the full cost of everything it triggered.

Admins can review cost across all pipelines from the admin reporting surface:

Report	Answers
Cost summary	Per-pipeline cost over a chosen period
Adapter breakdown	Which integration is the most expensive
Anomalies	Pipelines whose recent cost has spiked relative to their baseline

Anomaly detection compares each pipeline's recent cost against its rolling baseline and flags runaway pipelines, with a floor that suppresses noise from trivially cheap workloads. Anomaly access requires an admin telemetry permission.

Why it matters

Run history, structured logs, persisted tracebacks, before/after audit snapshots, write counters, and cost rollups give you a single, tenant-local source of truth for operating pipelines under governance: prove what ran, prove what it changed, see what it cost, and catch anomalies before they become a bill. Nothing leaves your Azure tenant to make that true.