Documentation / Pipelines

Pipelines Overview

Build sandboxed Python pipelines in Nexus to read, transform, and write data across your connected systems — run on demand, on a schedule, or chained together, all inside your own Azure tenant.

Nexus pipelines let you automate work across your connected systems with plain Python. A pipeline reads from your data sources, transforms the results, and writes them back — to another system, an internal workspace table, or a downloadable file. Pipelines are for data engineers, analysts, and platform teams who want repeatable, auditable automation that runs entirely inside their own Azure tenant, with no Stingray-hosted data plane in the path.

This page orients you: what a pipeline is, how the sandbox constrains it, and the authoring lifecycle from first draft to scheduled production run. The Pipeline SDK reference documents the ctx surface in full; Scheduling Pipelines and Monitoring & Auditing Pipelines cover triggers and observability.

What a pipeline is

A Nexus pipeline is a Python module with a single entrypoint: async def run(ctx). The module is stored on your Nexus server and executed by a sandboxed runtime. You never import database drivers, HTTP libraries, or credentials — every capability arrives through the injected ctx object.

async def run(ctx):
    rows = await ctx.adapters.netsuite.read({"collection": "invoices", "limit": 100})
    await ctx.logger.info("read invoices", count=len(rows))
    return {"rows": len(rows)}

That is a complete, runnable pipeline. Return a dict (or anything) and it is captured as the run result; raise an exception and the runtime records the full traceback so you can diagnose failures from real stack traces, not guesswork.

Nexus deliberately chose Python modules over a bespoke YAML DSL (ADR-0001). A DSL forces you to learn a second, weaker language and hides failures behind opaque template errors; a Python module gives you the whole language — comprehensions, functions, typed data handling — plus real exceptions and real tracebacks. The trade you accept for that power is the sandbox, described below.

What a pipeline is for

Pipelines are the workhorse for anything that moves or reshapes data on a cadence:

Cross-system sync — read from one connected system, transform, and write to another (for example a nightly NetSuite-to-SQL-Server ledger sync).
Staging and enrichment — pull raw records into the internal workspace, join and aggregate them there, then serve the curated table to reports.
Scheduled exports — produce a CSV/JSON/XLSX file on a schedule for downstream consumers.
Orchestration — a parent pipeline invokes child pipelines for multi-step workflows, with cost and identity attributed across the tree.

If you only need to read and shape data for a report or a dashboard, you may not need a pipeline at all — a saved UQL report may be enough. Reach for a pipeline when you need to write back to a system, chain steps, stage intermediate state, or run on a schedule.

The `ctx` SDK

Everything a pipeline can do is reached through ctx:

Attribute	What it does
`ctx.adapters.<name>`	Read, write, and discover schema on a connected source (e.g. `netsuite`, `mssql`, `postgres`, `monday`)
`ctx.secrets`	Read-only access to secrets in your Key Vault (`get`, `get_required`)
`ctx.http`	An SSRF-safe HTTP client (`get`, `post`, `put`, `delete`, `request`) for any API not covered by an adapter
`ctx.workspace`	Read and write tables in your tenant-private internal database
`ctx.files`	Write output blobs (CSV/JSON/XLSX) and read named file references
`ctx.nexus`	Invoke child pipelines and orchestrate multi-step runs
`ctx.logger`	Structured, async logging tied to the run (`info`, `warning`, `error`, `debug`)
`ctx.params`	Per-invocation inputs passed when the pipeline was started
`ctx.variables`	A per-run scratch namespace, preserved in run diagnostics

Adapter calls are bound to the run's identity and permissions, so a pipeline can only reach data the run is authorized to see. Credentials are resolved by the host against the invoking principal — a pipeline can never resolve another user's credentials. Everything on ctx is asynchronous — always await it. The full method-level reference is on the Pipeline SDK page.

The sandbox

Pipeline source runs under an import allowlist and a restricted set of builtins. The runtime parses your code into an abstract syntax tree and checks it before any of it executes, so a disallowed import or a stripped builtin is rejected at save/validate time rather than mid-run.

Allowed imports: the runtime SDK plus pure-data standard-library modules such as json, datetime, decimal, re, math, collections, itertools, functools, uuid, hashlib, base64, urllib.parse, and asyncio.
Reached through ctx instead: HTTP (ctx.http), databases (ctx.adapters / ctx.workspace), secrets (ctx.secrets), and file I/O (ctx.files).
Blocked imports: filesystem, process, and raw-network modules (os, subprocess, socket, pathlib), third-party HTTP clients (httpx, requests), database drivers, and dynamic-import machinery.
Stripped builtins: even at runtime, open, eval, exec, compile, input, breakpoint, __import__, globals, and locals are removed from the execution namespace. A pipeline that calls open(...) raises NameError — it does not read a file.

The allowlist is intentionally short, and extending it is a governed change: adding a module requires a security-review event, not a config toggle. The result is a predictable execution environment — pipelines reach only the systems Nexus grants them, and authors get clear, named errors (SandboxImportError: Disallowed import 'os') instead of silent failures.

The sandbox is one layer of the runtime's defense in depth, not the whole story. Per-run wall-clock timeouts, pipeline row-count constraints, and (on tenants where it is enabled) an env-scrubbed isolated worker process wrap execution as well. Those runtime guarantees and their exact limits are documented in Reliability & Operations.

Running pipelines

A pipeline can run several ways, and every path goes through the same sandboxed executor, so behavior is identical no matter how a run starts:

On demand — trigger a run manually from the desktop client, the REST API, or the MCP surface.
On a schedule — declare a cron trigger and Nexus fires it automatically on wall-clock-aligned ticks.
Chained — orchestrate child pipelines from a parent run via ctx.nexus for multi-step workflows.

Schedules and metadata are declared when you save the pipeline, or inline at the top of the file with # nexus: directives:

# nexus:trigger scheduled 0 9 * * *
# nexus:timeout 300

async def run(ctx):
    deals = await ctx.adapters.monday.read({"collection": "deals", "limit": 500})
    rows = [{"id": d["id"], "amount": d.get("amount", 0)} for d in deals]
    await ctx.workspace.insert("daily_deals", rows)
    await ctx.files.write("deals_export", rows, file_type="csv")
    return {"written": len(rows)}

# nexus:trigger scheduled <cron> attaches a cron schedule; # nexus:timeout <seconds> overrides the default per-run wall-clock timeout. Scheduling semantics — including how Nexus guarantees a cron tick fires exactly once across replicas — are covered in Scheduling Pipelines.

Authoring and debugging lifecycle

Pipelines are versioned and validated as you work. The lifecycle from draft to production looks like this:

Validate the source without saving to confirm it compiles under the sandbox and defines async def run(ctx). Disallowed imports and syntax errors surface here, before the pipeline can ever be scheduled.
Save — each save is committed to a per-tenant git version history you can diff and restore. You can always roll a pipeline back to a known-good revision.
Test with a dry run that intercepts writes, so you can exercise logic against real reads without mutating live data.
Execute a tracked run, then inspect structured logs, row counters, and the full traceback for any failure.
Schedule the pipeline for its recurring cadence, or wire an external trigger.
Gate writes (optional) behind an approval step so a reviewer signs off before any data-changing run commits.

Because every run captures its logs, counters, and (on failure) the complete Python traceback, you debug from exactly what happened — and version history means a bad change is one restore away from reverted.

Approval workflow

Reading data is safe to automate; writing it is where mistakes are expensive. Nexus can hold any pipeline run that would write data as a pending approval until an authorized reviewer inspects and approves it. Approval authority is governed by RBAC, and every approval or rejection is recorded as part of the audit trail. This gives you separation of duties — the person who authors or schedules a pipeline need not be the person who authorizes a production write. The full workflow is on the Scheduling Pipelines page.

Governance and identity

Pipelines run under the tenant's RBAC model. A run carries an identity — a user or an app principal — and every adapter read, adapter write, secret fetch, and HTTP call is bounded by that identity's grants. You can pin a curated execution role to a pipeline so it runs with exactly the permissions it needs and no standing elevation. The same RBAC and audit machinery governs pipelines invoked through the embedded MCP server by an AI agent — an agent acts strictly within the connecting identity's permissions.

What this does and doesn't do

Does: run sandboxed Python inside your tenant; bind every capability to the run's identity; persist status, counters, structured logs, and full tracebacks; version every save; enforce per-run timeouts and row-count constraints; gate writes behind human approval; fire cron schedules exactly once per tick.
Doesn't (today): automatically retry a failed run, or route failures to a dead-letter queue. Retry and DLQ are roadmap items — the honest reliability primitives that exist today, and the idempotent-pipeline patterns buyers use in their place, are documented in Reliability & Operations. Cancellation is best-effort: a run requests cancellation rather than guaranteeing immediate termination (see Cancellation).