Documentation / Reporting & Data

Files

One unified file store — small references inline in Postgres, larger uploads and outputs as Azure Blobs, chosen transparently by size — with REST and ctx.files access, all inside your own Azure tenant.

Files in Nexus are a single, unified store for the documents your data work touches: small inline text references you author by hand, uploads supplied to a running pipeline, and the outputs a pipeline produces. This page is for analysts and developers who build pipelines and reports and need to bring files in, write results out, and reuse prior outputs as input — and for the reviewer who wants to know exactly where each byte lives.

Every file lives in your own Azure tenant. Nothing leaves your subscription, and access is always governed by role-based permissions checked on every request.

One store, two storage backends

Nexus unifies what used to be two separate subsystems — inline text references and pipeline blob outputs — into one files concept with a single API, a single permission surface, and a single listing view. Under the hood it picks the storage backend transparently by size and kind, so you don't manage two systems:

Kind	Created by	Stored as	Editable
Reference	An author, by hand	Inline text in Postgres (CSV / Markdown / plain text, up to 1 MB)	Yes
Output / Upload	A pipeline run, or a file upload	Blob in tenant Azure storage, parsed on read	No (immutable record)

References are the named, programmatic {{file.NAME}} form — small templates, lookup lists, or configuration snippets you want a pipeline to read by name. Because they are small and text, they live inline in Postgres, versioned and editable.
Outputs and uploads are the larger, run-scoped artifacts: the CSV a pipeline emits, the spreadsheet someone uploads to an approval step. These live as blobs in your tenant's Azure storage, accessed through the deployment's managed identity, and are recorded as immutable rows in the files table.

The distinction is storage ∈ {inline, blob} and kind ∈ {reference, output, upload}. You address every file the same way regardless of backend; Nexus resolves inline vs blob for you.

Base64 is transport, not storage

A point worth stating for anyone integrating over the API: when you upload file bytes, the content may be base64-encoded on the wire so it can travel safely through JSON-RPC and REST. That encoding is a transport detail only. The bytes are stored decoded — text references as text in Postgres, binary uploads and outputs as raw blobs in Azure storage. Nexus never persists base64 as its storage format, and you should not treat the encoded form as anything but a way to move bytes across the API boundary. Dedicated upload paths exist precisely so large or binary files move as bytes rather than being inlined into a call.

Working with files in a pipeline

Pipelines are Python modules with an async def run(ctx) entry point. The file surface is available as ctx.files, scoped to the current run, so a pipeline reads inputs and writes outputs without having to supply an identity it could spoof.

async def run(ctx):
    # Read a named text reference by name.
    template = await ctx.files.read("monthly_template")

    rows = await ctx.adapters.sales.read(...)

    # Write an output blob (csv by default).
    result = await ctx.files.write("monthly_report", rows, file_type="csv")

    # Read an earlier output (or upload) back as PARSED data.
    prior = await ctx.files.read_output("monthly_report")
    # prior -> {"data": [...], "columns": [...], "row_count": ...}

    return {"file_id": result["file_id"], "rows": len(rows)}

The ctx.files methods:

write(name, data, *, file_type="csv") — write a list of rows as an output blob. The file type is validated against the allowed export types.
read(name) — return the text content of a named reference.
read_output(name) — find the most recent output or upload whose name matches and return its parsed data (data, columns, row_count). Scoped to the current pipeline by default; you can narrow it to a specific earlier run.
open(file_id) — read any file by id. References return their content; blob files return parsed data.
list(*, kind=None) — list this pipeline's files, optionally filtered by kind.

Because read_output and open return parsed data, the natural pattern is to feed a prior run's output — or a file someone uploaded to an approval step — straight back in as the next run's input. Run-scoping means every write is attributable to the run that made it, which is what keeps the audit trail honest.

Supported formats

Outputs are written as structured exports (CSV and similar). On read, Nexus parses a range of common formats into rows and columns, including CSV, JSON, Excel (XLSX), plain text, PDF, Word (DOCX), and email (MSG / EML), so an uploaded spreadsheet or document becomes usable data without extra glue code. The parse happens on read; the stored blob is always the original bytes.

Three things files are used for

The unified store deliberately treats these the same, so a pipeline or report can find, read, and build on any of them:

Attachments / inputs. A file a user uploads to a pipeline or an approval step — a spreadsheet to reconcile, a document to parse. Read it back as parsed data with read_output/open.
Outputs. What a run produces — the CSV export, the generated report. Immutable, blob-backed, and shown on the run's detail view.
Site assets. The static HTML/CSS/JS behind a Site or dashboard are uploaded and versioned as files too, which is why a dashboard change is a reviewable commit rather than an untracked deploy.

Keeping all three in one store means "a template I authored," "a file a user uploaded," and "what last night's run produced" are all reachable the same way — find it, read it, build on it.

Managing files outside pipelines

Files are also a first-class resource over the REST API and in the desktop app's Documents view. You can:

List and filter files by kind, pipeline, run, direction, or a text search.
View a file — references return their content inline; blobs return a short-lived signed download link.
Download the raw bytes of any file.
Create, edit, and delete references (display name, description, content). Output and upload blobs are immutable records; deleting one also removes its underlying blob.

The same operations are available over MCP (uql_list_files, uql_get_file, uql_read_file, uql_create_file, uql_update_file, uql_delete_file), so an AI assistant manages files under the same permission checks as everything else.

Access and governance

Access is governed by role-based permissions: a read permission to list, view, and download, and a manage permission to create, edit, or delete. Like the rest of Nexus, these checks run on every request — there is no unguarded file endpoint. Blob downloads are served through short-lived signed links rather than a public URL, so a link cannot be replayed indefinitely or shared outside the permission boundary. Because storage is your tenant's own Azure Blob storage reached via managed identity, there are no standing storage credentials to leak, and every byte stays inside your subscription.

Limits & guarantees

References are ≤ 1 MB text and editable; anything larger or binary is a blob.
Outputs and uploads are immutable records. You don't edit them in place — you write a new one. Deleting an output also deletes its blob.
ctx.files is run-scoped by default, which is what makes every write attributable; you can widen a read to a specific earlier run when you need prior output.
Backend is chosen for you. You never pick inline vs blob — Nexus resolves it by size and kind, and you address files uniformly.
Binary fidelity on download depends on the client path. Text/CSV/JSON/Markdown round-trip exactly; some clients may relay binary blobs (xlsx/pdf) through a JSON path — use the raw download or signed link for byte-exact binary.

Why it works this way

Keeping references and run artifacts in one store means a report or pipeline can treat "a template I authored," "a file a user uploaded," and "what last night's run produced" the same way — find it, read it, build on it. Run-scoping on ctx.files keeps each run's writes attributable, transparent inline-vs-blob storage keeps small config cheap and large artifacts durable, and tenant-resident storage with managed-identity access keeps every byte inside your Azure boundary.