Documentation / Reporting & Data

Data Sync

Keep a target system in sync with a source in Nexus — discover the target, map fields, preview the exact inserts and updates, then run governed, auditable syncs by hand or on a schedule.

A Nexus data sync writes records from a source into a target system — for example, pushing rows from a reporting query or a SQL table into a CRM, an ERP, or another database. It is for admins who need a repeatable, reviewable way to keep two systems aligned, with a clear preview of what will change before anything is written. Syncs run entirely inside your own Azure tenant, against the bridges you have already configured.

This page covers the discover → map → preview → execute flow, the difference between saved and ad-hoc syncs, and the two-tier serving pattern — a scheduled pipeline landing data into the internal workspace, with reports serving it fast — that Nexus uses for real production workloads today.

How a sync is built

Setting up a sync follows four steps. The Nexus Desktop App walks you through them, and the same actions are available over the MCP and REST surfaces for automation.

1. Discover the target

Nexus inspects the target bridge and collection so you can see the available fields and their types. You map against the target system's real schema rather than guessing column names, and type information catches obvious mismatches early. Discovery uses the same discover_schema capability every adapter exposes.

2. Map the fields

Once Nexus knows the source columns and the target columns, it suggests a column mapping by matching field names. You review and adjust — rename, remap, or drop fields — until each source field lands in the right target field. The mapping is part of the saved sync definition, so it is reused on every run and is itself version-controlled alongside your other content.

3. Preview the diff

Before writing anything, Nexus computes a diff between the incoming data and the current state of the target. The preview is categorised so you can see exactly what a run will do:

Category	Meaning
Inserts	New records that do not exist in the target yet
Updates	Existing records whose mapped fields have changed
Unchanged	Records already in sync (no write needed)

You can page through the detail rows in each category to confirm the change set is what you expect. Nothing is written during preview — it is a pure read-and-compare. This is the safety property that makes a sync auditable up front: you approve a concrete insert/update set, not a black-box job.

4. Execute

When the diff looks right, you execute the sync. Nexus applies the inserts and updates to the target through the configured bridge and records the outcome. Because the runtime tracks rows created, read, and updated, every run produces an auditable record of what changed. Updates match on the mapped key so a re-run of the same source is idempotent for unchanged rows — they show as Unchanged and are not rewritten.

Saved syncs and ad-hoc syncs

Nexus supports two modes:

Saved syncs — a named, reusable configuration (source, target bridge and collection, and the column mapping). Save it once, then preview and execute it repeatedly as the source data changes. Saved syncs are the unit you schedule and audit.
Ad-hoc syncs — preview or execute a sync definition on the fly without saving it, useful for one-off loads or for testing a mapping before committing to it.

Both modes use the same discover → map → preview → execute flow and the same diff engine, so an ad-hoc preview behaves identically to a saved one. A common workflow is to iterate a mapping with ad-hoc previews, then save it once it produces the diff you expect.

The two-tier serving pattern

For sources that are slow, rate-limited, or expensive to read live — most REST and ERP APIs — the right architecture is not to point dashboards and the Excel add-in directly at the source. It is a two-tier pattern that Nexus runs in production:

Tier 1 — a scheduled pipeline lands data into the internal workspace. The pipeline reads the source (often pre-aggregating at the source, where that adapter can push down GROUP BY), then writes the result into per-tenant workspace tables. The workspace is a baked-in Postgres scratch database with its own managed identity, structurally isolated from Nexus's operational tables.
Tier 2 — reports serve from the workspace. Dashboards, exports, and the Excel add-in run reports addressed at "system": "postgres", "bridge_name": "workspace". Those reads are fast local Postgres queries, not round-trips to a throttled API, so the serving surface stays live-on-load while the expensive read happens overnight.

This separation means the source system is hit on a controlled cadence you own, the serving layer is fast and always available, and the two concerns — extraction and presentation — evolve independently.

A real example: nightly NetSuite → workspace

Nexus runs exactly this pattern in production for a live tenant's financial reporting. A set of Excel financial reports (trial balance, account balance, chart of accounts, and account detail) were ported to the two-tier model:

A nightly scheduled pipeline reads NetSuite over its REST / SuiteQL surface, running grouped SuiteQL so NetSuite itself does the aggregation (native sum(amount) per posting period and subsidiary) rather than dragging millions of transaction lines across the wire. It writes compact pre-aggregated balances plus refreshed dimensions (accounts, subsidiaries, currencies, periods, exchange rates) into workspace tables. State is tracked with a modified-date watermark so each run only pulls what changed.
Workspace-backed reports then serve the numbers: filtered, joined, and FX-adjusted at report time from the small pre-aggregated tables. Because the heavy lifting is pre-computed, a trial balance that would time out against the live API returns in milliseconds — and the figures reconcile penny-exact against the source of truth.
The presentation surface (the Excel add-in) points at the same report IDs it always used, so nothing downstream changed.

The takeaway for your own build: when a source can't push down the aggregation you need at serving speed, do the aggregation on a schedule into the workspace, and serve reports from there.

Running on a schedule

Syncs do not have to be run by hand. Because a sync executes through the Nexus pipeline runtime, you drive it from a scheduled pipeline — a Python module that supplies the source rows and triggers the sync — so a target system stays current automatically. A scheduled run produces the same structured logs and row counters as a manual run, so you keep full visibility into every automated write. Scheduling uses the same cron machinery as any pipeline, with exactly-once firing per tick (see Pipeline scheduling). An on-demand "refresh this date range" pipeline is a common companion to a nightly sync, letting an operator re-pull a specific window without waiting for the next scheduled run.

Governance and safety

Admin-gated. Previewing and executing syncs requires the Admin role. Building and saving a sync configuration is an administrative action, keeping write access to external systems controlled.
In-tenant. Source reads and target writes all happen inside your Azure subscription, through bridges you control. Data is not routed through a third party.
Preview before write. The diff step means no run is a black box — you see the precise insert and update set before any record leaves the source.
Auditable. Each run captures structured logs and row-level counters (created, read, updated), giving you a record of every sync in the audit log.
Service-bridge credentials. Writes go through the target bridge's own configured credentials, never a user's personal connection — the same credential-isolation rule as the rest of the platform.

Limits & guarantees

Inserts and updates, matched on the mapped key. A sync reconciles the mapped fields; it does not delete target records that vanished from the source unless your pipeline explicitly handles deletes.
Preview is a point-in-time read. The diff reflects the target as of preview; if the target changes between preview and execute, the executed diff is recomputed at execute time.
Re-runs are idempotent for unchanged rows. Matching rows show as Unchanged and are not rewritten, so re-running a sync over the same source is safe.
Best-effort, not transactional across systems. A sync applies writes through the target adapter; it is not a distributed transaction, so on a partial failure you get an auditable record of what was written and can re-run to reconcile. (See the reliability guidance on idempotent design.)
Workspace reports cap at 1,000 rows unless the serving report sets an explicit limit — worth remembering when the workspace is your serving tier.

Putting it together

A typical setup looks like this:

Configure the source (a report or a table on a bridge) and the target bridge.
Discover the target collection and let Nexus suggest a mapping.
Adjust the mapping and save the sync.
Preview the diff, confirm the inserts and updates, and execute.
Schedule a pipeline to refresh the source and re-run the sync on a cadence — or, for slow sources, land data into the workspace nightly and serve reports from there.

The result is a target system that stays aligned with your source data, with every change reviewable up front and recorded after the fact.