Imports & schedules

How data is imported from connected systems, the shared sync-schedule model with its 15-minute cadence floor, and how imports feed the Expected-vs-Actual model.

Imports are how the platform learns the current state of a connected system — who has accounts, which entitlements exist, and who holds them. Each system runs on its own independent schedule, and the same schedule primitive that drives imports also drives scheduled workflows. This page explains the import model, the schedule model, and how imported data becomes the Actual side of the reconciliation comparison.

What an import is

An import is a read from a connected system. It pulls the system's current objects — accounts, entitlements, and the assignments between them — into the platform so they can be governed. Imports are strictly read-only run types: syncing, health-checking, probing, and discovery. Writing back to a system (creating accounts, granting access) is a separate concern — a provisioning operation, tracked as a provisioning job — and is never part of an import.

The distinction matters: imports tell the platform what the system actually looks like right now. Governance decides what it should look like. Reconciliation compares the two.

💡 Tip

Business data — identities, accounts, entitlements, and assignments — must arrive through imports from live systems, never be seeded directly. Human-resources data flows in as identities; target systems flow in as accounts, entitlements, and assignments. Only configuration (connectors, account-type blueprints, policies, rules) is set up directly.

The sync-schedule model

Every connected system carries its own sync schedule, evaluated independently against that system's own last-run time. One system's cadence is never altered by another's. A schedule has a kind:

Kind	Meaning
interval	Run every N minutes
daily	Run once a day at a set time
weekly	Run on chosen days of the week at a set time
monthly	Run on chosen days of the month (or an "nth weekday", e.g. the second Tuesday) at a set time
manual	Never run automatically — imports only when triggered by hand

Each schedule carries its own timezone, and richer schedules can add a validity window (a start and end date), an intraday time window for interval schedules ("only within work hours"), and a "repeat every N" multiplier for cadences like biweekly or quarterly.

The 15-minute cadence floor

A sync interval may not be faster than 15 minutes. A value of 0 means manual-only and is always allowed; anything from 1 to 14 minutes is rejected. The reason is performance, not cost: the scheduler tick is about a minute and syncs within a tick run one after another, so a sub-15-minute cadence adds sweep load that slows other systems without any meaningful gain in freshness.

The floor is enforced centrally in the schedule parser, so it applies uniformly to every consumer of the schedule primitive. The configuration surface reflects this — presets start at hourly, and the custom input is clamped at 15 with a soft advisory in the 15-to-59-minute range: allowed, but not recommended.

The shared schedule primitive

The schedule model is a single shared primitive, not a per-feature reimplementation. Two real consumers use the identical schedule shape and the identical parser:

• Connector sync — a system's sync_schedule, driving when it imports.
• Scheduled workflows — a workflow's trigger schedule, driving when a trigger_type=scheduled workflow fires.

Because both go through the same parser, the 15-minute floor, the timezone handling, and every schedule kind behave the same everywhere. A multi-type timeline can therefore project the next fire times of both connectors and scheduled workflows onto one view — so "what runs when, across both imports and workflows" is a fact on screen rather than something inferred from a list of one-line summaries.

Event-triggered workflows are a separate mechanism (they react to an event, not a clock) and are outside the schedule primitive. A workflow can also start one run per person on a date of that person or on a per-person schedule; see workflows.

Ordering and dependencies between systems

Schedules are independent, but imports can declare a data dependency on another system — for example, an Entra sync that needs Active Directory to resolve samAccountName first. When two systems with a declared dependency both come due in the same scheduler tick, the platform orders them so the upstream system imports before the downstream one. A dependency only has teeth when both ends are due at once; it never changes either system's own cadence. Each system's import also runs in isolation — one broken system's failure is caught and logged, and the others continue.

When something else already owns the write

Two related questions come up as soon as a second system is connected, and both are about not fighting a sync that already exists.

Is there an upstream source? During onboarding a vendor-gated step asks whether another directory syncs into the system you are connecting — for Entra, whether an on-premises Active Directory feeds the tenant. Answering yes narrows what is imported to the objects that system actually owns, so the platform does not try to manage what the upstream sync already manages. Without that, two writers race over the same object and the loser is whichever ran last.

Who should receive a write? A sync relationship declares that system A is the source for system B, scoped to group memberships, identities, or both. When a grant or revoke on B matches the declared pattern, the platform does not write to B. It records the change and routes the action to the source side instead, where the existing sync will carry it downstream.

The canonical case is AD feeding Entra through Connect Sync: a group membership belongs on the AD side, and writing it directly into Entra would be undone — silently — the next time the two reconcile with each other. Declaring the relationship makes that explicit rather than leaving it as folklore held by whoever configured it.

Provenance is not one global authority matrix

An imported object retains the connector instance that observed it, and mappings declare how source fields enter the canonical model. Correlation keys, write mappings and upstream sync relationships each make authority explicit at their own boundary.

RapidValue does not currently claim one universal matrix that resolves every property conflict across every object and identity type. Evaluate authority per connector, mapping and operation. The distinction—and the safe conflict behaviour for API identity pushes—is documented in Source provenance and data authority.

How imports feed the Expected-vs-Actual model

An import produces the Actual state: exactly what exists in the connected system as of that run. The governance model produces the Expected state: what should exist, according to access grant policy matches, approved access requests, role-based assignments, and explicit keeps.

Reconciliation compares the two, per identity, per system:

• Actual comes from the most recent import.
• Expected comes from the governance model.
• Any difference is drift — an attribute changed by hand, an assignment that exists in the system but is not expected, or an account the model expects that is missing.

So imports are strictly upstream of reconciliation. They are not a competing view of access — they are the sensor that tells reconciliation what is really out there. A stale or failed import means reconciliation is comparing against old Actual data, which is why import health and schedules are worth watching. Throughout the product this comparison is labelled Expected / Actual.

One import at a time, and never wedged

A connector runs one sync at a time. Two imports of the same system racing each other read and write the same rows in an order neither controls, and the losing one silently overwrites the winner's work — so a manual "Sync now" while a scheduled run is in progress, or while another request for the same connector is open, is refused (HTTP 409) rather than queued behind it, and the scheduler skips instead of stacking.

"Sync now" is a request, not a wait. The call answers 202 with a request id and a status link, and the scheduler process runs the sync; the portal polls GET /connectors/{id}/sync/requests/{request_id} until it ends. A 50,000-record sync therefore no longer holds a web connection open. At most one request per connector is open (queued or running). Scheduled and requested syncs share one budget of 3 concurrent syncs, and a scheduled sync never takes the last slot, so a request does not wait behind a storm of due syncs. A request that was never picked up, or whose runner is gone, is reported as failed after 60 minutes. There is no synchronous form any more: ?wait=true was removed on 25 October 2026.

That guard creates a second problem that has to be solved in the same breath: if a sync process is killed halfway — a container restart, an out-of-memory kill — it leaves a run recorded as still running, and a naive guard would then block that connector's imports forever. There is no heartbeat to consult; the only thing known about a dead run is when it started. So a run still marked running for more than 60 minutes (it was 30 until October 2026) is treated as dead, closed as failed with an explanation, and got out of the way. That clearing happens before the concurrency check, on the manual path and the scheduled one, and a background sweep does it too — so a wedged run clears even if nobody tries that connector again.

The pattern is worth naming because it generalises: a guard that can be poisoned by a crash is a guard that eventually becomes an outage. If you add one, you owe it a way out.

A big source sync commits per chunk

A source sync whose delta is above 2,000 changes (default; an operator setting, 0 turns it off) is applied in chunks, each its own commit, instead of one transaction for the whole run.

• Ownership first. The people the run attaches, takes over or transfers, with their own updates, form chunk 1 and are never split.
• The leaver guard is judged once on the whole delta before chunk 1, so a large share of leavers cannot hide by being spread over small chunks. JML events go out per chunk, after that chunk's commit.
• A failure part-way ends the run as partial. What landed stays. The run summary records the chunk size, the total, how many are done and which failed. The next run computes its difference from the database and applies only the remainder. A first-time connector keeps its last-sync time empty. If chunk 1 itself fails, the run fails.
• Limits. A released breaker or leaver override is spent before chunk 1; if the run stops half-way the next run judges the remainder afresh. This is source sync only; the target sync is still one transaction.

A read that could not see everything is not an empty read

A sync treats every record a bulk read does not return as gone, so a read that quietly returned less than the target holds would revoke real access. Failed reads therefore stop the sync or hold only what they could not see:

• A failed or truncated read on a generic connector fails the sync. For REST, SQL and SFTP-CSV connectors, a failed nested request, a page walk cut short, or a configuration that does not parse raises an error and the run applies nothing, instead of reading as an empty system. The wizard's live test and previews keep their lenient, limited reads.
• A failed read of one group's members holds that group. For Entra and Google Workspace, a group whose membership could not be read keeps its stored grants, the run ends partial with the failed scopes on its summary, and reconciliation holds adds and revokes on those scopes until a later complete read.
• Agent-mode list reads have a ceiling per connector. Each is bounded by Records per agent read (default 100,000, at most 250,000) and Seconds per agent read (default 600, at most 1,800), set on the Connection tab; a value above the ceiling is refused. A source larger than the bound is cut, reported as incomplete, and derives nothing from what it did not read. All the reads of one run together may wait at most 55 minutes. Very large values need memory (about 1 GiB per 100,000 records).
• Agent-mode connectors report the same. From agent version 0.2.37 a read cut short reports what it missed and the sync holds that scope. Below that version every revoke, disable and delete on the system is held, with a finding that says to update the agent; adds still go.

A held import, and what releasing or discarding it does, is described in Safety limits. Discarding a held import closes the trip that held it and resumes the sync that trip paused, unless you untick Resume syncing afterwards.

A people file instead of an HR connection

An organisation without an HR integration can upload a CSV of its employees as a real authoritative source of people. In the system onboarding wizard it is the People file choice under "Which connector?"; the wizard then asks for the file, the columns and a check against Entra instead of a connection, credentials and a base URL. After activation the system has a People file tab.

• Strict before anything runs. A person who is missing from a source is a leaver, so a file that cannot be read completely and cleanly is refused at upload, before any run: no guessing at the encoding, no ragged rows, no silently cut file. The limits are 10 MB and 50,000 rows. The person id, e-mail and name must be filled and unique in the file, as must the column you join on.
• Nothing changes until you apply it. An upload stores and checks the file; it never creates or changes a person. A new file waits until an administrator has seen its preview and applied it, and the same leaver guard and breaker as any source sync can hold it. There is no schedule: a people file is never synced by the clock.
• One current file. Only the applied file is kept, reduced to the columns you mapped. A replaced file keeps only its fingerprint, counts and who uploaded it.
• The check against Entra is advice. It says how many of the file's e-mails match a synced Entra account, which usually catches a file mapped onto the wrong column.
• The join column is measured. Which column matches the file with people other sources already hold is measured on the uploaded file, with the counts, and preselected for a first mapping; the administrator keeps or changes it.
• E-mail authority is pinned to the file, and recorded in the audit log, so the file's e-mail addresses are not refused as a change from a source that may not set them. A pin that another system holds is never taken over silently.
• Administrators only. Uploading and mapping need the administrator role; the history shows metadata, never the file's contents, to administrators and auditors. A people file runs in the platform itself: no agent, no credentials, and it never writes back.

What a run tells you afterwards

A run used to report only its own diff — so many records read, so many changed. That answers "did it work", but not "is this system in the state I expect", and the second question is the one that catches problems.

Each completed run now also records where the system stands: how many accounts, entitlements and assignments it holds; how many of those accounts are correlated to a real identity; and how many carry a classification. Alongside that sits the change since the previous import, computed against the last completed run rather than against live data — so it reads as "since the last time we looked", which is the comparison a human is actually making.

Two things make that number trustworthy. A first import has no predecessor, so it shows no delta at all rather than presenting its entire contents as a sudden change. And computing any of this is best-effort: if the enrichment fails, the run still succeeds. Insight about an import must never be able to break the import.

Read together, those figures answer the questions that matter after a sync: a correlated-account count that stops tracking the total means new accounts are arriving unowned, and a sharp negative delta is either a real deprovisioning event or a filter someone narrowed by accident — either way, visible in the run rather than discovered a month later in a review.

How long these run logs are kept is one tenant-wide setting, with how often the clean-up runs. An administrator sets both under Environment & connections › Sync run logs: a retention of 1 to 3,650 days and an interval of 1 to 168 hours. Until the first save the platform defaults apply, and the screen says so.

Further reading

• Reconciliation
• Provisioning jobs
• Source provenance and data authority