Tier-3 Hybrid Architecture
Agent / tunnel / in-process execution — connector reach and credential custody.
The architecture in one picture
The control plane (software-as-a-service) and the agent (running in the customer's virtual private cloud, or VPC) are the only two processes that talk to each other over the internet, and the agent is always the one that opens the connection: it polls for work and holds a WebSocket it initiated itself, so nothing has to be exposed inbound to reach it. System credentials stay in the agent vault — the payloads sent from the control plane to the agent never contain secrets.
The credential-custody problem
Customers may want the scale of a SaaS identity governance product without sending system credentials (API keys, LDAP bind DNs and OAuth secrets) to its control plane. A control-plane connector and a customer-hosted product are not the only two choices.
RapidValue's answer is the tier-3 hybrid model: the control plane (the interface, workflows, advisor, and audit trail) runs as SaaS, a lightweight agent inside the customer VPC handles all system input/output, and every secret stays in the customer's own vault.
Three execution modes per system
Every connected system has an execution_mode setting:
-
in_process — the control plane runs system calls directly. This is the default for SaaS-only customers with no local credential-custody requirement (test and development tenants, marketing-side integrations).
-
agent — a full proxy: every system method (listing users, granting a group, and so on) goes through the task queue to the agent, which rebuilds the connector locally and runs it there. This is the default for proof-of-concept (POC) tenants and all on-premises integrations. The agent resolves credentials locally from its own vault file.
-
tunnel — the system's code runs on the control plane (REST or SCIM), but the HTTP transport is routed through the agent. This suits targets whose API is only reachable from inside the customer VPC, while the system logic itself does does not need to execute locally.
Tunnel mode bridges HTTP and nothing else, so it is refused rather than degraded wherever there is no HTTP layer to bridge. LDAP and Active Directory, SQL, SFTP and SMB/NTFS are raw TCP: building one of those in tunnel mode fails loudly at build time instead of quietly falling back to running from the control plane. The same applies to any engine that cannot accept an injected transport — today only the REST and SCIM engines can. For everything else, the local-custody answer is full agent mode.
The secret-handling rule
Secrets NEVER travel over the wire from the control plane to the agent. The
payload for a task carries only configuration (engine type, base URL, engine
config). Authentication is resolved on the agent side through SecretRef
references that the agent looks up in its vault.
Since bring-your-own-vault v2 there is a per-tenant registry of secret
backends (zero or more per tenant); a system binds to one backend. agent_local
backends never store secrets centrally — resolution happens inside the tier-3
agent, and the secret never crosses the wire. read_only backends let the
customer supply a reference themselves, which the platform only resolves. The API
never returns a secret value — only its resolution status.
⚠ Note
If you ever see credentials in a log line taken from the wire payload during debugging, something is structurally wrong. Open a security ticket — this violates the documented credential-custody boundary.
Tunnel server-side request forgery guard
Every http_request task carries an allowed_base_url (equal to the system's
base_url), and the agent refuses anything that falls outside it — preventing a
compromised control plane from using the agent as a general-purpose proxy into
internal services (a server-side request forgery, or SSRF, attack).
The comparison is structural, not a string prefix: scheme, host and port must
match exactly, the path must sit under the base path, and a URL carrying
credentials in its authority is rejected outright. A prefix check reads as
equivalent and is not. https://api.acme.com.evil.com/ starts with the allowed
base as text, and https://api.acme.com@internal.corp/ puts the real destination
after an @ where a prefix check never looks. Both are refused here because the
host is parsed and compared rather than matched as characters.
Trusting a private certificate authority
Internal systems routinely present a certificate signed by the organisation's own certificate authority, or a self-signed one. Verification is on by default, and there are exactly two ways to make such a target verify — one safe and reusable, one an explicit escape hatch:
| Knob | Level | Effect |
|---|---|---|
| Trusted TLS certificates | tenant | Outbound connector TLS verifies against the system CA bundle plus the certificates the administrator added once |
skip_verify |
one connector | No verification at all, for that connector only — behind an explicit warning in the interface |
The trust store is tenant-level rather than per-connector deliberately: an internal CA is added once and every system behind it verifies, instead of the same certificate being pasted into each connector separately. A malformed entry in the list is skipped and logged rather than breaking TLS for every other connector.
One shared resolution serves all three execution modes, so trust does not quietly
mean something different depending on where the call is made from. That matters
most in tunnel mode, where the agent performs the TLS handshake, not the
control plane: the trusted certificates and the skip flag therefore travel in the
http_request task itself — they are public certificates and a boolean, not
secrets — and the agent applies exactly the same rules. In agent mode they ride
along in the connector configuration for the same reason.
How the agent proves who it is
The agent starts with an enrolment token, and that token is deliberately a single-purpose credential rather than a standing one. It exists so a fresh agent can prove itself once and register a public key. It is not the agent's long-term identity.
On first boot the agent generates an ECDSA P-256 keypair and registers the public half. From then on it authenticates by challenge-response: it asks for a one-time nonce (valid for two minutes), signs it with its private key, and exchanges the signed assertion for a short-lived access token — four hours, refreshed before it lapses. The nonce is consumed on use, so a captured assertion cannot be replayed.
Once a keypair is registered, the enrolment token stops being accepted. That is the point of the design. Previously a leaked enrolment token granted full agent authority indefinitely, in parallel with the keypair, which made the keypair story decorative: the only kill switch was revoking the whole agent. Refusing the bearer once a key exists also closes a key-overwrite path, where someone holding the token could replace the registered public key with their own and then mint themselves legitimate tokens. The refusal is enforced where the token is resolved rather than in a single route handler, so it holds on every path that accepts an agent credential — the WebSocket and the download endpoints included.
The enrolment window is also time-boxed: seven days by default, configurable, and switchable off entirely for an air-gapped installation with a long lead time. Agents that registered before the window existed carry no deadline and are not retro-actively locked out — a deployment must not disqualify a healthy fleet. Both dead ends have a named remedy, and each refusal says which one applies rather than a bare "refused":
| Situation | The enrolment token is | Remedy |
|---|---|---|
| A keypair is registered | Refused — the agent holds a better credential | Authenticate with the key; reset the keypair only if the private key is genuinely gone |
| Window closed, no keypair | Refused — the token was for enrolling, and enrolment never happened | Re-issue the enrolment token |
Re-issuing rotates rather than extends: it mints a fresh token and kills the old one. Extending would hand another week of life to precisely the token that has been sitting in a ticket queue.
Resetting the keypair is break-glass, and it revokes rather than waits. It exists for a lost private key, which would otherwise lock an agent out permanently — it cannot sign a challenge and its bearer is refused. But reset has a second trigger: a stolen private key, and that is the only case where the operation does real security work. So it does three things in one step — clears the registered key, re-opens the enrolment window so the operator's existing install still works, and revokes outstanding access tokens instead of letting a thief's four-hour session drain. Clearing the key while leaving live sessions running would reproduce, one layer up, exactly the weakness the bearer refusal closed: a credential outliving the act of revoking it. The mechanism is the existing one — a revocation watermark compared against the token's issue time, the same shape used for human sessions — not a second one invented for agents. Every reset records its actor and reason.
A running agent recovers from this by itself. A rejected call makes it drop the dead token, retry the challenge-response, and — if the control plane no longer holds its public key, which is what a reset looks like from the agent's side — re-enrol using the bearer that the reset just re-enabled. Without that step, revoking sessions would have produced a silently dead agent, which is worse than the lockout the reset was fixing.
A client certificate can be bound to an agent as an additional layer. Where one is set, a request that does not present that certificate's fingerprint is refused even with an otherwise valid credential, tying the agent to one machine as well as to one key. It is optional and additive: it narrows who may authenticate, and never replaces the keypair.
Updating the agent without trusting the channel
An agent that updates itself is only as safe as its check on what it downloaded. The control plane holds one Ed25519 signing key for the whole fleet and signs the agent source with it; the agent verifies that signature over the bytes it actually received, before writing anything to disk.
It verifies against a key it pinned on first contact. The public key is handed over at registration and repeated on every heartbeat, and the agent writes it once. If a different key arrives later, the pinned one is kept and the mismatch is logged — a compromised control plane must not be able to rotate the key it will be judged against. A legitimate rotation is therefore a deliberate, out-of-band act rather than something the channel can do to itself.
Three refusals matter, and all three leave the agent running its current version rather than trusting unverified code: no key pinned yet, no signature offered, or a signature that does not verify. It also refuses source that does not look like the agent at all, or that does not contain the version it was told to install.
Updates can be pushed as ordinary queued work rather than only polled, so an administrator can roll the fleet and see each agent's outcome: the update attempt is a task with a status like any other.
An older mechanism still exists alongside it, and it is worth naming rather than implying it is gone. Before the fleet-wide keypair there was a heartbeat-driven update verified with a shared symmetric key configured on both the control plane and the agent host. It is still there as a transition path for agents that predate the signed one, and it refuses just as firmly — no key configured locally, or no matching digest, and it declines to update. But a per-agent shared secret is exactly what drifts: rotate one side, or rebuild a host without it, and an agent quietly stops being able to update at all. One asymmetric key for the whole fleet has nothing per-agent left to fall out of step, which is why it is the path to be on.
Either way, an update that installs but does not survive is reverted without an operator. The previous version is kept before anything is written, and the new one only becomes the accepted version after it has run for a minute without falling over. If it crashes before that, the next start finds the old version still sitting there unconfirmed and re-executes it. A bad update is therefore a restart, not a site visit.
Where the connector code comes from
Agent mode rebuilds the connector locally, which means the agent host needs the connector code — and a customer host has only the agent script and a short list of Python dependencies. So the control plane serves a connector bundle: the slice of platform code the connector path actually needs, generated from the control plane's own source at the moment it is served. An engine running on the agent is therefore the same engine as the one running on the control plane, not a re-implementation that can drift.
The slice is computed by following imports from the connector entry points, and it follows an edge only into connector code, shared utilities, and domain data definitions. It deliberately stops at domain services: connectors are the lowest layer and depend on data models, never on services, so a connector reaching for one would be a layering fault to fix rather than a reason to widen what gets shipped onto customer hosts. Holding that boundary is also what keeps the agent's dependency list short.
Lockstep is by content hash. The bundle's version is a hash of its contents, advertised on every heartbeat; when the agent's copy does not match — or is missing, or no longer imports — it fetches and unpacks a fresh one and drops the stale modules from memory. Rolling the control plane re-provisions the fleet without an operator step.
The bundle carries code and only code — public connector logic. It never contains a vault file, an environment file, or a resolved credential, and a test asserts exactly that on every build. Secrets are still resolved agent-side from the local vault, which is the entire point of the model. Tunnel mode needs none of this: there the engine is built on the control plane and only HTTP is bridged.
Running the agent
Three operational capabilities sit alongside the security ones:
- WebSocket push — the agent opens a connection and the control plane notifies it the moment work exists, so task latency drops from roughly 25 seconds of long-poll to near-zero. Long-poll remains the fallback, and the connection is always outbound.
- Supervisor process — automatic restart on crash with exponential backoff and clean signal forwarding, so a crash is a blip rather than an outage.
- Provisioning thresholds — cumulative create, update and delete counters. When a threshold is exceeded the agent refuses further operations of that type without crashing itself, which turns a runaway loop into a bounded one.
Task execution is auditable and ends in an explicit completed, failed or timeout state. WebSocket push reduces latency; it is not the delivery guarantee. The reliable fallback is polling, and reconciliation plus the Failed Jobs queue resolve uncertainty after an interrupted or ambiguous operation.
Known edges, published rather than discovered
-
No exactly-once or transparent agent failover claim. The current queue is addressed to an agent and execution is synchronous. Instance groups are labels, not completed high-availability routing. A timeout must be reconciled against the target before retrying.
-
Tunnel mode is HTTP-only, and that is structural. Nothing planned changes it: LDAP, AD, SQL, SFTP and SMB have no HTTP layer to bridge. For those, agent mode is the local-execution answer, not a workaround.
- The TLS trust store does not yet reach every connector. Both knobs flow through the connectors built on a shared HTTP client. Connectors built on a vendor SDK — Entra ID, ServiceNow, Salesforce, Google Workspace, Box — construct their own clients and do not read them today, so the option is deliberately hidden for those rather than offered and silently ignored.
- Signed push-updates are a deployment step, not an automatic property. The asymmetric signature exists only where the control plane has been given a signing key. Without one it advertises no signature, the push path refuses, and a fleet still carrying the older shared-key configuration keeps updating over that path instead. Nothing unverified is installed either way — but "every update is signed with a fleet key" is a statement about a configured deployment, not about the software on its own.
- The enrolment remedies are administrative. An agent that is both keyless and past its window cannot recover on its own; it needs an administrator. That is deliberate: the alternative is a credential that renews itself, which is the thing being removed.
- None of this is externally certified. Nothing on this page has been certified or attested by a third party. What exists instead is mechanism you can inspect: the refusals are enforced where credentials are resolved rather than at one convenient route, the signature is checked against a key your own agent pinned, and every reset and re-issue leaves an immutable audit record.
Further reading: