RapidValueDocs RapidValue.eu
Docs/Running it
8 min readRunning itOwner: RapidValue ProductProduct baseline 5440d975

Failed Jobs Queue

Provisioning failures are recorded honestly as jobs, classified by why they failed — and retry is gated differently for each class.

Every write this platform makes to a target system is a provisioning job: a durable record of what was attempted, against which system, on behalf of which identity, and how it ended. Most succeed and are never looked at. The ones that do not become a queue that a person works.

There used to be a second mechanism — a failed write would also spawn a manual task for someone to go and do by hand. That is gone. A failure produces exactly one artefact, the job itself, carrying a structured explanation of what went wrong and a snapshot of the configuration that was in force at the time. One record, one place to look, one thing to close.

ℹ A healthy queue is not an empty queue.

Real target systems rate-limit, time out and go down for maintenance. Those failures are recorded rather than swallowed, so a queue with items in it is usually the platform working correctly. What matters is the mix and the age — see operating rhythm for how to read it week to week.

What the record actually says

A failure carries a reason from a fixed set — configuration error, lookup failed, missing dependency, data quality, target API error, target unavailable, rate limited, policy blocked, or unknown — and a detail structure shaped to that reason. The shape is not cosmetic: it is what makes a failure diagnosable without opening a log.

A configuration-shaped failure names the mapping and the lookup that went wrong, and points at the editor that owns it:

{
  "kind": "lookup_failed",
  "lookup_business_id": "lookup-cost-centre",
  "lookup_args": { "cost_centre": "FIN-4471" },
  "filter_template_rendered": "(&(objectClass=group)(cn=CC-FIN-4471))",
  "target_response": { "matched": 0 },
  "suggested_action": "No group matches this cost centre. Create it in the directory, or narrow the lookup so unmatched identities are skipped.",
  "config_deep_link": "/systems/ad-corp/lookups/lookup-cost-centre"
}

A policy-blocked failure looks deliberately different, and the difference is the point:

{
  "kind": "policy_blocked",
  "operation": "update_identity",
  "policy": "attribute_drift_action",
  "blocked_by": "connector policy set to review",
  "what_would_have_been_sent": { "department": "Finance" },
  "suggested_action": "Change the connector's drift policy, or record this job as won't-fix."
}

There are no response fields here, because there was no response. No call left the platform. An earlier version of this record wore HTTP-shaped fields on a policy-blocked row — a status of zero and a response body that was actually our own sentence — describing an exchange that never happened. what_would_have_been_sent is the intent that was held back, not a payload that went over the wire. The same discipline applies in the other direction: when a call was made, the record carries the real request and response as captured from the connector, redacted, and an unknown status is absent rather than reported as zero.

Three retry classes, not one gate

The interesting question about a failed write is not did it fail but would trying again help. Three answers, and each needs different handling:

Class What it means Retry behaviour
Never retryable A governance gate refused the write before it reached a connector. Refused, permanently.
Transient Nothing in the configuration would change the outcome; the condition passes on its own. Allowed freely, no gate.
Configuration-gated Something in the platform's configuration is wrong. Refused until the configuration actually changes.

Never retryable covers policy-blocked failures. A production-write gate, a connector set to review attribute drift rather than act on it — these stopped the write on purpose. No configuration edit anywhere unblocks them, because the block was not a configuration error; it was the policy doing its job. Retrying is guaranteed to block again, so the platform refuses and names the remedy: change the policy, or record the job as won't-fix. The refusal is enforced on the server, not merely hidden in the interface.

Transient covers target API errors, unreachable targets, and rate limiting. Here the platform's configuration is not at fault and there is nothing to fix, so gating retry behind a configuration change would make retry permanently impossible for exactly the failures where retry is the correct and only response. (Rate limiting is included even though the brake is the platform's own: the window passes, and no mapping edit would have helped.)

Configuration-gated is everything else — a broken mapping expression, a lookup that returned nothing or several things, a required attribute the identity does not have, a dependency on an account that does not exist, and unclassified failures. For these, the job holds a hash of the configuration snapshot taken at failure time, and a retry is refused while that hash is unchanged. This is the anti-quick-fix rule, and it is the reason the queue drains rather than churns: click retry until it works is not a strategy, and an operations team that can do it will.

A single universal gate was wrong in both directions at once. Applied to transient failures it froze them permanently — the queue would fill with target outages that nobody could clear, and a bulk "retry all" would quietly refuse every one of them. Applied to policy-blocked failures it was too lenient: it implied that some configuration change, somewhere, would eventually let the write through, and it would not. The taxonomy is not a refinement of the gate; it is the recognition that "can this be retried" has three answers.

One decision, surfaced on the record

Eligibility is computed by a single function, and both callers use it: the retry endpoint that enforces it, and the serialisation that puts retry_eligible and the blocking reason on the job itself.

That is a deliberate structural choice rather than tidiness. When the interface derives its own view of what is retryable, it drifts, and the drift always shows up the same way: an enabled button that returns an error when pressed. Here the button cannot be enabled unless the server would allow the action, because both read the same verdict — and when it is disabled, the record says why, in a machine-readable code that the interface turns into an explanation rather than showing a raw exception.

The same single-source rule runs one level up. The per-system root-cause view groups failures by reason and labels each cluster — the target's fault, the source data's fault, our configuration's fault, or none of those — and derives "is this fixable in the configuration editor" from the same canonical set the queue uses. Before that, the two disagreed: the same job could be offered a "fix configuration" prompt in one view while the queue correctly showed it as not fixable in another.

What retry actually does — and does not

Retrying clears the job back to pending, increments its attempt count, and re-baselines the configuration snapshot so that the next failure is measured against the corrected configuration rather than the original broken one. It is recorded in the audit trail with the failure reason it is superseding.

It does not itself call the target. The write is re-attempted the next time the platform converges that identity's access, because reconciliation is the engine that drives provisioning; clearing the job takes it out of the queue and moves the baseline the next attempt will be judged against. Nothing polls failed jobs and re-runs them on a timer: a failed job is never retried automatically. Someone has to fix the cause and say so.

One status behaves differently by design. A job blocked on a missing dependency — a write that needs an account on another system which does not exist yet — is held rather than failed, and resolves when the dependency lands. It sits in the same queue because it is still work waiting on something, but it is not a failure and is not counted as one.

Bulk actions, and closing something you will not fix

Bulk retry and bulk won't-fix take an explicit list of jobs — never a filter, so what is being acted on is what was selected — and return a per-job result: which succeeded, which did not, and for each refusal the same blocking code a single retry would have given. A job that is ineligible does not stop the rest, and it does not disappear into a summary. This matters more than it sounds: the earlier behaviour collected the outcomes and reported success regardless, so a "retry all" against a connector that was down reported cheerfully while refusing every job in the selection.

Won't-fix is the honest close for a failure that will not be repaired — legacy data, a customer-specific exception, a policy that is correct and blocking. It requires a justification string; an empty one is rejected. The justification, the actor and the failure reason it supersedes are written to the immutable audit trail, because "we decided not to fix this" is a governance decision and needs to survive the person who made it. The bulk form applies one shared justification across the selection and records it per job.

Every action on this queue — retry, won't-fix, bulk or single — requires the platform administrator role. An earlier, narrower operations role was retired and deliberately not reinstated.

What this does not do

  • No automatic retry, and no backoff schedule. There is no worker draining failed jobs. A transient failure clears when the underlying write is attempted again during normal convergence, not because something re-queued it.
  • Retry does not validate your fix. The configuration hash proves that something changed, not that the change was correct. A wrong fix produces a second failure with a fresh baseline.
  • The queue does not tell you what to change beyond the suggested action and deep link carried on the record. Those are per-reason hints, not diagnosis.
  • A structured detail is only as good as what the connector reported. Where a target returns an opaque error, the record says so rather than inventing a cause.

Further reading:

Did this answer your question?One click records the page; add detail by email if something is missing.

Try “tenant isolation”, “role mining” or “Entra”.