Failed Jobs Queue

Provisioning failures are recorded honestly as jobs, classified by why they failed — and retry is gated differently for each class.

Every write attempt this platform makes to a target system leaves evidence: what was sent, against which system, on behalf of which identity, and what came back. A provisioning job is the work-queue entry on top of that evidence, filed when a write needs a person's attention. A write that succeeds needs no triage and is never looked at. The ones that do not succeed become a queue that a person works.

There used to be a second mechanism — a failed write would also spawn a manual task for someone to go and do by hand. That is gone. A failure produces exactly one artefact, the job itself, carrying a structured explanation of what went wrong and a snapshot of the configuration that was in force at the time. One record, one place to look, one thing to close.

ℹ A healthy queue is not an empty queue.

Real target systems rate-limit, time out and go down for maintenance. Those failures are recorded rather than swallowed, so a queue with items in it is usually the platform working correctly. What matters is the mix and the age — see operating rhythm for how to read it week to week.

The administrator's Home shows the same failures grouped by system and error class: per group, how many jobs and how many rejected writes, how many distinct accounts, when it started and the HTTP status of the newest error. It counts failed and blocked jobs and also writes that were rejected without ever becoming a job, so a system whose writes are all refused shows up even when the queue itself is empty. The raw error text stays on the record; Home shows only the class. When anything fails, the bar at the top names the largest group ("Provisioning to system has failed since …") and how many other systems fail; when nothing fails, the table says so in one line.

What the record actually says

A failure carries a reason from a fixed set — configuration error, lookup failed, missing dependency, data quality, target API error, target unavailable, rate limited, policy blocked, or unknown. (A write operation that the system's owner switched off in its write permissions is not a failure and files no job; the platform refuses it before anything is sent, and the Reconciliation view shows it as "writes switched off". Older jobs of that kind were closed with their own cause, "writes switched off for this system".) — and a detail structure shaped to that reason. The shape is not cosmetic: it is what makes a failure diagnosable without opening a log.

A configuration-shaped failure names the mapping and the lookup that went wrong, and points at the editor that owns it:

{
  "kind": "lookup_failed",
  "lookup_business_id": "lookup-cost-centre",
  "lookup_args": { "cost_centre": "FIN-4471" },
  "filter_template_rendered": "(&(objectClass=group)(cn=CC-FIN-4471))",
  "target_response": { "matched": 0 },
  "suggested_action": "No group matches this cost centre. Create it in the directory, or narrow the lookup so unmatched identities are skipped.",
  "config_deep_link": "/systems/ad-corp/lookups/lookup-cost-centre"
}

A policy-blocked failure looks deliberately different, and the difference is the point:

{
  "kind": "policy_blocked",
  "operation": "update_identity",
  "policy": "attribute_drift_action",
  "blocked_by": "connector policy set to review",
  "what_would_have_been_sent": { "department": "Finance" },
  "suggested_action": "Change the connector's drift policy, or record this job as won't-fix."
}

There are no response fields here, because there was no response. No call left the platform. An earlier version of this record wore HTTP-shaped fields on a policy-blocked row — a status of zero and a response body that was actually our own sentence — describing an exchange that never happened. what_would_have_been_sent is the intent that was held back, not a payload that went over the wire. The same discipline applies in the other direction: when a call was made, the record carries the real request and response as captured from the connector, redacted, and an unknown status is absent rather than reported as zero.

Three retry classes, not one gate

The interesting question about a failed write is not did it fail but would trying again help. Three answers, and each needs different handling:

Class	What it means	Retry behaviour
Never retryable	A governance gate refused the write before it reached a connector.	Refused, permanently.
Transient	Nothing in the configuration would change the outcome; the condition passes on its own.	Allowed freely, no gate.
Configuration-gated	Something in the platform's configuration is wrong.	Refused until the configuration actually changes.

Never retryable covers policy-blocked failures (and the closed switched-off rows above). A production-write gate, a connector set to review attribute drift rather than act on it — these stopped the write on purpose. No configuration edit anywhere unblocks them, because the block was not a configuration error; it was the policy doing its job. Retrying is guaranteed to block again, so the platform refuses and names the remedy: change the policy, or record the job as won't-fix. The refusal is enforced on the server, not merely hidden in the interface.

Transient covers target API errors, unreachable targets, and rate limiting. Here the platform's configuration is not at fault and there is nothing to fix, so gating retry behind a configuration change would make retry permanently impossible for exactly the failures where retry is the correct and only response. (Rate limiting is included even though the brake is the platform's own: the window passes, and no mapping edit would have helped.)

Configuration-gated is everything else — a broken mapping expression, a lookup that returned nothing or several things, a required attribute the identity does not have, a dependency on an account that does not exist, and unclassified failures. For these, the job holds a hash of the configuration snapshot taken at failure time, and a retry is refused while that hash is unchanged. This is the anti-quick-fix rule, and it is the reason the queue drains rather than churns: click retry until it works is not a strategy, and an operations team that can do it will.

A single universal gate was wrong in both directions at once. Applied to transient failures it froze them permanently — the queue would fill with target outages that nobody could clear, and a bulk "retry all" would quietly refuse every one of them. Applied to policy-blocked failures it was too lenient: it implied that some configuration change, somewhere, would eventually let the write through, and it would not. The taxonomy is not a refinement of the gate; it is the recognition that "can this be retried" has three answers.

One decision, surfaced on the record

Eligibility is computed by a single function, and both callers use it: the retry endpoint that enforces it, and the serialisation that puts retry_eligible and the blocking reason on the job itself.

That is a deliberate structural choice rather than tidiness. When the interface derives its own view of what is retryable, it drifts, and the drift always shows up the same way: an enabled button that returns an error when pressed. Here the button cannot be enabled unless the server would allow the action, because both read the same verdict — and when it is disabled, the record says why, in a machine-readable code that the interface turns into an explanation rather than showing a raw exception.

The same single-source rule runs one level up. The per-system root-cause view groups failures by reason and labels each cluster — the target's fault, the source data's fault, our configuration's fault, or none of those — and derives "is this fixable in the configuration editor" from the same canonical set the queue uses. Before that, the two disagreed: the same job could be offered a "fix configuration" prompt in one view while the queue correctly showed it as not fixable in another.

One job per demand, and jobs that close themselves

A persistently failing write produces one job, not one per reconciliation pass. Jobs are keyed on system, identity, account, entitlement and operation; a repeat failure refreshes the open job and counts an occurrence on it. If reconciliation had cancelled a job and the same failure returns within seven days, the cancelled job is reopened rather than a sibling created. A job an administrator resolved is never reopened.

Reconciliation also closes jobs whose cause has gone away, in the same run that proves it, and records which of two outcomes it was:

• Success: a grant job whose entitlement is now live in the target, or a revoke job whose entitlement is no longer there. The original failure detail is kept on the closed job.
• Cancelled: a grant job that is no longer wanted, or a revoke job that is no longer wanted because the access is expected again. A grant job raised by an access request is never cancelled this way, because an approved request must not read as silently withdrawn.

A job that an administrator archived or marked won't-fix is left as it is. Separately, when a target answers a write on an existing account with a clear "that account does not exist", the platform records the account as no longer present in the target and cancels its open jobs with that reason; a 404 on the endpoint or on some other object does not do this.

What retry actually does — and does not

Retrying clears the job back to pending, increments its attempt count, and re-baselines the configuration snapshot so that the next failure is measured against the corrected configuration rather than the original broken one. It is recorded in the audit trail with the failure reason it is superseding.

It does not itself call the target. The write is re-attempted the next time the platform converges that identity's access, because reconciliation is the engine that drives provisioning; clearing the job takes it out of the queue and moves the baseline the next attempt will be judged against. Nothing polls failed jobs and re-runs them on a timer: a failed job is never retried automatically. Someone has to fix the cause and say so.

One status behaves differently by design. A job blocked on a missing dependency — a write that needs an account on another system which does not exist yet — is held rather than failed, and resolves when the dependency lands. It sits in the same queue because it is still work waiting on something, but it is not a failure and is not counted as one.

Bulk actions, and closing something you will not fix

Bulk retry and bulk won't-fix take an explicit list of jobs — never a filter, so what is being acted on is what was selected — and return a per-job result: which succeeded, which did not, and for each refusal the same blocking code a single retry would have given. A job that is ineligible does not stop the rest, and it does not disappear into a summary. This matters more than it sounds: the earlier behaviour collected the outcomes and reported success regardless, so a "retry all" against a connector that was down reported cheerfully while refusing every job in the selection.

Won't-fix is the honest close for a failure that will not be repaired — legacy data, a customer-specific exception, a policy that is correct and blocking. It requires a justification string; an empty one is rejected. The justification, the actor and the failure reason it supersedes are written to the immutable audit trail, because "we decided not to fix this" is a governance decision and needs to survive the person who made it. The bulk form applies one shared justification across the selection and records it per job.

Cancelling a retry that is stuck

A job that was retried returns to pending and waits for the next convergence. When that never comes, the job showed as "Dispatching" for weeks, and none of the closes above fitted: won't-fix and archive refuse a pending job, and marking it resolved would claim a write that never happened. An administrator can now cancel the retry: one job, a selection, or every pending retry untouched for a number of days ("Clean up stale retries", 7 by default, counted first and confirmed afterwards, at most 500 per run). A reason is required. The job becomes cancelled with the resolution "admin cancelled", leaves the queue and the inbox, and the audit trail records the reason and that nothing was written to the target. A running job cannot be cancelled, and old pending retries do not expire on their own.

Every action on this queue — retry, won't-fix, cancel, bulk or single — requires the platform administrator role. An earlier, narrower operations role was retired and deliberately not reinstated.

What this does not do

• No automatic retry, and no backoff schedule. There is no worker draining failed jobs. A transient failure clears when the underlying write is attempted again during normal convergence, not because something re-queued it. Reconciliation can close a job whose cause has gone away (see above), but it does not re-run the write because a job exists.
• Retry does not validate your fix. The configuration hash proves that something changed, not that the change was correct. A wrong fix produces a second failure with a fresh baseline.
• The queue does not tell you what to change beyond the suggested action and deep link carried on the record. Those are per-reason hints, not diagnosis.
• A structured detail is only as good as what the connector reported. Where a target returns an opaque error, the record says so rather than inventing a cause.

---

Further reading:

• Operating rhythm — how to read this queue weekly, and what a healthy mix looks like
• Platform health — where the failure counts roll up
• Reconciliation — the engine that re-attempts the write
• Platform Advisor — findings that are state rather than failed writes
• Provisioning Mappings — where configuration-shaped failures are fixed
• Read/Write Parity — the class of misconfiguration that causes them