Recovery machinery exists to answer one question: when something fails, what is allowed to happen next — and for how long. The answer is a nested budget hierarchy, an escalation ladder, bounded regeneration with a circuit breaker, and checkpoint-based resume. Every mechanism is bounded; none can loop indefinitely.
The budget hierarchy
| Level | Bound | Exhaustion behavior |
|---|---|---|
| Iterations per attempt | Planner-verifier cycles | Attempt fails, critique loop ends |
| Retries per step | Execution attempts | Escalation ladder begins |
| Escalations | Human signals before park | Workflow parks with diagnosis |
| Regeneration cap | Child generations per SUB_SOP | Parent-side escalation, then park |
| Global timeout | Wall clock, max_global_workflow_timeout_ms |
Run terminates; unconsumed HITL responses stripped |
Budgets bind the workflow, not the incarnation. A resume-from restart, a new generation, a re-entered attempt — none of them reset what the run has already consumed. This is the property that makes budgets real: consumption is recorded in the ledger, and the ledger is the run's memory.
The escalation ladder
Each tier signals a human with the step's diagnosis and the inputs already consumed. A response is consumed exactly once, by the first escalation that reads it; an unconsumed response is stripped at hard-cap termination and cannot leak into a later, unrelated gate. The ladder's terminal state is the park.
The circuit breaker
Regeneration exists for transient failure. A deterministic failure — identical signature, identical inputs, no state change between attempts — is not transient, and regenerating against it produces a storm. The breaker enforces the distinction: consecutive generations with the same signature halt regeneration immediately, and the workflow parks with the signature and diagnosis attached. Full mechanics in sub-SOP orchestration.
Resume-from
| Mechanism | Semantics |
|---|---|
| Checkpoint | Completed prefix, artifact references, budget consumption |
| Re-entry | The failed step re-enters the pipeline at hydration — the completed prefix does not re-execute |
| Seeding | Terminal artifacts from the prefix hydrate into downstream context as before |
Park resolution classes
| Class | Semantics | Effect |
|---|---|---|
| Data provision | A human supplies the missing input | Fresh planning iteration with the data in scope |
| Instruction | A human corrects the step's directive | Fresh planning iteration under the correction |
| Gap acceptance | The input gap is confirmed real | Zero-case exemption honored; step completes on the acknowledged gap |
| Reject | The step's goal is wrong | Terminal — compensation path |
A generic retry response against a data-level gap re-enters planning with unchanged context and fails identically — the input still contains nothing to select. The engine surfaces this: the escalation payload's diagnosis states which condition holds, and the park distinguishes data-level gaps from transient conditions. The distinction is not advisory; the correct resolution class follows from the diagnosis.
One-shot consumption
A HITL response is consumed exactly once. On consumption, the step's execution state is cleared and planning rerun with the response in context — a fresh iteration, never a blind re-run of the failed attempt. The semantics are specified in HITL gates.
NOTE — The response classes and the budget-persistence rules above are specified from incident post-mortems on the running system: a park whose only offered resolution was retry against a zero-selection input, and escalation counters that reset per generation. Both behaviors are defects of the same class — budgets and resolutions that do not survive a context boundary — and both are closed in the current baseline.