Processes are disposable; rollout state is resumable. When the collector dies,
the next one reconciles each running rollout from durable state instead of
quarantining or losing it:
- execution: a process of an earlier boot on this host is DEAD; a top-level
reaper whose owner (the collector) died still proves its subtree empty
(scopes record the owner that started them); reconcile() pauses a rollout
with a proven-clean crash checkpoint as CONTROLLER_RESTARTED and marks its
running tool calls INTERRUPTED; resume() across a reboot rebases monotonic
clocks by wall time and credits the downtime as a pause
- stageworker.settle_interrupted: an admitted step whose executor is gone ends
as STAGE_EXECUTOR_EXIT (free compile) and its handle is told directly; a
board step without a clean board receipt still quarantines; nothing replays
- collection_runtime.crash_checkpoint: capture the real system prompt, freeze
the session at the last saved message (mid-work phases only); configure_pi
skips a generation ID the ledger already completed
- pi_session.js: checkpoint after every saved message (and compaction); on a
recovered resume drop an unfinished trailing reply and answer open tool
calls with an honest "interrupted by a platform restart" result
- scheduler: a paused rollout's local slot held by a dead holder is released
once every process is proven gone
- pipeline: SIGUSR1 = planned handover (stop claiming, let board steps
finish, shorten in-flight request holds, exit); SIGTERM still drains
- export: bench_recovery closes the cut segment; the dropped unfinished reply
is excluded (UNFINISHED_BEFORE_RESTART)
- tests: test_controller_restart (real Pi on Linux: kill -9 mid-command,
kill -9 mid-reply, SIGUSR1 handover, power loss + reboot), test_settle_
interrupted; three tests of the old quarantine-everything rule now assert
the new outcome while keeping their guarantees (no replay, unrelated PID
untouched, board lease kept)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three r20260929-06 rollouts sat in PAUSED_INFRA although their stage had
finished (status DONE/FAIL, no process left): the completion replay in
stageworker._validate_recovered_receipt raised ValueError, so the operation
stayed DISPATCHED and the execution was quarantined (one-way).
- Nested scopes: a deadline kill of the outer stage leaves the inner reaper
with owner_lost=True although both levels reaped via waitpid(ECHILD).
owned_processes() already accepted that when the enclosing verified receipt
names the exact supervisor; receipt validation demanded VERIFIED. Both now
share execution.scope_cleaned().
- Board stages killed by QUEUE_DEADLINE_EXCEEDED never held a board slot
(phase_running fires only under the board lock) and so have no board
receipt; that missing receipt no longer counts as unconfirmed cleanup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cancellation can end a compiler parent before its nested reaper. The nested
receipt then reports owner loss even when both reapers reached waitpid(ECHILD).
Recognize the verified enclosing receipt only when execution, epoch, immediate
parent, and full supervisor birth identity match and the nested children were
fully reaped. Preserve the original receipts and fail closed without that proof.
Add eight regression tests, including identity mismatch, incomplete cleanup,
missing parent proof, and live-process preservation. The positive regression
fails on the previous code and passes on the patch; recorded incident receipts
also distinguish the two implementations without changing the cancelled outcome.