Processes are disposable; rollout state is resumable. When the collector dies,
the next one reconciles each running rollout from durable state instead of
quarantining or losing it:
- execution: a process of an earlier boot on this host is DEAD; a top-level
reaper whose owner (the collector) died still proves its subtree empty
(scopes record the owner that started them); reconcile() pauses a rollout
with a proven-clean crash checkpoint as CONTROLLER_RESTARTED and marks its
running tool calls INTERRUPTED; resume() across a reboot rebases monotonic
clocks by wall time and credits the downtime as a pause
- stageworker.settle_interrupted: an admitted step whose executor is gone ends
as STAGE_EXECUTOR_EXIT (free compile) and its handle is told directly; a
board step without a clean board receipt still quarantines; nothing replays
- collection_runtime.crash_checkpoint: capture the real system prompt, freeze
the session at the last saved message (mid-work phases only); configure_pi
skips a generation ID the ledger already completed
- pi_session.js: checkpoint after every saved message (and compaction); on a
recovered resume drop an unfinished trailing reply and answer open tool
calls with an honest "interrupted by a platform restart" result
- scheduler: a paused rollout's local slot held by a dead holder is released
once every process is proven gone
- pipeline: SIGUSR1 = planned handover (stop claiming, let board steps
finish, shorten in-flight request holds, exit); SIGTERM still drains
- export: bench_recovery closes the cut segment; the dropped unfinished reply
is excluded (UNFINISHED_BEFORE_RESTART)
- tests: test_controller_restart (real Pi on Linux: kill -9 mid-command,
kill -9 mid-reply, SIGUSR1 handover, power loss + reboot), test_settle_
interrupted; three tests of the old quarantine-everything rule now assert
the new outcome while keeping their guarantees (no replay, unrelated PID
untouched, board lease kept)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- quota: once an UNKNOWN attempt is a full token window past its local
deadline, abandon it via the operator path (evidence kept, ended_at NULL,
generation fenced); the group then gets one probe.
- quota: generations paused at admission by that group pause may resume
when none of their own attempts is unended.
- pipeline: isolate an identifiable unreconciled rollout and keep
collecting; unidentifiable reports still stop. Drive broker reconcile
every loop so a fully paused group can recover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reuse execution, stage and review evidence to recover database registration and
SFT publication without replaying completed work. Keep finished futures during
SQLite contention, and hand durable results to startup recovery when draining.
Release proven local leases independently of unresolved board isolation.
Keep UNKNOWN review retries explicit and archived, publish completed receipts
after transient export failures, and prevent a malformed sample from aborting
the entire dataset. Scope routine reconciliation to newly finished attempts.
Add failure-injection coverage and the production acceptance plan.