52 Commits

Author SHA1 Message Date
44f643d290 fix: stop false OPERATION_STATE_UNKNOWN on finished stages
Three r20260929-06 rollouts sat in PAUSED_INFRA although their stage had
finished (status DONE/FAIL, no process left): the completion replay in
stageworker._validate_recovered_receipt raised ValueError, so the operation
stayed DISPATCHED and the execution was quarantined (one-way).

- Nested scopes: a deadline kill of the outer stage leaves the inner reaper
  with owner_lost=True although both levels reaped via waitpid(ECHILD).
  owned_processes() already accepted that when the enclosing verified receipt
  names the exact supervisor; receipt validation demanded VERIFIED. Both now
  share execution.scope_cleaned().
- Board stages killed by QUEUE_DEADLINE_EXCEEDED never held a board slot
  (phase_running fires only under the board lock) and so have no board
  receipt; that missing receipt no longer counts as unconfirmed cleanup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 09:50:23 +08:00
f61d2efebb fix: replay board ssh calls dropped before authentication
Board ssh ports are public and constantly hold 15-35 unauthenticated
scanner connections; whenever that exceeds sshd MaxStartups, our handshake
is closed with kex_exchange_identification. Only upload steps retried, so
a single drop failed the bench call itself (exit 255) or lost the cleanup
receipt fetch, which then quarantined the board.

run() and guarded_run() now replay a call up to twice when ssh reports
kex_exchange_identification: the connection closed before auth, so the
board never saw the command and replay cannot duplicate work. Any other
failure is returned unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 09:18:48 +08:00
f315496d18 fix: clear a board-side quarantine once its operation has no survivors
board_guard writes .board-inference.quarantine.json before spawning the
inference child. If the guard is killed (ssh drop, SIGKILL) the file stays
and every later 07-board/08-eval on that board exits 78 until a human
deletes it; rdk-sandbox-40009 was blocked this way for 1.5 days.

The guard already holds the board lock every admitted child inherits. It
now also checks /proc for any process carrying the old operation token;
with none left, the record is stale: it is removed and logged as
stale_quarantine_cleared, then the normal occupancy check still applies.
Survivors keep the board quarantined. Unverified-cleanup receipts now
store the token so the same rule covers them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:44:20 +08:00
80a8e246c0 fix: release board slots whose holder died without a cleanup receipt
When board ssh drops during cleanup, no receipt is written and the board
lease stayed quarantined forever; all 8 boards ended up locked and every
rollout timed out in the board queue (QUEUE_DEADLINE_EXCEEDED).

reconcile_leases now probes the board when the holder is dead, the
execution is terminal and no local process remains. If the board runs
nothing but sshd/ps the slot is released and board-release-<slot>.json
records the evidence; busy or unreachable boards stay quarantined.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:07:22 +08:00
2f54d407cb fix: recover from unknown request outcomes without stopping the collector
- quota: once an UNKNOWN attempt is a full token window past its local
  deadline, abandon it via the operator path (evidence kept, ended_at NULL,
  generation fenced); the group then gets one probe.
- quota: generations paused at admission by that group pause may resume
  when none of their own attempts is unended.
- pipeline: isolate an identifiable unreconciled rollout and keep
  collecting; unidentifiable reports still stop. Drive broker reconcile
  every loop so a fully paused group can recover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:18:11 +08:00
2eb29caa83 fix: let dead, quarantined failed executions stop blocking unrelated work
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:17:40 +08:00
10663a800f fix: preserve completed calibration when later compilation is cancelled 2026-09-29 19:04:50 +08:00
13f1e99aee fix: distinguish empty documentation searches from tool failures 2026-09-29 18:29:59 +08:00
3a30aac297 fix: persist board dispatch evidence before remote execution 2026-09-29 16:33:16 +08:00
498187569c fix: close rollout finalization and review publication gaps
Reuse execution, stage and review evidence to recover database registration and
SFT publication without replaying completed work. Keep finished futures during
SQLite contention, and hand durable results to startup recovery when draining.
Release proven local leases independently of unresolved board isolation.

Keep UNKNOWN review retries explicit and archived, publish completed receipts
after transient export failures, and prevent a malformed sample from aborting
the entire dataset. Scope routine reconciliation to newly finished attempts.
Add failure-injection coverage and the production acceptance plan.
2026-09-29 16:09:01 +08:00
e1e73816af fix: recover stage registration and reclaim completed local leases 2026-09-29 12:55:35 +08:00
2da29f8050 fix: preserve deployed recovery and board cleanup behavior
Record the deployed continuation, process reaping, recovery, and cleanup fixes with the current board registry. Add focused regression coverage and document the shared production branch and existing startup configuration.
2026-09-29 01:27:19 +08:00
8bb1694499 fix: retry transient tool event reads once 2026-09-24 02:00:12 +08:00
b398495d54 fix: allow 32768 tokens for content review 2026-09-23 23:29:04 +08:00
6b62baf1fd fix: scope execution readiness to the requested run 2026-09-23 22:34:41 +08:00
Codex
9af7952ab5 fix: consume tool boundaries under lock and diagnose transport errors 2026-09-23 22:00:17 +08:00
Codex
e1d5a96252 fix: match fixed top-down poses by source identity 2026-09-23 21:57:25 +08:00
Codex
b81bc80875 fix: reuse managed waits and drain accepted stage work 2026-09-23 21:57:25 +08:00
Codex
d35c82a8bf fix: integrate managed content review into dataset refresh 2026-09-23 21:30:40 +08:00
Codex
1c21193221 fix: parse proc bytes and honor quota holds before claim 2026-09-23 01:28:21 +08:00
7440ba8cb8 fix: verify nested reaper cleanup through enclosing receipts
Cancellation can end a compiler parent before its nested reaper. The nested
receipt then reports owner loss even when both reapers reached waitpid(ECHILD).
Recognize the verified enclosing receipt only when execution, epoch, immediate
parent, and full supervisor birth identity match and the nested children were
fully reaped. Preserve the original receipts and fail closed without that proof.

Add eight regression tests, including identity mismatch, incomplete cleanup,
missing parent proof, and live-process preservation. The positive regression
fails on the previous code and passes on the patch; recorded incident receipts
also distinguish the two implementations without changing the cancelled outcome.
2026-09-22 20:29:45 +08:00
f611ef29ab fix: keep managed API requests within the broker lifecycle
Pi requests could hit Node fetch's independent five-minute header deadline,
and queued requests could outlive their local client or terminal rollout.
Set header/body deadlines only for the loopback broker origin, cancel scopes
when clients disconnect or executions terminate, and retain uncertain upstream
reservations instead of retrying after cancellation. Resumable pauses remain open.

Validation: real Pi SDK accepted a response delayed 310 seconds (314.139 s test);
quota, Pi session, acceptance, cancellation, and lifecycle regression coverage.
2026-09-22 20:28:42 +08:00
Codex
068c56043f Merge managed natural-agent runtime into retained benchmark mainline 2026-09-22 15:35:28 +08:00
Codex
5884a6d7c6 Snapshot existing adapters and environment before extension merge 2026-09-22 14:56:41 +08:00
bench
d309109f2b pre-grind snapshot 2026-09-19: batches 1-7 adapters/probes/recipes + eval additive changes 2026-09-19 17:52:03 +08:00
bench
ee4c7fce1c reward v0.4: board_offline 不加分只免罚;board-backfill 支持 --rollout 用现成 HBM 补测
- export/reward.py: G3 unverified 且最后一次 07/08 为 board_offline → G2/G3 权重 0、G1 保持 1.0、免 TIMEOUT 罚分,
  返回 unverifiable=board_offline;RECIPE v0.4。实测 r20260916-01 regnet.y.800mf −0.97 → +0.03
- export/to_agentrl.py: 记录 reward_unverifiable
- harness/pipeline.py: board_backfill_rollout:用 rollouts/kk/artifacts 的 HBM 补跑 07/08(lane=backfill)后 verify_rollout
- harness/cli.py: bench board-backfill --rollout JOB_ID[:KK]
- 单测 scripts/test_reward_board_offline.py、scripts/test_rollout_backfill.py;CONTRACT §9 / README 同步
2026-09-16 15:58:24 +08:00
bench
668a7b1d16 Schedule board stages across independent S100P devices 2026-09-16 11:04:30 +08:00
bench
3e722f3e94 Collect trajectories across models before repeated sampling 2026-09-16 08:23:00 +08:00
bench
cd220cc81e fix: preserve deployment outcomes and export completed run data 2026-09-16 07:55:44 +08:00
bench
864c20a087 fix: regenerate configured repairs and use glm-5.3 for collection 2026-09-16 02:32:06 +08:00
bench
4a0e7d521f fix: run S100P board accuracy through the existing pipeline 2026-09-16 02:17:36 +08:00
bench
c717d01d97 sync 2026-09-15_12:46:30 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:46:30 +08:00
bench
cb68619267 sync 2026-09-15_12:42:44 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:42:44 +08:00
bench
ed9c7381c0 sync 2026-09-15_12:39:05 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:39:05 +08:00
bench
3dce78e49d sync 2026-09-15_12:38:10 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:38:10 +08:00
bench
5ad7d4b05c sync 2026-09-15_12:36:44 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:36:44 +08:00
bench
546ec40019 sync 2026-09-15_12:35:36 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:35:36 +08:00
bench
1fd42dff20 sync 2026-09-15_12:32:50 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:32:50 +08:00
bench
b55efc94ce sync 2026-09-15_12:30:11 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:30:12 +08:00
bench
04190b7b4c sync 2026-09-15_12:27:43 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:27:43 +08:00
bench
5bafe1d7f4 sync 2026-09-15_12:26:22 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:26:22 +08:00
bench
1838ceb36d sync 2026-09-15_12:24:58 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:24:58 +08:00
bench
0e61344b34 sync 2026-09-15_12:23:21 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:23:21 +08:00
bench
50edd8c2a1 sync 2026-09-15_12:21:38 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:21:38 +08:00
bench
ba1e1a7bd1 sync 2026-09-15_11:34:17 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 11:34:17 +08:00
bench
d191b68b88 sync 2026-09-15_11:26:25 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 11:26:25 +08:00
bench
c88303e69b sync 2026-09-15_09:50:25 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 09:50:25 +08:00
bench
9d1741b9fa sync 2026-09-15_09:42:56 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 09:42:56 +08:00
bench
0a2f71f18a sync 2026-09-15_09:37:55 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 09:37:55 +08:00
bench
998870db74 sync 2026-09-15_08:14:50 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 08:14:50 +08:00