91 Commits

Author SHA1 Message Date
bench
91ed6add7c docs(memory): opencode rerun results on a separate staging run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 08:33:48 +08:00
bench
161d0a27e0 docs(memory): outcome of the 7 resumed 04 rollouts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 02:27:56 +08:00
bench
69954d7c0d docs(memory): breakdown of export rollouts missing a verdict
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 21:52:09 +08:00
bench
b98113c8af docs(memory): Salty CLI 0.2.6 fixes job list parse failure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 20:19:04 +08:00
bench
2d164cd2a4 docs(memory): opencode on staging, fixes and pitfalls
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:41:24 +08:00
bench
064e1a025f docs(memory): 04 paused rollouts resumed or cancelled after Qwen redeploy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 14:26:18 +08:00
bench
216d5d0a05 docs(memory): reboot fixes shipped and 7 quarantined rollouts cancelled
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 14:13:58 +08:00
bench
e3d14ad820 docs(memory): 10-21 Mac mini reboot and recovery
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:22:20 +08:00
bench
60c85d87e5 docs(memory): Qwen service context length, concurrency and stall pattern
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 01:50:31 +08:00
bench
93a09d2a83 docs(memory): dev OOM memory doubling shipped to production
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:43:10 +08:00
bench
16b99ee1ed docs(memory): 06-hbm artifact return fix and two rollouts re-verified
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:47:19 +08:00
bench
e50bf15278 docs(memory): offload and staging reaper restarted on job-first cleanup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:26:59 +08:00
bench
8064feb2fc docs(memory): Salty job-list outage and v23 SFT chain review
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:52:10 +08:00
bench
eab2dbcc37 docs(memory): board lease fix hot-swapped to production
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 16:28:08 +08:00
bench
aad8d5b597 docs(memory): production switched to the new code on 10-20
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 15:23:13 +08:00
bench
3fa2e236f6 docs(memory): branches renamed to salty-prod / salty-test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 15:14:04 +08:00
bench
a6e2ce0ccb docs(memory): production hotfix of the one-shot Salty cleanup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 14:47:30 +08:00
bench
6badb5d44a docs(memory): Salty 0.2.4 on staging, background delete retries
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 14:30:32 +08:00
bench
2c5a08015f docs(memory): overnight staging supervision results
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 05:25:04 +08:00
bench
98d85800ea docs(memory): Salty storage rm 405 incident and the one-shot cleanup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:01:35 +08:00
bench
12bbc542dd docs(memory): staging collector, the three fixes verified there
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:00:15 +08:00
bench
0da66edde0 Resume rollouts after the collector dies (crash, kill, reboot, planned handover)
Processes are disposable; rollout state is resumable. When the collector dies,
the next one reconciles each running rollout from durable state instead of
quarantining or losing it:

- execution: a process of an earlier boot on this host is DEAD; a top-level
  reaper whose owner (the collector) died still proves its subtree empty
  (scopes record the owner that started them); reconcile() pauses a rollout
  with a proven-clean crash checkpoint as CONTROLLER_RESTARTED and marks its
  running tool calls INTERRUPTED; resume() across a reboot rebases monotonic
  clocks by wall time and credits the downtime as a pause
- stageworker.settle_interrupted: an admitted step whose executor is gone ends
  as STAGE_EXECUTOR_EXIT (free compile) and its handle is told directly; a
  board step without a clean board receipt still quarantines; nothing replays
- collection_runtime.crash_checkpoint: capture the real system prompt, freeze
  the session at the last saved message (mid-work phases only); configure_pi
  skips a generation ID the ledger already completed
- pi_session.js: checkpoint after every saved message (and compaction); on a
  recovered resume drop an unfinished trailing reply and answer open tool
  calls with an honest "interrupted by a platform restart" result
- scheduler: a paused rollout's local slot held by a dead holder is released
  once every process is proven gone
- pipeline: SIGUSR1 = planned handover (stop claiming, let board steps
  finish, shorten in-flight request holds, exit); SIGTERM still drains
- export: bench_recovery closes the cut segment; the dropped unfinished reply
  is excluded (UNFINISHED_BEFORE_RESTART)
- tests: test_controller_restart (real Pi on Linux: kill -9 mid-command,
  kill -9 mid-reply, SIGUSR1 handover, power loss + reboot), test_settle_
  interrupted; three tests of the old quarantine-everything rule now assert
  the new outcome while keeping their guarantees (no replay, unrelated PID
  untouched, board lease kept)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 09:29:26 +08:00
bench
f5cfd414eb docs(memory): v23 上线完成(10-18 15:43)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 15:50:08 +08:00
bench
18379401e4 v23: hand context compaction back to Pi, cut process rules from the prompt
- broker: a context-window 400 fails its generation (fences nothing) and goes
  back to Pi as context_length_exceeded, so Pi compacts and resends
- runner: new generation ID after compaction, wait for Pi's resumed run,
  record kept_messages; drop agent/context-mask.js and context_mask_chars
- models.json: contextWindow large so Pi compacts only on a real overflow,
  never on an estimate right before a pause (resume would not match)
- export: compactions[] gives what the model actually saw after each one
- prompt: keep facts and acceptance, drop how-to rules and coaching
  (SYSTEM.md 89->64 lines, e2e.md 27->20); stage status == stage wait

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:04:58 +08:00
bench
e3697f0ddb docs(memory): 10-17 删减清单与索引补齐
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:04:31 +08:00
bench
aca965ac9e docs/memory: 10-17 停 06(账号 B 没额度)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 11:30:56 +08:00
bench
2beb36a085 docs/memory: 能否当强化学习环境的评估与收尾清单
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 03:21:01 +08:00
bench
8703cd1b89 docs/memory: 10-17 底层复查(流程漏斗、平台复测缺速度、Qwen 续跑必死的根因)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 02:24:18 +08:00
bench
250f85eefb docs/memory: 10-16 断电停机与重启;AGENTS.md 索引补齐到与 MEMORY.md 一致
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:25:20 +08:00
bench
c1ecae2130 docs/memory: 稳定性体检复核更正(监控已有但有盲区、白跑时长实为约 1660 小时等)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 18:56:08 +08:00
bench
cb0ddb1625 docs(memory): 10-16 稳定性体检
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 18:06:41 +08:00
bench
a7f28230a8 docs(memory): v18+ 0 通过的原因与值得注意的行为
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:55:30 +08:00
bench
58d522a825 docs(memory): v16–17 下的 PASS/SFT 数
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:45:12 +08:00
bench
f481c3bc42 docs(memory): 10-16 按 collector 的 PASS/SFT 与自主性统计
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:42:11 +08:00
bench
5621eb1c95 docs(memory): 10-16 05 换 GLM key 与新账号组
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:23:40 +08:00
bench
e2741cb851 docs(memory): 10-16 17:00 复查(v18–21 行为与结果、S600 账号 429)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 16:55:34 +08:00
bench
ab2d95f4ce docs(memory): 10-16 下午 v22 热部署与重启
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 16:23:59 +08:00
bench
f7b9f3238c docs(memory): v20/v21、原图来源、dev 排队、stage wait
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 06:40:15 +08:00
bench
8122c30b88 docs(memory): v18 夜间情况、v19 板上自测与板卡隔离实测
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 01:19:35 +08:00
bench
b2cde2b3c3 memory: v17 autonomy check 2026-10-15
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 19:38:09 +08:00
bench
71e888cfc1 docs(memory): 热部署后核实、Salty 清理结果、dev 缓存泄露修复;同步 SFT 抽样细读记录
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 19:24:51 +08:00
bench
a526af7d78 docs(memory): 10-15 下午重启前清查出的底层 bug、v17/判分 v0.8、重启过程与核实
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 14:39:29 +08:00
bench
5a8e69d191 docs(memory): 派发前误判修复上线、待办总表、热部署前必查
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:21:10 +08:00
bench
6f3377b28a docs(memory): 评测文件被冻结不能热改(11:08 改坏三条)、暂停名额修复、第一条真自带图 PASS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:48:20 +08:00
bench
32a8e21f34 docs(memory): 板上评测按字节分批已上线(f56ac2f)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:08:34 +08:00
bench
ea8362e545 docs(memory): 自带图结果:4 条真改图未出结论,superpoint 的自带图是复制的
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 10:44:24 +08:00
bench
4495899f03 docs(memory): 板上清扫上线核实:3 块板删 31 个旧目录,板上无失败
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 10:40:45 +08:00
bench
427d5fe7cc docs(memory): v16 下 Qwen 成功率变化的原因
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 09:43:54 +08:00
bench
d48152f417 docs(memory): 板上会话目录修复已上线(a263374),以及不能删活目录的原因
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 09:42:27 +08:00
bench
4d00ff71f7 docs(memory): 10-14 HBM 步骤瓶颈、10-15 v16 轨迹质量对比
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 09:35:28 +08:00