report.py read grade.stderr with the default 1 MiB cap only to keep the
last line of a FAIL. A SPIFFS grader logged 1.1 MiB of proof data there,
the read raised ValueError, and the whole batch was rejected mid-run,
interrupting the attempts still in flight. Read it with the trace cap
instead, and fall back to the verdict reason if it is larger still.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured at f20f11c after a top-up: Qwen passed 3 of 4 valid attempts.
The task digest matches the measured one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured at 7f01d64 after a top-up: Qwen failed all 3 valid attempts;
3 more fetched the upstream fix and were voided. The task digest
matches the measured one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HARD: Mynewt FCB media results and SPIFFS append truncate space
(batch 28 47298b7, after a top-up); TinyUSB CH34x control completion,
TLV320 output gain encoding and ArduinoJSON decimal carry (batch 26
f20f11c); NetX Duo PPP buffered burst (batch 28 7f01d64).
MEDIUM: RT-Thread ringbuffer DMA peek (batch 28 7f01d64).
Task digests match the measured ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both were measured at a7fca39 after a top-up. LVGL encoded line results
passed 1 of 4 valid attempts; NuttX vi tail transitions failed all 3.
Task digests match the measured ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 28 commit a7fca39 gave ESP event loop accounting as MEDIUM: 2 of
3 attempts passed and the failure was a wrong fix in a complete session.
The task comes from the commit Qwen measured, its digest matches, and it
is still present at the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 29 commit 8731df1 gave NuttX ARP negative cache retry and Zephyr
pipe poll readiness as HARD: all 3 attempts of each ended in a complete
session with a wrong fix. Each task comes from the commit Qwen measured,
its digest matches, and it is still present at the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 28 commit 3847b17 gave ESP ringbuf reserve/publish (1 of 3 passed)
and RT-Thread dataqueue wrap reset (0 of 3) as HARD, and RT-Thread
condition wait transitions and NuttX NSH quoted script flow (2 of 3) as
MEDIUM. Each task comes from the commit Qwen measured, its digest
matches, and it is still present at the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a top-up, batch 28 commit b7fd210 gave Mynewt FCB2 reverse
alignment (2 of 3 valid attempts passed) and SPIFFS append error
progress (3 of 4) as MEDIUM. Each task comes from the commit Qwen
measured, its digest matches, and it is still present at the batch
branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 28 commit b7fd210 gave ESP NVS iterator versions and RT-Thread
workqueue timer state as HARD: all 3 attempts of each ran out of model
context, which counts as a failure. Each task comes from the commit Qwen
measured, its digest matches, and it is still present at the batch
branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 29 commit 97ead2b gave Zephyr ZMS restart records, NuttX TCP retry
closure and Zephyr eventfd lock progress (1 of 3 passed each) and Zephyr
telnet close handoff (0 of 3) as HARD. Each task comes from the commit
Qwen measured, its digest matches, and it is still present at the batch
branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 28 commit ca63702 gave RT-Thread mutex priority ceilings (1 of 3
passed), LVGL repeat cadence and RT-Thread soft-timer wakeup (0 of 3) as
HARD. Each task comes from the commit Qwen measured, its digest matches,
and it is still present at the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With top-up attempts from batch 26 commit 5726468 added, MPack zero-length
writes passed 2 of 3 valid attempts and tinycbor JSON value fidelity 3 of
4. Each task's digest matches the measurement and it is still present at
the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With top-up attempts from batch 26 commit b57e094 added, littlefs path
resolution failed 3 of 3 valid attempts, CANopen socket backpressure 4 of
4 and MQTT receive progress 3 of 4, mostly by exhausting the context.
Each task's digest matches the measurement and it is still present at
the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With its top-up attempts from batch 28 commit de6f2ac added, the task
passed 3 of 4 valid attempts. Its digest matches the measurement and it
is still present at the batch branch head. The libuv spawn task stays
unmeasured: 4 of its 10 attempts fetched the upstream fix and were voided.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 26 commits 5726468 and b57e094 add NuttX time zone sources, ELF
DWARF indirect attributes, Contiki MQTT segments and FreeRTOS TCP overlap
(HARD), plus canmatrix DBC, Contiki NAT64 and UDPard frames (MEDIUM).
Batch 27's last commit cbf719b adds token macros, screen terminal and
homogeneous raster (HARD), plus modal motion, XMODEM and directed
hysteresis (MEDIUM).
These runs finished hours ago but the watcher missed them: it counted
only from each restart. Each task comes from the commit Qwen measured,
its digest matches, and it is still present at the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With top-up attempts from batch 28 commit b804230 added, ESP NVS blob
version cleanup failed all 3 valid attempts (each ran out of context) and
LVGL bidi numeric runs passed 1 of 3. Each task's digest matches the
measurement and it is still present at the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With its top-up attempts from batch 29 commit 97ce4dd added, the task
passed 1 of 3 valid attempts. Its digest matches the measurement and it
is still present at the batch branch head.
Progress also records that in batch 28's latest top-up Qwen fetched the
upstream fix in half of its attempts; the contamination check voided all
of them, which halves the measurement yield on route D tasks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 28 commit de6f2ac gave ESP FAT image short names and libuv timer
turns as MEDIUM; the ESP HTTP response task is being topped up. Batch 28
has 3 MEDIUM and no HARD in 20 measured so far, below batches 26 and 29.
Each task comes from the commit Qwen measured, its digest matches, and it
is still present at the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 29 commit 97ce4dd gave Zephyr work-queue deadline handoff as HARD
(Qwen finished all three sessions with a wrong fix) and NuttX clock slew
domains as MEDIUM. The Zephyr CoAP observe task is being topped up.
Each task comes from the commit Qwen measured, its digest matches, and it
is still present at the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 29 (Zephyr and NuttX fixes) commit 2bb22bb gave NuttX mounted-path
transactions and Zephyr NVS recovery compaction as HARD and NuttX ELF
module bindings as MEDIUM. Batch 28 commit b804230 gave Mynewt JSON
arrays as MEDIUM; four of its tasks are being topped up.
Each task comes from the commit Qwen measured, its digest matches, and it
is still present at the batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With top-up attempts from batch 26 commit 7f99538 added, lwIP UDP route
and validation passed 1 of 3 valid attempts and NuttX floating format
layout 0 of 3; Qwen ran out of context in most failures. Each comes from
the measured commit, its digest matches, and it is still present at the
batch branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With their top-up attempts added, soft FMA (batch 27) passed 2 of 4
valid attempts and antelope index-load rollback (batch 26) 2 of 3. Each
comes from the commit Qwen measured, its digest matches, and it is still
present at the batch branch head.
One FlashDB attempt reached GitHub for the upstream fix; the
contamination check excluded it as designed, so it does not count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 26 commit 7f99538 measured 1 HARD (FreeRTOS loopback and mDNS
state, where Qwen ran out of context in all three attempts), 3 MEDIUM
(canmatrix JSON round trip, Zephyr byte-value formatting, Contiki event
timer sleep) and 4 ROUTINE; two tasks still need more valid attempts.
From batch 27 the lazy integer expression task becomes MEDIUM once its
top-up attempts are added.
Each task comes from the commit Qwen measured, its digest matches, and it
is still present at the batch branch head. Attempts made while /home was
below the 20 GiB sampling floor are recorded as disk_low and left out.
Batches 26, 28 and 29 are stopped: after a login refresh on 11-03 the
Codex account no longer offers the sol models they use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Route D (real fixes in embedded open-source projects) keeps producing the
hard tasks: of 64 measured so far, 15 are HARD and 4 MEDIUM, mostly because
Qwen exhausts its context in the larger real workspaces. This adds
CANopen SRDO and ASCII timing, FreeRTOS DHCP, lwIP MQTT fragments, printf
extremal exponents, Zephyr packet buffers and heatshrink (HARD), plus
lwjson, RFM9x, SFUD and nanopb proto3 (MEDIUM).
From batch 27 (spec-dense implementation tasks) come one HARD and one
MEDIUM; that batch is winding down after 30 tasks gave 1 HARD and 3 MEDIUM.
Each task comes from the commit Qwen measured, its digest matches, and it
is still present at the batch branch head. Agent timeouts stay excluded:
a pyserial attempt timed out on Qwen's own blocking read yet graded PASS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user wants 2000 tasks, most of them ones Qwen cannot solve. New batch
tasks are now measured on their branch first (3 Qwen trials each), and only
HARD or MEDIUM tasks reach main; ROUTINE ones stay on the batch branches.
A session cut by context_limit or output_limit counts as a failure for
difficulty (user decision 2026-11-03); the trajectory stays incomplete.
- Batch 26 (route D, real upstream fixes in embedded open-source projects):
7 HARD of 20 (coreJSON, littlefs, CANopen, coreMQTT, LwM2M, Zephyr ring,
lwIP SNTP). Most failures are Qwen exhausting its context in the larger
workspaces.
- Batch 25 (HWE mutation repairs): 1 HARD and 1 MEDIUM of 56.
- Batch 27 (spec-dense implementation tasks): 2 MEDIUM of the first 10.
Each task comes from the exact commit Qwen measured, and its digest matches
the measured one. Failure reasons are not yet reviewed, so difficulty is
the automatic label.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 24 (20261102-C-bos-seed-24) screened all 3078 Bos seed queries.
Only 23 themes were in scope (pure user-space reliability, at most one
task per theme); the rest were single operations or explanations,
installs and networking, system administration, ROS2, board-only or GUI
requests. Two finished drafts were withdrawn as too close to existing
tasks. Seeds supply direction only: all sources, contracts and tests are
new, and no task shares more than 14 characters with its seed text.
Each task passed the local gate and the trace6 Docker gate (5 runs per
sample class). Claude rebuilt the catalog from the calibration receipts:
23/23 USABLE. 16 of 23 are C/C++.
Only the task directories are taken; the batch worktree's lowered disk
floor in rollout/core.py stays out of main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 21 (20261027-A-bos-redo-21) rebuilt 40 of the 53 rejected Bos-qiao
environments: hidden assertions that ran inside the candidate's process
now run in a trusted parent that reads observations from a low-UID
worker; gold bytes are unchanged. 9 were excluded for scope (ROS2,
oversize, security core), 3 are blocked because the source requires an
exec bit the platform does not freeze, 1 lacks requests/bs4 in trace6.
Batch 18 (20261026-C-hwe-codegen-18) adds its last 34 of 134 tasks; the
first 100 landed in 2f32624d.
Claude reviewed 74/74 USABLE against current digests, canonical
families, no ID or source overlap with main, plain public docs. Only the
new task directories are taken; the stale pre-batch-20 copies of
concur-release-capsule and resource-temp-link on the batch 21 branch and
both branches' progress notes stay as main has them. Difficulty is
unmeasured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers the six parts, the disk and key-cap pauses, cold migrations, the FAT gateway
probe and the 16 tasks left without a complete trajectory, with reasons.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
36's sshd reset new connections during key exchange for a few minutes; three jobs in
a row gave up after one try per host and paused the batch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every attempt kept result.tar, its unpacked tree and the copies in the attempt, so a
third of the hot disk went to a tree nothing reads again. result.tar and its manifest
still record exactly what the job returned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A downloader task's agent served test files from 127.0.0.1 inside its own job; the rule
the user set is about the outside network. Look-alike hosts such as 127.0.0.1.evil.io
and localhost@evil.io still count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rollout.salty_run keeps the gateway, grading and evidence on 36 and moves only the
agent step into a Salty job. The gateway listens on 127.0.0.1 with a per-attempt
bearer token; the job reaches it over port 22 with a restricted one-time key that is
removed after each attempt. rollout.batch gains --backend salty and repeatable
--credential-file, and writes pass/fail SFT exports through rollout.sft at the end.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rollout.dashboard serves catalog counts, attempts and complete trajectories, and the public
prompt and template of each task. catalog.controllers()/complete() are split out so the
dashboard can reuse receipts it has already read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
100 tasks from the HWE arm-linux code-gen pool (outside the lk/man/pth
range given to batch 22), each with at least one independent
contract-edge defect and plain public docs. Claude reviewed 100/100
USABLE against current digests; no ID or source overlap with main or
batch 22. Difficulty is unmeasured.
progress.md records the wrap-up: all Codex sessions and Qwen schedulers
stopped for the user's rollout, Qwen results so far (57 HARD of 373),
and what is left on each batch branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the 101 lk/man/pth code-gen sources: 63 tasks (C11 36, C++17 27,
11 canonical families), 38 skipped for scope or duplication. Each task
has at least one independent contract-edge defect, boundary types
rotate, and public docs state trap clauses as plain contract prose.
Claude reviewed 63/63 USABLE against current digests. Difficulty is
unmeasured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only author-side spec.md files change, so no task digest moves. The specs of
concur-release-capsule and resource-temp-link follow batch 20's family and
version. The 53 rejected Bos environments wait: JuiceFS returns EIO for the
whole source tree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
210 batch-15 tasks used 119 ad-hoc family names. Each now carries one of the
canonical root-cause families (11 of the 19 are used), with version + 1 and a
spec.md note, and was recalibrated through both gates. Claude checked that only
family, version and spec.md changed, that every family is canonical, and the
new trace6 Docker receipts against current digests (210/210 USABLE).
Squashed from the 36-local branch batch20-family.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One user-space repair task per eligible HWE debug-diag / compile-deploy base
ID: 320 admitted (C 218, C++ 97, build scripts 5), 1 skipped by the content
filter. Each ships a public spec with a check-to-basis table, at least two
defects with single-fix fixtures, a symptom-masking fixture and an alternative
correct implementation. Claude re-checked all 320: structure, the HWE quality
items and trace6 Docker calibration receipts against current digests
(320/320 USABLE). Difficulty is unmeasured. Known follow-up: 212 tasks use 120
ad-hoc family names; the next batch maps them to the canonical list.
The batch's 1,256-line progress log stays in its factory report and the
36-local branch batch15-hwe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the importer accepted fixture scripts that rewrite a target (dd69720),
54 of the 55 environments batch 16 could not import loaded and were imported
with graders narrowed to the declared files and READMEs stating the delivery
boundary. Claude re-checked all 54 packages and their trace6 Docker
calibration receipts against current digests (54/54 USABLE). Difficulty is
unmeasured. Squashed from the 36-local branch batch19-rglob2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 191 Bos environments whose graders rglob the workspace were imported under
the new rule: graders read only the declared candidate files and the READMEs
state that only those files are delivered. After the import-scan fix, 135 were
approved; 55 are still refused because a fixture script writes the same file
twice, and 1 has a target missing from its template. Claude re-checked all 135
packages, the delivery-boundary wording and their trace6 Docker calibration
receipts against current digests (135/135 USABLE). Difficulty is unmeasured.
Squashed from the 36-local branch batch16-rglob.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 72 tasks that had no difficulty conclusion were measured with
qwen3.8-27b-nvfp4 on trace6: 68 ROUTINE, 3 MEDIUM, 1 held (S500_n3_100,
insufficient basis), none HARD. 172 raw attempts, 148 valid. Root review also
flagged 14 tasks whose README and grader disagree or whose grader misses a
README requirement; batch 17 fixes and re-measures them. No task changes;
records only. Squashed from the 36-local branch batch13-screen.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
192 unused Bos n1/a environments inventoried in the multi-agent mode: 159 are
rglob-type and 6 ROS2-type (not importable under the current contract), 13 were
rejected by the importer or missing targets, 14 imported. Claude re-checked all
14 packages and their trace6 Docker calibration receipts against current
digests (14/14 USABLE). Difficulty is unmeasured. Squashed from the 36-local
branch batch14-n1a.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
119 Bos n3 environments inventoried in the multi-agent mode: 69 imported with
graders rewritten to run candidates in subprocesses, 18 rejected because the
source gold fails its own contract, 32 rglob-type skipped. S500_n2_094 and
S500_n2_115 READMEs now state the oracle's integer, nesting and exception rules,
which lifts their holds. Claude re-checked all 71 packages: structure, isolation
and trace6 Docker calibration receipts against current digests (71/71 USABLE).
Difficulty is unmeasured. Squashed from the 36-local branch batch11-n3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
120 tasks screened with qwen3.8-27b-nvfp4 on trace6 in the multi-agent mode:
29 HARD, 13 MEDIUM, 75 ROUTINE, 3 held for ambiguous prompts (053/094/115).
364 raw attempts, 285 valid measurements. No task changes; records only.
Squashed from the 36-local branch batch10-screen; see docs/progress.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clarify four ambiguous prompts (S500_n2_053, S500_n2_096, S500_n2_103,
mem-mapping-lease), drop the unused tests.py from eleven rewritten n3 tasks,
add cheat_read_grader to two concurrency tasks and normalise their source
fields. All 18 recalibrated on trace6 (820/820 as expected).
Squashed from the 36-local branch batch12-cleanup; see docs/progress.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batches now commit on their own branch and land on main with git merge
--squash, so main gains one commit per batch. Record the four-step push.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>