11 Commits

Author SHA1 Message Date
amos
dd69720c15 Let Bos fixture scripts rewrite a target; the last write wins
The fixture grammar is straight-line heredocs with no branches, so a script that
writes the same target twice has a well-defined result: its last write. 55
rglob-type environments in batch 16 were refused only because their no-op
fixture writes a module twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 11:40:08 +08:00
amos
a729ebe27a Let rglob-type imports declare the extra modules their gold writes
Batch 16 found 55 environments whose gold splits code into new modules, which
the importer rejects as non-target writes. Declare those files as candidates
with empty template stubs and list them in the README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 02:30:21 +08:00
amos
d802952425 Allow importing rglob-type Bos tasks by narrowing the grader to declared files
The platform freezes only candidate_files, so a grader that rglobs the workspace
can wrongly fail a legitimate split. Instead of leaving 191 such environments
out, rewrite the grader to read the declared files and state the delivery
boundary in the README. Tasks with more than 16 targets stay out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 00:26:22 +08:00
Codex
30762e4d29 docs: allow evidence-based symptom task prompts 2026-10-17 01:39:03 +08:00
amos
05bd1e10c3 Re-author three blocked Bos graders so candidates only run in a subprocess
S500_n3_133 v3, S500_n3_102 v4 and S500_a2_426889b9a968 v3 move every
expected value and assertion into check.py; the candidate process only
receives inputs and prints raw observations. isolation-blocked.txt and the
in-process tests.py are removed and each task gains cheat_read_grader.
Formal trace5 calibration on 36 passed with these exact task digests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 01:32:04 +08:00
amos
de3755c14b fix: keep the Bos grader away from candidate processes
The Bos wrapper ran check.py and every candidate process as UID 65534, so a
candidate could read the grader copy through /proc/<checker>/cwd and pass
without implementing anything. check.py now runs as root on a 0700 grader
copy; its subprocesses drop to 65534 and only the work copy and mkdtemp
scratch directories are handed over. An audit hook ends the check as
UNVERIFIED if candidate code would run inside the checker.

36 tasks run hidden tests inside the candidate process (tests.py style and
S500_a2_426889b9a968); they carry isolation-blocked.txt and always grade
UNVERIFIED. The importer adds the marker for any extra root *.py. All 74 Bos
tasks get the new verify.py and a version bump.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:33:34 +08:00
amos
293c2c7a8e docs: task-making spec, Codex factory brief and storage layout
Rewrite task-authoring.md as one spec (four source routes, exclusions,
prompt/grader/fixture rules, four gates, difficulty pools), add the
brief that gives Codex the machines, storage layout, data map, commands
and red lines, and record the Bos-qiao raw-data audit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:22:36 +08:00
amos
94eaf64261 fix: private grade diagnostics and a public-contract admission check
- bos_verify still runs the original check.py unchanged, but failing
  child processes now copy their stderr tail into grade.stderr; report
  rows carry fail_detail (that line, else the grade reason).
- bos_import rejects envs whose hidden tests use candidate-module names
  the public prompt/template never mention; a repo test applies the same
  check to every admitted task.
- S500_n3_152 v2 publishes process(request): its v1 FAILs came from that
  hidden requirement, not from the models. All Bos tasks carry the new
  wrapper as version 2; 21 tasks recalibrated on trace5 (805 trials).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:27:29 +08:00
amos
1b73b3742b fix: keep host stdout complete and lift the GLM output cap
The first Bos pilot batch exposed two adapter faults on long responses:
- fs.writeSync(1, ...) ignored partial writes to a nonblocking stdout pipe,
  so host stdout lost the tail of 100 KB lines and could no longer match
  the native trace. writeAll now loops until every byte is written and
  waits briefly on EAGAIN.
- Pi SDK caps max_tokens at 32000 unless told otherwise; GLM-5.3 spent
  all of it reasoning and stopped with "length" before any edit. The
  adapter now passes the registered 131072 (accepted upstream).

Runtime trace3 is the trace2 digest plus this pi.mjs; 12/12 SDK checks
pass inside it and all 21 tasks were recalibrated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 22:10:12 +08:00
amos
acfa601a56 feat: multi-file task contract, model table and shared I/O helpers
- Tasks may declare up to 16 candidate files; the whole template becomes
  the Agent workspace while only candidate files are frozen and graded.
- Optional fixtures add more cheats or legal alternatives (five trials
  each); optional grader_files ship private tests next to verify.py.
- Tasks without an argv oracle export call_type=verifier ground truth.
- Upstream model routes live in one MODELS table.
- One bounded no-follow reader, one exclusive writer and one SIGTERM
  helper replace per-module copies; drop unused validate_task.

frame-decoder keeps its task digest, calibration receipts and exported
query. Verified on host 36: internal network probe, 20-trial calibration
and a real GLM rollout (PASS, evidence VERIFIED, cleanup confirmed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:25:09 +08:00
amos
e524d9eda5 docs: deployment map, task authoring standard and Qwen trial record
Add the current and target deployment diagrams, the task author standard
and copyable spec template, the five-field data contract, and record the
Qwen Responses trial and pending-create cleanup fix in progress and
acceptance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:06:06 +08:00