20 Commits

Author SHA1 Message Date
amos
5b5a63d267 Merge native trace prefix deltas for trace6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	docs/progress.md
2026-10-18 22:19:33 +08:00
amos
913d3c367f Add the usable task catalog and admit batch 6 tasks
All 161 task packages, including the 21 batch 6 C++17 tasks that passed
gates 1/2, are now in Git. rollout.catalog builds docs/task-catalog.{json,md}
from the task packages, a fresh trace5 calibration (128/128 PASS), every
verified controller receipt on the current task digest, documented holds
and digest-bound difficulty reviews. 123 tasks are usable (6 HARD),
5 are on hold and 33 still need a subprocess-only grader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 17:43:15 +08:00
amos
b773033658 Record model requests and payloads as prefix deltas in the native trace
Each model_request/model_payload repeated the full context, so the trace
grew with the square of the request count: batch 6 Qwen runs hit 64 MiB
at ~75 requests and 14 attempts (13 graded PASS) were discarded.
Requests now record only items after the common prefix with the previous
one (schema embedded-rollout.pi/2), and pi_run.expand rebuilds full
payloads before the unchanged broker/feedback checks. Needs a new image
(trace6) and recalibration before use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 10:50:56 +08:00
amos
b71d605d81 Lower the shared-disk floor to 30 GiB at the user's request
The other users of /home left 57 GiB free; the user chose to keep running
and stop only at 30 GiB. Batch start and runtime disk_low now share that floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 13:11:21 +08:00
Codex
c58d5cc345 fix: allow 900 seconds of upstream stream silence 2026-10-17 01:40:40 +08:00
amos
de3755c14b fix: keep the Bos grader away from candidate processes
The Bos wrapper ran check.py and every candidate process as UID 65534, so a
candidate could read the grader copy through /proc/<checker>/cwd and pass
without implementing anything. check.py now runs as root on a 0700 grader
copy; its subprocesses drop to 65534 and only the work copy and mkdtemp
scratch directories are handed over. An audit hook ends the check as
UNVERIFIED if candidate code would run inside the checker.

36 tasks run hidden tests inside the candidate process (tests.py style and
S500_a2_426889b9a968); they carry isolation-blocked.txt and always grade
UNVERIFIED. The importer adds the marker for any extra root *.py. All 74 Bos
tasks get the new verify.py and a version bump.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:33:34 +08:00
amos
141f3d06aa fix: cap disk use per attempt and refuse to start on a low output disk
Agent containers wrote to bind-mounted /workspace and /trace with no
size limit, so a runaway Agent could fill the shared /home. Containers
no longer start below 50 GiB free; a running Agent is stopped when the
disk drops below that floor or its attempt directory passes 2 GiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:22:35 +08:00
amos
94eaf64261 fix: private grade diagnostics and a public-contract admission check
- bos_verify still runs the original check.py unchanged, but failing
  child processes now copy their stderr tail into grade.stderr; report
  rows carry fail_detail (that line, else the grade reason).
- bos_import rejects envs whose hidden tests use candidate-module names
  the public prompt/template never mention; a repo test applies the same
  check to every admitted task.
- S500_n3_152 v2 publishes process(request): its v1 FAILs came from that
  hidden requirement, not from the models. All Bos tasks carry the new
  wrapper as version 2; 21 tasks recalibrated on trace5 (805 trials).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:27:29 +08:00
amos
f0c277a0d1 fix: controller owns model output limits; verify them on the wire
Output limits were baked into pi.mjs inside the runtime image, so every
change (GLM 32000 -> 131072, Qwen 8192 -> 65536) needed a new image and a
full 785-trial recalibration, and the Qwen baseline ran on an image the
repo could not rebuild.
- MODELS holds max_tokens and its wire field per model; pi_run passes
  --model and --max-tokens to the adapter, which has no defaults left.
- Receipts record model_max_tokens and model_sampling=service-default;
  evidence requires every payload to carry exactly that limit, which
  would have caught the SDK's silent 32000 cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:27:29 +08:00
amos
31c862cd3c fix: retry transient upstream failures and classify broken sessions
Two trace4 runs lost their whole session to one upstream HTTP 500 on the
first request (2 of 562 requests), and their controllers still reported
the template's FAIL grade.
- The proxy retries upstream 429/5xx and connection errors up to twice
  (2 s, 5 s) before any byte reaches Pi, records every status in
  upstream_statuses, and never retries while it is closing.
- A controller reports PASS or FAIL only with VERIFIED evidence and a
  clean Agent exit; otherwise the grade stays in task_verdict and the
  status is INFRA_ERROR when the model channel returned non-200, else
  UNVERIFIED.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 02:10:05 +08:00
amos
f20316fa4d fix: report counts controllers without a receipt instead of failing
A running or killed controller has no receipt yet; it is now an attempt
with status NO_RECEIPT and listed among the exclusions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 00:06:56 +08:00
amos
7071ecfa7f fix: raise the native trace limit to 64 MiB
Bos tasks with 131072-token reasoning produce ~110 bytes per streamed
token plus a full context per request; the trace3 batch hit 16 MiB
before finishing. Adapter and host share the new limit, and the size
tests now derive their data from MAX_TRACE_BYTES. Runtime trace4 carries
this adapter; all 21 tasks recalibrated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:05:39 +08:00
amos
3e19424391 feat: regrade frozen candidates and report rerun consistency
rollout.regrade checks the original task, image and candidate digests,
grades the frozen candidate again in a fresh grader container and records
whether the verdict matches. rollout.report counts confirmed regrades under
the batch's regrade/ directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:52:29 +08:00
amos
b19cfd4844 feat: Bos-qiao task importer, unprivileged check wrapper and batch report
- rollout.bos_import converts one Bos-qiao env into a task package. Fixture
  scripts are parsed, never run: only quoted cat heredocs, literal
  Path(...).write_text calls or the factory's literal files loop are
  accepted. cheat_exit_zero becomes noop, valid_* become extra PASS
  fixtures, every other cheat stays a FAIL fixture.
- rollout/bos_verify.py runs the original check.py unchanged as UID 65534
  after the same read-isolation probe as frame-decoder; modes are set
  before ownership is given away because the grader lacks CAP_FOWNER.
- rollout.report summarizes a batch of controllers; only PASS/FAIL with
  VERIFIED evidence, clean exit and confirmed cleanup count as complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:48:25 +08:00
amos
acfa601a56 feat: multi-file task contract, model table and shared I/O helpers
- Tasks may declare up to 16 candidate files; the whole template becomes
  the Agent workspace while only candidate files are frozen and graded.
- Optional fixtures add more cheats or legal alternatives (five trials
  each); optional grader_files ship private tests next to verify.py.
- Tasks without an argv oracle export call_type=verifier ground truth.
- Upstream model routes live in one MODELS table.
- One bounded no-follow reader, one exclusive writer and one SIGTERM
  helper replace per-module copies; drop unused validate_task.

frame-decoder keeps its task digest, calibration receipts and exported
query. Verified on host 36: internal network probe, 20-trial calibration
and a real GLM rollout (PASS, evidence VERIFIED, cleanup confirmed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:25:09 +08:00
amos
714f345b63 feat: add Qwen Responses route, five-field task export and pending-create cleanup
- Pi controller accepts a fixed Qwen qwen3.8-27b-nvfp4 Responses route next
  to GLM-5.3; the broker requires exactly one response.completed per call
  and evidence checks the native model identity.
- rollout.export writes the UltraData five-field JSONL from a frozen
  dataset.json snapshot; every archive now carries dataset.jsonl.
- A Docker CLI that did not finish cannot confirm container or network
  cleanup, and auto-calibration cleanup uncertainty reaches the controller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:05:56 +08:00
amos
d205c44249 fix: cut agent internet egress and loosen grader timeouts
The dedicated Pi network was a plain bridge, so the Agent could reach the
internet and fetch upstream fixes. Create it with --internal, make run
accept only internal networks, and default to --network none.

Verifier limits of 1s per case and 15s per grade turned host load into
false FAIL verdicts. Use 5s per case, 30s compile, 10s isolation probe and
timeout_seconds=60; frame-decoder task version becomes 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 20:30:00 +08:00
amos
fb837e5535 feat: seal verified rollout evidence archives 2026-10-12 11:15:45 +08:00
amos
563a3745df feat: integrate Pi rollout and protected model proxy 2026-10-12 11:15:44 +08:00
amos
e74867a46f feat: add calibrated Docker task harness 2026-10-12 11:15:44 +08:00