8 Commits

Author SHA1 Message Date
amos
c93aeb8fd8 Let a credential pick one of the four Qwen instances
The Qwen service runs four identical instances on 114.111.18.115:18080-18083. A
credential may now carry upstream_port, limited to the ports the route lists, so
rollout.batch's least-busy key choice spreads attempts across instances. The broker
log records the upstream each request used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 14:58:12 +08:00
amos
c64c79b31d Run the Pi agent in Salty jobs over a one-time SSH tunnel and export SFT rows
rollout.salty_run keeps the gateway, grading and evidence on 36 and moves only the
agent step into a Salty job. The gateway listens on 127.0.0.1 with a per-attempt
bearer token; the job reaches it over port 22 with a restricted one-time key that is
removed after each attempt. rollout.batch gains --backend salty and repeatable
--credential-file, and writes pass/fail SFT exports through rollout.sft at the end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 17:45:28 +08:00
amos
3adab31da7 Self-heal rollouts: retry stalled model streams and run batches from receipts
Batch 6 lost 26 of 103 attempts to Qwen requests that sent their preamble and
then stalled; Pi's own Node fetch gave up after 300 s and reported "terminated".

- model_proxy: hold the SSE preamble until the first generation event, retry a
  stall (180 s) within the existing retry budget, and send SSE keepalive
  comments so Pi never times out first; record first_event_seconds.
- report: name one exclusion reason per incomplete attempt and compute the
  JOBS lines still owed, with a per-line resample budget.
- batch: run a JOBS file to completion, resume from receipts, resample
  infrastructure failures, pause after three environment failures in a row,
  forward SIGTERM to running controllers, and refuse a second runner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 08:28:53 +08:00
Codex
c58d5cc345 fix: allow 900 seconds of upstream stream silence 2026-10-17 01:40:40 +08:00
amos
31c862cd3c fix: retry transient upstream failures and classify broken sessions
Two trace4 runs lost their whole session to one upstream HTTP 500 on the
first request (2 of 562 requests), and their controllers still reported
the template's FAIL grade.
- The proxy retries upstream 429/5xx and connection errors up to twice
  (2 s, 5 s) before any byte reaches Pi, records every status in
  upstream_statuses, and never retries while it is closing.
- A controller reports PASS or FAIL only with VERIFIED evidence and a
  clean Agent exit; otherwise the grade stays in task_verdict and the
  status is INFRA_ERROR when the model channel returned non-200, else
  UNVERIFIED.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 02:10:05 +08:00
amos
acfa601a56 feat: multi-file task contract, model table and shared I/O helpers
- Tasks may declare up to 16 candidate files; the whole template becomes
  the Agent workspace while only candidate files are frozen and graded.
- Optional fixtures add more cheats or legal alternatives (five trials
  each); optional grader_files ship private tests next to verify.py.
- Tasks without an argv oracle export call_type=verifier ground truth.
- Upstream model routes live in one MODELS table.
- One bounded no-follow reader, one exclusive writer and one SIGTERM
  helper replace per-module copies; drop unused validate_task.

frame-decoder keeps its task digest, calibration receipts and exported
query. Verified on host 36: internal network probe, 20-trial calibration
and a real GLM rollout (PASS, evidence VERIFIED, cleanup confirmed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:25:09 +08:00
amos
714f345b63 feat: add Qwen Responses route, five-field task export and pending-create cleanup
- Pi controller accepts a fixed Qwen qwen3.8-27b-nvfp4 Responses route next
  to GLM-5.3; the broker requires exactly one response.completed per call
  and evidence checks the native model identity.
- rollout.export writes the UltraData five-field JSONL from a frozen
  dataset.json snapshot; every archive now carries dataset.jsonl.
- A Docker CLI that did not finish cannot confirm container or network
  cleanup, and auto-calibration cleanup uncertainty reaches the controller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:05:56 +08:00
amos
563a3745df feat: integrate Pi rollout and protected model proxy 2026-10-12 11:15:44 +08:00