eust-w b47a07e8d2 chore: baseline snapshot of the Codex-delivered code before review
Imports the previously untracked working tree as-is so that the review
changes that follow (requirements -> design -> implementation) can be
reviewed as diffs and reverted if needed. No content changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 13:18:04 +08:00

13 KiB

Trusted AgentBundle evaluation and rollout

The evaluator is an independent process. The coding agent cannot set its benchmark, runner binary, signing key or trusted verification keys. Evaluator configuration is operator-owned; do not mount these files into a candidate workspace.

node --import tsx packages/evolution/src/cli.ts runs offline or shadow evaluation. With missing configuration it emits status: unconfigured and performs no model calls. No sample benchmark or fabricated score is installed in production.

Required evaluator environment:

  • WF_EVALUATION_JOB_FILE: operator-reviewed JSON with tenantId, candidateId, candidateDigest, baselineDigest, baseline, candidate, targetMetric and mode (offline or shadow). Bundle schemas are in src/bundle.ts.
  • WF_EVALUATION_BENCHMARK_FILE: protected versioned manifest. src/evaluate.ts defines its tasks, provenance and fixed environment/budget/model parameters. At least 50 distinct task IDs and inputs are required. Tenant opt-in provenance must be backed by operator-controlled consent records before making this file.
  • WF_EVALUATION_RUNNER: absolute path to an operator-owned VM/container runner. It reads one JSON EvaluationRequest from stdin, returns one JSON EvaluationResult on stdout, and isolates every attempt. The process adapter itself is not a sandbox. It must not be pointed at a coding-agent-generated executable.
  • WF_EVALUATION_RUNNER_ISOLATED=true: operator attestation that the runner enforces isolation, fixed budgets, read-only benchmark/verifier access, and no production credentials or write endpoints. This flag alone does not configure a sandbox.
  • WF_EVALUATION_RUNNER_VERSION: immutable version identifying the runner implementation.
  • WF_EVALUATION_PRIVATE_KEY_FILE and WF_EVALUATION_KEY_ID: Ed25519 signing key and ID.
  • Optional WF_EVALUATION_RUNNER_ENV_JSON: explicitly allowlisted provider credentials and runner settings. The parent process environment/signing-key configuration is not inherited. Never put production or evaluation-signing credentials here.
  • Optional WF_EVALUATION_OUTPUT_FILE: new local evidence file (exclusive creation, permissions 0600). Raw measurements contain task IDs; keep them in trusted storage.

Offline evaluation executes the same 50+ tasks three times for both bundles, with paired seeds, budgets and environment settings and alternating execution order. Success confidence intervals use paired task-cluster means and a 95% Student-t interval; repeats are not counted as independent tasks. This normal-approximation interval is an empirical estimate, not proof for unseen tasks. Costs and latency P95 come from measured runner output. All four hard gates must pass. The report includes content digests of tasks, fixed conditions and measurements, and its Ed25519 signature covers all metrics, target metric, tenant/candidate IDs and bundle versions.

The API accepts signed reports from an evaluator identity only. Configure WF_EVALUATOR_PUBLIC_KEYS as a JSON map of key IDs to PEM public keys. Reports are valid for seven days, reject future timestamps beyond clock skew, and cannot be reused for a different tenant, candidate or declared metric. Upload signed reports to POST /internal/evolution/:id/evaluation, signed shadow/canary observations to POST /internal/evolution/:id/observations (the HTTP controller defines the prefix). Store the complete output in protected evidence storage before uploading its report.

Bundles and rollout

bundleConfig permits only versioned prompt text and a model identifier. WF_EVOLUTION_ALLOWED_MODELS is an operator-owned comma-separated allowlist. No tool, permission, credential, executable, scoring rule or deployment instruction field is accepted. Prompt text cannot grant authority to the independently enforced tool policy. The candidate digest must equal bundleDigest(bundleConfig). Content is immutable.

The API routes only new runs. An active run resolves its saved bundle version even after promotion or rollback. Canary cohorts use a stable hash of tenant, project, run and candidate. The sequence is internal 5%, opted-in customer 5%, 25%, 100%, then promotion. Every stage, including 100%, requires 24 hours and 30 successful real runs. Signed observations reference actual terminal run IDs; integer count claims, repeated run IDs, stale-stage records, and reset-to-shadow bypasses are rejected. A hard-gate failure or failed observed run immediately stops candidate allocation.

Executable tool/platform code is not installed by this package. Its independent release system must provide signed artifact-bound release evidence using WF_PLATFORM_RELEASE_PUBLIC_KEYS, followed by a human approval. Its production rollout remains in that external system, and local advancement cannot manufacture its results. Trust-root changes are rejected from this automatic channel. Code rollout requests are persisted as externalAction records. A signed release receipt must bind deployment ID, source commit, image and previous compatible image. After maintainer approval, signed platform_canary observations supply individual task IDs and timestamps for each full observation window. Duplicate tasks do not increase the count. A failed gate freezes the candidate and requests rollback; only a matching signed platform_rollback receipt with healthy business checks can mark it rolled back. Requesting an external action is not deployment evidence.

npm run rsi:rollout performs one rollout admission pass; -- --watch runs a dedicated service loop when deliberately configured. It requires operator policy WF_EVOLUTION_AUTO_PROMOTION=true and an authorized evaluator/system identity. Absent configuration returns unconfigured. It installs no scheduler and does not replace the independent platform release controller. Endpoints are POST /internal/evolution/rollout/tick and GET /internal/evolution/external-actions.

Verification

npx vitest run tests/evolution.test.ts validates pairing, duplicate detection, report signature tampering, cross-candidate binding, no-production-write shadowing, deterministic opt-in cohorts, real-run observations, full observation windows and immutable-version rollback. Test measurements are explicitly fixtures, not deployment acceptance evidence. A real benchmark, runner, model authorization and isolated infrastructure remain prerequisites for an actual improvement claim.

Explicit failure-to-candidate meta loop

MetaEvolution.tick(actor) (apps/api/src/domain/meta-evolution.ts) performs one admission pass. The operator-facing API is POST /v1/evolution/tick. The one-shot client is:

node scripts/meta-evolution-tick.mjs --tick

The client requires WF_SYSTEM_TENANT, WF_META_TICK_TOKEN, and WF_META_CONTROL_PLANE_URL (or WF_CONTROL_PLANE_URL). It prints unconfigured without submitting anything when these are absent. It never creates a background loop or schedule. The tick credential is an authorized system operator credential; it is separate from the platform BYOK credential consumed by generated runs.

Configure server-side:

Setting Default / requirement
WF_SYSTEM_TENANT Explicit separately funded platform tenant
WF_META_PROJECT_ID Platform-owned repository with protected verification commands
WF_META_PROVIDER codex; codebuddy also supported when its executor is configured
WF_META_SCOPE prompt; prompt/orchestration generate bundles, tool/platform generate reviewed source proposals
WF_META_MIN_FAILURES 3 distinct failed source runs
WF_META_MAX_TOKENS 30000 reserved platform tokens per generated task
WF_META_MAX_MINUTES 15 active execution minutes; one repair round
WF_META_LOOKBACK_DAYS 7; at most 200 recent failed source runs inspected
WF_PROVISIONER_TARGETS_FILE Operator tenant routing map for the independently isolated execution service; a legacy URL/token must be bound to one tenant

The system tenant must hold its own active provider connection. The service does not read another tenant's provider, usage, runs, or experiences, and does not create an improvement run when this configuration is missing. Submission reuses the ordinary control-plane quota reservation, outbox, execution queue and lease limits.

A tick groups current-stable-bundle failures by provider, stage and normalized failure signature. It redacts credential-looking values and removes volatile URLs, UUIDs, timestamps and commit digests. Missing configuration/credentials/rate-limit outages are not treated as prompt-quality training examples. Legacy failure-event lookup is bounded to 5000 events per tick. A source run counts once; already consumed runs and failed evolution tasks do not trigger a recursive spending loop. Each explicit tick admits at most one bounded cluster.

An optional existing experience record can provide missing failure context only when it has kind: failure_observation, the same sourceTenantId as the system tenant, source IDs that all resolve to its failed runs, and a current authorization {purpose: platform_evolution, approvedBy, approvedAt} without revokedAt. The approving member must still be an owner/admin. Even an experience claiming consent is ignored if its source tenant is different. This first version does not import customer examples into the global loop. Revoked pending evidence requires operator review and is not resubmitted.

Prepared operations are auditable experience entities with kind: meta_task. Their idempotency binds sorted source-run IDs, stable baseline and scope. Project, provider, budget, prompts and source references are frozen before submission. If the submit commits but its response is lost, the next explicit tick repairs the same operation instead of creating or charging for a second task. The run is created atomically with purpose: evolution, noDeployment: true, source IDs, parent bundle, scope and meta-task ID. Public wish input cannot supply these fields.

For prompt/orchestration scopes, the coding agent commits a complete .wishforge/evolution-candidate.json bundle object. The independent verifier restores the Git checkpoint, runs protected checks, then reads this file with a workspace realpath boundary, 64 KiB limit, UTF-8 validation and strict bundle schema. Only that verifier's current fenced lease can submit the artifact to POST /internal/runs/:id/evolution-candidate; scope, baseline and candidate digest are derived/checked by the server. Coding agents cannot choose approval state or install permissions through the file.

For tool/platform scopes, the agent commits source changes instead. The independent verifier submits the exact checkpoint, bundle digest and validation evidence through POST /internal/runs/:id/evolution-code-candidate. The server derives source identity and checks the current requirement and verification policy. Protected-path changes escalate to trust_root; those proposals cannot use automatic promotion. Path classification alone does not prove semantic safety, so build, identity and release credentials remain outside the candidate workspace and under independent review.

Once extracted, the generation run records generatedEvolutionCandidateId, waits for independent evolution evaluation, and releases its unused reservation. Recovery recognizes that recorded ID and does not repeat the builder or candidate submission. It does not integrate the candidate branch, build an image, or request deployment. The release service rejects evolution-purpose runs even if called separately. Generated proposals still require the signed offline evaluation, shadow and full canary observation windows described above. Generated tool/platform proposals retain the verified branch/checkpoint and external release channel; generation never installs executable code or automatically grants production release permission.

tests/evolution-meta.test.ts covers clustering, provenance filtering, platform-only funding, concurrent deduplication, response-loss recovery, source authorization revocation, source-proposal scopes and rejection of trust-root auto-generation. Worker tests cover actual candidate-file reads, path escapes, oversized/extra-authority JSON, and recovery of an already generated candidate. These are contract fixtures. The local one-shot client was executed with missing platform settings and returned unconfigured; no real model call or scheduled task was made.

tests/evolution-source.test.ts and tests/evolution-code-rollout.test.ts additionally exercise verified source extraction, protected-path escalation, maintainer approval, all four canary windows, duplicate observation rejection and signed rollback receipts. Their clocks and measurements are fixtures, not completed 24-hour production canaries.