Imports the previously untracked working tree as-is so that the review changes that follow (requirements -> design -> implementation) can be reviewed as diffs and reverted if needed. No content changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Trusted AgentBundle evaluation and rollout
The evaluator is an independent process. The coding agent cannot set its benchmark, runner binary, signing key or trusted verification keys. Evaluator configuration is operator-owned; do not mount these files into a candidate workspace.
node --import tsx packages/evolution/src/cli.ts runs offline or shadow evaluation.
With missing configuration it emits status: unconfigured and performs no model calls.
No sample benchmark or fabricated score is installed in production.
Required evaluator environment:
WF_EVALUATION_JOB_FILE: operator-reviewed JSON withtenantId,candidateId,candidateDigest,baselineDigest,baseline,candidate,targetMetricandmode(offlineorshadow). Bundle schemas are insrc/bundle.ts.WF_EVALUATION_BENCHMARK_FILE: protected versioned manifest.src/evaluate.tsdefines its tasks, provenance and fixed environment/budget/model parameters. At least 50 distinct task IDs and inputs are required. Tenant opt-in provenance must be backed by operator-controlled consent records before making this file.WF_EVALUATION_RUNNER: absolute path to an operator-owned VM/container runner. It reads one JSONEvaluationRequestfrom stdin, returns one JSONEvaluationResulton stdout, and isolates every attempt. The process adapter itself is not a sandbox. It must not be pointed at a coding-agent-generated executable.WF_EVALUATION_RUNNER_ISOLATED=true: operator attestation that the runner enforces isolation, fixed budgets, read-only benchmark/verifier access, and no production credentials or write endpoints. This flag alone does not configure a sandbox.WF_EVALUATION_RUNNER_VERSION: immutable version identifying the runner implementation.WF_EVALUATION_PRIVATE_KEY_FILEandWF_EVALUATION_KEY_ID: Ed25519 signing key and ID.- Optional
WF_EVALUATION_RUNNER_ENV_JSON: explicitly allowlisted provider credentials and runner settings. The parent process environment/signing-key configuration is not inherited. Never put production or evaluation-signing credentials here. - Optional
WF_EVALUATION_OUTPUT_FILE: new local evidence file (exclusive creation, permissions 0600). Raw measurements contain task IDs; keep them in trusted storage.
Offline evaluation executes the same 50+ tasks three times for both bundles, with paired seeds, budgets and environment settings and alternating execution order. Success confidence intervals use paired task-cluster means and a 95% Student-t interval; repeats are not counted as independent tasks. This normal-approximation interval is an empirical estimate, not proof for unseen tasks. Costs and latency P95 come from measured runner output. All four hard gates must pass. The report includes content digests of tasks, fixed conditions and measurements, and its Ed25519 signature covers all metrics, target metric, tenant/candidate IDs and bundle versions.
The API accepts signed reports from an evaluator identity only. Configure
WF_EVALUATOR_PUBLIC_KEYS as a JSON map of key IDs to PEM public keys. Reports are
valid for seven days, reject future timestamps beyond clock skew, and cannot be reused
for a different tenant, candidate or declared metric. Upload signed reports to
POST /internal/evolution/:id/evaluation, signed shadow/canary observations to
POST /internal/evolution/:id/observations (the HTTP controller defines the prefix).
Store the complete output in protected evidence storage before uploading its report.
Bundles and rollout
bundleConfig permits only versioned prompt text and a model identifier.
WF_EVOLUTION_ALLOWED_MODELS is an operator-owned comma-separated allowlist. No tool,
permission, credential, executable, scoring rule or deployment instruction field is
accepted. Prompt text cannot grant authority to the independently enforced tool policy.
The candidate digest must equal bundleDigest(bundleConfig). Content is immutable.
The API routes only new runs. An active run resolves its saved bundle version even after promotion or rollback. Canary cohorts use a stable hash of tenant, project, run and candidate. The sequence is internal 5%, opted-in customer 5%, 25%, 100%, then promotion. Every stage, including 100%, requires 24 hours and 30 successful real runs. Signed observations reference actual terminal run IDs; integer count claims, repeated run IDs, stale-stage records, and reset-to-shadow bypasses are rejected. A hard-gate failure or failed observed run immediately stops candidate allocation.
Executable tool/platform code is not installed by this package. Its independent
release system must provide signed artifact-bound release evidence using
WF_PLATFORM_RELEASE_PUBLIC_KEYS, followed by a human approval. Its production
rollout remains in that external system, and local advancement cannot manufacture
its results. Trust-root changes are rejected from this automatic channel.
Code rollout requests are persisted as externalAction records. A signed release
receipt must bind deployment ID, source commit, image and previous compatible image.
After maintainer approval, signed platform_canary observations supply individual
task IDs and timestamps for each full observation window. Duplicate tasks do not
increase the count. A failed gate freezes the candidate and requests rollback;
only a matching signed platform_rollback receipt with healthy business checks
can mark it rolled back. Requesting an external action is not deployment evidence.
npm run rsi:rollout performs one rollout admission pass; -- --watch runs a
dedicated service loop when deliberately configured. It requires operator policy
WF_EVOLUTION_AUTO_PROMOTION=true and an authorized evaluator/system identity.
Absent configuration returns unconfigured. It installs no scheduler and does not
replace the independent platform release controller. Endpoints are
POST /internal/evolution/rollout/tick and GET /internal/evolution/external-actions.
Verification
npx vitest run tests/evolution.test.ts validates pairing, duplicate detection,
report signature tampering, cross-candidate binding, no-production-write shadowing,
deterministic opt-in cohorts, real-run observations, full observation windows and
immutable-version rollback. Test measurements are explicitly fixtures, not deployment
acceptance evidence. A real benchmark, runner, model authorization and isolated
infrastructure remain prerequisites for an actual improvement claim.
Explicit failure-to-candidate meta loop
MetaEvolution.tick(actor) (apps/api/src/domain/meta-evolution.ts) performs one
admission pass. The operator-facing API is POST /v1/evolution/tick. The one-shot
client is:
node scripts/meta-evolution-tick.mjs --tick
The client requires WF_SYSTEM_TENANT, WF_META_TICK_TOKEN, and
WF_META_CONTROL_PLANE_URL (or WF_CONTROL_PLANE_URL). It prints unconfigured
without submitting anything when these are absent. It never creates a background
loop or schedule. The tick credential is an authorized system operator credential;
it is separate from the platform BYOK credential consumed by generated runs.
Configure server-side:
| Setting | Default / requirement |
|---|---|
WF_SYSTEM_TENANT |
Explicit separately funded platform tenant |
WF_META_PROJECT_ID |
Platform-owned repository with protected verification commands |
WF_META_PROVIDER |
codex; codebuddy also supported when its executor is configured |
WF_META_SCOPE |
prompt; prompt/orchestration generate bundles, tool/platform generate reviewed source proposals |
WF_META_MIN_FAILURES |
3 distinct failed source runs |
WF_META_MAX_TOKENS |
30000 reserved platform tokens per generated task |
WF_META_MAX_MINUTES |
15 active execution minutes; one repair round |
WF_META_LOOKBACK_DAYS |
7; at most 200 recent failed source runs inspected |
WF_PROVISIONER_TARGETS_FILE |
Operator tenant routing map for the independently isolated execution service; a legacy URL/token must be bound to one tenant |
The system tenant must hold its own active provider connection. The service does not read another tenant's provider, usage, runs, or experiences, and does not create an improvement run when this configuration is missing. Submission reuses the ordinary control-plane quota reservation, outbox, execution queue and lease limits.
A tick groups current-stable-bundle failures by provider, stage and normalized failure signature. It redacts credential-looking values and removes volatile URLs, UUIDs, timestamps and commit digests. Missing configuration/credentials/rate-limit outages are not treated as prompt-quality training examples. Legacy failure-event lookup is bounded to 5000 events per tick. A source run counts once; already consumed runs and failed evolution tasks do not trigger a recursive spending loop. Each explicit tick admits at most one bounded cluster.
An optional existing experience record can provide missing failure context only
when it has kind: failure_observation, the same sourceTenantId as the system
tenant, source IDs that all resolve to its failed runs, and a current authorization
{purpose: platform_evolution, approvedBy, approvedAt} without revokedAt.
The approving member must still be an owner/admin. Even an experience claiming
consent is ignored if its source tenant is different. This first version does not
import customer examples into the global loop. Revoked pending evidence requires
operator review and is not resubmitted.
Prepared operations are auditable experience entities with kind: meta_task.
Their idempotency binds sorted source-run IDs, stable baseline and scope. Project,
provider, budget, prompts and source references are frozen before submission. If
the submit commits but its response is lost, the next explicit tick repairs the
same operation instead of creating or charging for a second task. The run is created
atomically with purpose: evolution, noDeployment: true, source IDs, parent bundle,
scope and meta-task ID. Public wish input cannot supply these fields.
For prompt/orchestration scopes, the coding agent commits a complete .wishforge/evolution-candidate.json bundle
object. The independent verifier restores the Git checkpoint, runs protected
checks, then reads this file with a workspace realpath boundary, 64 KiB limit,
UTF-8 validation and strict bundle schema. Only that verifier's current fenced lease
can submit the artifact to POST /internal/runs/:id/evolution-candidate; scope,
baseline and candidate digest are derived/checked by the server. Coding agents
cannot choose approval state or install permissions through the file.
For tool/platform scopes, the agent commits source changes instead. The independent
verifier submits the exact checkpoint, bundle digest and validation evidence through
POST /internal/runs/:id/evolution-code-candidate. The server derives source identity
and checks the current requirement and verification policy. Protected-path changes
escalate to trust_root; those proposals cannot use automatic promotion. Path
classification alone does not prove semantic safety, so build, identity and release
credentials remain outside the candidate workspace and under independent review.
Once extracted, the generation run records generatedEvolutionCandidateId, waits
for independent evolution evaluation, and releases its unused reservation. Recovery
recognizes that recorded ID and does not repeat the builder or candidate submission.
It does not integrate the candidate branch, build an image, or request deployment.
The release service rejects evolution-purpose runs even if called separately.
Generated proposals still require the signed offline evaluation, shadow and full
canary observation windows described above. Generated tool/platform proposals retain
the verified branch/checkpoint and external release channel; generation never
installs executable code or automatically grants production release permission.
tests/evolution-meta.test.ts covers clustering, provenance filtering, platform-only
funding, concurrent deduplication, response-loss recovery, source authorization
revocation, source-proposal scopes and rejection of trust-root auto-generation. Worker tests cover actual
candidate-file reads, path escapes, oversized/extra-authority JSON, and recovery of
an already generated candidate. These are contract fixtures. The local one-shot
client was executed with missing platform settings and returned unconfigured;
no real model call or scheduled task was made.
tests/evolution-source.test.ts and tests/evolution-code-rollout.test.ts additionally
exercise verified source extraction, protected-path escalation, maintainer approval,
all four canary windows, duplicate observation rejection and signed rollback receipts.
Their clocks and measurements are fixtures, not completed 24-hour production canaries.