eust-w b47a07e8d2 chore: baseline snapshot of the Codex-delivered code before review
Imports the previously untracked working tree as-is so that the review
changes that follow (requirements -> design -> implementation) can be
reviewed as diffs and reverted if needed. No content changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 13:18:04 +08:00

5.0 KiB

Run sandbox provisioner

This service renders and applies real Kubernetes resources through an explicit kubectl --context ... --namespace ... command. It has no local fallback and does not start without a tenant, cluster context, namespace, digest-pinned worker image, storage class, HTTPS control-plane URL and Temporal configuration.

POST /runs requires Authorization: Bearer <WF_PROVISIONER_TOKEN> and {tenantId, runId, generation, executorToken, verifierToken, resourceGrant}. The authenticated control plane issues the two short-lived run-scoped tokens. The provisioner checks their scope and lifetime without holding the token-signing key. The API independently verifies signatures when the workers use them. Returned status provisioned means Kubernetes accepted the resources; it does not mean either pod is scheduled or ready.

The independent HMAC resource grant binds the immutable runtime image and total CPU/memory/persistent-volume allowance for both roles. Each role receives half, rounded down. This grant does not include retained earlier-generation PVCs, IDE resources or temporary disk (currently an 8 GiB ephemeral limit and 4 GiB /tmp limit per role); it is not an aggregate tenant storage budget. The provisioner verifies the signature, expiry and its static caps, then renders the signed image only if listed in the current/previous runtime allowlist. Workers never receive the grant-signing key. See plans and admission.

Each run receives deterministic, idempotent names for two independent RWO PVCs, one per role, two separate capability Secrets, two unbound service accounts, two Kata Jobs and a run-scoped NetworkPolicy. Policy/identities are applied before jobs. Jobs get only their own role token and cannot mount a Kubernetes token, host path or Docker socket. Provider homes are private memory volumes. Only DNS plus labeled API and Temporal services are allowed by the generated network policy; provider/tool/build network access requires separately reviewed egress-proxy configuration.

Required environment:

WF_TENANT_ID=<one tenant>
WF_PROVISIONER_TOKEN=<separate generated service credential>
WF_RESOURCE_GRANT_SECRET=<independent API/provisioner shared signing secret>
WF_KUBE_CONTEXT=<explicit target context>
WF_KUBE_NAMESPACE=<preprovisioned tenant namespace>
WF_RUN_WORKER_IMAGE=<registry/image@sha256:64-hex-digest>
WF_RUN_PREVIOUS_IMAGES=<optional comma-separated prior immutable image digests>
WF_RUN_STORAGE_CLASS=<storage class>
WF_CONTROL_PLANE_URL=https://api.wishforge-control.svc:8443
WF_CONTROL_NAMESPACE=wishforge-control
TEMPORAL_ADDRESS=temporal.wishforge-control.svc:7233
TEMPORAL_NAMESPACE=wishforge

Optional WF_VERIFIER_DOCKER_CONFIG_SECRET references a separately provisioned verifier-only Docker/BuildKit client configuration. There is no default daemon, host socket or registry credential. An isolated builder plus its explicit egress and authentication still need provisioning and acceptance; without them the runtime must fail closed at build time. The worker image must include Node/runtime, the configured coding adapters and /usr/bin/bwrap; Kata and nested unprivileged bubblewrap support must be tested on the actual node pool.

The executor cannot mount the verifier's PVC. The verifier restores an exact Git bundle and validates its digest and source SHA in its own workspace. API leases and generation-bound capabilities independently fence stale evidence. Before a new generation starts, only this run's older managed Jobs are deleted with foreground propagation; their PVCs remain for diagnosis. A terminal Job in the same generation returns failed and requires an explicit new-generation resume. Tokens expire; jobs have a shorter hard deadline. Existing terminal Jobs are not silently deleted/recreated by retries. An operator-controlled retry/cleanup policy must reconcile expired jobs and Secrets; PVC deletion requires the applicable retention policy. No endpoint grants arbitrary namespace, image, command or cleanup control to callers.

POST /runs/release accepts {tenantId,runId,generation,resourceGrant} with the same service authentication. It checks owned generation labels, stops only matching Jobs, and reports whether compute has stopped; it never deletes persistent volumes or fabricates compute billing measurements. Provision/cleanup is serialized per run within this process. Run a single active provisioner per tenant namespace; this queue is not durable distributed fencing across a process restart. Unknown prior outcomes remain reserved until API-side operator reconciliation after cleanup.

Tests validate rendered resources and fake command invocation; no real cluster has been provisioned. Production acceptance requires actual RuntimeClass VM proof, Secret/role isolation, CNI denial, independent RWO storage, generation-bound queue routing, bubblewrap execution and end-to-end token revocation. WF_SANDBOX_ENFORCED=true records the intended runtime path; it is not independent evidence of isolation.