# Existing DroboticFlow POC delivery The local POC mode adopts the existing `droboticflow` Kubernetes namespace. The operator profile fixes the Gitea repository, `poc` branch, API server, namespace UID, three Deployment UIDs, container names, registry namespace, public URL, and pinned Kaniko executor. It is a private local file; web requests and repository manifests cannot replace it. `packages/runtime/src/poc-profile.ts` defines the profile. `PocRunnerDeploymentAdapter` plugs into the normal signed Runner queue. The complete `pocRelease` component set participates in the signed manifest; its API image and source commit must match the outer release. This mode does not recreate Services, Secrets, RBAC, PVCs, or application configuration. ## Source and build 1. The verifier commits and pushes the tested source to Gitea `poc` using the ordinary branch precondition. 2. `buildPocRelease` checks the operator repository, local HEAD, and live remote `poc` HEAD against the complete source SHA. It archives that exact commit, excluding untracked files. 3. Each selected component builds in a separate bounded Kaniko Pod with an `emptyDir`. The Pod mounts only the existing registry credential reference and has service-account token automount disabled. It has no business PVC or application secret mounts. 4. Kaniko records the pushed immutable digest through `/dev/termination-log`. The builder saves source-archive, component, Pod UID, and bundle evidence locally, then deletes its own Pod using a UID precondition. Logs remain in private operator state and are not returned to the website. 5. Components not selected for rebuilding remain pinned to their observed digests in the same release set. The API, Worker, and Coordinator are always included in the manifest. The component names accepted by the builder are `api`, `worker`, and `coordinator`. Default builds include all three. New feature builds must follow the target repository's versioning instructions; a preflight reconstruction of an existing commit is recorded separately from feature delivery. ## Apply and verification Before the first mutation, the adapter verifies the configured cluster server, namespace UID, each Deployment UID, current immutable images, readiness, and the preserved configuration fingerprints. Fingerprints cover each Deployment's complete `spec`, normalizing only its named container image and WishForge's own audit annotations. Replicas, sidecars, volumes, security settings, labels, other annotations, and environment configuration remain covered. Every image update is a JSON Patch with `test` operations for UID, resourceVersion, container name, and previous image. Updates touch only the named image and `wishforge.d-robotics.cc/*` audit annotations. The sequence is Coordinator, Worker, then API. Multiple Kubernetes Deployments are not a transaction; the local execution policy must reject incompatible schema or cross-service protocol changes before release. Completion requires all three rollouts, exact ready replica counts, live Pod digests, unchanged configuration fingerprints, public health/readiness/application responses, and the protected business smoke checks from the signed release. When the API changes, the public `/api/health.buildCommit` must equal the delivered source SHA. The pre-existing `DROBOTICFLOW_IMAGE` environment variable is preserved; Pod digests and the image-embedded build commit are authoritative for delivery. ## Unreliable links to the cluster The operator's machine may reach the cluster API over an unreliable link. A request that kubectl reports as unanswered (it could not connect, the connection broke, or kubectl hung) is repeated only where repeating is safe; a command that ran in a Pod and failed counts as answered. - Reads (`get`, `logs`) take up to six attempts, waiting 2, 4, 8, 16 and 30 seconds between them. - Before each attempt to create its Pod, the builder looks the Pod up by its attempt-unique name, so a create whose answer was lost is adopted instead of repeated. - A lost source upload is sent again. Unpacking runs at most once at a time, refuses an upload whose SHA-256 differs from the archive (gzip's checksum covers images without `sha256sum`), and marks the context ready only after a complete unpack, so Kaniko never starts from a partial context. A damaged upload is sent again, up to three times. - A dropped `rollout status` watch resumes for the rest of the rollout timeout instead of failing the release and triggering a rollback. - The public checks after a rollout (health, readiness, the entry page, the API build commit and the protected business smoke checks) only read, and are repeated up to six times, waiting 2 to 10 seconds: just after a rollout the public address may still reach a Pod that is shutting down. A release whose checks never pass is still rolled back. - Patches are never repeated; an uncertain patch is still reconciled by observation. Each repeat is announced: during a build in the run's timeline, during a release in the Runner's log. A link that stays down longer still fails the build or release; the run then waits for a person as an infrastructure fault, and continuing it starts a new build attempt. ## Durable state and recovery Release journals are written atomically with private permissions and filesystem synchronization. A release ID cannot be reused with another manifest. An environment lock prevents overlapping local deployments. Intent is persisted before each remote patch, so an interrupted write is reconciled by observation instead of repeated blindly. Possible final journal states are: | State | Meaning | | --- | --- | | `succeeded` | The desired component set passed runtime and business checks. | | `rolled_back` | An explicitly compatible prior image set was restored and passed runtime checks. | | `needs_reconciliation` | A receipt is uncertain, configuration changed, an external release appeared, or rollback compatibility was not declared. | Rollback only changes images still owned by the attempted release, and never overwrites a newer external image. It does not revert the Gitea commit or database changes. Configuration drift stops automatic compensation. After a process crash, an operator must inspect any stale lock before removing it; `reconcile` does not bypass active locks. Signed Runner recovery also requires the persisted release to match the signed component bundle exactly. ## Operator CLI ```sh npx tsx scripts/poc-release.ts inspect /absolute/operator/poc-profile.json /absolute/operator/poc-state npx tsx scripts/poc-release.ts build /absolute/operator/poc-profile.json /absolute/operator/poc-state /absolute/clean/checkout FULL_SHA RELEASE_ID npx tsx scripts/poc-release.ts deploy /absolute/operator/poc-profile.json /absolute/operator/poc-state /absolute/operator/poc-state/builds/RELEASE_ID/release.json npx tsx scripts/poc-release.ts reconcile /absolute/operator/poc-profile.json /absolute/operator/poc-state RELEASE_ID ``` The CLI is for local operator use. The website uses the signed Runner path. `PocRunnerDeploymentAdapter.render()` is read-only; deployment requires `{ execute: true }`. The dedicated POC HTTP exception belongs to the local operator binding, not the general application URL policy. Focused verification is `npx vitest run packages/runtime/test/poc-release.test.ts`. The tests exercise baseline drift, immutable image and configuration binding, JSON Patch preconditions, preserved resources, uncertain committed writes, compatible rollback, newer external releases, duplicate receipts, protected smoke failure, Runner evidence, and builds and rollouts through lost kubectl answers.