The public checks after a POC rollout (health, readiness, the entry page, the API build commit and the protected smoke checks) were each made once. Just after a rollout the public address can still reach a Pod that is shutting down, which answers with the old build commit, and the link can drop: either failed a healthy release and rolled it back. They now take up to six attempts, waiting 2 to 10 seconds; a release whose checks never pass is still rolled back. Retries were also silent: a build slowed down by a flaky link looked stuck. Every repeat now says what went unanswered and when the next attempt comes, in the run's timeline during a build and in the Runner's log during a release. Requests and checks share one retry helper. The third live test found kubectl cp and exec reporting a failed dial as "error: dial tcp ...: connect: operation timed out", without the "Unable to connect to the server" of get. That upload was not retried and the build stopped; dial failures and connection timeouts now count as unanswered, checked against the messages seen on the POC link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7.5 KiB
Existing DroboticFlow POC delivery
The local POC mode adopts the existing droboticflow Kubernetes namespace. The operator profile fixes the Gitea repository, poc branch, API server, namespace UID, three Deployment UIDs, container names, registry namespace, public URL, and pinned Kaniko executor. It is a private local file; web requests and repository manifests cannot replace it.
packages/runtime/src/poc-profile.ts defines the profile. PocRunnerDeploymentAdapter plugs into the normal signed Runner queue. The complete pocRelease component set participates in the signed manifest; its API image and source commit must match the outer release. This mode does not recreate Services, Secrets, RBAC, PVCs, or application configuration.
Source and build
- The verifier commits and pushes the tested source to Gitea
pocusing the ordinary branch precondition. buildPocReleasechecks the operator repository, local HEAD, and live remotepocHEAD against the complete source SHA. It archives that exact commit, excluding untracked files.- Each selected component builds in a separate bounded Kaniko Pod with an
emptyDir. The Pod mounts only the existing registry credential reference and has service-account token automount disabled. It has no business PVC or application secret mounts. - Kaniko records the pushed immutable digest through
/dev/termination-log. The builder saves source-archive, component, Pod UID, and bundle evidence locally, then deletes its own Pod using a UID precondition. Logs remain in private operator state and are not returned to the website. - Components not selected for rebuilding remain pinned to their observed digests in the same release set. The API, Worker, and Coordinator are always included in the manifest.
The component names accepted by the builder are api, worker, and coordinator. Default builds include all three. New feature builds must follow the target repository's versioning instructions; a preflight reconstruction of an existing commit is recorded separately from feature delivery.
Apply and verification
Before the first mutation, the adapter verifies the configured cluster server, namespace UID, each Deployment UID, current immutable images, readiness, and the preserved configuration fingerprints. Fingerprints cover each Deployment's complete spec, normalizing only its named container image and WishForge's own audit annotations. Replicas, sidecars, volumes, security settings, labels, other annotations, and environment configuration remain covered.
Every image update is a JSON Patch with test operations for UID, resourceVersion, container name, and previous image. Updates touch only the named image and wishforge.d-robotics.cc/* audit annotations. The sequence is Coordinator, Worker, then API. Multiple Kubernetes Deployments are not a transaction; the local execution policy must reject incompatible schema or cross-service protocol changes before release.
Completion requires all three rollouts, exact ready replica counts, live Pod digests, unchanged configuration fingerprints, public health/readiness/application responses, and the protected business smoke checks from the signed release. When the API changes, the public /api/health.buildCommit must equal the delivered source SHA. The pre-existing DROBOTICFLOW_IMAGE environment variable is preserved; Pod digests and the image-embedded build commit are authoritative for delivery.
Unreliable links to the cluster
The operator's machine may reach the cluster API over an unreliable link. A request that kubectl reports as unanswered (it could not connect, the connection broke, or kubectl hung) is repeated only where repeating is safe; a command that ran in a Pod and failed counts as answered.
- Reads (
get,logs) take up to six attempts, waiting 2, 4, 8, 16 and 30 seconds between them. - Before each attempt to create its Pod, the builder looks the Pod up by its attempt-unique name, so a create whose answer was lost is adopted instead of repeated.
- A lost source upload is sent again. Unpacking runs at most once at a time, refuses an upload whose SHA-256 differs from the archive (gzip's checksum covers images without
sha256sum), and marks the context ready only after a complete unpack, so Kaniko never starts from a partial context. A damaged upload is sent again, up to three times. - A dropped
rollout statuswatch resumes for the rest of the rollout timeout instead of failing the release and triggering a rollback. - The public checks after a rollout (health, readiness, the entry page, the API build commit and the protected business smoke checks) only read, and are repeated up to six times, waiting 2 to 10 seconds: just after a rollout the public address may still reach a Pod that is shutting down. A release whose checks never pass is still rolled back.
- Patches are never repeated; an uncertain patch is still reconciled by observation.
Each repeat is announced: during a build in the run's timeline, during a release in the Runner's log.
A link that stays down longer still fails the build or release; the run then waits for a person as an infrastructure fault, and continuing it starts a new build attempt.
Durable state and recovery
Release journals are written atomically with private permissions and filesystem synchronization. A release ID cannot be reused with another manifest. An environment lock prevents overlapping local deployments. Intent is persisted before each remote patch, so an interrupted write is reconciled by observation instead of repeated blindly.
Possible final journal states are:
| State | Meaning |
|---|---|
succeeded |
The desired component set passed runtime and business checks. |
rolled_back |
An explicitly compatible prior image set was restored and passed runtime checks. |
needs_reconciliation |
A receipt is uncertain, configuration changed, an external release appeared, or rollback compatibility was not declared. |
Rollback only changes images still owned by the attempted release, and never overwrites a newer external image. It does not revert the Gitea commit or database changes. Configuration drift stops automatic compensation. After a process crash, an operator must inspect any stale lock before removing it; reconcile does not bypass active locks. Signed Runner recovery also requires the persisted release to match the signed component bundle exactly.
Operator CLI
npx tsx scripts/poc-release.ts inspect /absolute/operator/poc-profile.json /absolute/operator/poc-state
npx tsx scripts/poc-release.ts build /absolute/operator/poc-profile.json /absolute/operator/poc-state /absolute/clean/checkout FULL_SHA RELEASE_ID
npx tsx scripts/poc-release.ts deploy /absolute/operator/poc-profile.json /absolute/operator/poc-state /absolute/operator/poc-state/builds/RELEASE_ID/release.json
npx tsx scripts/poc-release.ts reconcile /absolute/operator/poc-profile.json /absolute/operator/poc-state RELEASE_ID
The CLI is for local operator use. The website uses the signed Runner path. PocRunnerDeploymentAdapter.render() is read-only; deployment requires { execute: true }. The dedicated POC HTTP exception belongs to the local operator binding, not the general application URL policy.
Focused verification is npx vitest run packages/runtime/test/poc-release.test.ts. The tests exercise baseline drift, immutable image and configuration binding, JSON Patch preconditions, preserved resources, uncertain committed writes, compatible rollback, newer external releases, duplicate receipts, protected smoke failure, Runner evidence, and builds and rollouts through lost kubectl answers.