eust-w 56a845f53c 🐛 Ride out a flaky link to the POC cluster during builds and releases
In the 2026-10-26 POC run the Mac's link to the cluster API dropped
about once a minute. One unanswered kubectl call ended the whole build
twice, though the Kaniko Pod kept building, and each manual resume
re-ran the protected verification. The release path had the same
weakness with worse consequences: a dropped rollout watch counted as a
failed rollout and rolled a healthy release back.

Only requests that are safe to repeat are repeated, and only when the
cluster API left them unanswered (no connection, a broken connection,
kubectl hung), never when it refused them:

- Reads take up to six attempts, waiting 2 to 30 seconds between them.
- The builder looks its Pod up by its attempt-unique name before each
  create, adopting a Pod whose create answer was lost.
- A lost source upload is sent again. Unpacking runs once at a time,
  refuses an upload whose SHA-256 differs from the archive, and marks
  the context ready only when complete, so Kaniko never starts from a
  partial context; a damaged upload is sent again.
- A dropped rollout watch resumes for the rest of the rollout timeout.

Patches are still never repeated. A link that stays down longer still
stops the run as an infrastructure fault.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 02:25:34 +08:00
..