eust-w 30d916774c 🐛 Fail launches whose pod stays blocked instead of hanging the run
The reconciler used to be the only place that failed a node whose pod sat
in ImagePullBackOff/Unschedulable past WORKER_PENDING_FAILURE_SECONDS. Now
that it leaves runs with a live execution lease alone (0b365155), such a
node would wait for the full launch deadline.

Have Launch check the Job's pods (at most every 10s) and, once a pod has
been blocked for PendingFailureTimeout, delete the Job and return the
blocking reason and message. The executor passes the configured pending
threshold. On POC this surfaced builtin validator images that cannot be
pulled, which the reconciler race had been masking as missing Jobs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 16:25:24 +08:00
..