🐛 Repeat the POC acceptance checks a settling rollout can fail, and show every retry
The public checks after a POC rollout (health, readiness, the entry page, the API build commit and the protected smoke checks) were each made once. Just after a rollout the public address can still reach a Pod that is shutting down, which answers with the old build commit, and the link can drop: either failed a healthy release and rolled it back. They now take up to six attempts, waiting 2 to 10 seconds; a release whose checks never pass is still rolled back. Retries were also silent: a build slowed down by a flaky link looked stuck. Every repeat now says what went unanswered and when the next attempt comes, in the run's timeline during a build and in the Runner's log during a release. Requests and checks share one retry helper. The third live test found kubectl cp and exec reporting a failed dial as "error: dial tcp ...: connect: operation timed out", without the "Unable to connect to the server" of get. That upload was not retried and the build stopped; dial failures and connection timeouts now count as unanswered, checked against the messages seen on the POC link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
9394241212
commit
92a06dd3b2
@ -1,11 +1,13 @@
|
||||
import path from 'node:path';
|
||||
import {
|
||||
ComposeDeploymentAdapter,
|
||||
DEFAULT_PATIENCE,
|
||||
KubernetesDeploymentAdapter,
|
||||
DeploymentSecretBroker,
|
||||
loadPocProfile,
|
||||
PocRunnerDeploymentAdapter,
|
||||
} from '@wishforge/runtime';
|
||||
import { log } from './log.js';
|
||||
import type { RunnerConfig } from './runner.js';
|
||||
|
||||
/** An adopted POC runner only accepts the operator's fixed Kubernetes profile. */
|
||||
@ -14,7 +16,18 @@ export async function createRunnerAdapters(env: Record<string, string | undefine
|
||||
if (!path.isAbsolute(env.WF_POC_PROFILE_FILE)) throw new Error('WF_POC_PROFILE_FILE must be an absolute path');
|
||||
if (!env.WF_RUNNER_STATE_DIR) throw new Error('WF_RUNNER_STATE_DIR is required');
|
||||
const profile = await loadPocProfile(env.WF_POC_PROFILE_FILE);
|
||||
return { kubernetes: new PocRunnerDeploymentAdapter(profile, path.join(env.WF_RUNNER_STATE_DIR, 'poc')) };
|
||||
return {
|
||||
kubernetes: new PocRunnerDeploymentAdapter(
|
||||
profile,
|
||||
path.join(env.WF_RUNNER_STATE_DIR, 'poc'),
|
||||
undefined,
|
||||
undefined,
|
||||
{
|
||||
...DEFAULT_PATIENCE,
|
||||
onRetry: notice => log('warn', notice),
|
||||
},
|
||||
),
|
||||
};
|
||||
}
|
||||
return {
|
||||
compose: new ComposeDeploymentAdapter({
|
||||
|
||||
@ -4,6 +4,7 @@ import type { GitHubDeliveryState, Run } from '@wishforge/contracts';
|
||||
import {
|
||||
buildDockerImage,
|
||||
buildPocRelease,
|
||||
DEFAULT_PATIENCE,
|
||||
loadPocProfile,
|
||||
readLocalPocBinding,
|
||||
localPocForProject,
|
||||
@ -338,6 +339,8 @@ async function deliver(session: Session, integrated: Verified): Promise<Activity
|
||||
!changedPaths.some(file => /(^|\/)(migrations?|store|auth|security)(\/|$)/i.test(file)),
|
||||
signal: session.keeper.signal,
|
||||
onProgress: message => session.transition('build', message).then(() => undefined),
|
||||
// A flaky link to the cluster shows in the timeline instead of only as a slow build.
|
||||
patience: { ...DEFAULT_PATIENCE, onRetry: notice => session.transition('build', notice) },
|
||||
},
|
||||
session.execute,
|
||||
);
|
||||
|
||||
@ -89,4 +89,4 @@ LaunchAgent 随当前用户登录运行,退出登录后停止;电脑关机
|
||||
|
||||
本机实验套餐由 operator CLI 管理:`node --import tsx scripts/configure-local-poc-plan.ts 20000000` 设置每月 2000 万计量 Token 和单并发;这不购买模型额度或扣款。单次愿望默认 100 万 Token,包含 CLI 缓存读取。CLI 最终计量在回合结束时上报;被暂停、取消或超时打断的会话没有总量,改按它已收到的每次模型响应估算计入,时间线标为估算。超预算会阻止后续执行并保留检查点,不能视为供应商逐请求实时硬上限。
|
||||
|
||||
本机到 POC 集群 API 的连接不稳定时(例如经 Wi‑Fi 或企业网络客户端),构建与发布会自动重试没有得到应答的请求:读取最多试 6 次,间隔 2 至 30 秒;上传源码失败会重传,解压可安全重复;`rollout status` 断开后在发布时限内接着观察。连接中断超过这些重试时,运行仍停在「基础设施」等待,网络恢复后在网页上继续即可。可以用 `kubectl --kubeconfig <POC kubeconfig> --request-timeout=5s get --raw /readyz` 反复探测这条链路。
|
||||
本机到 POC 集群 API 的连接不稳定时(例如经 Wi‑Fi 或企业网络客户端),构建与发布会自动重试没有得到应答的请求:读取最多试 6 次,间隔 2 至 30 秒;上传源码失败会重传,解压可安全重复;`rollout status` 断开后在发布时限内接着观察;发布后的公网检查(健康、就绪、首页、API 的构建提交和受保护业务检查)也各试 6 次,间隔 2 至 10 秒,因为滚动更新刚结束时公网入口可能还连到正在退出的旧 Pod。每次重试都有记录:构建时写进运行时间线,发布时写进 `runner.log`。连接中断超过这些重试时,运行仍停在「基础设施」等待,网络恢复后在网页上继续即可。可以用 `kubectl --kubeconfig <POC kubeconfig> --request-timeout=5s get --raw /readyz` 反复探测这条链路。
|
||||
|
||||
@ -30,8 +30,11 @@ The operator's machine may reach the cluster API over an unreliable link. A requ
|
||||
- Before each attempt to create its Pod, the builder looks the Pod up by its attempt-unique name, so a create whose answer was lost is adopted instead of repeated.
|
||||
- A lost source upload is sent again. Unpacking runs at most once at a time, refuses an upload whose SHA-256 differs from the archive (gzip's checksum covers images without `sha256sum`), and marks the context ready only after a complete unpack, so Kaniko never starts from a partial context. A damaged upload is sent again, up to three times.
|
||||
- A dropped `rollout status` watch resumes for the rest of the rollout timeout instead of failing the release and triggering a rollback.
|
||||
- The public checks after a rollout (health, readiness, the entry page, the API build commit and the protected business smoke checks) only read, and are repeated up to six times, waiting 2 to 10 seconds: just after a rollout the public address may still reach a Pod that is shutting down. A release whose checks never pass is still rolled back.
|
||||
- Patches are never repeated; an uncertain patch is still reconciled by observation.
|
||||
|
||||
Each repeat is announced: during a build in the run's timeline, during a release in the Runner's log.
|
||||
|
||||
A link that stays down longer still fails the build or release; the run then waits for a person as an infrastructure fault, and continuing it starts a new build attempt.
|
||||
|
||||
## Durable state and recovery
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
import { normalizeApplicationUrl, type ApplicationAccessEvidence, isProduction } from '@wishforge/contracts';
|
||||
import { requireCommit, safeIdentifier } from './git.js';
|
||||
import { repeat, type Patience } from './patience.js';
|
||||
import { RuntimeError, type ReleaseManifest, type TrustedPocDeliveryPolicy } from './types.js';
|
||||
|
||||
export function validateRelease(
|
||||
@ -182,5 +183,40 @@ export async function verifyHttp(
|
||||
return { passed: true, checks: ['health', 'business-smoke'] };
|
||||
}
|
||||
|
||||
/**
|
||||
* {@link verifyHttp}, repeated until it passes or has used its attempts: just after a switch the
|
||||
* target may still be settling, and a dropped connection says nothing about the release. The
|
||||
* checks only read. The last attempt's outcome is returned, or its error thrown.
|
||||
*/
|
||||
export async function verifyHttpPatiently(
|
||||
release: ReleaseManifest,
|
||||
base: string | undefined,
|
||||
fetcher: typeof fetch,
|
||||
patience: Patience,
|
||||
options: { stable?: boolean } = {},
|
||||
): ReturnType<typeof verifyHttp> {
|
||||
let last: Awaited<ReturnType<typeof verifyHttp>> | undefined;
|
||||
try {
|
||||
return await repeat(
|
||||
async () => {
|
||||
last = undefined;
|
||||
last = await verifyHttp(release, base, fetcher, options);
|
||||
if (!last.passed) throw new Error(last.reason ?? 'The HTTP checks did not pass');
|
||||
return last;
|
||||
},
|
||||
patience,
|
||||
{
|
||||
maxDelayMs: 10_000,
|
||||
// A malformed target address cannot get better.
|
||||
retryable: error => !(error instanceof RuntimeError && error.code === 'INVALID_HEALTH_URL'),
|
||||
describe: error => (error instanceof Error ? error.message : 'The HTTP checks failed'),
|
||||
},
|
||||
);
|
||||
} catch (error) {
|
||||
if (last) return last;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
export { ComposeDeploymentAdapter, KubernetesDeploymentAdapter, environmentResourceName } from './bluegreen.js';
|
||||
export type { ComposeTarget, KubernetesTarget } from './bluegreen.js';
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
export * from './types.js';
|
||||
export * from './process.js';
|
||||
export * from './patience.js';
|
||||
export * from './git.js';
|
||||
export * from './verify.js';
|
||||
export * from './build.js';
|
||||
|
||||
46
packages/runtime/src/patience.ts
Normal file
46
packages/runtime/src/patience.ts
Normal file
@ -0,0 +1,46 @@
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
|
||||
/** How often to repeat a request or check that may fail for reasons outside it: a flaky link, a target still settling. */
|
||||
export interface Patience {
|
||||
/** Attempts in all, the first included. */
|
||||
attempts: number;
|
||||
/** The first wait between attempts; it doubles, up to a limit the caller sets. */
|
||||
delayMs: number;
|
||||
/** Told before each repeat, to show it in a run's timeline or a log; its failures are ignored. */
|
||||
onRetry?: (notice: string) => unknown;
|
||||
}
|
||||
/** Six attempts wait a minute in all between them (2, 4, 8, 16 and 30 seconds), besides their own time limits. */
|
||||
export const DEFAULT_PATIENCE: Patience = { attempts: 6, delayMs: 2000 };
|
||||
|
||||
/**
|
||||
* Runs `attempt` until it succeeds, `retryable` refuses its failure, or the attempts are used up;
|
||||
* the last failure is then thrown. Only for attempts that are safe to repeat.
|
||||
*/
|
||||
export async function repeat<T>(
|
||||
attempt: () => Promise<T>,
|
||||
patience: Patience,
|
||||
options: {
|
||||
/** A short account of a failure, for the notice before the next attempt. */
|
||||
describe: (error: unknown) => string;
|
||||
retryable?: (error: unknown) => boolean;
|
||||
maxDelayMs?: number;
|
||||
signal?: AbortSignal;
|
||||
},
|
||||
): Promise<T> {
|
||||
for (let tried = 1, wait = patience.delayMs; ; tried++, wait = Math.min(wait * 2, options.maxDelayMs ?? 30_000)) {
|
||||
try {
|
||||
return await attempt();
|
||||
} catch (error) {
|
||||
if (tried >= patience.attempts || !(options.retryable?.(error) ?? true)) throw error;
|
||||
const seconds = Math.max(1, Math.round(wait / 1000));
|
||||
await Promise.resolve()
|
||||
.then(() =>
|
||||
patience.onRetry?.(
|
||||
`${options.describe(error)}; trying again in ${seconds} s (attempt ${tried + 1} of ${patience.attempts})`,
|
||||
),
|
||||
)
|
||||
.catch(() => undefined);
|
||||
await delay(wait, undefined, { signal: options.signal });
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -3,13 +3,13 @@ import { mkdir, readFile, stat } from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { requireCommit, safeIdentifier } from './git.js';
|
||||
import type { Patience } from './patience.js';
|
||||
import { executeCommand, type CommandExecutor } from './process.js';
|
||||
import {
|
||||
KubectlError,
|
||||
PocDeploymentAdapter,
|
||||
pocReleaseSchema,
|
||||
writePocJson,
|
||||
type KubectlPatience,
|
||||
type PocImages,
|
||||
type PocRelease,
|
||||
} from './poc-release.js';
|
||||
@ -39,8 +39,8 @@ export interface PocBuildInput {
|
||||
rollbackCompatible?: boolean;
|
||||
signal?: AbortSignal;
|
||||
onProgress?: (message: string) => void | Promise<void>;
|
||||
/** How long to keep repeating requests the cluster API leaves unanswered. */
|
||||
patience?: KubectlPatience;
|
||||
/** How often to repeat requests the cluster API leaves unanswered, and whom to tell. */
|
||||
patience?: Patience;
|
||||
}
|
||||
|
||||
interface BuildPod {
|
||||
|
||||
@ -4,6 +4,7 @@ import { createHash } from 'node:crypto';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { canonicalJson } from '@wishforge/contracts';
|
||||
import { z } from 'zod';
|
||||
import { DEFAULT_PATIENCE, repeat, type Patience } from './patience.js';
|
||||
import { CommandStopped, executeCommand, type CommandExecutor } from './process.js';
|
||||
import {
|
||||
pocComponentNames,
|
||||
@ -121,9 +122,13 @@ export class KubectlError extends Error {
|
||||
this.name = 'KubectlError';
|
||||
}
|
||||
}
|
||||
/** How kubectl reports a request the cluster API never answered. */
|
||||
/**
|
||||
* How kubectl reports a request the cluster API never answered. `get` says "Unable to connect to
|
||||
* the server"; `cp` and `exec` name the failed dial or read directly ("error: dial tcp …: connect:
|
||||
* operation timed out" on macOS, "connection timed out" on Linux).
|
||||
*/
|
||||
const UNANSWERED =
|
||||
/Unable to connect to the server|i\/o timeout|connection refused|connection reset by peer|broken pipe|no route to host|network is unreachable|TLS handshake timeout|context deadline exceeded|Client\.Timeout exceeded|http2: (?:client connection lost|server sent GOAWAY)|unexpected EOF|error dialing backend|the server was unable to return a response in the time allotted|the server is currently unable to handle the request|etcdserver: (?:request timed out|leader changed)/i;
|
||||
/Unable to connect to the server|dial tcp|i\/o timeout|operation timed out|connection timed out|connection refused|connection reset by peer|broken pipe|use of closed network connection|no route to host|network is unreachable|TLS handshake timeout|context deadline exceeded|Client\.Timeout exceeded|request canceled while waiting for connection|http2: (?:client connection lost|server sent GOAWAY)|unexpected EOF|error dialing backend|Timeout occurred|the server was unable to return a response in the time allotted|the server is currently unable to handle the request|has asked us to try again later|etcdserver: (?:request timed out|leader changed)/i;
|
||||
|
||||
/**
|
||||
* The cluster API did not answer: connecting failed, the connection broke or kubectl hung.
|
||||
@ -138,15 +143,11 @@ export function unanswered(error: unknown): boolean {
|
||||
!/command terminated with exit code/.test(error.stderr)
|
||||
);
|
||||
}
|
||||
/** How often to repeat a request the cluster API left unanswered, for a flaky link. */
|
||||
export interface KubectlPatience {
|
||||
/** Attempts in all, the first included. */
|
||||
attempts: number;
|
||||
/** The first wait between attempts; it doubles, up to 30 seconds. */
|
||||
delayMs: number;
|
||||
}
|
||||
/** Six attempts wait a minute in all between them, besides their own time limits. */
|
||||
const PATIENCE: KubectlPatience = { attempts: 6, delayMs: 2000 };
|
||||
/** What went unanswered, for a retry notice; kubectl's own first line says why. */
|
||||
const silence = (error: unknown) =>
|
||||
error instanceof KubectlError
|
||||
? `kubectl ${error.verb} got no answer from the cluster API (${error.stderr.trim().split('\n')[0]!.slice(0, 160)})`
|
||||
: 'kubectl got no answer from the cluster API in time';
|
||||
|
||||
/** Adoption updates only fixed container images and audit annotations, guarded by UID/RV/image tests. */
|
||||
export class PocDeploymentAdapter {
|
||||
@ -156,7 +157,7 @@ export class PocDeploymentAdapter {
|
||||
readonly stateDirectory: string,
|
||||
readonly execute: CommandExecutor = executeCommand,
|
||||
readonly fetcher: typeof fetch = fetch,
|
||||
readonly patience: KubectlPatience = PATIENCE,
|
||||
readonly patience: Patience = DEFAULT_PATIENCE,
|
||||
) {
|
||||
this.profile = pocProfileSchema.parse(profile);
|
||||
}
|
||||
@ -185,14 +186,12 @@ export class PocDeploymentAdapter {
|
||||
}
|
||||
/** Repeats `request` while the cluster API leaves it unanswered; only for requests safe to repeat. */
|
||||
async whileUnanswered<T>(request: () => Promise<T>): Promise<T> {
|
||||
for (let attempt = 1, wait = this.patience.delayMs; ; attempt++, wait = Math.min(wait * 2, 30_000)) {
|
||||
try {
|
||||
return await request();
|
||||
} catch (error) {
|
||||
if (!unanswered(error) || attempt >= this.patience.attempts) throw error;
|
||||
await delay(wait);
|
||||
}
|
||||
}
|
||||
return repeat(request, this.patience, { retryable: unanswered, describe: silence });
|
||||
}
|
||||
private async notify(notice: string) {
|
||||
await Promise.resolve()
|
||||
.then(() => this.patience.onRetry?.(notice))
|
||||
.catch(() => undefined);
|
||||
}
|
||||
async assertIdentity(): Promise<void> {
|
||||
const config = JSON.parse(await this.kubectl(['config', 'view', '--minify', '-o', 'json'])) as {
|
||||
@ -346,6 +345,9 @@ export class PocDeploymentAdapter {
|
||||
break;
|
||||
} catch (error) {
|
||||
if (!unanswered(error) || Date.now() + wait >= until) throw error;
|
||||
await this.notify(
|
||||
`${silence(error)} while watching the ${component} rollout; watching again in ${Math.round(wait / 1000)} s`,
|
||||
);
|
||||
await delay(wait);
|
||||
}
|
||||
}
|
||||
@ -378,18 +380,28 @@ export class PocDeploymentAdapter {
|
||||
)
|
||||
throw Error(`POC ${component} Pod digest/readiness mismatch`);
|
||||
}
|
||||
for (const endpoint of ['/api/health', '/api/ready', '/']) {
|
||||
const response = await this.fetcher(new URL(endpoint, this.profile.applicationUrl), {
|
||||
signal: AbortSignal.timeout(15_000),
|
||||
redirect: 'error',
|
||||
});
|
||||
if (response.status !== 200) throw Error(`POC ${endpoint} returned ${response.status}`);
|
||||
if (endpoint === '/api/health' && expectedApiSourceCommit) {
|
||||
const health = (await response.json()) as { buildCommit?: string };
|
||||
if (health.buildCommit !== expectedApiSourceCommit)
|
||||
throw Error('POC public API source commit does not match the deployed artifact');
|
||||
} else await response.body?.cancel();
|
||||
}
|
||||
// Just after a rollout the public address may still reach a Pod that is shutting down, and the
|
||||
// link may drop: a check fails only once it has failed every attempt.
|
||||
for (const endpoint of ['/api/health', '/api/ready', '/'])
|
||||
await repeat(
|
||||
async () => {
|
||||
const response = await this.fetcher(new URL(endpoint, this.profile.applicationUrl), {
|
||||
signal: AbortSignal.timeout(15_000),
|
||||
redirect: 'error',
|
||||
});
|
||||
if (response.status !== 200) {
|
||||
await response.body?.cancel();
|
||||
throw Error(`POC ${endpoint} returned ${response.status}`);
|
||||
}
|
||||
if (endpoint === '/api/health' && expectedApiSourceCommit) {
|
||||
const health = (await response.json()) as { buildCommit?: string };
|
||||
if (health.buildCommit !== expectedApiSourceCommit)
|
||||
throw Error('POC public API source commit does not match the deployed artifact');
|
||||
} else await response.body?.cancel();
|
||||
},
|
||||
this.patience,
|
||||
{ maxDelayMs: 10_000, describe: error => (error instanceof Error ? error.message : `POC ${endpoint} failed`) },
|
||||
);
|
||||
return {
|
||||
checkedAt: new Date().toISOString(),
|
||||
checks: [
|
||||
|
||||
@ -1,6 +1,7 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
import { canonicalJson } from '@wishforge/contracts';
|
||||
import { verifyHttp } from './deploy.js';
|
||||
import { verifyHttpPatiently } from './deploy.js';
|
||||
import { DEFAULT_PATIENCE, type Patience } from './patience.js';
|
||||
import { executeCommand, type CommandExecutor } from './process.js';
|
||||
import { PocDeploymentAdapter, pocReleaseSchema, type PocJournal, type PocRelease } from './poc-release.js';
|
||||
import { profileDigest, type PocProfile } from './poc-profile.js';
|
||||
@ -20,8 +21,10 @@ export class PocRunnerDeploymentAdapter implements DeploymentAdapter {
|
||||
stateDirectory: string,
|
||||
execute: CommandExecutor = executeCommand,
|
||||
private readonly fetcher: typeof fetch = fetch,
|
||||
/** For requests and checks a flaky link or a settling rollout can fail; tells the Runner's log. */
|
||||
private readonly patience: Patience = DEFAULT_PATIENCE,
|
||||
) {
|
||||
this.adapter = new PocDeploymentAdapter(profile, stateDirectory, execute, fetcher);
|
||||
this.adapter = new PocDeploymentAdapter(profile, stateDirectory, execute, fetcher, patience);
|
||||
}
|
||||
private manifest(release: ReleaseManifest): PocRelease {
|
||||
const inner = pocReleaseSchema.parse((release as ReleaseManifest & { pocRelease?: unknown }).pocRelease);
|
||||
@ -65,7 +68,7 @@ export class PocRunnerDeploymentAdapter implements DeploymentAdapter {
|
||||
);
|
||||
}
|
||||
private async verifyBusiness(release: ReleaseManifest): Promise<void> {
|
||||
const http = await verifyHttp(release, this.profile.applicationUrl, this.fetcher);
|
||||
const http = await verifyHttpPatiently(release, this.profile.applicationUrl, this.fetcher, this.patience);
|
||||
if (!http.passed) throw Error(http.reason ?? 'POC protected business smoke failed');
|
||||
}
|
||||
private async result(release: ReleaseManifest, journal: PocJournal): Promise<DeploymentResult> {
|
||||
@ -85,7 +88,7 @@ export class PocRunnerDeploymentAdapter implements DeploymentAdapter {
|
||||
bundleDigest: pocReleaseDigest(journal.release),
|
||||
},
|
||||
};
|
||||
const http = await verifyHttp(release, this.profile.applicationUrl, this.fetcher);
|
||||
const http = await verifyHttpPatiently(release, this.profile.applicationUrl, this.fetcher, this.patience);
|
||||
if (!http.passed)
|
||||
return {
|
||||
releaseId: release.id,
|
||||
|
||||
96
packages/runtime/test/patience.test.ts
Normal file
96
packages/runtime/test/patience.test.ts
Normal file
@ -0,0 +1,96 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { repeat } from '../src/patience.js';
|
||||
import { verifyHttpPatiently } from '../src/deploy.js';
|
||||
import type { ReleaseManifest } from '../src/types.js';
|
||||
|
||||
describe('repeating what a flaky link or a settling target can fail', () => {
|
||||
it('repeats a failing attempt until it passes, telling about each repeat', async () => {
|
||||
const notices: string[] = [];
|
||||
let calls = 0;
|
||||
const value = await repeat(
|
||||
async () => {
|
||||
if (++calls < 3) throw new Error(`no answer ${calls}`);
|
||||
return 'done';
|
||||
},
|
||||
{ attempts: 3, delayMs: 1, onRetry: notice => notices.push(notice) },
|
||||
{ describe: error => (error as Error).message },
|
||||
);
|
||||
expect(value).toBe('done');
|
||||
expect(notices).toEqual([
|
||||
'no answer 1; trying again in 1 s (attempt 2 of 3)',
|
||||
'no answer 2; trying again in 1 s (attempt 3 of 3)',
|
||||
]);
|
||||
});
|
||||
it('gives up after the last attempt, or at once on a failure it may not repeat', async () => {
|
||||
let calls = 0;
|
||||
const down = async () => {
|
||||
calls++;
|
||||
throw new Error('down');
|
||||
};
|
||||
await expect(repeat(down, { attempts: 3, delayMs: 1 }, { describe: () => 'down' })).rejects.toThrow('down');
|
||||
expect(calls).toBe(3);
|
||||
calls = 0;
|
||||
await expect(
|
||||
repeat(down, { attempts: 3, delayMs: 1 }, { describe: () => 'down', retryable: () => false }),
|
||||
).rejects.toThrow('down');
|
||||
expect(calls).toBe(1);
|
||||
});
|
||||
it('is not stopped by a notice that cannot be delivered', async () => {
|
||||
let calls = 0;
|
||||
const value = await repeat(
|
||||
async () => {
|
||||
if (++calls < 2) throw new Error('blip');
|
||||
return calls;
|
||||
},
|
||||
{
|
||||
attempts: 2,
|
||||
delayMs: 1,
|
||||
onRetry: () => {
|
||||
throw new Error('timeline unavailable');
|
||||
},
|
||||
},
|
||||
{ describe: () => 'blip' },
|
||||
);
|
||||
expect(value).toBe(2);
|
||||
});
|
||||
it('repeats HTTP checks until they pass, and reports the last failure when they never do', async () => {
|
||||
const release: ReleaseManifest = {
|
||||
schemaVersion: 1,
|
||||
id: 'release',
|
||||
tenantId: 'tenant',
|
||||
environmentId: 'env',
|
||||
sourceCommit: 'a'.repeat(40),
|
||||
image: `registry.example/web@sha256:${'b'.repeat(64)}`,
|
||||
containerPort: 3000,
|
||||
healthPath: '/health',
|
||||
smokeChecks: [{ path: '/catalog', expectedStatus: 200 }],
|
||||
};
|
||||
let health = 0;
|
||||
// The first health check meets a dropped connection, the second a target still starting.
|
||||
const settling: typeof fetch = async url => {
|
||||
if (new URL(String(url)).pathname === '/health' && ++health === 1) throw new TypeError('fetch failed');
|
||||
return new Response('ok', { status: health === 2 ? 503 : 200 });
|
||||
};
|
||||
const patience = { attempts: 3, delayMs: 1 };
|
||||
expect(await verifyHttpPatiently(release, 'https://app.example.test/', settling, patience)).toMatchObject({
|
||||
passed: true,
|
||||
});
|
||||
expect(health).toBe(3);
|
||||
let calls = 0;
|
||||
const broken: typeof fetch = async () => {
|
||||
calls++;
|
||||
return new Response('', { status: 503 });
|
||||
};
|
||||
expect(await verifyHttpPatiently(release, 'https://app.example.test/', broken, patience)).toEqual({
|
||||
passed: false,
|
||||
checks: [],
|
||||
reason: 'Health returned 503',
|
||||
});
|
||||
expect(calls).toBe(patience.attempts);
|
||||
// A target address that is not a web address cannot get better.
|
||||
await expect(verifyHttpPatiently(release, 'ftp://app.example.test/', broken, patience)).rejects.toMatchObject({
|
||||
code: 'INVALID_HEALTH_URL',
|
||||
});
|
||||
expect(calls).toBe(patience.attempts);
|
||||
});
|
||||
});
|
||||
@ -365,6 +365,7 @@ describe('the POC build and release on a flaky link to the cluster', () => {
|
||||
const digest = `sha256:${'d'.repeat(64)}`;
|
||||
const pods = new Map<string, any>();
|
||||
const requests: string[][] = [];
|
||||
const notices: string[] = [];
|
||||
const faults: Record<
|
||||
string,
|
||||
{ exitCode?: number; stderr: string; lands?: boolean; when?: (args: string[]) => boolean }[]
|
||||
@ -424,7 +425,7 @@ describe('the POC build and release on a flaky link to the cluster', () => {
|
||||
releaseId: 'run-1-build',
|
||||
stateDirectory: f.root,
|
||||
components: ['api'],
|
||||
patience,
|
||||
patience: { ...patience, onRetry: notice => notices.push(notice) },
|
||||
},
|
||||
execute,
|
||||
);
|
||||
@ -436,6 +437,7 @@ describe('the POC build and release on a flaky link to the cluster', () => {
|
||||
pods,
|
||||
faults,
|
||||
requests,
|
||||
notices,
|
||||
count: (verb: string) => requests.filter(args => args[0] === verb).length,
|
||||
};
|
||||
}
|
||||
@ -448,6 +450,16 @@ describe('the POC build and release on a flaky link to the cluster', () => {
|
||||
expect(
|
||||
unanswered(new CommandStopped('TIMEOUT', 'Command exceeded its time limit', { stdout: '', stderr: '' })),
|
||||
).toBe(true);
|
||||
// As kubectl cp reported them on the POC link, without the "Unable to connect" prefix of get.
|
||||
for (const stderr of [
|
||||
'error: dial tcp 120.48.181.105:6443: connect: operation timed out',
|
||||
'E1026 01:06:17.504268 6886 v2.go:129] "Unhandled Error" err="next reader: read tcp 192.168.6.207:59375->120.48.181.105:6443: i/o timeout"',
|
||||
'error: dial tcp 10.0.0.1:6443: connect: connection timed out',
|
||||
])
|
||||
expect(unanswered(new KubectlError('cp', 1, stderr))).toBe(true);
|
||||
expect(unanswered(new KubectlError('exec', 1, 'error: unable to upgrade connection: container not found'))).toBe(
|
||||
false,
|
||||
);
|
||||
expect(unanswered(new KubectlError('get', 1, 'Error from server (NotFound): pods "build" not found'))).toBe(false);
|
||||
expect(unanswered(new KubectlError('rollout', 1, 'error: timed out waiting for the condition'))).toBe(false);
|
||||
// The command in the Pod ran; what it printed does not make its failure a lost answer.
|
||||
@ -473,6 +485,11 @@ describe('the POC build and release on a flaky link to the cluster', () => {
|
||||
expect(c.count('cp')).toBe(2);
|
||||
expect(c.count('exec')).toBe(3);
|
||||
expect(c.pods.size).toBe(0);
|
||||
// Each repeat is told, for the run's timeline.
|
||||
expect(c.notices).toContain(
|
||||
`kubectl get got no answer from the cluster API (${unreachable}); trying again in 1 s (attempt 2 of 4)`,
|
||||
);
|
||||
expect(c.notices).toHaveLength(4);
|
||||
});
|
||||
it('repeats a lookup that never gets through only as often as the patience allows', async () => {
|
||||
const c = await cluster();
|
||||
@ -519,4 +536,62 @@ describe('the POC build and release on a flaky link to the cluster', () => {
|
||||
expect(dropped).toBe(600);
|
||||
expect(resumed).toBeLessThanOrEqual(600);
|
||||
});
|
||||
it('accepts a release once the public address serves the new build, not at its first answer', async () => {
|
||||
const f = await fixture();
|
||||
const notices: string[] = [];
|
||||
let health = 0;
|
||||
// Right after the rollout the public address first fails, then still reaches the old Pod.
|
||||
const settling: typeof fetch = async url => {
|
||||
if (new URL(String(url)).pathname !== '/api/health') return new Response('ok');
|
||||
health++;
|
||||
if (health === 1) return new Response('', { status: 502 });
|
||||
return Response.json({ buildCommit: health === 2 ? 'f'.repeat(40) : 'a'.repeat(40) });
|
||||
};
|
||||
const adapter = new PocDeploymentAdapter(profile, f.root, f.execute, settling, {
|
||||
...patience,
|
||||
onRetry: notice => notices.push(notice),
|
||||
});
|
||||
const result = await adapter.deploy(release());
|
||||
expect(result.state).toBe('succeeded');
|
||||
expect(notices).toEqual([
|
||||
'POC /api/health returned 502; trying again in 1 s (attempt 2 of 4)',
|
||||
'POC public API source commit does not match the deployed artifact; trying again in 1 s (attempt 3 of 4)',
|
||||
]);
|
||||
});
|
||||
it('still rolls back a release whose public API never serves the new build', async () => {
|
||||
const f = await fixture();
|
||||
const stale: typeof fetch = async () => Response.json({ buildCommit: 'f'.repeat(40) });
|
||||
const adapter = new PocDeploymentAdapter(profile, f.root, f.execute, stale, patience);
|
||||
const result = await adapter.deploy(release());
|
||||
expect(result.state).toBe('rolled_back');
|
||||
expect(result.error).toContain('source commit does not match');
|
||||
});
|
||||
it('lets the Runner repeat its protected checks while the release settles', async () => {
|
||||
const f = await fixture();
|
||||
const inner = release({ expectedTemplateDigests: (await f.adapter.snapshot()).templateDigests });
|
||||
const manifest: ReleaseManifest & { pocRelease: PocRelease } = {
|
||||
schemaVersion: 1,
|
||||
id: 'outer-release-1',
|
||||
tenantId: 'tenant',
|
||||
environmentId: 'poc',
|
||||
sourceCommit: inner.sourceCommit,
|
||||
image: inner.images.api,
|
||||
architecture: 'linux/amd64',
|
||||
containerPort: 3001,
|
||||
healthPath: '/api/health',
|
||||
applicationUrl: profile.applicationUrl,
|
||||
smokeChecks: [{ path: '/api/ready', expectedStatus: 200 }],
|
||||
pocRelease: inner,
|
||||
};
|
||||
let ready = 0;
|
||||
// The adapter's own readiness check and the Runner's protected smoke check each meet one 503.
|
||||
const fetcher: typeof fetch = async url => {
|
||||
if (new URL(String(url)).pathname === '/api/ready' && [1, 3].includes(++ready))
|
||||
return new Response('', { status: 503 });
|
||||
return Response.json({ buildCommit: 'a'.repeat(40) });
|
||||
};
|
||||
const wrapper = new PocRunnerDeploymentAdapter(profile, f.root, f.execute, fetcher, patience);
|
||||
expect((await wrapper.deploy(manifest, { execute: true })).status).toBe('verified');
|
||||
expect(ready).toBe(5);
|
||||
});
|
||||
});
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user