🐛 Treat POC images deployed by tag as of unknown source
Someone deployed the POC api and worker outside WishForge by mutable tags, and every build then stopped with a raw validation error: the cluster snapshot only accepted digest-pinned images. A component running a tag is now of unknown source, as one set by hand already was: the timeline names the component and its tag, the build makes it again, and the release needs a person's approval. A release still deploys digests only; what it replaces may be a tag of the same repository, and such a release never declares itself rollback-compatible, since what the tag pointed at may be gone. An image neither pinned nor tagged, or not of the component's repository, stops the build with a message naming it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
f9eb1f5dfd
commit
44b1bee723
@ -1,6 +1,6 @@
|
||||
import type { Project, PocReleaseSet } from '@wishforge/contracts';
|
||||
import { DomainError } from '@wishforge/contracts';
|
||||
import { readLocalPocBinding, localPocForProject, loadPocProfile, profileDigest } from '@wishforge/runtime';
|
||||
import { imageOf, readLocalPocBinding, localPocForProject, loadPocProfile, profileDigest } from '@wishforge/runtime';
|
||||
|
||||
export function boundPocProject(project: Project, env: NodeJS.ProcessEnv) {
|
||||
return localPocForProject(readLocalPocBinding(env.WF_LOCAL_POC_BINDING_FILE, env), project);
|
||||
@ -30,9 +30,12 @@ export async function assertPocBuild(
|
||||
worker: 'droboticflow-worker',
|
||||
coordinator: 'drosync-coordinator',
|
||||
})) {
|
||||
for (const images of [release.images, release.expectedImages]) {
|
||||
if (!images[component as keyof typeof images].startsWith(`${profile.registry}/${repository}@sha256:`))
|
||||
throw new DomainError('POC_IMAGE_SCOPE', 'A POC component image is outside the fixed registry repository');
|
||||
}
|
||||
const key = component as keyof typeof release.images;
|
||||
// A release deploys digests only; what it replaces may be a tag someone deployed by hand.
|
||||
if (
|
||||
!release.images[key].startsWith(`${profile.registry}/${repository}@sha256:`) ||
|
||||
!imageOf(release.expectedImages[key], `${profile.registry}/${repository}`)
|
||||
)
|
||||
throw new DomainError('POC_IMAGE_SCOPE', 'A POC component image is outside the fixed registry repository');
|
||||
}
|
||||
}
|
||||
|
||||
@ -134,5 +134,5 @@ LaunchAgent 随当前用户登录运行,退出登录后停止;电脑关机
|
||||
- 推到 `poc` 的提交让受保护检查失败时:合入的合并结果检查失败后,会再单独检查不带本次改动的 `poc` 和运行的起点。确认是之后推送的提交弄坏了 `poc`,运行停在「目标分支自身未通过检查」等待,不让 Claude 去改别人的代码,也不消耗修复轮数;修好 `poc` 后点「继续」,运行把已验证的改动合并到最新的 `poc` 上并重新检查。这些对照检查每次约 2.5 分钟,只在合并结果失败时才跑。
|
||||
- 运行合入之后有人推了 `poc`:构建照常进行,构建的是这个运行合入的提交,时间线会说明 `poc` 已经前进;更新的提交由之后的运行部署。只有 `poc` 被强推、不再包含这个提交时,运行才停在「仓库」等待(`POC_SOURCE_REWRITTEN`),这时取消运行,需要的话重新提交愿望。不要强推 `poc`:进行中的运行合入时会把删掉的提交重新带回来。
|
||||
- 发布的风险按这次要替换的组件当前运行的代码到这次发布之间的全部变化计算,直接推到 `poc` 的改动同样会触发审批(迁移、鉴权、Dockerfile、依赖等);沿用原镜像的组件这次不变,不计入。
|
||||
- 每次发布在 Deployment 上记下组件镜像构建自哪个提交和镜像本身。构建前按这些记录逐个判断:worker、coordinator 运行的代码与本次源码之间只有网页文件不同时沿用原镜像,否则重新构建。没有记录、记录已被回滚覆盖、或镜像被人用 `kubectl set image` 改过的组件视为来源不明:会重新构建,并且这次发布必须人工审批、不自动回滚。
|
||||
- 每次发布在 Deployment 上记下组件镜像构建自哪个提交和镜像本身。构建前按这些记录逐个判断:worker、coordinator 运行的代码与本次源码之间只有网页文件不同时沿用原镜像,否则重新构建。没有记录、记录已被回滚覆盖、或镜像被人用 `kubectl set image` 改过的组件视为来源不明:会重新构建,并且这次发布必须人工审批、不自动回滚。用 tag 而不是摘要部署的镜像同样视为来源不明(发布记录里的 `expectedImages` 可以是同一镜像仓库的 tag,`images` 仍只能是摘要);既不是 tag 也不是摘要、或不属于该组件镜像仓库的镜像,构建会停下并说明是哪个组件。
|
||||
- 因此升级到这个版本后的第一次 POC 发布会重建全部三个组件,并等待一次人工审批:此前 WishForge 只重建过 api,worker 和 coordinator 仍运行接入 POC 之前的镜像。
|
||||
|
||||
@ -417,7 +417,7 @@ POC 打开了 Seatbelt 沙箱,受保护检查在 macOS 沙箱里运行:读
|
||||
- **api**:每次都重建;
|
||||
- **worker、coordinator**:只有它们实际运行的代码与这次的代码之间,除了网页文件还有别的不同时,才重新构建,否则沿用原来的镜像。
|
||||
|
||||
每次发布都会在集群上记下每个组件的镜像构建自哪个提交。没有记录、被回滚覆盖、或者有人用 `kubectl set image` 手动改过镜像的组件,会被当作「来源不明」:重新构建,并且这次发布必须人工审批。
|
||||
每次发布都会在集群上记下每个组件的镜像构建自哪个提交。没有记录、被回滚覆盖、或者有人用 `kubectl set image` 手动改过镜像的组件,会被当作「来源不明」:重新构建,并且这次发布必须人工审批。有人在 WishForge 之外用 tag(例如 `droboticflow-api:0c776c04-poc-…`,而不是 `@sha256:` 摘要)部署的镜像同样当作来源不明,时间线里会写明是哪个组件、哪个 tag;这次发布部署的仍是摘要固定的镜像,并且失败时不会自动回滚到那个 tag(tag 指向的内容可能已经变了)。
|
||||
|
||||
### 6.5 发布与验收
|
||||
|
||||
@ -534,7 +534,7 @@ POC 环境默认没有勾选「每次发布都需人工确认」,所以只改
|
||||
|
||||
1. **不要强推(force push)`poc`**。进行中的运行合入时会把你删掉的提交重新带回来;已经合入的运行会停在「项目仓库未就绪」,只能取消。
|
||||
2. **推 `poc` 之前自己先跑一遍检查**,保持分支是好的,否则别人的运行会停在「目标分支自身未通过检查」。
|
||||
3. **不要手动改 POC 集群里的镜像**。改过的组件会被当作来源不明,下次发布要全部重建并人工审批。
|
||||
3. **不要手动改 POC 集群里的镜像**。改过的组件(包括用 tag 部署的)会被当作来源不明,下次发布要重建并人工审批,而且那次发布不能自动回滚。确实需要手动部署时,事后让 WishForge 再发布一次,组件就会回到有记录的状态。
|
||||
4. 想知道线上运行的是不是 `poc` 的最新提交,让维护者跑一次 `doctor`,看 `application.runsBranchHead`。
|
||||
|
||||
---
|
||||
|
||||
@ -1,6 +1,10 @@
|
||||
import { z } from 'zod';
|
||||
|
||||
const immutableImage = z.string().regex(/^[a-zA-Z0-9][a-zA-Z0-9._:/-]*@sha256:[a-f0-9]{64}$/);
|
||||
/** What a component ran when the release was built: a digest, or a tag someone deployed outside WishForge. */
|
||||
const deployedImage = z
|
||||
.string()
|
||||
.regex(/^[a-zA-Z0-9][a-zA-Z0-9._:/-]*(?:@sha256:[a-f0-9]{64}|:[A-Za-z0-9_][A-Za-z0-9._-]{0,127})$/);
|
||||
const imageSet = z.object({ api: immutableImage, worker: immutableImage, coordinator: immutableImage }).strict();
|
||||
const digest = z.string().regex(/^sha256:[a-f0-9]{64}$/);
|
||||
/** The entire component set, including unchanged components, participates in approval and runner signatures. */
|
||||
@ -11,7 +15,7 @@ export const pocReleaseInput = z
|
||||
sourceCommit: z.string().regex(/^[a-f0-9]{40}$/),
|
||||
profileDigest: digest,
|
||||
images: imageSet,
|
||||
expectedImages: imageSet,
|
||||
expectedImages: z.object({ api: deployedImage, worker: deployedImage, coordinator: deployedImage }).strict(),
|
||||
expectedTemplateDigests: z.object({ api: digest, worker: digest, coordinator: digest }).strict().optional(),
|
||||
rollbackCompatible: z.boolean(),
|
||||
})
|
||||
|
||||
@ -7,6 +7,7 @@ import type { Patience } from './patience.js';
|
||||
import { executeCommand, type CommandExecutor } from './process.js';
|
||||
import { RuntimeError } from './types.js';
|
||||
import {
|
||||
imageOf,
|
||||
KubectlError,
|
||||
PocDeploymentAdapter,
|
||||
pocReleaseSchema,
|
||||
@ -230,6 +231,19 @@ export async function buildPocRelease(
|
||||
const baseline = await adapter.snapshot();
|
||||
if (pocComponentNames.some(c => !baseline.deployments[c].ready))
|
||||
throw Error('POC must be fully ready before a build');
|
||||
// Someone deployed these outside WishForge by a tag: their source is unknown, so the plan builds
|
||||
// them again, and the release needs a person's approval and cannot roll back to them.
|
||||
const unpinned = pocComponentNames.filter(c => !baseline.deployments[c].pinned);
|
||||
for (const component of unpinned) {
|
||||
const repository = `${input.profile.registry}/${pocComponents[component].repository}`;
|
||||
if (!imageOf(baseline.images[component], repository))
|
||||
throw Error(
|
||||
`POC ${component} runs ${baseline.images[component]}, which is not an image of ${repository}; deploy one of its images, then continue`,
|
||||
);
|
||||
await input.onProgress?.(
|
||||
`${component} runs ${baseline.images[component]}, which someone deployed outside WishForge by a tag: its source is unknown, so it is built again, and this release needs approval and cannot roll back to it`,
|
||||
);
|
||||
}
|
||||
const buildDirectory = path.join(input.stateDirectory, 'builds', input.releaseId);
|
||||
// Reusing an ID with an incomplete build is deliberately refused; preserve the evidence for inspection.
|
||||
await mkdir(path.dirname(buildDirectory), { recursive: true, mode: 0o700 });
|
||||
@ -287,6 +301,9 @@ export async function buildPocRelease(
|
||||
rollbackCompatible: input.rollbackCompatible ?? false,
|
||||
};
|
||||
const components = requireComponents(plan.components);
|
||||
const kept = unpinned.filter(c => !components.includes(c));
|
||||
if (kept.length)
|
||||
throw Error(`POC ${kept.join(', ')} runs an image deployed by a tag, which a release cannot keep; build it again`);
|
||||
const archive = path.join(buildDirectory, 'source.tar.gz');
|
||||
await command('git', ['archive', '--format=tar.gz', `--output=${archive}`, input.sourceCommit]);
|
||||
if ((await stat(archive)).size > 512 * 1024 * 1024) throw Error('POC source archive exceeds 512 MiB');
|
||||
@ -424,7 +441,7 @@ export async function buildPocRelease(
|
||||
images,
|
||||
expectedImages: baseline.images,
|
||||
expectedTemplateDigests: baseline.templateDigests,
|
||||
rollbackCompatible: plan.rollbackCompatible,
|
||||
rollbackCompatible: plan.rollbackCompatible && !unpinned.length,
|
||||
});
|
||||
evidence.completedAt = new Date().toISOString();
|
||||
evidence.bundleDigest = pocReleaseDigest(release);
|
||||
|
||||
@ -16,7 +16,22 @@ import {
|
||||
} from './poc-profile.js';
|
||||
|
||||
const digestImage = z.string().regex(/^[a-zA-Z0-9][a-zA-Z0-9._:/-]*@sha256:[a-f0-9]{64}$/);
|
||||
/**
|
||||
* What a component may run when a release starts: an image pinned by digest, or one a person
|
||||
* deployed outside WishForge by a mutable tag. A release only ever deploys digests.
|
||||
*/
|
||||
const deployedImage = z
|
||||
.string()
|
||||
.regex(/^[a-zA-Z0-9][a-zA-Z0-9._:/-]*(?:@sha256:[a-f0-9]{64}|:[A-Za-z0-9_][A-Za-z0-9._-]{0,127})$/);
|
||||
/** Whether an image names its exact content: a tag can point at anything later. */
|
||||
export const pinnedImage = (image: string) => digestImage.safeParse(image).success;
|
||||
/** Whether an image is one of `repository`'s, by digest or by tag. */
|
||||
export const imageOf = (image: string, repository: string) =>
|
||||
image.startsWith(`${repository}@sha256:`) ||
|
||||
(image.startsWith(`${repository}:`) &&
|
||||
/^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$/.test(image.slice(repository.length + 1)));
|
||||
const imageSet = z.object({ api: digestImage, worker: digestImage, coordinator: digestImage }).strict();
|
||||
const deployedImageSet = z.object({ api: deployedImage, worker: deployedImage, coordinator: deployedImage }).strict();
|
||||
const templateDigestSet = z
|
||||
.object({
|
||||
api: z.string().regex(/^sha256:[a-f0-9]{64}$/),
|
||||
@ -31,7 +46,7 @@ export const pocReleaseSchema = z
|
||||
sourceCommit: z.string().regex(/^[a-f0-9]{40}$/),
|
||||
profileDigest: z.string().regex(/^sha256:[a-f0-9]{64}$/),
|
||||
images: imageSet,
|
||||
expectedImages: imageSet,
|
||||
expectedImages: deployedImageSet,
|
||||
expectedTemplateDigests: templateDigestSet.optional(),
|
||||
rollbackCompatible: z.boolean(),
|
||||
})
|
||||
@ -59,6 +74,7 @@ interface Deployment {
|
||||
};
|
||||
}
|
||||
export interface PocSnapshot {
|
||||
/** What each component runs; see deployments[component].pinned for whether that is a digest. */
|
||||
images: PocImages;
|
||||
templateDigests: z.infer<typeof templateDigestSet>;
|
||||
deployments: Record<
|
||||
@ -68,6 +84,8 @@ export interface PocSnapshot {
|
||||
resourceVersion: string;
|
||||
generation: number;
|
||||
ready: boolean;
|
||||
/** False when it runs a tag someone deployed outside WishForge: its source is unknown. */
|
||||
pinned: boolean;
|
||||
/** The commit its image was built from, while it runs the image a release put there. */
|
||||
sourceCommit?: string;
|
||||
}
|
||||
@ -246,17 +264,25 @@ export class PocDeploymentAdapter {
|
||||
const deployments = {} as PocSnapshot['deployments'];
|
||||
for (const component of pocComponentNames) {
|
||||
const item = await this.deployment(component);
|
||||
images[component] = digestImage.parse(this.image(component, item));
|
||||
const image = this.image(component, item);
|
||||
if (!deployedImage.safeParse(image).success)
|
||||
throw Error(
|
||||
`POC ${component} runs image ${JSON.stringify(image).slice(0, 300)}, which is pinned neither by digest nor by tag; deploy a tagged or digest-pinned image of it, then continue`,
|
||||
);
|
||||
images[component] = image;
|
||||
templateDigests[component] = pocTemplateDigest(component, item);
|
||||
const annotations = item.spec.template.metadata?.annotations ?? {};
|
||||
const sourceCommit = annotations[`${annotationPrefix}/source-commit`] ?? '';
|
||||
const pinned = pinnedImage(image);
|
||||
deployments[component] = {
|
||||
uid: item.metadata.uid,
|
||||
resourceVersion: item.metadata.resourceVersion,
|
||||
generation: item.metadata.generation,
|
||||
ready: this.ready(item),
|
||||
// A person's `kubectl set image` leaves the annotations as they were; the image tells.
|
||||
...(/^[0-9a-f]{40}$/.test(sourceCommit) && annotations[`${annotationPrefix}/image`] === images[component]
|
||||
pinned,
|
||||
// A person's `kubectl set image` leaves the annotations as they were; the image tells. A
|
||||
// tag tells nothing: what it pointed at when deployed is unknown.
|
||||
...(pinned && /^[0-9a-f]{40}$/.test(sourceCommit) && annotations[`${annotationPrefix}/image`] === image
|
||||
? { sourceCommit }
|
||||
: {}),
|
||||
};
|
||||
@ -274,10 +300,16 @@ export class PocDeploymentAdapter {
|
||||
pocReleaseSchema.parse(release);
|
||||
if (release.profileDigest !== profileDigest(this.profile)) throw Error('Release profile digest mismatch');
|
||||
for (const component of pocComponentNames) {
|
||||
const prefix = `${this.profile.registry}/${pocComponents[component].repository}@sha256:`;
|
||||
if (!release.images[component].startsWith(prefix) || !release.expectedImages[component].startsWith(prefix))
|
||||
const repository = `${this.profile.registry}/${pocComponents[component].repository}`;
|
||||
if (
|
||||
!release.images[component].startsWith(`${repository}@sha256:`) ||
|
||||
!imageOf(release.expectedImages[component], repository)
|
||||
)
|
||||
throw Error(`POC ${component} image repository mismatch`);
|
||||
}
|
||||
// What a tag pointed at is gone once replaced: nothing can roll back to it.
|
||||
if (release.rollbackCompatible && pocComponentNames.some(c => !pinnedImage(release.expectedImages[c])))
|
||||
throw Error('A release that replaces an image deployed by tag cannot roll back to it');
|
||||
}
|
||||
journalPath(id: string): string {
|
||||
if (!/^[a-z0-9][a-z0-9-]{0,62}$/.test(id)) throw Error('Invalid release ID');
|
||||
|
||||
@ -669,6 +669,58 @@ describe('what a POC build builds after people changed the branch or the cluster
|
||||
expect(evidence.retainedImages).toEqual({ worker: images('a').worker });
|
||||
expect(release).toMatchObject({ rollbackCompatible: true, expectedImages: images('a') });
|
||||
});
|
||||
it('builds a component someone deployed by a tag again, and never rolls back to the tag', async () => {
|
||||
const c = await cluster();
|
||||
const tag = `${profile.registry}/${pocComponents.worker.repository}:30d91677-poc-20261027162554`;
|
||||
// A person deployed a tag over the worker; even annotations naming it do not vouch for a tag.
|
||||
annotate(c.state.worker, 'b'.repeat(40), tag);
|
||||
c.state.worker.spec.template.spec.containers[0].image = tag;
|
||||
const snapshot = await c.adapter.snapshot();
|
||||
expect(snapshot.images.worker).toBe(tag);
|
||||
expect(snapshot.deployments.worker).toMatchObject({ pinned: false });
|
||||
expect(snapshot.deployments.worker.sourceCommit).toBeUndefined();
|
||||
expect(snapshot.deployments.api).toMatchObject({ pinned: true });
|
||||
// A plan that keeps the tag is refused: a release deploys digests only.
|
||||
await expect(
|
||||
c.build({ components: undefined, plan: async () => ({ components: ['api'], rollbackCompatible: true }) }),
|
||||
).rejects.toThrow('deployed by a tag');
|
||||
let running: Partial<Record<PocComponent, string>> | undefined;
|
||||
const { release } = await c.build({
|
||||
releaseId: 'run-2-build',
|
||||
components: undefined,
|
||||
plan: async seen => {
|
||||
running = seen;
|
||||
return { components: ['api', 'worker'], rollbackCompatible: true };
|
||||
},
|
||||
});
|
||||
expect(running?.worker).toBeUndefined();
|
||||
expect(release).toMatchObject({ rollbackCompatible: false, expectedImages: { ...images('a'), worker: tag } });
|
||||
expect(release.images.worker).toBe(`${profile.registry}/${pocComponents.worker.repository}@${c.digest}`);
|
||||
expect(
|
||||
c.progress.some(message =>
|
||||
message.startsWith(`worker runs ${tag}, which someone deployed outside WishForge by a tag`),
|
||||
),
|
||||
).toBe(true);
|
||||
// Deploying replaces exactly the tag that ran, and the components run digests again.
|
||||
const deployer = new PocDeploymentAdapter(profile, path.join(c.root, 'deployments'), c.execute, async () =>
|
||||
Response.json({ buildCommit: c.commit }),
|
||||
);
|
||||
expect((await deployer.deploy(release)).state).toBe('succeeded');
|
||||
expect(c.state.worker.spec.template.spec.containers[0].image).toBe(release.images.worker);
|
||||
expect((await c.adapter.snapshot()).deployments.worker).toMatchObject({ pinned: true, sourceCommit: c.commit });
|
||||
// A release that would roll back to a tag is refused before anything changes.
|
||||
await expect(deployer.deploy({ ...release, id: 'run-3', rollbackCompatible: true })).rejects.toThrow(
|
||||
'cannot roll back',
|
||||
);
|
||||
});
|
||||
it('stops at an image that is neither pinned nor tagged, or not one of the POC repositories', async () => {
|
||||
const c = await cluster();
|
||||
c.state.coordinator.spec.template.spec.containers[0].image = `${profile.registry}/${pocComponents.coordinator.repository}`;
|
||||
await expect(c.adapter.snapshot()).rejects.toThrow('pinned neither by digest nor by tag');
|
||||
c.state.coordinator.spec.template.spec.containers[0].image = 'docker.io/library/nginx:latest';
|
||||
await expect(c.build({ components: ['api', 'coordinator'] })).rejects.toThrow('is not an image of');
|
||||
expect(c.count('create')).toBe(0);
|
||||
});
|
||||
it('keeps the records of earlier builds, and only their recent source archives', async () => {
|
||||
const c = await cluster();
|
||||
const builds = path.join(c.root, 'builds');
|
||||
|
||||
@ -5,6 +5,7 @@ import path from 'node:path';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import {
|
||||
matchesApplicationAccess,
|
||||
pocReleaseInput,
|
||||
type Actor,
|
||||
type Release,
|
||||
type Run,
|
||||
@ -355,6 +356,23 @@ describe('operator-bound local POC delivery', () => {
|
||||
f.evidence({ type: 'build', passed: true, imageDigest: images('c').api, pocRelease: f.poc }),
|
||||
).rejects.toMatchObject({ code: 'POC_BUILD_MISMATCH' });
|
||||
});
|
||||
it('accepts a release replacing a tag someone deployed, only of the POC repositories', async () => {
|
||||
const f = await fixture();
|
||||
const tag = `${f.profile.registry}/droboticflow-worker:30d91677-poc-20261027162554`;
|
||||
const replacing = { ...f.poc, expectedImages: { ...f.poc.expectedImages, worker: tag }, rollbackCompatible: false };
|
||||
expect(pocReleaseInput.safeParse(replacing).success).toBe(true);
|
||||
await expect(assertPocBuild(f.project, f.environment.id, replacing, f.env)).resolves.toBeUndefined();
|
||||
// A release deploys digests only.
|
||||
expect(pocReleaseInput.safeParse({ ...replacing, images: { ...f.poc.images, worker: tag } }).success).toBe(false);
|
||||
await expect(
|
||||
assertPocBuild(
|
||||
f.project,
|
||||
f.environment.id,
|
||||
{ ...replacing, expectedImages: { ...replacing.expectedImages, worker: 'docker.io/library/nginx:latest' } },
|
||||
f.env,
|
||||
),
|
||||
).rejects.toMatchObject({ code: 'POC_IMAGE_SCOPE' });
|
||||
});
|
||||
it('invalidates a release when the operator profile changes after build', async () => {
|
||||
const f = await fixture();
|
||||
const release = await f.prepare();
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user