diff --git a/AGENTS.md b/AGENTS.md index cebe35c2..fa214eae 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,6 +21,11 @@ - When push is approved, push the user's personal feature branch only; never push `dev` or `poc` directly. - Do not stage or commit unrelated user changes already present in the working tree. +## Release Workflow + +- For release preparation, deployment, rollback, or resuming release work, read `docs/release/README.md` first and run `python3 scripts/release.py status ` when a record exists. Follow the recorded state and evidence; a successful rollout alone is not business acceptance. +- Fanyu image releases use `scripts/release.py`. Configuration changes or migrations outside its supported scope require a separately reviewed plan and the corresponding user authorization. + ## Credentials - Never hardcode credentials (BOS AK/SK, Mongo, Gitea token, cluster kubeconfig) in any tracked file. diff --git a/docs/release/README.md b/docs/release/README.md new file mode 100644 index 00000000..6a065db5 --- /dev/null +++ b/docs/release/README.md @@ -0,0 +1,155 @@ +# 泛宇发布试行流程 + +## 开始或恢复工作 + +发布、部署故障、回退、上下文恢复时先读本页。状态的权威来源是 +`local-data/releases//release.json`。先运行 `python3 scripts/release.py status` 列出记录, +再用 `python3 scripts/release.py status ` 查看缺失证据与下一步。 +历史聊天和“Pod 已 Ready”不代表业务验收完成。使用命令更新记录,保留失败和未验证项。 + +首版只支持 **Fanyu 现有 API 与切片处理器的同一不可变镜像更新**,并同步容器的 +`DROBOTICFLOW_BUILD_COMMIT`。保留 ConfigMap、Secret、PVC、Service、资源和副本数。 +需要更改配置、其他组件或数据库时,单独设计并审查变更计划;首版不提供绕过参数。 +`deploy-k8s.sh` 的 Fanyu 全量 apply 入口已阻断,其他环境的原流程不变。 + +这是命令行防误操作流程,不是安全边界:操作者仍可以直接运行 kubectl。团队级强制执行 +需后续收敛集群写权限。首版没有自动回退、自动重试、压测或自动创建测试数据。 + +## 授权与数据边界 + +- 本地 prepare/check/plan 不构成部署、PR 或客户数据写入授权。 +- `check --cluster` / `verify` 只读集群资源;实际执行也须符合本次会话的环境访问授权。 +- 展示 plan 后,获得用户对该计划的明确部署授权,再执行 deploy。`--approval-ref` 记录 + 用户消息引用,不能由代理虚构,也不是系统自动验证的授权凭据。计划变化后重新授权。 +- 应用启动本身可能执行迁移,镜像更新不能保证零业务写入。部署前必须审查候选版本启动 + 行为,记录 change-safety 证据;有新增迁移或客户数据改写时,停止并取得精确操作授权。 +- 生产业务验证默认只读。领取、保存、送检等写测试须另外批准隔离测试项目、账号和范围。 +- 记录、日志、回退材料均在 Git 忽略的 local-data;不要填写凭证或复制客户正文到记录。 + +## 顺序与完成条件 + +### 1. Prepare:绑定候选版本和目标身份(离线) + +使用干净、已提交的工作树;其他人的未提交改动保留在原工作树,使用隔离工作树发布。 +镜像必须使用 `repository@sha256:...`,不接受可变 tag。先构建、核对镜像来源,再准备发布。 +目标 server、context、namespace UID 从已核实的集群身份取得,不能从一次错误连接中直接认可。 +工具检查 namespace UID 及 API server,namespace 名字相同也不能蒙混过关。 + +```sh +python3 scripts/release.py prepare --id \ + --image \ + --kubeconfig --context \ + --server --namespace-uid \ + --purpose '本次问题和修改范围' \ + --rollback '旧镜像、兼容性结论及需要先核对的任务状态' +``` + +完成条件:记录有 commit、镜像摘要、策略摘要、目标身份、目的和回退说明;未访问集群。 +记录绑定的是当前 checkout commit。镜像与源码的对应关系仍须独立构建记录验证。 + +### 2. Check:本地检查与线上只读快照 + +```sh +python3 scripts/release.py check --local +python3 scripts/release.py check --cluster +``` + +本地检查命令和环境约束以 [fanyu-policy.json](fanyu-policy.json) 为准。首版运行流程工具测试、 +前端类型/构建及切片恢复定向回归,不冒充全套产品测试通过。任何失败阻断,不支持忽略开关。 +命令执行结果及输出摘要写入记录,不保存或回显原始输出;排查失败时单独重跑对应命令。 + +集群检查解析容器 env/envFrom/ConfigMap,核对数据库名称、会话认证、API external 模式、 +处理器并发/资源、存储连接、PVC 及当前 rollout 状态。只校验 Mongo Secret 的绑定,不读取 +其值;**不证明实际 Mongo 主机正确、凭证有效或视频能正常播放**,由业务证据补齐。 +公网带宽作为观察项展示,不以固定数值判断并发容量充足。 + +完成条件:两类检查均成功、源码和策略未变化。再次 check 会废弃旧计划和证据。 + +### 3. Plan:形成可审查的计划 + +```sh +python3 scripts/release.py plan +``` + +显示旧/新镜像、构建标记、目标、带宽观察值、限制与回退说明,并生成计划摘要。 +核对两个部署的修改范围。计划不更新 ConfigMap、Secret、存储或网络。 + +部署前需要两份人工审查证据: + +- `image-provenance`:构建记录将本次源码 commit 与镜像 digest 对应,候选镜像的验证情况明确。 +- `change-safety`:审查启动迁移/兼容性,检查在途切片与送检工作,确定维护窗口或安全切换方式, + 并说明回退旧版本是否安全。工具不自行读取或更改业务数据库来判断任务状态。 + +```sh +python3 scripts/release.py evidence --check image-provenance \ + --result passed --file --note '构建和镜像核对结论' +python3 scripts/release.py evidence --check change-safety \ + --result passed --file --note '迁移、在途任务及回退核对结论' +``` + +证据保存文件路径、SHA256、时间和计划摘要。工具验证文件未变,**不自动判断内容是否真实充分**。 +人工证据应明确环境、版本、步骤、实际结果、限制和审查人;不能把空模板标记为 passed。 + +### 4. Deploy:本次授权后的有界变更 + +```sh +python3 scripts/release.py deploy \ + --confirm-plan --approval-ref +``` + +部署前再次核对源码、策略、证据及线上快照,集群预检超过一小时需重做。每次 patch 前重新 +核对两个部署及关联配置;用 UID/resourceVersion 条件更新镜像与构建标记。 +写请求发送前先落盘 intent。任何超时或中断按结果未知处理;不重复发、不自动回滚。 + +两个 Deployment 不是事务,也没有跨主机锁。中间失败可能只更新一个组件,必须 verify +核对实际状态再制定恢复计划。JSON 条件更新防止覆盖同一资源的并发修改,但不能保证跨资源 +原子性;正式部署应使用团队约定的维护窗口,后续可增加服务端发布锁。 + +完成条件:仅代表 patch 已确认,状态仍为 `deployed_pending_verification`。 + +### 5. Verify:分开基础设施与业务验收 + +```sh +python3 scripts/release.py verify +``` + +核对目标镜像、构建标记、rollout、配置/PVC/服务身份以及未批准的部署变化。 +未 Ready 可稍后重新运行只读 verify,不重新 deploy。部署配置符合后仍为 +`deployed_pending_acceptance`,不等于已经登录、播放、切片或送检成功。 + +按策略中 requiredEvidence 逐项记录验收:五角色登录及业务API、多供应商权限、标注与交付 +(普通/舍弃/无效/空白)、质检/终审、切片重启恢复、视频首帧/缓冲/实际播放速度。 +所有写入测试都遵循前述隔离与授权要求,不能占用生产客户任务。 + +```sh +python3 scripts/release.py evidence --check video-playback \ + --result passed --file --note '角色、并发、网络和播放实测结果' +python3 scripts/release.py verify +``` + +完成条件:基础设施复核通过且所有必需证据有效,才进入 `accepted_with_evidence`(包含人工 +证据的验收),不是“全自动端到端通过”。未测填 unverified,失败填 failed,不得代填通过。 + +## 故障、回退和紧急修复 + +- `status` 显示最后一步、写请求状态、缺失证据和 checkout。换上下文后先运行它。 +- 源码/策略变化重新 prepare;部署前同事更新了线上,重新 check/plan 并取得新授权。 +- `outcome_unknown` 或 `deploying`:先 verify,不能再次 deploy 同一记录。部分更新不能靠 + 修改 JSON 清掉历史;保存调查记录并制定新的恢复计划。 +- 回退须新的明确授权,核对当前版本与数据兼容性,生成条件式变更。禁止直接 apply 旧完整 + Deployment、restore 整库或覆盖客户新增数据。首版不自动执行回退。 +- 紧急修复写明“已解决 / 未解决 / 临时取舍 / 后续任务”,关联本地问题票;未验证部分仍保留, + 不因上线紧急而自动验收通过。旧规格由新规则显式替代,不同时保留两份有效相反结论。 + +## 本地试运行与后续范围 + +```sh +python3 -m unittest discover -s tests/release -p 'test_*.py' +python3 scripts/release.py --help +``` + +测试使用模拟 Kubernetes,不连接集群、不创建账号或任务。覆盖数据库/认证/处理器回归、 +陈旧计划、同事更新、部分失败、证据失效、锁冲突和 legacy 入口阻断。 + +后续再扩展:服务端部署互斥、OCI provenance 自动核对、隔离环境真实业务验收、受审查的配置 +变更及回退执行器。试行阶段避免把这些未实现能力写成已有保证。 diff --git a/docs/release/fanyu-policy.json b/docs/release/fanyu-policy.json new file mode 100644 index 00000000..118f246b --- /dev/null +++ b/docs/release/fanyu-policy.json @@ -0,0 +1,23 @@ +{ + "schemaVersion": 1, + "namespace": "droboticflow-fanyu", + "database": "drobodata-fanyu", + "bucket": "drobodata-fanyu", + "workspaceMount": "/workspace", + "workspacePVC": "droboticflow-workspace", + "mongoSecret": "droboticflow-mongo-secret", + "deployments": { + "droboticflow-api": "api", + "droboticflow-annotation-processor": "processor" + }, + "requiredEvidence": [ + "image-provenance", "change-safety", "role-login", "supplier-permissions", + "annotation-delivery", "review-acceptance", "slice-recovery", "video-playback" + ], + "localChecks": [ + ["python3", "-m", "unittest", "discover", "-s", "tests/release", "-p", "test_*.py"], + ["npm", "run", "lint:web"], + ["npm", "run", "build:web"], + ["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery", "-count=1"] + ] +} diff --git a/scripts/deploy-k8s.sh b/scripts/deploy-k8s.sh index ec91a1bd..3bd0da4a 100755 --- a/scripts/deploy-k8s.sh +++ b/scripts/deploy-k8s.sh @@ -14,6 +14,10 @@ export DEPLOY_ENV source "$ROOT_DIR/scripts/load-deploy-env.sh" DEPLOY_ENV="${DEPLOY_ENV:-}" # shellcheck source=scripts/lib/build-metadata.sh +if [[ "$DEPLOY_ENV" == "fanyu" || "${K8S_NAMESPACE:-}" == "droboticflow-fanyu" ]]; then + echo "ERROR: Fanyu releases use python3 scripts/release.py; see docs/release/README.md. The legacy broad apply path is blocked before any cluster operation." >&2 + exit 2 +fi source "$ROOT_DIR/scripts/lib/build-metadata.sh" K8S_NAMESPACE="${K8S_NAMESPACE:-droboticflow}" diff --git a/scripts/release.py b/scripts/release.py new file mode 100644 index 00000000..ad41d807 --- /dev/null +++ b/scripts/release.py @@ -0,0 +1,455 @@ +#!/usr/bin/env python3 +"""Fanyu image/build-stamp release journal. Secret values are never persisted.""" +import argparse +import contextlib +import copy +import datetime as dt +import fcntl +import hashlib +import json +import os +from pathlib import Path +import re +import subprocess +import sys +import uuid + +ROOT = Path(__file__).resolve().parents[1] +RECORDS = ROOT / 'local-data/releases' +POLICY = ROOT / 'docs/release/fanyu-policy.json' +SAFE_ENV = ('MONGO_DB', 'DROBOTICFLOW_REQUIRE_SESSION_AUTH', + 'ANNOTATION_PROCESSOR_MODE', 'ANNOTATION_PROCESSOR_CONCURRENCY', + 'ANNOTATION_INDEX_BACKFILL', 'ANNOTATION_WORKSPACE_CONNECTION_ID', + 'ANNOTATION_WORKSPACE_BUCKET', 'WORKSPACE_MOUNT_PATH') + + +class Blocked(Exception): + pass + + +def require(ok, message): + if not ok: + raise Blocked(message) + + +def now(): + return dt.datetime.now(dt.timezone.utc).isoformat() + + +def digest(value): + return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(',', ':')).encode()).hexdigest() + + +def run(argv, *, stdin=None, timeout=60): + try: + p = subprocess.run(argv, cwd=ROOT, input=stdin, text=True, capture_output=True, timeout=timeout) + except (OSError, subprocess.TimeoutExpired): + raise Blocked('Command unavailable or timed out; outcome must be checked before retry.') from None + # Do not echo kubectl output, URLs or credential-bearing errors. + require(p.returncode == 0, 'Command failed: ' + argv[0] + ' (output withheld; no success recorded).') + return p.stdout + + +def source(): + return {'commit': run(['git', 'rev-parse', 'HEAD']).strip(), + 'dirty': bool(run(['git', 'status', '--porcelain', '--untracked-files=normal']).strip())} + + +def policy(): + return json.loads(POLICY.read_text()) + + +def record_path(release_id): + require(bool(re.fullmatch(r'[a-zA-Z0-9][a-zA-Z0-9_-]{0,79}', release_id)), 'Invalid release id.') + return RECORDS / release_id / 'release.json' + + +def save(r): + path = record_path(r['id']) + path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + temp = path.with_suffix('.tmp') + with os.fdopen(os.open(temp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600), 'w') as f: + json.dump(r, f, indent=2, ensure_ascii=False) + f.write('\n') + os.replace(temp, path) + + +def event(r, name): + r['events'].append({'at': now(), 'event': name}) + save(r) + + +@contextlib.contextmanager +def locked(release_id): + path = record_path(release_id) + require(path.exists(), 'Release record not found; use prepare first.') + with path.with_suffix('.lock').open('a') as f: + try: + fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + raise Blocked('Another command is updating this release.') from None + yield json.loads(path.read_text()) + + +def fresh(r): + s = source() + require(not s['dirty'], 'Checkout is dirty; commit task changes in a clean checkout first.') + require(s['commit'] == r['commit'], 'Checkout commit changed; prepare a new release.') + require(digest(policy()) == r['policyHash'], 'Release policy changed; prepare a new release.') + + +def kubectl(r, args, *, stdin=None): + t = r['target'] + return run(['kubectl', '--kubeconfig', t['kubeconfig'], '--context', t['context'], + '--request-timeout=20s', '-n', t['namespace']] + args, stdin=stdin, timeout=60) + + +def get(r, kind, name): + return json.loads(kubectl(r, ['get', kind, name, '-o', 'json'])) + + +def env_values(container, configs): + values = {} + for entry in container.get('envFrom', []): + ref = entry.get('configMapRef') + require(bool(ref), 'Secret/unknown envFrom is unsupported by this preflight.') + for key, value in configs[ref['name']].get('data', {}).items(): + values[entry.get('prefix', '') + key] = value + for entry in container.get('env', []): + key = entry['name'] + if 'value' in entry: + values[key] = entry['value'] + elif 'configMapKeyRef' in entry.get('valueFrom', {}): + ref = entry['valueFrom']['configMapKeyRef'] + require(ref['key'] in configs[ref['name']].get('data', {}), 'Missing ConfigMap key.') + values[key] = configs[ref['name']]['data'][ref['key']] + else: + # Explicit unknowns must override envFrom rather than silently use its value. + values[key] = None + return values + + +def inspect_deployment(deployment, container_name, configs, p): + spec = deployment['spec'] + pod = spec['template']['spec'] + containers = pod['containers'] + matches = [(i, c) for i, c in enumerate(containers) if c['name'] == container_name] + require(len(matches) == 1, 'Expected deployment container missing.') + index, c = matches[0] + e = env_values(c, configs) + require(e.get('MONGO_DB') == p['database'], 'Database name mismatch.') + require(e.get('ANNOTATION_WORKSPACE_BUCKET') == p['bucket'], 'Workspace bucket mismatch.') + require(bool(e.get('ANNOTATION_WORKSPACE_CONNECTION_ID')), 'Missing workspace storage connection.') + require(e.get('WORKSPACE_MOUNT_PATH') == p['workspaceMount'], 'Workspace mount mismatch.') + mongo = next((x for x in c.get('env', []) if x['name'] == 'MONGO_URI'), {}) + require(mongo.get('valueFrom', {}).get('secretKeyRef') == {'name': p['mongoSecret'], 'key': 'MONGO_URI'}, + 'Mongo secret binding mismatch; preflight does not read secret values.') + mounts = {v['name']: v['mountPath'] for v in c.get('volumeMounts', [])} + claims = [v for v in pod.get('volumes', []) if mounts.get(v['name']) == p['workspaceMount']] + require(len(claims) == 1 and claims[0].get('persistentVolumeClaim', {}).get('claimName') == p['workspacePVC'], + 'Workspace PVC binding mismatch.') + require(spec.get('replicas', 1) > 0, 'Deployment has no replicas.') + if container_name == 'api': + require(e.get('DROBOTICFLOW_REQUIRE_SESSION_AUTH') in ('true', '1'), 'Session authentication disabled.') + require(e.get('ANNOTATION_PROCESSOR_MODE') == 'external', 'API must use external processor.') + else: + require(e.get('ANNOTATION_PROCESSOR_MODE') == 'processor', 'Processor mode mismatch.') + require(e.get('ANNOTATION_PROCESSOR_CONCURRENCY') == '1', 'First release profile requires concurrency 1.') + require(e.get('ANNOTATION_INDEX_BACKFILL') == '0', 'Historical backfill must remain disabled.') + require(c.get('resources', {}).get('limits', {}).get('memory') == '8Gi', 'First release profile requires processor memory limit 8Gi.') + require(spec.get('replicas') == 1 and spec.get('strategy', {}).get('type') == 'Recreate', + 'Processor must have one replica and Recreate strategy.') + status = deployment.get('status', {}) + ready = (status.get('observedGeneration', 0) >= deployment['metadata'].get('generation', 1) + and status.get('updatedReplicas', 0) == spec.get('replicas', 1) + and status.get('readyReplicas', 0) == spec.get('replicas', 1) + and status.get('replicas', 0) == spec.get('replicas', 1)) + m = deployment['metadata'] + neutral = copy.deepcopy(spec) + nc = neutral['template']['spec']['containers'][index] + nc['image'] = '' + nc['env'] = [x for x in nc.get('env', []) if x['name'] != 'DROBOTICFLOW_BUILD_COMMIT'] + stamp = next((i for i, x in enumerate(c.get('env', [])) if x['name'] == 'DROBOTICFLOW_BUILD_COMMIT'), None) + return {'uid': m['uid'], 'specHash': digest(spec), 'nonReleaseHash': digest(neutral), + 'buildCommitIndex': stamp, 'buildCommit': e.get('DROBOTICFLOW_BUILD_COMMIT'), 'image': c['image'], + 'resourceVersion': m['resourceVersion'], 'containerIndex': index, + 'ready': ready, 'env': {k: e.get(k) for k in SAFE_ENV}} + + +def snapshot(r): + t, p = r['target'], policy() + view = json.loads(kubectl(r, ['config', 'view', '--minify', '-o', 'json'])) + require(view['clusters'][0]['cluster']['server'] == t['server'], 'Kube API server mismatch.') + ns = get(r, 'namespace', t['namespace']) + require(ns['metadata']['uid'] == t['namespaceUID'], 'Namespace UID mismatch.') + deployments = {n: get(r, 'deployment', n) for n in p['deployments']} + config_names = set() + for d in deployments.values(): + for c in d['spec']['template']['spec']['containers']: + config_names.update(x['configMapRef']['name'] for x in c.get('envFrom', []) if 'configMapRef' in x) + config_names.update(x['valueFrom']['configMapKeyRef']['name'] for x in c.get('env', []) + if 'configMapKeyRef' in x.get('valueFrom', {})) + configs = {n: get(r, 'configmap', n) for n in sorted(config_names)} + deps = {n: inspect_deployment(d, p['deployments'][n], configs, p) for n, d in deployments.items()} + envs = [d['env'] for d in deps.values()] + require(len({e['ANNOTATION_WORKSPACE_CONNECTION_ID'] for e in envs}) == 1, 'API/processor storage connections differ.') + pvc = get(r, 'pvc', p['workspacePVC']) + require(pvc.get('status', {}).get('phase') == 'Bound', 'Workspace PVC is not Bound.') + svc = get(r, 'service', 'droboticflow-api-public-eip') + ann = svc['metadata'].get('annotations', {}) + return {'at': now(), 'deployments': deps, + 'configs': {n: {'uid': x['metadata']['uid'], 'hash': digest(x.get('data', {}))} for n, x in configs.items()}, + 'pvc': {'uid': pvc['metadata']['uid'], 'specHash': digest(pvc['spec'])}, + 'publicService': {'uid': svc['metadata']['uid'], 'specHash': digest(svc['spec']), + 'annotationHash': digest(ann), + 'bandwidthMbps': ann.get('service.beta.kubernetes.io/cce-elastic-ip-bandwidth-in-mbps', 'unknown')}} + + +def stable(s): + return {**{k: s[k] for k in ('configs', 'pvc', 'publicService')}, + 'deployments': {n: {k: d[k] for k in ('uid', 'specHash', 'image')} for n, d in s['deployments'].items()}} + + +def plan_data(r): + return {k: r[k] for k in ('id', 'commit', 'image', 'policyHash', 'target', 'baseline', 'localChecks', 'purpose', 'rollback')} + + +def prepare(a): + p, s = policy(), source() + branch = run(['git', 'branch', '--show-current']).strip() + require(not re.search(r'(^|/)poc($|[-/])', branch), 'POC branches cannot prepare Fanyu releases.') + require(not s['dirty'], 'Prepare requires a clean checkout (use an isolated checkout for unrelated edits).') + require(bool(re.fullmatch(r'[^\s@]+@sha256:[0-9a-f]{64}', a.image)), 'Use an immutable repository@sha256 image.') + require(a.server.startswith('https://') and '@' not in a.server and '?' not in a.server, 'Use a credential-free HTTPS Kubernetes server URL.') + release_id = a.id or dt.datetime.now(dt.timezone.utc).strftime('%Y%m%dT%H%M%SZ-') + uuid.uuid4().hex[:8] + require(not record_path(release_id).exists(), 'Release id already exists.') + r = {'schemaVersion': 1, 'id': release_id, 'createdAt': now(), 'commit': s['commit'], + 'image': a.image, 'policyHash': digest(p), 'purpose': a.purpose, 'rollback': a.rollback, + 'target': {'kubeconfig': str(Path(a.kubeconfig).resolve()), 'context': a.context, + 'server': a.server.rstrip('/'), 'namespace': p['namespace'], 'namespaceUID': a.namespace_uid}, + 'state': 'prepared', 'localChecks': [], 'baseline': None, 'planHash': None, + 'operations': [], 'evidence': {}, 'events': []} + event(r, 'prepared; no network access') + print(json.dumps({'id': r['id'], 'state': r['state']}, ensure_ascii=False)) + + +def check(r, a): + fresh(r) + require(not r['operations'], 'A deployment was attempted; check outcome with verify, never rewrite its plan.') + r['planHash'] = None + r['evidence'] = {} + r['state'] = 'checking' + event(r, 'check started; previous plan invalidated') + if a.local: + r['localChecks'] = [] + for argv in policy()['localChecks']: + started = now() + try: + output = run(argv, timeout=1800) + except Blocked: + r['localChecks'].append({'argv': argv, 'startedAt': started, 'result': 'failed'}) + r['state'] = 'blocked' + event(r, 'local check failed') + raise + r['localChecks'].append({'argv': argv, 'startedAt': started, 'result': 'passed', 'outputHash': digest(output)}) + save(r) + fresh(r) + if a.cluster: + # Never preserve an earlier successful snapshot when a new read fails. + r['baseline'] = None + save(r) + try: + s = snapshot(r) + require(all(d['ready'] for d in s['deployments'].values()), 'Current deployments are not fully ready.') + r['baseline'] = s + except Blocked: + r['state'] = 'blocked' + event(r, 'cluster check failed') + raise + r['state'] = 'checked_partial' + event(r, 'checks recorded; plan must still be generated') + + +def make_plan(r): + fresh(r) + require(not r['operations'], 'Deployment already attempted.') + require([x['argv'] for x in r['localChecks']] == policy()['localChecks'] + and all(x['result'] == 'passed' for x in r['localChecks']), 'Required local checks incomplete.') + require(r['baseline'] is not None, 'Read-only cluster check required.') + r['planHash'] = digest(plan_data(r)) + r['state'] = 'awaiting_approval' + event(r, 'plan prepared; no authorization inferred') + print(json.dumps({'planHash': r['planHash'], 'target': r['target'], 'changes': [ + {'deployment': n, 'from': d['image'], 'to': r['image'], 'buildCommit': r['commit']} for n, d in r['baseline']['deployments'].items()], + 'bandwidthMbps': r['baseline']['publicService']['bandwidthMbps'], + 'warnings': ['Only image/build stamp changes; startup migrations require change-safety evidence.', + 'Image-to-source provenance requires independent evidence.', + 'Bandwidth adequacy and business acceptance remain manual evidence.'], + 'rollback': r['rollback']}, ensure_ascii=False, indent=2)) + + +def deploy(r, a): + fresh(r) + require(r['state'] == 'awaiting_approval' and not r['operations'], 'No fresh deployable plan; inspect status.') + require(r['planHash'] == a.confirm_plan == digest(plan_data(r)), 'Plan hash does not match.') + require(bool(a.approval_ref.strip()), 'Record the explicit user approval reference for this exact plan.') + age = (dt.datetime.now(dt.timezone.utc) - dt.datetime.fromisoformat(r['baseline']['at'])).total_seconds() + require(0 <= age <= 3600, 'Cluster check is older than one hour; recheck and obtain a new plan approval.') + for key in ('image-provenance', 'change-safety'): + require(key in r['evidence'] and evidence_valid(r['evidence'][key], r), key + ' evidence required before deployment.') + current = snapshot(r) + require(stable(current) == stable(r['baseline']), 'Live resources changed since check; prepare/check a new plan.') + require(all(d['ready'] for d in current['deployments'].values()), 'Live deployment not ready.') + r['approval'] = {'reference': a.approval_ref, 'at': now(), 'planHash': r['planHash']} + r['state'] = 'deploying' + event(r, 'explicit approval reference recorded; deployment starting') + for name in current['deployments']: + latest = snapshot(r) + require(all(latest[k] == current[k] for k in ('configs', 'pvc', 'publicService')), 'Configuration changed during deployment; inspect partial outcome.') + for n, live in latest['deployments'].items(): + old = current['deployments'][n] + done = any(o['deployment'] == n and o['result'] == 'acknowledged' for o in r['operations']) + require(live['uid'] == old['uid'] and live['nonReleaseHash'] == old['nonReleaseHash'], 'Deployment changed during release.') + require((live['image'] == r['image'] and live['buildCommit'] == r['commit']) if done else live['specHash'] == old['specHash'], 'Image/configuration drift during release.') + dep = latest['deployments'][name] + patch = [{'op': 'test', 'path': '/metadata/resourceVersion', 'value': dep['resourceVersion']}, + {'op': 'test', 'path': '/metadata/uid', 'value': dep['uid']}, + {'op': 'test', 'path': f"/spec/template/spec/containers/{dep['containerIndex']}/image", 'value': dep['image']}, + {'op': 'replace', 'path': f"/spec/template/spec/containers/{dep['containerIndex']}/image", 'value': r['image']}] + stamp = {'name': 'DROBOTICFLOW_BUILD_COMMIT', 'value': r['commit']} + env_path = f"/spec/template/spec/containers/{dep['containerIndex']}/env/" + patch.append({'op': 'add' if dep['buildCommitIndex'] is None else 'replace', + 'path': env_path + ('-' if dep['buildCommitIndex'] is None else str(dep['buildCommitIndex'])), 'value': stamp}) + op = {'deployment': name, 'before': dep['image'], 'desired': r['image'], 'at': now(), 'result': 'unknown'} + r['operations'].append(op) + save(r) # Persist intent before sending any mutation. + try: + kubectl(r, ['patch', 'deployment', name, '--type=json', '--patch-file=/dev/stdin'], stdin=json.dumps(patch)) + except Blocked: + r['state'] = 'outcome_unknown' + event(r, 'patch outcome unknown; verify before any further operation') + raise + op['result'] = 'acknowledged' + save(r) + r['state'] = 'deployed_pending_verification' + event(r, 'image patches acknowledged; rollout and acceptance not yet verified') + + +def record_evidence(r, a): + require(r['planHash'] is not None, 'Generate a plan before attaching evidence.') + require(a.check in policy()['requiredEvidence'], 'Unknown evidence category.') + require(a.check in ('image-provenance', 'change-safety') or r['state'] in ('deployed_pending_acceptance', 'accepted_with_evidence'), + 'Business evidence can be attached only after infrastructure verification.') + path = Path(a.file).resolve() + require(path.is_file() and path.stat().st_size > 0, 'Evidence must be a nonempty local file.') + r['evidence'][a.check] = {'result': a.result, 'path': str(path), + 'sha256': hashlib.sha256(path.read_bytes()).hexdigest(), + 'at': now(), 'planHash': r['planHash'], 'note': a.note, + 'kind': 'operator-attested; content not automatically evaluated'} + if a.result != 'passed' and r['state'] == 'accepted_with_evidence': + r['state'] = 'deployed_pending_acceptance' + event(r, 'evidence recorded: ' + a.check) + + +def evidence_valid(e, r): + path = Path(e['path']) + return (e['result'] == 'passed' and e['planHash'] == r['planHash'] and path.is_file() + and hashlib.sha256(path.read_bytes()).hexdigest() == e['sha256']) + + +def verify(r): + require(bool(r['operations']), 'No deployment attempt in this record.') + # Failure cannot retain a stale accepted state. + r['state'] = 'deployed_pending_verification' + save(r) + require(digest(policy()) == r['policyHash'], 'Policy changed; verify with the original policy revision.') + s = snapshot(r) + r['lastObservation'] = s + expected = r['baseline'] + require(all(s[k] == expected[k] for k in ('configs', 'pvc', 'publicService')), 'Configuration drift detected.') + for n, d in s['deployments'].items(): + require(d['uid'] == expected['deployments'][n]['uid'], 'Deployment was replaced.') + require(d['nonReleaseHash'] == expected['deployments'][n]['nonReleaseHash'], 'Deployment settings drifted beyond the approved image/build stamp.') + require(d['image'] == r['image'] and d['buildCommit'] == r['commit'] and d['ready'], 'Target image rollout incomplete; outcome remains pending.') + r['state'] = 'deployed_pending_acceptance' + missing = [k for k in policy()['requiredEvidence'] if k not in r['evidence'] or not evidence_valid(r['evidence'][k], r)] + if not missing: + r['state'] = 'accepted_with_evidence' + event(r, 'infrastructure verified; remaining evidence: ' + ', '.join(missing)) + + +def status(r): + p = policy() + missing = [k for k in p['requiredEvidence'] if k not in r['evidence'] or not evidence_valid(r['evidence'][k], r)] + state = 'acceptance_evidence_invalid' if r['state'] == 'accepted_with_evidence' and missing else r['state'] + print(json.dumps({'id': r['id'], 'state': state, 'recordedState': r['state'], 'commit': r['commit'], 'image': r['image'], + 'policyMatches': digest(p) == r['policyHash'], 'checkout': source(), + 'planHash': r['planHash'], 'missingEvidence': missing, + 'operations': r['operations'], 'lastEvent': r['events'][-1], + 'next': 'Resume from recorded state; fresh approval is required for any new deployment or rollback.'}, + indent=2, ensure_ascii=False)) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest='command', required=True) + prep = sub.add_parser('prepare', help='Offline: create a release record from a clean committed checkout.') + prep.add_argument('--id') + for name in ('image', 'kubeconfig', 'context', 'server', 'namespace-uid', 'purpose', 'rollback'): + prep.add_argument('--' + name, required=True) + for cmd in ('check', 'plan', 'deploy', 'verify', 'status', 'evidence'): + c = sub.add_parser(cmd) + if cmd == 'status': + c.add_argument('id', nargs='?') + else: + c.add_argument('id') + if cmd == 'check': + c.add_argument('--local', action='store_true') + c.add_argument('--cluster', action='store_true') + elif cmd == 'deploy': + c.add_argument('--confirm-plan', required=True) + c.add_argument('--approval-ref', required=True) + elif cmd == 'evidence': + c.add_argument('--check', required=True) + c.add_argument('--result', choices=['passed', 'failed', 'unverified'], required=True) + c.add_argument('--file', required=True) + c.add_argument('--note', required=True) + a = parser.parse_args() + try: + if a.command == 'prepare': + prepare(a) + return + if a.command == 'status' and not a.id: + summaries = [] + for path in sorted(RECORDS.glob('*/release.json')): + item = json.loads(path.read_text()) + summaries.append({k: item[k] for k in ('id', 'state', 'commit', 'createdAt')}) + print(json.dumps(summaries, indent=2, ensure_ascii=False)) + return + with locked(a.id) as r: + if a.command == 'check': + require(a.local or a.cluster, 'Choose --local and/or --cluster.') + check(r, a) + elif a.command == 'plan': + make_plan(r) + elif a.command == 'deploy': + try: + deploy(r, a) + except Blocked: + if r['state'] == 'deploying': + r['state'] = 'outcome_unknown' + event(r, 'interrupted; inspect partial outcome before further changes') + raise + elif a.command == 'verify': + verify(r) + elif a.command == 'evidence': + record_evidence(r, a) + status(r) + except (Blocked, KeyError, ValueError, OSError) as e: + # Validation errors are authored here; generic errors omit possibly sensitive values. + print('BLOCKED: ' + (str(e) if isinstance(e, Blocked) else type(e).__name__), file=sys.stderr) + sys.exit(2) + + +if __name__ == '__main__': + main() diff --git a/tests/release/test_release.py b/tests/release/test_release.py new file mode 100644 index 00000000..f4956ae4 --- /dev/null +++ b/tests/release/test_release.py @@ -0,0 +1,332 @@ +"""No cluster or customer credentials: exercise release transitions with fake Kubernetes.""" +import argparse +import copy +import importlib.util +import json +import os +from pathlib import Path +import subprocess +import shutil +import sys +import tempfile +import unittest +from unittest.mock import patch + +ROOT = Path(__file__).resolve().parents[2] +spec = importlib.util.spec_from_file_location('release', ROOT / 'scripts/release.py') +m = importlib.util.module_from_spec(spec) +spec.loader.exec_module(m) +IMAGE = 'registry.example/api@sha256:' + 'a' * 64 + + +def fixture(name, container): + config = {'data': {'MONGO_DB': 'drobodata-fanyu', 'DROBOTICFLOW_REQUIRE_SESSION_AUTH': 'true', + 'ANNOTATION_PROCESSOR_MODE': 'external', 'ANNOTATION_WORKSPACE_BUCKET': 'drobodata-fanyu', + 'ANNOTATION_WORKSPACE_CONNECTION_ID': 'storage-example', 'WORKSPACE_MOUNT_PATH': '/workspace'}, + 'metadata': {'uid': 'config-uid'}} + env = [{'name': 'MONGO_URI', 'valueFrom': {'secretKeyRef': {'name': 'droboticflow-mongo-secret', 'key': 'MONGO_URI'}}}] + if container == 'processor': + env += [{'name': k, 'value': v} for k, v in { + 'ANNOTATION_PROCESSOR_MODE': 'processor', 'ANNOTATION_PROCESSOR_CONCURRENCY': '1', + 'ANNOTATION_INDEX_BACKFILL': '0'}.items()] + d = {'metadata': {'name': name, 'uid': name + '-uid', 'resourceVersion': '1', 'generation': 1}, + 'spec': {'replicas': 1, 'strategy': {'type': 'Recreate'}, 'template': {'spec': { + 'containers': [{'name': container, 'image': 'registry.example/api@sha256:' + 'b' * 64, + 'envFrom': [{'configMapRef': {'name': 'config'}}], 'env': env, + 'resources': {'limits': {'memory': '8Gi'}}, + 'volumeMounts': [{'name': 'workspace', 'mountPath': '/workspace'}]}], + 'volumes': [{'name': 'workspace', 'persistentVolumeClaim': {'claimName': 'droboticflow-workspace'}}]}}}, + 'status': {'observedGeneration': 1, 'replicas': 1, 'updatedReplicas': 1, 'readyReplicas': 1}} + return d, config + + +class ReleaseTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.records = Path(self.tmp.name) / 'records' + self.p = m.policy() + self.p['localChecks'] = [['fake-check']] + self.start_patch('RECORDS', self.records) + self.start_patch('policy', lambda: copy.deepcopy(self.p)) + self.start_patch('source', lambda: {'commit': 'c' * 40, 'dirty': False}) + self.start_patch('run', lambda *args, **kwargs: 'feature/release') + self.ds = {} + for n, c in self.p['deployments'].items(): + self.ds[n], config = fixture(n, c) + self.configs = {'config': config} + self.writes = [] + self.fail_patch = None + self.before_snapshot = None + self.start_patch('snapshot', self.snapshot) + self.start_patch('kubectl', self.kubectl) + m.prepare(argparse.Namespace(id='test', image=IMAGE, context='fanyu', server='https://cluster.example', + namespace_uid='namespace-uid', kubeconfig='/example/config', purpose='test', rollback='review old image')) + self.r = self.load() + + def start_patch(self, name, value): + p = patch.object(m, name, value) + p.start() + self.addCleanup(p.stop) + + def load(self): + return json.loads(m.record_path('test').read_text()) + + def snapshot(self, r): + if self.before_snapshot: + self.before_snapshot() + return {'at': m.now(), 'deployments': { + n: m.inspect_deployment(d, self.p['deployments'][n], self.configs, self.p) for n, d in self.ds.items()}, + 'configs': {'config': {'uid': 'config-uid', 'hash': m.digest(self.configs)}}, + 'pvc': {'uid': 'pvc-uid'}, 'publicService': {'bandwidthMbps': '5'}} + + def kubectl(self, r, args, *, stdin=None): + self.assertEqual(args[:2], ['patch', 'deployment']) + name = args[2] + self.writes.append(name) + if self.fail_patch == len(self.writes): + raise m.Blocked('simulated timeout') + ops = json.loads(stdin) + self.assertEqual(ops[0]['path'], '/metadata/resourceVersion') + d = self.ds[name] + self.assertEqual(ops[0]['value'], d['metadata']['resourceVersion']) + c = d['spec']['template']['spec']['containers'][0] + c['image'] = ops[3]['value'] + if ops[4]['op'] == 'add': + c['env'].append(ops[4]['value']) + else: + c['env'][int(ops[4]['path'].split('/')[-1])] = ops[4]['value'] + d['metadata']['resourceVersion'] = str(int(d['metadata']['resourceVersion']) + 1) + return '{}' + + def plan(self): + m.check(self.r, argparse.Namespace(local=True, cluster=True)) + m.make_plan(self.r) + + def evidence(self, key, result='passed'): + f = Path(self.tmp.name) / (key + '.txt') + f.write_text('synthetic test evidence for current release') + m.record_evidence(self.r, argparse.Namespace(check=key, file=str(f), result=result, note='test evidence only')) + return f + + def approve(self): + self.evidence('image-provenance') + self.evidence('change-safety') + return argparse.Namespace(confirm_plan=self.r['planHash'], approval_ref='user-message:test') + + def test_prepare_is_offline_and_private(self): + self.assertEqual(self.r['state'], 'prepared') + self.assertEqual(m.record_path('test').stat().st_mode & 0o777, 0o600) + self.assertFalse(self.writes) + + def test_cannot_plan_without_both_checks(self): + with self.assertRaises(m.Blocked): + m.make_plan(self.r) + m.check(self.r, argparse.Namespace(local=True, cluster=False)) + with self.assertRaises(m.Blocked): + m.make_plan(self.r) + + def test_local_failure_cannot_pass(self): + with patch.object(m, 'run', side_effect=m.Blocked('failure')): + with self.assertRaises(m.Blocked): + m.check(self.r, argparse.Namespace(local=True, cluster=False)) + self.assertEqual(self.load()['state'], 'blocked') + + def test_recheck_invalidates_previous_plan_and_failed_snapshot(self): + self.plan() + with patch.object(m, 'snapshot', side_effect=m.Blocked('offline')): + with self.assertRaises(m.Blocked): + m.check(self.r, argparse.Namespace(local=False, cluster=True)) + saved = self.load() + self.assertIsNone(saved['baseline']) + self.assertIsNone(saved['planHash']) + + def test_source_and_policy_change_block_deployment(self): + self.plan() + a = self.approve() + for s in [{'commit': 'd' * 40, 'dirty': False}, {'commit': 'c' * 40, 'dirty': True}]: + with patch.object(m, 'source', return_value=s), self.assertRaises(m.Blocked): + m.deploy(self.r, a) + self.p['database'] = 'another' + with self.assertRaises(m.Blocked): + m.deploy(self.r, a) + self.assertFalse(self.writes) + + def test_exact_plan_and_explicit_reference_required(self): + self.plan() + a = self.approve() + a.confirm_plan = 'wrong' + with self.assertRaises(m.Blocked): + m.deploy(self.r, a) + a.confirm_plan = self.r['planHash'] + a.approval_ref = '' + with self.assertRaises(m.Blocked): + m.deploy(self.r, a) + self.assertFalse(self.writes) + + def test_changed_evidence_rejected(self): + self.plan() + a = self.approve() + Path(self.r['evidence']['image-provenance']['path']).write_text('changed') + with self.assertRaises(m.Blocked): + m.deploy(self.r, a) + self.assertFalse(self.writes) + + def test_colleague_live_change_blocks_before_patch(self): + self.plan() + a = self.approve() + self.ds['droboticflow-api']['spec']['replicas'] = 3 + with self.assertRaises(m.Blocked): + m.deploy(self.r, a) + self.assertFalse(self.writes) + + def test_config_change_between_patches_stops_second(self): + self.plan() + a = self.approve() + def change(): + if len(self.writes) == 1: + self.configs['config']['data']['unrelated'] = 'changed' + self.before_snapshot = change + with self.assertRaises(m.Blocked): + m.deploy(self.r, a) + self.assertEqual(len(self.writes), 1) + self.assertEqual(len(self.load()['operations']), 1) + + def test_timeout_preserves_unknown_intent_and_cannot_redeploy(self): + self.plan() + a = self.approve() + self.fail_patch = 2 + with self.assertRaises(m.Blocked): + m.deploy(self.r, a) + saved = self.load() + self.assertEqual(saved['state'], 'outcome_unknown') + self.assertEqual([o['result'] for o in saved['operations']], ['acknowledged', 'unknown']) + with self.assertRaises(m.Blocked): + m.deploy(saved, a) + with self.assertRaises(m.Blocked): + m.verify(saved) + self.assertNotEqual(self.load()['state'], 'accepted_with_evidence') + + def test_deploy_and_ready_are_not_business_acceptance(self): + self.plan() + m.deploy(self.r, self.approve()) + self.assertEqual(self.r['state'], 'deployed_pending_verification') + m.verify(self.r) + self.assertEqual(self.r['state'], 'deployed_pending_acceptance') + for key in self.p['requiredEvidence']: + if key not in self.r['evidence']: + self.evidence(key) + m.verify(self.r) + self.assertEqual(self.r['state'], 'accepted_with_evidence') + Path(self.r['evidence']['video-playback']['path']).unlink() + m.verify(self.r) + self.assertEqual(self.r['state'], 'deployed_pending_acceptance') + + def test_no_predeployment_business_evidence(self): + self.plan() + with self.assertRaises(m.Blocked): + self.evidence('video-playback') + + def test_verify_rejects_unplanned_settings(self): + self.plan() + m.deploy(self.r, self.approve()) + self.ds['droboticflow-api']['spec']['template']['spec']['containers'][0]['resources'] = {'limits': {'memory': '1Gi'}} + with self.assertRaises(m.Blocked): + m.verify(self.r) + self.assertEqual(self.load()['state'], 'deployed_pending_verification') + + def test_configuration_regressions_block(self): + for key, value in [('MONGO_DB', 'droboticflow'), ('DROBOTICFLOW_REQUIRE_SESSION_AUTH', 'false'), + ('ANNOTATION_PROCESSOR_MODE', 'inprocess'), ('ANNOTATION_WORKSPACE_BUCKET', ''), + ('WORKSPACE_MOUNT_PATH', '/wrong')]: + configs = copy.deepcopy(self.configs) + configs['config']['data'][key] = value + with self.subTest(key=key), self.assertRaises(m.Blocked): + m.inspect_deployment(self.ds['droboticflow-api'], 'api', configs, self.p) + + def test_unknown_explicit_env_does_not_fall_back_to_envfrom(self): + d = copy.deepcopy(self.ds['droboticflow-api']) + d['spec']['template']['spec']['containers'][0]['env'].append({ + 'name': 'MONGO_DB', 'valueFrom': {'secretKeyRef': {'name': 'other', 'key': 'db'}}}) + with self.assertRaises(m.Blocked): + m.inspect_deployment(d, 'api', self.configs, self.p) + + def test_snapshot_identity_and_no_secret_reads(self): + r = self.r + replies = {'namespace': {'metadata': {'uid': 'namespace-uid'}}, + 'pvc': {'metadata': {'uid': 'pvc'}, 'spec': {}, 'status': {'phase': 'Bound'}}, + 'service': {'metadata': {'uid': 'svc'}, 'spec': {}}} + reads = [] + def fake_get(_, kind, name): + reads.append(kind) + if kind == 'deployment': return self.ds[name] + if kind == 'configmap': return self.configs[name] + return replies[kind] + # setUp replaced snapshot; recover implementation through a second module import. + real = importlib.util.module_from_spec(spec) + spec.loader.exec_module(real) + with patch.object(real, 'policy', return_value=self.p), patch.object(real, 'get', side_effect=fake_get), patch.object(real, 'kubectl', return_value=json.dumps({'clusters': [{'cluster': {'server': r['target']['server']}}]})): + real.snapshot(r) + self.assertNotIn('secret', reads) + replies['namespace']['metadata']['uid'] = 'wrong' + with self.assertRaises(real.Blocked): real.snapshot(r) + + def test_stale_cluster_check_blocks_deploy(self): + self.plan() + self.r['baseline']['at'] = '2000-01-01T00:00:00+00:00' + m.make_plan(self.r) + with self.assertRaises(m.Blocked): m.deploy(self.r, self.approve()) + self.assertFalse(self.writes) + + def test_record_path_traversal_rejected(self): + with self.assertRaises(m.Blocked): m.record_path('../other') + + def test_record_lock_prevents_two_writers(self): + with m.locked('test'): + with self.assertRaises(m.Blocked): + with m.locked('test'): pass + + +class EntrypointTests(unittest.TestCase): + def test_cli_persists_and_resumes_offline(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / 'scripts').mkdir() + (root / 'docs/release').mkdir(parents=True) + shutil.copy(ROOT / 'scripts/release.py', root / 'scripts/release.py') + shutil.copy(ROOT / 'docs/release/fanyu-policy.json', root / 'docs/release/fanyu-policy.json') + (root / '.gitignore').write_text('local-data/\n') + for args in [ ['git', 'init', '-q'], ['git', 'add', '.'], + ['git', '-c', 'user.name=ReleaseTest', '-c', 'user.email=test@example.invalid', + 'commit', '-qm', 'fixture'] ]: + subprocess.run(args, cwd=root, check=True, capture_output=True) + def cli(*args): + return subprocess.run([sys.executable, 'scripts/release.py', *args], cwd=root, + text=True, capture_output=True) + r = cli('prepare', '--id', 'offline-demo', '--image', IMAGE, '--context', 'example', + '--kubeconfig', '/does-not-exist', '--server', 'https://cluster.example', + '--namespace-uid', 'example-uid', '--purpose', 'offline smoke', '--rollback', 'not deployed') + self.assertEqual(r.returncode, 0, r.stderr) + resumed = cli('status', 'offline-demo') + self.assertEqual(resumed.returncode, 0, resumed.stderr) + self.assertEqual(json.loads(resumed.stdout)['state'], 'prepared') + blocked = cli('deploy', 'offline-demo', '--confirm-plan', 'unprepared', '--approval-ref', 'none') + self.assertEqual(blocked.returncode, 2) + self.assertIn('No fresh deployable plan', blocked.stderr) + + def test_legacy_fanyu_deploy_stops_before_kubectl(self): + with tempfile.TemporaryDirectory() as tmp: + fake = Path(tmp) / 'kubectl' + called = Path(tmp) / 'called' + fake.write_text('#!/bin/sh\ntouch "' + str(called) + '"\nexit 99\n') + fake.chmod(0o755) + env = dict(os.environ, PATH=tmp + ':' + os.environ['PATH'], DEPLOY_ENV='fanyu', + DROBOTICFLOW_DEPLOY_ENV_LOADED='test', K8S_NAMESPACE='droboticflow-fanyu') + p = subprocess.run(['bash', 'scripts/deploy-k8s.sh'], cwd=ROOT, env=env, text=True, capture_output=True) + self.assertEqual(p.returncode, 2) + self.assertIn('scripts/release.py', p.stderr) + self.assertFalse(called.exists()) + + +if __name__ == '__main__': + unittest.main()