🛡️ Add resumable Fanyu release workflow and deployment guards

This commit is contained in:
d-robotics 2026-09-26 07:30:35 -07:00
parent 2379bd586c
commit a2aad99de2
6 changed files with 974 additions and 0 deletions

View File

@ -21,6 +21,11 @@
- When push is approved, push the user's personal feature branch only; never push `dev` or `poc` directly.
- Do not stage or commit unrelated user changes already present in the working tree.
## Release Workflow
- For release preparation, deployment, rollback, or resuming release work, read `docs/release/README.md` first and run `python3 scripts/release.py status <release-id>` when a record exists. Follow the recorded state and evidence; a successful rollout alone is not business acceptance.
- Fanyu image releases use `scripts/release.py`. Configuration changes or migrations outside its supported scope require a separately reviewed plan and the corresponding user authorization.
## Credentials
- Never hardcode credentials (BOS AK/SK, Mongo, Gitea token, cluster kubeconfig) in any tracked file.

155
docs/release/README.md Normal file
View File

@ -0,0 +1,155 @@
# 泛宇发布试行流程
## 开始或恢复工作
发布、部署故障、回退、上下文恢复时先读本页。状态的权威来源是
`local-data/releases/<id>/release.json`。先运行 `python3 scripts/release.py status` 列出记录,
再用 `python3 scripts/release.py status <id>` 查看缺失证据与下一步。
历史聊天和“Pod 已 Ready”不代表业务验收完成。使用命令更新记录,保留失败和未验证项。
首版只支持 **Fanyu 现有 API 与切片处理器的同一不可变镜像更新**,并同步容器的
`DROBOTICFLOW_BUILD_COMMIT`。保留 ConfigMap、Secret、PVC、Service、资源和副本数。
需要更改配置、其他组件或数据库时,单独设计并审查变更计划;首版不提供绕过参数。
`deploy-k8s.sh` 的 Fanyu 全量 apply 入口已阻断,其他环境的原流程不变。
这是命令行防误操作流程,不是安全边界:操作者仍可以直接运行 kubectl。团队级强制执行
需后续收敛集群写权限。首版没有自动回退、自动重试、压测或自动创建测试数据。
## 授权与数据边界
- 本地 prepare/check/plan 不构成部署、PR 或客户数据写入授权。
- `check --cluster` / `verify` 只读集群资源;实际执行也须符合本次会话的环境访问授权。
- 展示 plan 后,获得用户对该计划的明确部署授权,再执行 deploy。`--approval-ref` 记录
用户消息引用,不能由代理虚构,也不是系统自动验证的授权凭据。计划变化后重新授权。
- 应用启动本身可能执行迁移,镜像更新不能保证零业务写入。部署前必须审查候选版本启动
行为,记录 change-safety 证据;有新增迁移或客户数据改写时,停止并取得精确操作授权。
- 生产业务验证默认只读。领取、保存、送检等写测试须另外批准隔离测试项目、账号和范围。
- 记录、日志、回退材料均在 Git 忽略的 local-data;不要填写凭证或复制客户正文到记录。
## 顺序与完成条件
### 1. Prepare:绑定候选版本和目标身份(离线)
使用干净、已提交的工作树;其他人的未提交改动保留在原工作树,使用隔离工作树发布。
镜像必须使用 `repository@sha256:...`,不接受可变 tag。先构建、核对镜像来源,再准备发布。
目标 server、context、namespace UID 从已核实的集群身份取得,不能从一次错误连接中直接认可。
工具检查 namespace UID 及 API server,namespace 名字相同也不能蒙混过关。
```sh
python3 scripts/release.py prepare --id <release-id> \
--image <registry/repository@sha256:digest> \
--kubeconfig <local-kubeconfig-path> --context <verified-context> \
--server <verified-https-api-server> --namespace-uid <verified-uid> \
--purpose '本次问题和修改范围' \
--rollback '旧镜像、兼容性结论及需要先核对的任务状态'
```
完成条件:记录有 commit、镜像摘要、策略摘要、目标身份、目的和回退说明;未访问集群。
记录绑定的是当前 checkout commit。镜像与源码的对应关系仍须独立构建记录验证。
### 2. Check:本地检查与线上只读快照
```sh
python3 scripts/release.py check <release-id> --local
python3 scripts/release.py check <release-id> --cluster
```
本地检查命令和环境约束以 [fanyu-policy.json](fanyu-policy.json) 为准。首版运行流程工具测试、
前端类型/构建及切片恢复定向回归,不冒充全套产品测试通过。任何失败阻断,不支持忽略开关。
命令执行结果及输出摘要写入记录,不保存或回显原始输出;排查失败时单独重跑对应命令。
集群检查解析容器 env/envFrom/ConfigMap,核对数据库名称、会话认证、API external 模式、
处理器并发/资源、存储连接、PVC 及当前 rollout 状态。只校验 Mongo Secret 的绑定,不读取
其值;**不证明实际 Mongo 主机正确、凭证有效或视频能正常播放**,由业务证据补齐。
公网带宽作为观察项展示,不以固定数值判断并发容量充足。
完成条件:两类检查均成功、源码和策略未变化。再次 check 会废弃旧计划和证据。
### 3. Plan:形成可审查的计划
```sh
python3 scripts/release.py plan <release-id>
```
显示旧/新镜像、构建标记、目标、带宽观察值、限制与回退说明,并生成计划摘要。
核对两个部署的修改范围。计划不更新 ConfigMap、Secret、存储或网络。
部署前需要两份人工审查证据:
- `image-provenance`:构建记录将本次源码 commit 与镜像 digest 对应,候选镜像的验证情况明确。
- `change-safety`:审查启动迁移/兼容性,检查在途切片与送检工作,确定维护窗口或安全切换方式,
并说明回退旧版本是否安全。工具不自行读取或更改业务数据库来判断任务状态。
```sh
python3 scripts/release.py evidence <release-id> --check image-provenance \
--result passed --file <local-evidence-file> --note '构建和镜像核对结论'
python3 scripts/release.py evidence <release-id> --check change-safety \
--result passed --file <local-evidence-file> --note '迁移、在途任务及回退核对结论'
```
证据保存文件路径、SHA256、时间和计划摘要。工具验证文件未变,**不自动判断内容是否真实充分**。
人工证据应明确环境、版本、步骤、实际结果、限制和审查人;不能把空模板标记为 passed。
### 4. Deploy:本次授权后的有界变更
```sh
python3 scripts/release.py deploy <release-id> \
--confirm-plan <plan-hash> --approval-ref <user-message-reference>
```
部署前再次核对源码、策略、证据及线上快照,集群预检超过一小时需重做。每次 patch 前重新
核对两个部署及关联配置;用 UID/resourceVersion 条件更新镜像与构建标记。
写请求发送前先落盘 intent。任何超时或中断按结果未知处理;不重复发、不自动回滚。
两个 Deployment 不是事务,也没有跨主机锁。中间失败可能只更新一个组件,必须 verify
核对实际状态再制定恢复计划。JSON 条件更新防止覆盖同一资源的并发修改,但不能保证跨资源
原子性;正式部署应使用团队约定的维护窗口,后续可增加服务端发布锁。
完成条件:仅代表 patch 已确认,状态仍为 `deployed_pending_verification`。
### 5. Verify:分开基础设施与业务验收
```sh
python3 scripts/release.py verify <release-id>
```
核对目标镜像、构建标记、rollout、配置/PVC/服务身份以及未批准的部署变化。
未 Ready 可稍后重新运行只读 verify,不重新 deploy。部署配置符合后仍为
`deployed_pending_acceptance`,不等于已经登录、播放、切片或送检成功。
按策略中 requiredEvidence 逐项记录验收:五角色登录及业务API、多供应商权限、标注与交付
(普通/舍弃/无效/空白)、质检/终审、切片重启恢复、视频首帧/缓冲/实际播放速度。
所有写入测试都遵循前述隔离与授权要求,不能占用生产客户任务。
```sh
python3 scripts/release.py evidence <release-id> --check video-playback \
--result passed --file <local-evidence-file> --note '角色、并发、网络和播放实测结果'
python3 scripts/release.py verify <release-id>
```
完成条件:基础设施复核通过且所有必需证据有效,才进入 `accepted_with_evidence`(包含人工
证据的验收),不是“全自动端到端通过”。未测填 unverified,失败填 failed,不得代填通过。
## 故障、回退和紧急修复
- `status` 显示最后一步、写请求状态、缺失证据和 checkout。换上下文后先运行它。
- 源码/策略变化重新 prepare;部署前同事更新了线上,重新 check/plan 并取得新授权。
- `outcome_unknown` 或 `deploying`:先 verify,不能再次 deploy 同一记录。部分更新不能靠
修改 JSON 清掉历史;保存调查记录并制定新的恢复计划。
- 回退须新的明确授权,核对当前版本与数据兼容性,生成条件式变更。禁止直接 apply 旧完整
Deployment、restore 整库或覆盖客户新增数据。首版不自动执行回退。
- 紧急修复写明“已解决 / 未解决 / 临时取舍 / 后续任务”,关联本地问题票;未验证部分仍保留,
不因上线紧急而自动验收通过。旧规格由新规则显式替代,不同时保留两份有效相反结论。
## 本地试运行与后续范围
```sh
python3 -m unittest discover -s tests/release -p 'test_*.py'
python3 scripts/release.py --help
```
测试使用模拟 Kubernetes,不连接集群、不创建账号或任务。覆盖数据库/认证/处理器回归、
陈旧计划、同事更新、部分失败、证据失效、锁冲突和 legacy 入口阻断。
后续再扩展:服务端部署互斥、OCI provenance 自动核对、隔离环境真实业务验收、受审查的配置
变更及回退执行器。试行阶段避免把这些未实现能力写成已有保证。

View File

@ -0,0 +1,23 @@
{
"schemaVersion": 1,
"namespace": "droboticflow-fanyu",
"database": "drobodata-fanyu",
"bucket": "drobodata-fanyu",
"workspaceMount": "/workspace",
"workspacePVC": "droboticflow-workspace",
"mongoSecret": "droboticflow-mongo-secret",
"deployments": {
"droboticflow-api": "api",
"droboticflow-annotation-processor": "processor"
},
"requiredEvidence": [
"image-provenance", "change-safety", "role-login", "supplier-permissions",
"annotation-delivery", "review-acceptance", "slice-recovery", "video-playback"
],
"localChecks": [
["python3", "-m", "unittest", "discover", "-s", "tests/release", "-p", "test_*.py"],
["npm", "run", "lint:web"],
["npm", "run", "build:web"],
["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery", "-count=1"]
]
}

View File

@ -14,6 +14,10 @@ export DEPLOY_ENV
source "$ROOT_DIR/scripts/load-deploy-env.sh"
DEPLOY_ENV="${DEPLOY_ENV:-}"
# shellcheck source=scripts/lib/build-metadata.sh
if [[ "$DEPLOY_ENV" == "fanyu" || "${K8S_NAMESPACE:-}" == "droboticflow-fanyu" ]]; then
echo "ERROR: Fanyu releases use python3 scripts/release.py; see docs/release/README.md. The legacy broad apply path is blocked before any cluster operation." >&2
exit 2
fi
source "$ROOT_DIR/scripts/lib/build-metadata.sh"
K8S_NAMESPACE="${K8S_NAMESPACE:-droboticflow}"

455
scripts/release.py Normal file
View File

@ -0,0 +1,455 @@
#!/usr/bin/env python3
"""Fanyu image/build-stamp release journal. Secret values are never persisted."""
import argparse
import contextlib
import copy
import datetime as dt
import fcntl
import hashlib
import json
import os
from pathlib import Path
import re
import subprocess
import sys
import uuid
ROOT = Path(__file__).resolve().parents[1]
RECORDS = ROOT / 'local-data/releases'
POLICY = ROOT / 'docs/release/fanyu-policy.json'
SAFE_ENV = ('MONGO_DB', 'DROBOTICFLOW_REQUIRE_SESSION_AUTH',
'ANNOTATION_PROCESSOR_MODE', 'ANNOTATION_PROCESSOR_CONCURRENCY',
'ANNOTATION_INDEX_BACKFILL', 'ANNOTATION_WORKSPACE_CONNECTION_ID',
'ANNOTATION_WORKSPACE_BUCKET', 'WORKSPACE_MOUNT_PATH')
class Blocked(Exception):
pass
def require(ok, message):
if not ok:
raise Blocked(message)
def now():
return dt.datetime.now(dt.timezone.utc).isoformat()
def digest(value):
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(',', ':')).encode()).hexdigest()
def run(argv, *, stdin=None, timeout=60):
try:
p = subprocess.run(argv, cwd=ROOT, input=stdin, text=True, capture_output=True, timeout=timeout)
except (OSError, subprocess.TimeoutExpired):
raise Blocked('Command unavailable or timed out; outcome must be checked before retry.') from None
# Do not echo kubectl output, URLs or credential-bearing errors.
require(p.returncode == 0, 'Command failed: ' + argv[0] + ' (output withheld; no success recorded).')
return p.stdout
def source():
return {'commit': run(['git', 'rev-parse', 'HEAD']).strip(),
'dirty': bool(run(['git', 'status', '--porcelain', '--untracked-files=normal']).strip())}
def policy():
return json.loads(POLICY.read_text())
def record_path(release_id):
require(bool(re.fullmatch(r'[a-zA-Z0-9][a-zA-Z0-9_-]{0,79}', release_id)), 'Invalid release id.')
return RECORDS / release_id / 'release.json'
def save(r):
path = record_path(r['id'])
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
temp = path.with_suffix('.tmp')
with os.fdopen(os.open(temp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600), 'w') as f:
json.dump(r, f, indent=2, ensure_ascii=False)
f.write('\n')
os.replace(temp, path)
def event(r, name):
r['events'].append({'at': now(), 'event': name})
save(r)
@contextlib.contextmanager
def locked(release_id):
path = record_path(release_id)
require(path.exists(), 'Release record not found; use prepare first.')
with path.with_suffix('.lock').open('a') as f:
try:
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
raise Blocked('Another command is updating this release.') from None
yield json.loads(path.read_text())
def fresh(r):
s = source()
require(not s['dirty'], 'Checkout is dirty; commit task changes in a clean checkout first.')
require(s['commit'] == r['commit'], 'Checkout commit changed; prepare a new release.')
require(digest(policy()) == r['policyHash'], 'Release policy changed; prepare a new release.')
def kubectl(r, args, *, stdin=None):
t = r['target']
return run(['kubectl', '--kubeconfig', t['kubeconfig'], '--context', t['context'],
'--request-timeout=20s', '-n', t['namespace']] + args, stdin=stdin, timeout=60)
def get(r, kind, name):
return json.loads(kubectl(r, ['get', kind, name, '-o', 'json']))
def env_values(container, configs):
values = {}
for entry in container.get('envFrom', []):
ref = entry.get('configMapRef')
require(bool(ref), 'Secret/unknown envFrom is unsupported by this preflight.')
for key, value in configs[ref['name']].get('data', {}).items():
values[entry.get('prefix', '') + key] = value
for entry in container.get('env', []):
key = entry['name']
if 'value' in entry:
values[key] = entry['value']
elif 'configMapKeyRef' in entry.get('valueFrom', {}):
ref = entry['valueFrom']['configMapKeyRef']
require(ref['key'] in configs[ref['name']].get('data', {}), 'Missing ConfigMap key.')
values[key] = configs[ref['name']]['data'][ref['key']]
else:
# Explicit unknowns must override envFrom rather than silently use its value.
values[key] = None
return values
def inspect_deployment(deployment, container_name, configs, p):
spec = deployment['spec']
pod = spec['template']['spec']
containers = pod['containers']
matches = [(i, c) for i, c in enumerate(containers) if c['name'] == container_name]
require(len(matches) == 1, 'Expected deployment container missing.')
index, c = matches[0]
e = env_values(c, configs)
require(e.get('MONGO_DB') == p['database'], 'Database name mismatch.')
require(e.get('ANNOTATION_WORKSPACE_BUCKET') == p['bucket'], 'Workspace bucket mismatch.')
require(bool(e.get('ANNOTATION_WORKSPACE_CONNECTION_ID')), 'Missing workspace storage connection.')
require(e.get('WORKSPACE_MOUNT_PATH') == p['workspaceMount'], 'Workspace mount mismatch.')
mongo = next((x for x in c.get('env', []) if x['name'] == 'MONGO_URI'), {})
require(mongo.get('valueFrom', {}).get('secretKeyRef') == {'name': p['mongoSecret'], 'key': 'MONGO_URI'},
'Mongo secret binding mismatch; preflight does not read secret values.')
mounts = {v['name']: v['mountPath'] for v in c.get('volumeMounts', [])}
claims = [v for v in pod.get('volumes', []) if mounts.get(v['name']) == p['workspaceMount']]
require(len(claims) == 1 and claims[0].get('persistentVolumeClaim', {}).get('claimName') == p['workspacePVC'],
'Workspace PVC binding mismatch.')
require(spec.get('replicas', 1) > 0, 'Deployment has no replicas.')
if container_name == 'api':
require(e.get('DROBOTICFLOW_REQUIRE_SESSION_AUTH') in ('true', '1'), 'Session authentication disabled.')
require(e.get('ANNOTATION_PROCESSOR_MODE') == 'external', 'API must use external processor.')
else:
require(e.get('ANNOTATION_PROCESSOR_MODE') == 'processor', 'Processor mode mismatch.')
require(e.get('ANNOTATION_PROCESSOR_CONCURRENCY') == '1', 'First release profile requires concurrency 1.')
require(e.get('ANNOTATION_INDEX_BACKFILL') == '0', 'Historical backfill must remain disabled.')
require(c.get('resources', {}).get('limits', {}).get('memory') == '8Gi', 'First release profile requires processor memory limit 8Gi.')
require(spec.get('replicas') == 1 and spec.get('strategy', {}).get('type') == 'Recreate',
'Processor must have one replica and Recreate strategy.')
status = deployment.get('status', {})
ready = (status.get('observedGeneration', 0) >= deployment['metadata'].get('generation', 1)
and status.get('updatedReplicas', 0) == spec.get('replicas', 1)
and status.get('readyReplicas', 0) == spec.get('replicas', 1)
and status.get('replicas', 0) == spec.get('replicas', 1))
m = deployment['metadata']
neutral = copy.deepcopy(spec)
nc = neutral['template']['spec']['containers'][index]
nc['image'] = '<release-image>'
nc['env'] = [x for x in nc.get('env', []) if x['name'] != 'DROBOTICFLOW_BUILD_COMMIT']
stamp = next((i for i, x in enumerate(c.get('env', [])) if x['name'] == 'DROBOTICFLOW_BUILD_COMMIT'), None)
return {'uid': m['uid'], 'specHash': digest(spec), 'nonReleaseHash': digest(neutral),
'buildCommitIndex': stamp, 'buildCommit': e.get('DROBOTICFLOW_BUILD_COMMIT'), 'image': c['image'],
'resourceVersion': m['resourceVersion'], 'containerIndex': index,
'ready': ready, 'env': {k: e.get(k) for k in SAFE_ENV}}
def snapshot(r):
t, p = r['target'], policy()
view = json.loads(kubectl(r, ['config', 'view', '--minify', '-o', 'json']))
require(view['clusters'][0]['cluster']['server'] == t['server'], 'Kube API server mismatch.')
ns = get(r, 'namespace', t['namespace'])
require(ns['metadata']['uid'] == t['namespaceUID'], 'Namespace UID mismatch.')
deployments = {n: get(r, 'deployment', n) for n in p['deployments']}
config_names = set()
for d in deployments.values():
for c in d['spec']['template']['spec']['containers']:
config_names.update(x['configMapRef']['name'] for x in c.get('envFrom', []) if 'configMapRef' in x)
config_names.update(x['valueFrom']['configMapKeyRef']['name'] for x in c.get('env', [])
if 'configMapKeyRef' in x.get('valueFrom', {}))
configs = {n: get(r, 'configmap', n) for n in sorted(config_names)}
deps = {n: inspect_deployment(d, p['deployments'][n], configs, p) for n, d in deployments.items()}
envs = [d['env'] for d in deps.values()]
require(len({e['ANNOTATION_WORKSPACE_CONNECTION_ID'] for e in envs}) == 1, 'API/processor storage connections differ.')
pvc = get(r, 'pvc', p['workspacePVC'])
require(pvc.get('status', {}).get('phase') == 'Bound', 'Workspace PVC is not Bound.')
svc = get(r, 'service', 'droboticflow-api-public-eip')
ann = svc['metadata'].get('annotations', {})
return {'at': now(), 'deployments': deps,
'configs': {n: {'uid': x['metadata']['uid'], 'hash': digest(x.get('data', {}))} for n, x in configs.items()},
'pvc': {'uid': pvc['metadata']['uid'], 'specHash': digest(pvc['spec'])},
'publicService': {'uid': svc['metadata']['uid'], 'specHash': digest(svc['spec']),
'annotationHash': digest(ann),
'bandwidthMbps': ann.get('service.beta.kubernetes.io/cce-elastic-ip-bandwidth-in-mbps', 'unknown')}}
def stable(s):
return {**{k: s[k] for k in ('configs', 'pvc', 'publicService')},
'deployments': {n: {k: d[k] for k in ('uid', 'specHash', 'image')} for n, d in s['deployments'].items()}}
def plan_data(r):
return {k: r[k] for k in ('id', 'commit', 'image', 'policyHash', 'target', 'baseline', 'localChecks', 'purpose', 'rollback')}
def prepare(a):
p, s = policy(), source()
branch = run(['git', 'branch', '--show-current']).strip()
require(not re.search(r'(^|/)poc($|[-/])', branch), 'POC branches cannot prepare Fanyu releases.')
require(not s['dirty'], 'Prepare requires a clean checkout (use an isolated checkout for unrelated edits).')
require(bool(re.fullmatch(r'[^\s@]+@sha256:[0-9a-f]{64}', a.image)), 'Use an immutable repository@sha256 image.')
require(a.server.startswith('https://') and '@' not in a.server and '?' not in a.server, 'Use a credential-free HTTPS Kubernetes server URL.')
release_id = a.id or dt.datetime.now(dt.timezone.utc).strftime('%Y%m%dT%H%M%SZ-') + uuid.uuid4().hex[:8]
require(not record_path(release_id).exists(), 'Release id already exists.')
r = {'schemaVersion': 1, 'id': release_id, 'createdAt': now(), 'commit': s['commit'],
'image': a.image, 'policyHash': digest(p), 'purpose': a.purpose, 'rollback': a.rollback,
'target': {'kubeconfig': str(Path(a.kubeconfig).resolve()), 'context': a.context,
'server': a.server.rstrip('/'), 'namespace': p['namespace'], 'namespaceUID': a.namespace_uid},
'state': 'prepared', 'localChecks': [], 'baseline': None, 'planHash': None,
'operations': [], 'evidence': {}, 'events': []}
event(r, 'prepared; no network access')
print(json.dumps({'id': r['id'], 'state': r['state']}, ensure_ascii=False))
def check(r, a):
fresh(r)
require(not r['operations'], 'A deployment was attempted; check outcome with verify, never rewrite its plan.')
r['planHash'] = None
r['evidence'] = {}
r['state'] = 'checking'
event(r, 'check started; previous plan invalidated')
if a.local:
r['localChecks'] = []
for argv in policy()['localChecks']:
started = now()
try:
output = run(argv, timeout=1800)
except Blocked:
r['localChecks'].append({'argv': argv, 'startedAt': started, 'result': 'failed'})
r['state'] = 'blocked'
event(r, 'local check failed')
raise
r['localChecks'].append({'argv': argv, 'startedAt': started, 'result': 'passed', 'outputHash': digest(output)})
save(r)
fresh(r)
if a.cluster:
# Never preserve an earlier successful snapshot when a new read fails.
r['baseline'] = None
save(r)
try:
s = snapshot(r)
require(all(d['ready'] for d in s['deployments'].values()), 'Current deployments are not fully ready.')
r['baseline'] = s
except Blocked:
r['state'] = 'blocked'
event(r, 'cluster check failed')
raise
r['state'] = 'checked_partial'
event(r, 'checks recorded; plan must still be generated')
def make_plan(r):
fresh(r)
require(not r['operations'], 'Deployment already attempted.')
require([x['argv'] for x in r['localChecks']] == policy()['localChecks']
and all(x['result'] == 'passed' for x in r['localChecks']), 'Required local checks incomplete.')
require(r['baseline'] is not None, 'Read-only cluster check required.')
r['planHash'] = digest(plan_data(r))
r['state'] = 'awaiting_approval'
event(r, 'plan prepared; no authorization inferred')
print(json.dumps({'planHash': r['planHash'], 'target': r['target'], 'changes': [
{'deployment': n, 'from': d['image'], 'to': r['image'], 'buildCommit': r['commit']} for n, d in r['baseline']['deployments'].items()],
'bandwidthMbps': r['baseline']['publicService']['bandwidthMbps'],
'warnings': ['Only image/build stamp changes; startup migrations require change-safety evidence.',
'Image-to-source provenance requires independent evidence.',
'Bandwidth adequacy and business acceptance remain manual evidence.'],
'rollback': r['rollback']}, ensure_ascii=False, indent=2))
def deploy(r, a):
fresh(r)
require(r['state'] == 'awaiting_approval' and not r['operations'], 'No fresh deployable plan; inspect status.')
require(r['planHash'] == a.confirm_plan == digest(plan_data(r)), 'Plan hash does not match.')
require(bool(a.approval_ref.strip()), 'Record the explicit user approval reference for this exact plan.')
age = (dt.datetime.now(dt.timezone.utc) - dt.datetime.fromisoformat(r['baseline']['at'])).total_seconds()
require(0 <= age <= 3600, 'Cluster check is older than one hour; recheck and obtain a new plan approval.')
for key in ('image-provenance', 'change-safety'):
require(key in r['evidence'] and evidence_valid(r['evidence'][key], r), key + ' evidence required before deployment.')
current = snapshot(r)
require(stable(current) == stable(r['baseline']), 'Live resources changed since check; prepare/check a new plan.')
require(all(d['ready'] for d in current['deployments'].values()), 'Live deployment not ready.')
r['approval'] = {'reference': a.approval_ref, 'at': now(), 'planHash': r['planHash']}
r['state'] = 'deploying'
event(r, 'explicit approval reference recorded; deployment starting')
for name in current['deployments']:
latest = snapshot(r)
require(all(latest[k] == current[k] for k in ('configs', 'pvc', 'publicService')), 'Configuration changed during deployment; inspect partial outcome.')
for n, live in latest['deployments'].items():
old = current['deployments'][n]
done = any(o['deployment'] == n and o['result'] == 'acknowledged' for o in r['operations'])
require(live['uid'] == old['uid'] and live['nonReleaseHash'] == old['nonReleaseHash'], 'Deployment changed during release.')
require((live['image'] == r['image'] and live['buildCommit'] == r['commit']) if done else live['specHash'] == old['specHash'], 'Image/configuration drift during release.')
dep = latest['deployments'][name]
patch = [{'op': 'test', 'path': '/metadata/resourceVersion', 'value': dep['resourceVersion']},
{'op': 'test', 'path': '/metadata/uid', 'value': dep['uid']},
{'op': 'test', 'path': f"/spec/template/spec/containers/{dep['containerIndex']}/image", 'value': dep['image']},
{'op': 'replace', 'path': f"/spec/template/spec/containers/{dep['containerIndex']}/image", 'value': r['image']}]
stamp = {'name': 'DROBOTICFLOW_BUILD_COMMIT', 'value': r['commit']}
env_path = f"/spec/template/spec/containers/{dep['containerIndex']}/env/"
patch.append({'op': 'add' if dep['buildCommitIndex'] is None else 'replace',
'path': env_path + ('-' if dep['buildCommitIndex'] is None else str(dep['buildCommitIndex'])), 'value': stamp})
op = {'deployment': name, 'before': dep['image'], 'desired': r['image'], 'at': now(), 'result': 'unknown'}
r['operations'].append(op)
save(r) # Persist intent before sending any mutation.
try:
kubectl(r, ['patch', 'deployment', name, '--type=json', '--patch-file=/dev/stdin'], stdin=json.dumps(patch))
except Blocked:
r['state'] = 'outcome_unknown'
event(r, 'patch outcome unknown; verify before any further operation')
raise
op['result'] = 'acknowledged'
save(r)
r['state'] = 'deployed_pending_verification'
event(r, 'image patches acknowledged; rollout and acceptance not yet verified')
def record_evidence(r, a):
require(r['planHash'] is not None, 'Generate a plan before attaching evidence.')
require(a.check in policy()['requiredEvidence'], 'Unknown evidence category.')
require(a.check in ('image-provenance', 'change-safety') or r['state'] in ('deployed_pending_acceptance', 'accepted_with_evidence'),
'Business evidence can be attached only after infrastructure verification.')
path = Path(a.file).resolve()
require(path.is_file() and path.stat().st_size > 0, 'Evidence must be a nonempty local file.')
r['evidence'][a.check] = {'result': a.result, 'path': str(path),
'sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
'at': now(), 'planHash': r['planHash'], 'note': a.note,
'kind': 'operator-attested; content not automatically evaluated'}
if a.result != 'passed' and r['state'] == 'accepted_with_evidence':
r['state'] = 'deployed_pending_acceptance'
event(r, 'evidence recorded: ' + a.check)
def evidence_valid(e, r):
path = Path(e['path'])
return (e['result'] == 'passed' and e['planHash'] == r['planHash'] and path.is_file()
and hashlib.sha256(path.read_bytes()).hexdigest() == e['sha256'])
def verify(r):
require(bool(r['operations']), 'No deployment attempt in this record.')
# Failure cannot retain a stale accepted state.
r['state'] = 'deployed_pending_verification'
save(r)
require(digest(policy()) == r['policyHash'], 'Policy changed; verify with the original policy revision.')
s = snapshot(r)
r['lastObservation'] = s
expected = r['baseline']
require(all(s[k] == expected[k] for k in ('configs', 'pvc', 'publicService')), 'Configuration drift detected.')
for n, d in s['deployments'].items():
require(d['uid'] == expected['deployments'][n]['uid'], 'Deployment was replaced.')
require(d['nonReleaseHash'] == expected['deployments'][n]['nonReleaseHash'], 'Deployment settings drifted beyond the approved image/build stamp.')
require(d['image'] == r['image'] and d['buildCommit'] == r['commit'] and d['ready'], 'Target image rollout incomplete; outcome remains pending.')
r['state'] = 'deployed_pending_acceptance'
missing = [k for k in policy()['requiredEvidence'] if k not in r['evidence'] or not evidence_valid(r['evidence'][k], r)]
if not missing:
r['state'] = 'accepted_with_evidence'
event(r, 'infrastructure verified; remaining evidence: ' + ', '.join(missing))
def status(r):
p = policy()
missing = [k for k in p['requiredEvidence'] if k not in r['evidence'] or not evidence_valid(r['evidence'][k], r)]
state = 'acceptance_evidence_invalid' if r['state'] == 'accepted_with_evidence' and missing else r['state']
print(json.dumps({'id': r['id'], 'state': state, 'recordedState': r['state'], 'commit': r['commit'], 'image': r['image'],
'policyMatches': digest(p) == r['policyHash'], 'checkout': source(),
'planHash': r['planHash'], 'missingEvidence': missing,
'operations': r['operations'], 'lastEvent': r['events'][-1],
'next': 'Resume from recorded state; fresh approval is required for any new deployment or rollback.'},
indent=2, ensure_ascii=False))
def main():
parser = argparse.ArgumentParser(description=__doc__)
sub = parser.add_subparsers(dest='command', required=True)
prep = sub.add_parser('prepare', help='Offline: create a release record from a clean committed checkout.')
prep.add_argument('--id')
for name in ('image', 'kubeconfig', 'context', 'server', 'namespace-uid', 'purpose', 'rollback'):
prep.add_argument('--' + name, required=True)
for cmd in ('check', 'plan', 'deploy', 'verify', 'status', 'evidence'):
c = sub.add_parser(cmd)
if cmd == 'status':
c.add_argument('id', nargs='?')
else:
c.add_argument('id')
if cmd == 'check':
c.add_argument('--local', action='store_true')
c.add_argument('--cluster', action='store_true')
elif cmd == 'deploy':
c.add_argument('--confirm-plan', required=True)
c.add_argument('--approval-ref', required=True)
elif cmd == 'evidence':
c.add_argument('--check', required=True)
c.add_argument('--result', choices=['passed', 'failed', 'unverified'], required=True)
c.add_argument('--file', required=True)
c.add_argument('--note', required=True)
a = parser.parse_args()
try:
if a.command == 'prepare':
prepare(a)
return
if a.command == 'status' and not a.id:
summaries = []
for path in sorted(RECORDS.glob('*/release.json')):
item = json.loads(path.read_text())
summaries.append({k: item[k] for k in ('id', 'state', 'commit', 'createdAt')})
print(json.dumps(summaries, indent=2, ensure_ascii=False))
return
with locked(a.id) as r:
if a.command == 'check':
require(a.local or a.cluster, 'Choose --local and/or --cluster.')
check(r, a)
elif a.command == 'plan':
make_plan(r)
elif a.command == 'deploy':
try:
deploy(r, a)
except Blocked:
if r['state'] == 'deploying':
r['state'] = 'outcome_unknown'
event(r, 'interrupted; inspect partial outcome before further changes')
raise
elif a.command == 'verify':
verify(r)
elif a.command == 'evidence':
record_evidence(r, a)
status(r)
except (Blocked, KeyError, ValueError, OSError) as e:
# Validation errors are authored here; generic errors omit possibly sensitive values.
print('BLOCKED: ' + (str(e) if isinstance(e, Blocked) else type(e).__name__), file=sys.stderr)
sys.exit(2)
if __name__ == '__main__':
main()

View File

@ -0,0 +1,332 @@
"""No cluster or customer credentials: exercise release transitions with fake Kubernetes."""
import argparse
import copy
import importlib.util
import json
import os
from pathlib import Path
import subprocess
import shutil
import sys
import tempfile
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[2]
spec = importlib.util.spec_from_file_location('release', ROOT / 'scripts/release.py')
m = importlib.util.module_from_spec(spec)
spec.loader.exec_module(m)
IMAGE = 'registry.example/api@sha256:' + 'a' * 64
def fixture(name, container):
config = {'data': {'MONGO_DB': 'drobodata-fanyu', 'DROBOTICFLOW_REQUIRE_SESSION_AUTH': 'true',
'ANNOTATION_PROCESSOR_MODE': 'external', 'ANNOTATION_WORKSPACE_BUCKET': 'drobodata-fanyu',
'ANNOTATION_WORKSPACE_CONNECTION_ID': 'storage-example', 'WORKSPACE_MOUNT_PATH': '/workspace'},
'metadata': {'uid': 'config-uid'}}
env = [{'name': 'MONGO_URI', 'valueFrom': {'secretKeyRef': {'name': 'droboticflow-mongo-secret', 'key': 'MONGO_URI'}}}]
if container == 'processor':
env += [{'name': k, 'value': v} for k, v in {
'ANNOTATION_PROCESSOR_MODE': 'processor', 'ANNOTATION_PROCESSOR_CONCURRENCY': '1',
'ANNOTATION_INDEX_BACKFILL': '0'}.items()]
d = {'metadata': {'name': name, 'uid': name + '-uid', 'resourceVersion': '1', 'generation': 1},
'spec': {'replicas': 1, 'strategy': {'type': 'Recreate'}, 'template': {'spec': {
'containers': [{'name': container, 'image': 'registry.example/api@sha256:' + 'b' * 64,
'envFrom': [{'configMapRef': {'name': 'config'}}], 'env': env,
'resources': {'limits': {'memory': '8Gi'}},
'volumeMounts': [{'name': 'workspace', 'mountPath': '/workspace'}]}],
'volumes': [{'name': 'workspace', 'persistentVolumeClaim': {'claimName': 'droboticflow-workspace'}}]}}},
'status': {'observedGeneration': 1, 'replicas': 1, 'updatedReplicas': 1, 'readyReplicas': 1}}
return d, config
class ReleaseTests(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
self.records = Path(self.tmp.name) / 'records'
self.p = m.policy()
self.p['localChecks'] = [['fake-check']]
self.start_patch('RECORDS', self.records)
self.start_patch('policy', lambda: copy.deepcopy(self.p))
self.start_patch('source', lambda: {'commit': 'c' * 40, 'dirty': False})
self.start_patch('run', lambda *args, **kwargs: 'feature/release')
self.ds = {}
for n, c in self.p['deployments'].items():
self.ds[n], config = fixture(n, c)
self.configs = {'config': config}
self.writes = []
self.fail_patch = None
self.before_snapshot = None
self.start_patch('snapshot', self.snapshot)
self.start_patch('kubectl', self.kubectl)
m.prepare(argparse.Namespace(id='test', image=IMAGE, context='fanyu', server='https://cluster.example',
namespace_uid='namespace-uid', kubeconfig='/example/config', purpose='test', rollback='review old image'))
self.r = self.load()
def start_patch(self, name, value):
p = patch.object(m, name, value)
p.start()
self.addCleanup(p.stop)
def load(self):
return json.loads(m.record_path('test').read_text())
def snapshot(self, r):
if self.before_snapshot:
self.before_snapshot()
return {'at': m.now(), 'deployments': {
n: m.inspect_deployment(d, self.p['deployments'][n], self.configs, self.p) for n, d in self.ds.items()},
'configs': {'config': {'uid': 'config-uid', 'hash': m.digest(self.configs)}},
'pvc': {'uid': 'pvc-uid'}, 'publicService': {'bandwidthMbps': '5'}}
def kubectl(self, r, args, *, stdin=None):
self.assertEqual(args[:2], ['patch', 'deployment'])
name = args[2]
self.writes.append(name)
if self.fail_patch == len(self.writes):
raise m.Blocked('simulated timeout')
ops = json.loads(stdin)
self.assertEqual(ops[0]['path'], '/metadata/resourceVersion')
d = self.ds[name]
self.assertEqual(ops[0]['value'], d['metadata']['resourceVersion'])
c = d['spec']['template']['spec']['containers'][0]
c['image'] = ops[3]['value']
if ops[4]['op'] == 'add':
c['env'].append(ops[4]['value'])
else:
c['env'][int(ops[4]['path'].split('/')[-1])] = ops[4]['value']
d['metadata']['resourceVersion'] = str(int(d['metadata']['resourceVersion']) + 1)
return '{}'
def plan(self):
m.check(self.r, argparse.Namespace(local=True, cluster=True))
m.make_plan(self.r)
def evidence(self, key, result='passed'):
f = Path(self.tmp.name) / (key + '.txt')
f.write_text('synthetic test evidence for current release')
m.record_evidence(self.r, argparse.Namespace(check=key, file=str(f), result=result, note='test evidence only'))
return f
def approve(self):
self.evidence('image-provenance')
self.evidence('change-safety')
return argparse.Namespace(confirm_plan=self.r['planHash'], approval_ref='user-message:test')
def test_prepare_is_offline_and_private(self):
self.assertEqual(self.r['state'], 'prepared')
self.assertEqual(m.record_path('test').stat().st_mode & 0o777, 0o600)
self.assertFalse(self.writes)
def test_cannot_plan_without_both_checks(self):
with self.assertRaises(m.Blocked):
m.make_plan(self.r)
m.check(self.r, argparse.Namespace(local=True, cluster=False))
with self.assertRaises(m.Blocked):
m.make_plan(self.r)
def test_local_failure_cannot_pass(self):
with patch.object(m, 'run', side_effect=m.Blocked('failure')):
with self.assertRaises(m.Blocked):
m.check(self.r, argparse.Namespace(local=True, cluster=False))
self.assertEqual(self.load()['state'], 'blocked')
def test_recheck_invalidates_previous_plan_and_failed_snapshot(self):
self.plan()
with patch.object(m, 'snapshot', side_effect=m.Blocked('offline')):
with self.assertRaises(m.Blocked):
m.check(self.r, argparse.Namespace(local=False, cluster=True))
saved = self.load()
self.assertIsNone(saved['baseline'])
self.assertIsNone(saved['planHash'])
def test_source_and_policy_change_block_deployment(self):
self.plan()
a = self.approve()
for s in [{'commit': 'd' * 40, 'dirty': False}, {'commit': 'c' * 40, 'dirty': True}]:
with patch.object(m, 'source', return_value=s), self.assertRaises(m.Blocked):
m.deploy(self.r, a)
self.p['database'] = 'another'
with self.assertRaises(m.Blocked):
m.deploy(self.r, a)
self.assertFalse(self.writes)
def test_exact_plan_and_explicit_reference_required(self):
self.plan()
a = self.approve()
a.confirm_plan = 'wrong'
with self.assertRaises(m.Blocked):
m.deploy(self.r, a)
a.confirm_plan = self.r['planHash']
a.approval_ref = ''
with self.assertRaises(m.Blocked):
m.deploy(self.r, a)
self.assertFalse(self.writes)
def test_changed_evidence_rejected(self):
self.plan()
a = self.approve()
Path(self.r['evidence']['image-provenance']['path']).write_text('changed')
with self.assertRaises(m.Blocked):
m.deploy(self.r, a)
self.assertFalse(self.writes)
def test_colleague_live_change_blocks_before_patch(self):
self.plan()
a = self.approve()
self.ds['droboticflow-api']['spec']['replicas'] = 3
with self.assertRaises(m.Blocked):
m.deploy(self.r, a)
self.assertFalse(self.writes)
def test_config_change_between_patches_stops_second(self):
self.plan()
a = self.approve()
def change():
if len(self.writes) == 1:
self.configs['config']['data']['unrelated'] = 'changed'
self.before_snapshot = change
with self.assertRaises(m.Blocked):
m.deploy(self.r, a)
self.assertEqual(len(self.writes), 1)
self.assertEqual(len(self.load()['operations']), 1)
def test_timeout_preserves_unknown_intent_and_cannot_redeploy(self):
self.plan()
a = self.approve()
self.fail_patch = 2
with self.assertRaises(m.Blocked):
m.deploy(self.r, a)
saved = self.load()
self.assertEqual(saved['state'], 'outcome_unknown')
self.assertEqual([o['result'] for o in saved['operations']], ['acknowledged', 'unknown'])
with self.assertRaises(m.Blocked):
m.deploy(saved, a)
with self.assertRaises(m.Blocked):
m.verify(saved)
self.assertNotEqual(self.load()['state'], 'accepted_with_evidence')
def test_deploy_and_ready_are_not_business_acceptance(self):
self.plan()
m.deploy(self.r, self.approve())
self.assertEqual(self.r['state'], 'deployed_pending_verification')
m.verify(self.r)
self.assertEqual(self.r['state'], 'deployed_pending_acceptance')
for key in self.p['requiredEvidence']:
if key not in self.r['evidence']:
self.evidence(key)
m.verify(self.r)
self.assertEqual(self.r['state'], 'accepted_with_evidence')
Path(self.r['evidence']['video-playback']['path']).unlink()
m.verify(self.r)
self.assertEqual(self.r['state'], 'deployed_pending_acceptance')
def test_no_predeployment_business_evidence(self):
self.plan()
with self.assertRaises(m.Blocked):
self.evidence('video-playback')
def test_verify_rejects_unplanned_settings(self):
self.plan()
m.deploy(self.r, self.approve())
self.ds['droboticflow-api']['spec']['template']['spec']['containers'][0]['resources'] = {'limits': {'memory': '1Gi'}}
with self.assertRaises(m.Blocked):
m.verify(self.r)
self.assertEqual(self.load()['state'], 'deployed_pending_verification')
def test_configuration_regressions_block(self):
for key, value in [('MONGO_DB', 'droboticflow'), ('DROBOTICFLOW_REQUIRE_SESSION_AUTH', 'false'),
('ANNOTATION_PROCESSOR_MODE', 'inprocess'), ('ANNOTATION_WORKSPACE_BUCKET', ''),
('WORKSPACE_MOUNT_PATH', '/wrong')]:
configs = copy.deepcopy(self.configs)
configs['config']['data'][key] = value
with self.subTest(key=key), self.assertRaises(m.Blocked):
m.inspect_deployment(self.ds['droboticflow-api'], 'api', configs, self.p)
def test_unknown_explicit_env_does_not_fall_back_to_envfrom(self):
d = copy.deepcopy(self.ds['droboticflow-api'])
d['spec']['template']['spec']['containers'][0]['env'].append({
'name': 'MONGO_DB', 'valueFrom': {'secretKeyRef': {'name': 'other', 'key': 'db'}}})
with self.assertRaises(m.Blocked):
m.inspect_deployment(d, 'api', self.configs, self.p)
def test_snapshot_identity_and_no_secret_reads(self):
r = self.r
replies = {'namespace': {'metadata': {'uid': 'namespace-uid'}},
'pvc': {'metadata': {'uid': 'pvc'}, 'spec': {}, 'status': {'phase': 'Bound'}},
'service': {'metadata': {'uid': 'svc'}, 'spec': {}}}
reads = []
def fake_get(_, kind, name):
reads.append(kind)
if kind == 'deployment': return self.ds[name]
if kind == 'configmap': return self.configs[name]
return replies[kind]
# setUp replaced snapshot; recover implementation through a second module import.
real = importlib.util.module_from_spec(spec)
spec.loader.exec_module(real)
with patch.object(real, 'policy', return_value=self.p), patch.object(real, 'get', side_effect=fake_get), patch.object(real, 'kubectl', return_value=json.dumps({'clusters': [{'cluster': {'server': r['target']['server']}}]})):
real.snapshot(r)
self.assertNotIn('secret', reads)
replies['namespace']['metadata']['uid'] = 'wrong'
with self.assertRaises(real.Blocked): real.snapshot(r)
def test_stale_cluster_check_blocks_deploy(self):
self.plan()
self.r['baseline']['at'] = '2000-01-01T00:00:00+00:00'
m.make_plan(self.r)
with self.assertRaises(m.Blocked): m.deploy(self.r, self.approve())
self.assertFalse(self.writes)
def test_record_path_traversal_rejected(self):
with self.assertRaises(m.Blocked): m.record_path('../other')
def test_record_lock_prevents_two_writers(self):
with m.locked('test'):
with self.assertRaises(m.Blocked):
with m.locked('test'): pass
class EntrypointTests(unittest.TestCase):
def test_cli_persists_and_resumes_offline(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
(root / 'scripts').mkdir()
(root / 'docs/release').mkdir(parents=True)
shutil.copy(ROOT / 'scripts/release.py', root / 'scripts/release.py')
shutil.copy(ROOT / 'docs/release/fanyu-policy.json', root / 'docs/release/fanyu-policy.json')
(root / '.gitignore').write_text('local-data/\n')
for args in [ ['git', 'init', '-q'], ['git', 'add', '.'],
['git', '-c', 'user.name=ReleaseTest', '-c', 'user.email=test@example.invalid',
'commit', '-qm', 'fixture'] ]:
subprocess.run(args, cwd=root, check=True, capture_output=True)
def cli(*args):
return subprocess.run([sys.executable, 'scripts/release.py', *args], cwd=root,
text=True, capture_output=True)
r = cli('prepare', '--id', 'offline-demo', '--image', IMAGE, '--context', 'example',
'--kubeconfig', '/does-not-exist', '--server', 'https://cluster.example',
'--namespace-uid', 'example-uid', '--purpose', 'offline smoke', '--rollback', 'not deployed')
self.assertEqual(r.returncode, 0, r.stderr)
resumed = cli('status', 'offline-demo')
self.assertEqual(resumed.returncode, 0, resumed.stderr)
self.assertEqual(json.loads(resumed.stdout)['state'], 'prepared')
blocked = cli('deploy', 'offline-demo', '--confirm-plan', 'unprepared', '--approval-ref', 'none')
self.assertEqual(blocked.returncode, 2)
self.assertIn('No fresh deployable plan', blocked.stderr)
def test_legacy_fanyu_deploy_stops_before_kubectl(self):
with tempfile.TemporaryDirectory() as tmp:
fake = Path(tmp) / 'kubectl'
called = Path(tmp) / 'called'
fake.write_text('#!/bin/sh\ntouch "' + str(called) + '"\nexit 99\n')
fake.chmod(0o755)
env = dict(os.environ, PATH=tmp + ':' + os.environ['PATH'], DEPLOY_ENV='fanyu',
DROBOTICFLOW_DEPLOY_ENV_LOADED='test', K8S_NAMESPACE='droboticflow-fanyu')
p = subprocess.run(['bash', 'scripts/deploy-k8s.sh'], cwd=ROOT, env=env, text=True, capture_output=True)
self.assertEqual(p.returncode, 2)
self.assertIn('scripts/release.py', p.stderr)
self.assertFalse(called.exists())
if __name__ == '__main__':
unittest.main()