forked from longtao.wu/droboticflow
🛡️ Add resumable Fanyu release workflow and deployment guards
This commit is contained in:
parent
2379bd586c
commit
a2aad99de2
@ -21,6 +21,11 @@
|
||||
- When push is approved, push the user's personal feature branch only; never push `dev` or `poc` directly.
|
||||
- Do not stage or commit unrelated user changes already present in the working tree.
|
||||
|
||||
## Release Workflow
|
||||
|
||||
- For release preparation, deployment, rollback, or resuming release work, read `docs/release/README.md` first and run `python3 scripts/release.py status <release-id>` when a record exists. Follow the recorded state and evidence; a successful rollout alone is not business acceptance.
|
||||
- Fanyu image releases use `scripts/release.py`. Configuration changes or migrations outside its supported scope require a separately reviewed plan and the corresponding user authorization.
|
||||
|
||||
## Credentials
|
||||
|
||||
- Never hardcode credentials (BOS AK/SK, Mongo, Gitea token, cluster kubeconfig) in any tracked file.
|
||||
|
||||
155
docs/release/README.md
Normal file
155
docs/release/README.md
Normal file
@ -0,0 +1,155 @@
|
||||
# 泛宇发布试行流程
|
||||
|
||||
## 开始或恢复工作
|
||||
|
||||
发布、部署故障、回退、上下文恢复时先读本页。状态的权威来源是
|
||||
`local-data/releases/<id>/release.json`。先运行 `python3 scripts/release.py status` 列出记录,
|
||||
再用 `python3 scripts/release.py status <id>` 查看缺失证据与下一步。
|
||||
历史聊天和“Pod 已 Ready”不代表业务验收完成。使用命令更新记录,保留失败和未验证项。
|
||||
|
||||
首版只支持 **Fanyu 现有 API 与切片处理器的同一不可变镜像更新**,并同步容器的
|
||||
`DROBOTICFLOW_BUILD_COMMIT`。保留 ConfigMap、Secret、PVC、Service、资源和副本数。
|
||||
需要更改配置、其他组件或数据库时,单独设计并审查变更计划;首版不提供绕过参数。
|
||||
`deploy-k8s.sh` 的 Fanyu 全量 apply 入口已阻断,其他环境的原流程不变。
|
||||
|
||||
这是命令行防误操作流程,不是安全边界:操作者仍可以直接运行 kubectl。团队级强制执行
|
||||
需后续收敛集群写权限。首版没有自动回退、自动重试、压测或自动创建测试数据。
|
||||
|
||||
## 授权与数据边界
|
||||
|
||||
- 本地 prepare/check/plan 不构成部署、PR 或客户数据写入授权。
|
||||
- `check --cluster` / `verify` 只读集群资源;实际执行也须符合本次会话的环境访问授权。
|
||||
- 展示 plan 后,获得用户对该计划的明确部署授权,再执行 deploy。`--approval-ref` 记录
|
||||
用户消息引用,不能由代理虚构,也不是系统自动验证的授权凭据。计划变化后重新授权。
|
||||
- 应用启动本身可能执行迁移,镜像更新不能保证零业务写入。部署前必须审查候选版本启动
|
||||
行为,记录 change-safety 证据;有新增迁移或客户数据改写时,停止并取得精确操作授权。
|
||||
- 生产业务验证默认只读。领取、保存、送检等写测试须另外批准隔离测试项目、账号和范围。
|
||||
- 记录、日志、回退材料均在 Git 忽略的 local-data;不要填写凭证或复制客户正文到记录。
|
||||
|
||||
## 顺序与完成条件
|
||||
|
||||
### 1. Prepare:绑定候选版本和目标身份(离线)
|
||||
|
||||
使用干净、已提交的工作树;其他人的未提交改动保留在原工作树,使用隔离工作树发布。
|
||||
镜像必须使用 `repository@sha256:...`,不接受可变 tag。先构建、核对镜像来源,再准备发布。
|
||||
目标 server、context、namespace UID 从已核实的集群身份取得,不能从一次错误连接中直接认可。
|
||||
工具检查 namespace UID 及 API server,namespace 名字相同也不能蒙混过关。
|
||||
|
||||
```sh
|
||||
python3 scripts/release.py prepare --id <release-id> \
|
||||
--image <registry/repository@sha256:digest> \
|
||||
--kubeconfig <local-kubeconfig-path> --context <verified-context> \
|
||||
--server <verified-https-api-server> --namespace-uid <verified-uid> \
|
||||
--purpose '本次问题和修改范围' \
|
||||
--rollback '旧镜像、兼容性结论及需要先核对的任务状态'
|
||||
```
|
||||
|
||||
完成条件:记录有 commit、镜像摘要、策略摘要、目标身份、目的和回退说明;未访问集群。
|
||||
记录绑定的是当前 checkout commit。镜像与源码的对应关系仍须独立构建记录验证。
|
||||
|
||||
### 2. Check:本地检查与线上只读快照
|
||||
|
||||
```sh
|
||||
python3 scripts/release.py check <release-id> --local
|
||||
python3 scripts/release.py check <release-id> --cluster
|
||||
```
|
||||
|
||||
本地检查命令和环境约束以 [fanyu-policy.json](fanyu-policy.json) 为准。首版运行流程工具测试、
|
||||
前端类型/构建及切片恢复定向回归,不冒充全套产品测试通过。任何失败阻断,不支持忽略开关。
|
||||
命令执行结果及输出摘要写入记录,不保存或回显原始输出;排查失败时单独重跑对应命令。
|
||||
|
||||
集群检查解析容器 env/envFrom/ConfigMap,核对数据库名称、会话认证、API external 模式、
|
||||
处理器并发/资源、存储连接、PVC 及当前 rollout 状态。只校验 Mongo Secret 的绑定,不读取
|
||||
其值;**不证明实际 Mongo 主机正确、凭证有效或视频能正常播放**,由业务证据补齐。
|
||||
公网带宽作为观察项展示,不以固定数值判断并发容量充足。
|
||||
|
||||
完成条件:两类检查均成功、源码和策略未变化。再次 check 会废弃旧计划和证据。
|
||||
|
||||
### 3. Plan:形成可审查的计划
|
||||
|
||||
```sh
|
||||
python3 scripts/release.py plan <release-id>
|
||||
```
|
||||
|
||||
显示旧/新镜像、构建标记、目标、带宽观察值、限制与回退说明,并生成计划摘要。
|
||||
核对两个部署的修改范围。计划不更新 ConfigMap、Secret、存储或网络。
|
||||
|
||||
部署前需要两份人工审查证据:
|
||||
|
||||
- `image-provenance`:构建记录将本次源码 commit 与镜像 digest 对应,候选镜像的验证情况明确。
|
||||
- `change-safety`:审查启动迁移/兼容性,检查在途切片与送检工作,确定维护窗口或安全切换方式,
|
||||
并说明回退旧版本是否安全。工具不自行读取或更改业务数据库来判断任务状态。
|
||||
|
||||
```sh
|
||||
python3 scripts/release.py evidence <release-id> --check image-provenance \
|
||||
--result passed --file <local-evidence-file> --note '构建和镜像核对结论'
|
||||
python3 scripts/release.py evidence <release-id> --check change-safety \
|
||||
--result passed --file <local-evidence-file> --note '迁移、在途任务及回退核对结论'
|
||||
```
|
||||
|
||||
证据保存文件路径、SHA256、时间和计划摘要。工具验证文件未变,**不自动判断内容是否真实充分**。
|
||||
人工证据应明确环境、版本、步骤、实际结果、限制和审查人;不能把空模板标记为 passed。
|
||||
|
||||
### 4. Deploy:本次授权后的有界变更
|
||||
|
||||
```sh
|
||||
python3 scripts/release.py deploy <release-id> \
|
||||
--confirm-plan <plan-hash> --approval-ref <user-message-reference>
|
||||
```
|
||||
|
||||
部署前再次核对源码、策略、证据及线上快照,集群预检超过一小时需重做。每次 patch 前重新
|
||||
核对两个部署及关联配置;用 UID/resourceVersion 条件更新镜像与构建标记。
|
||||
写请求发送前先落盘 intent。任何超时或中断按结果未知处理;不重复发、不自动回滚。
|
||||
|
||||
两个 Deployment 不是事务,也没有跨主机锁。中间失败可能只更新一个组件,必须 verify
|
||||
核对实际状态再制定恢复计划。JSON 条件更新防止覆盖同一资源的并发修改,但不能保证跨资源
|
||||
原子性;正式部署应使用团队约定的维护窗口,后续可增加服务端发布锁。
|
||||
|
||||
完成条件:仅代表 patch 已确认,状态仍为 `deployed_pending_verification`。
|
||||
|
||||
### 5. Verify:分开基础设施与业务验收
|
||||
|
||||
```sh
|
||||
python3 scripts/release.py verify <release-id>
|
||||
```
|
||||
|
||||
核对目标镜像、构建标记、rollout、配置/PVC/服务身份以及未批准的部署变化。
|
||||
未 Ready 可稍后重新运行只读 verify,不重新 deploy。部署配置符合后仍为
|
||||
`deployed_pending_acceptance`,不等于已经登录、播放、切片或送检成功。
|
||||
|
||||
按策略中 requiredEvidence 逐项记录验收:五角色登录及业务API、多供应商权限、标注与交付
|
||||
(普通/舍弃/无效/空白)、质检/终审、切片重启恢复、视频首帧/缓冲/实际播放速度。
|
||||
所有写入测试都遵循前述隔离与授权要求,不能占用生产客户任务。
|
||||
|
||||
```sh
|
||||
python3 scripts/release.py evidence <release-id> --check video-playback \
|
||||
--result passed --file <local-evidence-file> --note '角色、并发、网络和播放实测结果'
|
||||
python3 scripts/release.py verify <release-id>
|
||||
```
|
||||
|
||||
完成条件:基础设施复核通过且所有必需证据有效,才进入 `accepted_with_evidence`(包含人工
|
||||
证据的验收),不是“全自动端到端通过”。未测填 unverified,失败填 failed,不得代填通过。
|
||||
|
||||
## 故障、回退和紧急修复
|
||||
|
||||
- `status` 显示最后一步、写请求状态、缺失证据和 checkout。换上下文后先运行它。
|
||||
- 源码/策略变化重新 prepare;部署前同事更新了线上,重新 check/plan 并取得新授权。
|
||||
- `outcome_unknown` 或 `deploying`:先 verify,不能再次 deploy 同一记录。部分更新不能靠
|
||||
修改 JSON 清掉历史;保存调查记录并制定新的恢复计划。
|
||||
- 回退须新的明确授权,核对当前版本与数据兼容性,生成条件式变更。禁止直接 apply 旧完整
|
||||
Deployment、restore 整库或覆盖客户新增数据。首版不自动执行回退。
|
||||
- 紧急修复写明“已解决 / 未解决 / 临时取舍 / 后续任务”,关联本地问题票;未验证部分仍保留,
|
||||
不因上线紧急而自动验收通过。旧规格由新规则显式替代,不同时保留两份有效相反结论。
|
||||
|
||||
## 本地试运行与后续范围
|
||||
|
||||
```sh
|
||||
python3 -m unittest discover -s tests/release -p 'test_*.py'
|
||||
python3 scripts/release.py --help
|
||||
```
|
||||
|
||||
测试使用模拟 Kubernetes,不连接集群、不创建账号或任务。覆盖数据库/认证/处理器回归、
|
||||
陈旧计划、同事更新、部分失败、证据失效、锁冲突和 legacy 入口阻断。
|
||||
|
||||
后续再扩展:服务端部署互斥、OCI provenance 自动核对、隔离环境真实业务验收、受审查的配置
|
||||
变更及回退执行器。试行阶段避免把这些未实现能力写成已有保证。
|
||||
23
docs/release/fanyu-policy.json
Normal file
23
docs/release/fanyu-policy.json
Normal file
@ -0,0 +1,23 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"namespace": "droboticflow-fanyu",
|
||||
"database": "drobodata-fanyu",
|
||||
"bucket": "drobodata-fanyu",
|
||||
"workspaceMount": "/workspace",
|
||||
"workspacePVC": "droboticflow-workspace",
|
||||
"mongoSecret": "droboticflow-mongo-secret",
|
||||
"deployments": {
|
||||
"droboticflow-api": "api",
|
||||
"droboticflow-annotation-processor": "processor"
|
||||
},
|
||||
"requiredEvidence": [
|
||||
"image-provenance", "change-safety", "role-login", "supplier-permissions",
|
||||
"annotation-delivery", "review-acceptance", "slice-recovery", "video-playback"
|
||||
],
|
||||
"localChecks": [
|
||||
["python3", "-m", "unittest", "discover", "-s", "tests/release", "-p", "test_*.py"],
|
||||
["npm", "run", "lint:web"],
|
||||
["npm", "run", "build:web"],
|
||||
["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery", "-count=1"]
|
||||
]
|
||||
}
|
||||
@ -14,6 +14,10 @@ export DEPLOY_ENV
|
||||
source "$ROOT_DIR/scripts/load-deploy-env.sh"
|
||||
DEPLOY_ENV="${DEPLOY_ENV:-}"
|
||||
# shellcheck source=scripts/lib/build-metadata.sh
|
||||
if [[ "$DEPLOY_ENV" == "fanyu" || "${K8S_NAMESPACE:-}" == "droboticflow-fanyu" ]]; then
|
||||
echo "ERROR: Fanyu releases use python3 scripts/release.py; see docs/release/README.md. The legacy broad apply path is blocked before any cluster operation." >&2
|
||||
exit 2
|
||||
fi
|
||||
source "$ROOT_DIR/scripts/lib/build-metadata.sh"
|
||||
|
||||
K8S_NAMESPACE="${K8S_NAMESPACE:-droboticflow}"
|
||||
|
||||
455
scripts/release.py
Normal file
455
scripts/release.py
Normal file
@ -0,0 +1,455 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fanyu image/build-stamp release journal. Secret values are never persisted."""
|
||||
import argparse
|
||||
import contextlib
|
||||
import copy
|
||||
import datetime as dt
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
RECORDS = ROOT / 'local-data/releases'
|
||||
POLICY = ROOT / 'docs/release/fanyu-policy.json'
|
||||
SAFE_ENV = ('MONGO_DB', 'DROBOTICFLOW_REQUIRE_SESSION_AUTH',
|
||||
'ANNOTATION_PROCESSOR_MODE', 'ANNOTATION_PROCESSOR_CONCURRENCY',
|
||||
'ANNOTATION_INDEX_BACKFILL', 'ANNOTATION_WORKSPACE_CONNECTION_ID',
|
||||
'ANNOTATION_WORKSPACE_BUCKET', 'WORKSPACE_MOUNT_PATH')
|
||||
|
||||
|
||||
class Blocked(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def require(ok, message):
|
||||
if not ok:
|
||||
raise Blocked(message)
|
||||
|
||||
|
||||
def now():
|
||||
return dt.datetime.now(dt.timezone.utc).isoformat()
|
||||
|
||||
|
||||
def digest(value):
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(',', ':')).encode()).hexdigest()
|
||||
|
||||
|
||||
def run(argv, *, stdin=None, timeout=60):
|
||||
try:
|
||||
p = subprocess.run(argv, cwd=ROOT, input=stdin, text=True, capture_output=True, timeout=timeout)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
raise Blocked('Command unavailable or timed out; outcome must be checked before retry.') from None
|
||||
# Do not echo kubectl output, URLs or credential-bearing errors.
|
||||
require(p.returncode == 0, 'Command failed: ' + argv[0] + ' (output withheld; no success recorded).')
|
||||
return p.stdout
|
||||
|
||||
|
||||
def source():
|
||||
return {'commit': run(['git', 'rev-parse', 'HEAD']).strip(),
|
||||
'dirty': bool(run(['git', 'status', '--porcelain', '--untracked-files=normal']).strip())}
|
||||
|
||||
|
||||
def policy():
|
||||
return json.loads(POLICY.read_text())
|
||||
|
||||
|
||||
def record_path(release_id):
|
||||
require(bool(re.fullmatch(r'[a-zA-Z0-9][a-zA-Z0-9_-]{0,79}', release_id)), 'Invalid release id.')
|
||||
return RECORDS / release_id / 'release.json'
|
||||
|
||||
|
||||
def save(r):
|
||||
path = record_path(r['id'])
|
||||
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
temp = path.with_suffix('.tmp')
|
||||
with os.fdopen(os.open(temp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600), 'w') as f:
|
||||
json.dump(r, f, indent=2, ensure_ascii=False)
|
||||
f.write('\n')
|
||||
os.replace(temp, path)
|
||||
|
||||
|
||||
def event(r, name):
|
||||
r['events'].append({'at': now(), 'event': name})
|
||||
save(r)
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def locked(release_id):
|
||||
path = record_path(release_id)
|
||||
require(path.exists(), 'Release record not found; use prepare first.')
|
||||
with path.with_suffix('.lock').open('a') as f:
|
||||
try:
|
||||
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
raise Blocked('Another command is updating this release.') from None
|
||||
yield json.loads(path.read_text())
|
||||
|
||||
|
||||
def fresh(r):
|
||||
s = source()
|
||||
require(not s['dirty'], 'Checkout is dirty; commit task changes in a clean checkout first.')
|
||||
require(s['commit'] == r['commit'], 'Checkout commit changed; prepare a new release.')
|
||||
require(digest(policy()) == r['policyHash'], 'Release policy changed; prepare a new release.')
|
||||
|
||||
|
||||
def kubectl(r, args, *, stdin=None):
|
||||
t = r['target']
|
||||
return run(['kubectl', '--kubeconfig', t['kubeconfig'], '--context', t['context'],
|
||||
'--request-timeout=20s', '-n', t['namespace']] + args, stdin=stdin, timeout=60)
|
||||
|
||||
|
||||
def get(r, kind, name):
|
||||
return json.loads(kubectl(r, ['get', kind, name, '-o', 'json']))
|
||||
|
||||
|
||||
def env_values(container, configs):
|
||||
values = {}
|
||||
for entry in container.get('envFrom', []):
|
||||
ref = entry.get('configMapRef')
|
||||
require(bool(ref), 'Secret/unknown envFrom is unsupported by this preflight.')
|
||||
for key, value in configs[ref['name']].get('data', {}).items():
|
||||
values[entry.get('prefix', '') + key] = value
|
||||
for entry in container.get('env', []):
|
||||
key = entry['name']
|
||||
if 'value' in entry:
|
||||
values[key] = entry['value']
|
||||
elif 'configMapKeyRef' in entry.get('valueFrom', {}):
|
||||
ref = entry['valueFrom']['configMapKeyRef']
|
||||
require(ref['key'] in configs[ref['name']].get('data', {}), 'Missing ConfigMap key.')
|
||||
values[key] = configs[ref['name']]['data'][ref['key']]
|
||||
else:
|
||||
# Explicit unknowns must override envFrom rather than silently use its value.
|
||||
values[key] = None
|
||||
return values
|
||||
|
||||
|
||||
def inspect_deployment(deployment, container_name, configs, p):
|
||||
spec = deployment['spec']
|
||||
pod = spec['template']['spec']
|
||||
containers = pod['containers']
|
||||
matches = [(i, c) for i, c in enumerate(containers) if c['name'] == container_name]
|
||||
require(len(matches) == 1, 'Expected deployment container missing.')
|
||||
index, c = matches[0]
|
||||
e = env_values(c, configs)
|
||||
require(e.get('MONGO_DB') == p['database'], 'Database name mismatch.')
|
||||
require(e.get('ANNOTATION_WORKSPACE_BUCKET') == p['bucket'], 'Workspace bucket mismatch.')
|
||||
require(bool(e.get('ANNOTATION_WORKSPACE_CONNECTION_ID')), 'Missing workspace storage connection.')
|
||||
require(e.get('WORKSPACE_MOUNT_PATH') == p['workspaceMount'], 'Workspace mount mismatch.')
|
||||
mongo = next((x for x in c.get('env', []) if x['name'] == 'MONGO_URI'), {})
|
||||
require(mongo.get('valueFrom', {}).get('secretKeyRef') == {'name': p['mongoSecret'], 'key': 'MONGO_URI'},
|
||||
'Mongo secret binding mismatch; preflight does not read secret values.')
|
||||
mounts = {v['name']: v['mountPath'] for v in c.get('volumeMounts', [])}
|
||||
claims = [v for v in pod.get('volumes', []) if mounts.get(v['name']) == p['workspaceMount']]
|
||||
require(len(claims) == 1 and claims[0].get('persistentVolumeClaim', {}).get('claimName') == p['workspacePVC'],
|
||||
'Workspace PVC binding mismatch.')
|
||||
require(spec.get('replicas', 1) > 0, 'Deployment has no replicas.')
|
||||
if container_name == 'api':
|
||||
require(e.get('DROBOTICFLOW_REQUIRE_SESSION_AUTH') in ('true', '1'), 'Session authentication disabled.')
|
||||
require(e.get('ANNOTATION_PROCESSOR_MODE') == 'external', 'API must use external processor.')
|
||||
else:
|
||||
require(e.get('ANNOTATION_PROCESSOR_MODE') == 'processor', 'Processor mode mismatch.')
|
||||
require(e.get('ANNOTATION_PROCESSOR_CONCURRENCY') == '1', 'First release profile requires concurrency 1.')
|
||||
require(e.get('ANNOTATION_INDEX_BACKFILL') == '0', 'Historical backfill must remain disabled.')
|
||||
require(c.get('resources', {}).get('limits', {}).get('memory') == '8Gi', 'First release profile requires processor memory limit 8Gi.')
|
||||
require(spec.get('replicas') == 1 and spec.get('strategy', {}).get('type') == 'Recreate',
|
||||
'Processor must have one replica and Recreate strategy.')
|
||||
status = deployment.get('status', {})
|
||||
ready = (status.get('observedGeneration', 0) >= deployment['metadata'].get('generation', 1)
|
||||
and status.get('updatedReplicas', 0) == spec.get('replicas', 1)
|
||||
and status.get('readyReplicas', 0) == spec.get('replicas', 1)
|
||||
and status.get('replicas', 0) == spec.get('replicas', 1))
|
||||
m = deployment['metadata']
|
||||
neutral = copy.deepcopy(spec)
|
||||
nc = neutral['template']['spec']['containers'][index]
|
||||
nc['image'] = '<release-image>'
|
||||
nc['env'] = [x for x in nc.get('env', []) if x['name'] != 'DROBOTICFLOW_BUILD_COMMIT']
|
||||
stamp = next((i for i, x in enumerate(c.get('env', [])) if x['name'] == 'DROBOTICFLOW_BUILD_COMMIT'), None)
|
||||
return {'uid': m['uid'], 'specHash': digest(spec), 'nonReleaseHash': digest(neutral),
|
||||
'buildCommitIndex': stamp, 'buildCommit': e.get('DROBOTICFLOW_BUILD_COMMIT'), 'image': c['image'],
|
||||
'resourceVersion': m['resourceVersion'], 'containerIndex': index,
|
||||
'ready': ready, 'env': {k: e.get(k) for k in SAFE_ENV}}
|
||||
|
||||
|
||||
def snapshot(r):
|
||||
t, p = r['target'], policy()
|
||||
view = json.loads(kubectl(r, ['config', 'view', '--minify', '-o', 'json']))
|
||||
require(view['clusters'][0]['cluster']['server'] == t['server'], 'Kube API server mismatch.')
|
||||
ns = get(r, 'namespace', t['namespace'])
|
||||
require(ns['metadata']['uid'] == t['namespaceUID'], 'Namespace UID mismatch.')
|
||||
deployments = {n: get(r, 'deployment', n) for n in p['deployments']}
|
||||
config_names = set()
|
||||
for d in deployments.values():
|
||||
for c in d['spec']['template']['spec']['containers']:
|
||||
config_names.update(x['configMapRef']['name'] for x in c.get('envFrom', []) if 'configMapRef' in x)
|
||||
config_names.update(x['valueFrom']['configMapKeyRef']['name'] for x in c.get('env', [])
|
||||
if 'configMapKeyRef' in x.get('valueFrom', {}))
|
||||
configs = {n: get(r, 'configmap', n) for n in sorted(config_names)}
|
||||
deps = {n: inspect_deployment(d, p['deployments'][n], configs, p) for n, d in deployments.items()}
|
||||
envs = [d['env'] for d in deps.values()]
|
||||
require(len({e['ANNOTATION_WORKSPACE_CONNECTION_ID'] for e in envs}) == 1, 'API/processor storage connections differ.')
|
||||
pvc = get(r, 'pvc', p['workspacePVC'])
|
||||
require(pvc.get('status', {}).get('phase') == 'Bound', 'Workspace PVC is not Bound.')
|
||||
svc = get(r, 'service', 'droboticflow-api-public-eip')
|
||||
ann = svc['metadata'].get('annotations', {})
|
||||
return {'at': now(), 'deployments': deps,
|
||||
'configs': {n: {'uid': x['metadata']['uid'], 'hash': digest(x.get('data', {}))} for n, x in configs.items()},
|
||||
'pvc': {'uid': pvc['metadata']['uid'], 'specHash': digest(pvc['spec'])},
|
||||
'publicService': {'uid': svc['metadata']['uid'], 'specHash': digest(svc['spec']),
|
||||
'annotationHash': digest(ann),
|
||||
'bandwidthMbps': ann.get('service.beta.kubernetes.io/cce-elastic-ip-bandwidth-in-mbps', 'unknown')}}
|
||||
|
||||
|
||||
def stable(s):
|
||||
return {**{k: s[k] for k in ('configs', 'pvc', 'publicService')},
|
||||
'deployments': {n: {k: d[k] for k in ('uid', 'specHash', 'image')} for n, d in s['deployments'].items()}}
|
||||
|
||||
|
||||
def plan_data(r):
|
||||
return {k: r[k] for k in ('id', 'commit', 'image', 'policyHash', 'target', 'baseline', 'localChecks', 'purpose', 'rollback')}
|
||||
|
||||
|
||||
def prepare(a):
|
||||
p, s = policy(), source()
|
||||
branch = run(['git', 'branch', '--show-current']).strip()
|
||||
require(not re.search(r'(^|/)poc($|[-/])', branch), 'POC branches cannot prepare Fanyu releases.')
|
||||
require(not s['dirty'], 'Prepare requires a clean checkout (use an isolated checkout for unrelated edits).')
|
||||
require(bool(re.fullmatch(r'[^\s@]+@sha256:[0-9a-f]{64}', a.image)), 'Use an immutable repository@sha256 image.')
|
||||
require(a.server.startswith('https://') and '@' not in a.server and '?' not in a.server, 'Use a credential-free HTTPS Kubernetes server URL.')
|
||||
release_id = a.id or dt.datetime.now(dt.timezone.utc).strftime('%Y%m%dT%H%M%SZ-') + uuid.uuid4().hex[:8]
|
||||
require(not record_path(release_id).exists(), 'Release id already exists.')
|
||||
r = {'schemaVersion': 1, 'id': release_id, 'createdAt': now(), 'commit': s['commit'],
|
||||
'image': a.image, 'policyHash': digest(p), 'purpose': a.purpose, 'rollback': a.rollback,
|
||||
'target': {'kubeconfig': str(Path(a.kubeconfig).resolve()), 'context': a.context,
|
||||
'server': a.server.rstrip('/'), 'namespace': p['namespace'], 'namespaceUID': a.namespace_uid},
|
||||
'state': 'prepared', 'localChecks': [], 'baseline': None, 'planHash': None,
|
||||
'operations': [], 'evidence': {}, 'events': []}
|
||||
event(r, 'prepared; no network access')
|
||||
print(json.dumps({'id': r['id'], 'state': r['state']}, ensure_ascii=False))
|
||||
|
||||
|
||||
def check(r, a):
|
||||
fresh(r)
|
||||
require(not r['operations'], 'A deployment was attempted; check outcome with verify, never rewrite its plan.')
|
||||
r['planHash'] = None
|
||||
r['evidence'] = {}
|
||||
r['state'] = 'checking'
|
||||
event(r, 'check started; previous plan invalidated')
|
||||
if a.local:
|
||||
r['localChecks'] = []
|
||||
for argv in policy()['localChecks']:
|
||||
started = now()
|
||||
try:
|
||||
output = run(argv, timeout=1800)
|
||||
except Blocked:
|
||||
r['localChecks'].append({'argv': argv, 'startedAt': started, 'result': 'failed'})
|
||||
r['state'] = 'blocked'
|
||||
event(r, 'local check failed')
|
||||
raise
|
||||
r['localChecks'].append({'argv': argv, 'startedAt': started, 'result': 'passed', 'outputHash': digest(output)})
|
||||
save(r)
|
||||
fresh(r)
|
||||
if a.cluster:
|
||||
# Never preserve an earlier successful snapshot when a new read fails.
|
||||
r['baseline'] = None
|
||||
save(r)
|
||||
try:
|
||||
s = snapshot(r)
|
||||
require(all(d['ready'] for d in s['deployments'].values()), 'Current deployments are not fully ready.')
|
||||
r['baseline'] = s
|
||||
except Blocked:
|
||||
r['state'] = 'blocked'
|
||||
event(r, 'cluster check failed')
|
||||
raise
|
||||
r['state'] = 'checked_partial'
|
||||
event(r, 'checks recorded; plan must still be generated')
|
||||
|
||||
|
||||
def make_plan(r):
|
||||
fresh(r)
|
||||
require(not r['operations'], 'Deployment already attempted.')
|
||||
require([x['argv'] for x in r['localChecks']] == policy()['localChecks']
|
||||
and all(x['result'] == 'passed' for x in r['localChecks']), 'Required local checks incomplete.')
|
||||
require(r['baseline'] is not None, 'Read-only cluster check required.')
|
||||
r['planHash'] = digest(plan_data(r))
|
||||
r['state'] = 'awaiting_approval'
|
||||
event(r, 'plan prepared; no authorization inferred')
|
||||
print(json.dumps({'planHash': r['planHash'], 'target': r['target'], 'changes': [
|
||||
{'deployment': n, 'from': d['image'], 'to': r['image'], 'buildCommit': r['commit']} for n, d in r['baseline']['deployments'].items()],
|
||||
'bandwidthMbps': r['baseline']['publicService']['bandwidthMbps'],
|
||||
'warnings': ['Only image/build stamp changes; startup migrations require change-safety evidence.',
|
||||
'Image-to-source provenance requires independent evidence.',
|
||||
'Bandwidth adequacy and business acceptance remain manual evidence.'],
|
||||
'rollback': r['rollback']}, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
def deploy(r, a):
|
||||
fresh(r)
|
||||
require(r['state'] == 'awaiting_approval' and not r['operations'], 'No fresh deployable plan; inspect status.')
|
||||
require(r['planHash'] == a.confirm_plan == digest(plan_data(r)), 'Plan hash does not match.')
|
||||
require(bool(a.approval_ref.strip()), 'Record the explicit user approval reference for this exact plan.')
|
||||
age = (dt.datetime.now(dt.timezone.utc) - dt.datetime.fromisoformat(r['baseline']['at'])).total_seconds()
|
||||
require(0 <= age <= 3600, 'Cluster check is older than one hour; recheck and obtain a new plan approval.')
|
||||
for key in ('image-provenance', 'change-safety'):
|
||||
require(key in r['evidence'] and evidence_valid(r['evidence'][key], r), key + ' evidence required before deployment.')
|
||||
current = snapshot(r)
|
||||
require(stable(current) == stable(r['baseline']), 'Live resources changed since check; prepare/check a new plan.')
|
||||
require(all(d['ready'] for d in current['deployments'].values()), 'Live deployment not ready.')
|
||||
r['approval'] = {'reference': a.approval_ref, 'at': now(), 'planHash': r['planHash']}
|
||||
r['state'] = 'deploying'
|
||||
event(r, 'explicit approval reference recorded; deployment starting')
|
||||
for name in current['deployments']:
|
||||
latest = snapshot(r)
|
||||
require(all(latest[k] == current[k] for k in ('configs', 'pvc', 'publicService')), 'Configuration changed during deployment; inspect partial outcome.')
|
||||
for n, live in latest['deployments'].items():
|
||||
old = current['deployments'][n]
|
||||
done = any(o['deployment'] == n and o['result'] == 'acknowledged' for o in r['operations'])
|
||||
require(live['uid'] == old['uid'] and live['nonReleaseHash'] == old['nonReleaseHash'], 'Deployment changed during release.')
|
||||
require((live['image'] == r['image'] and live['buildCommit'] == r['commit']) if done else live['specHash'] == old['specHash'], 'Image/configuration drift during release.')
|
||||
dep = latest['deployments'][name]
|
||||
patch = [{'op': 'test', 'path': '/metadata/resourceVersion', 'value': dep['resourceVersion']},
|
||||
{'op': 'test', 'path': '/metadata/uid', 'value': dep['uid']},
|
||||
{'op': 'test', 'path': f"/spec/template/spec/containers/{dep['containerIndex']}/image", 'value': dep['image']},
|
||||
{'op': 'replace', 'path': f"/spec/template/spec/containers/{dep['containerIndex']}/image", 'value': r['image']}]
|
||||
stamp = {'name': 'DROBOTICFLOW_BUILD_COMMIT', 'value': r['commit']}
|
||||
env_path = f"/spec/template/spec/containers/{dep['containerIndex']}/env/"
|
||||
patch.append({'op': 'add' if dep['buildCommitIndex'] is None else 'replace',
|
||||
'path': env_path + ('-' if dep['buildCommitIndex'] is None else str(dep['buildCommitIndex'])), 'value': stamp})
|
||||
op = {'deployment': name, 'before': dep['image'], 'desired': r['image'], 'at': now(), 'result': 'unknown'}
|
||||
r['operations'].append(op)
|
||||
save(r) # Persist intent before sending any mutation.
|
||||
try:
|
||||
kubectl(r, ['patch', 'deployment', name, '--type=json', '--patch-file=/dev/stdin'], stdin=json.dumps(patch))
|
||||
except Blocked:
|
||||
r['state'] = 'outcome_unknown'
|
||||
event(r, 'patch outcome unknown; verify before any further operation')
|
||||
raise
|
||||
op['result'] = 'acknowledged'
|
||||
save(r)
|
||||
r['state'] = 'deployed_pending_verification'
|
||||
event(r, 'image patches acknowledged; rollout and acceptance not yet verified')
|
||||
|
||||
|
||||
def record_evidence(r, a):
|
||||
require(r['planHash'] is not None, 'Generate a plan before attaching evidence.')
|
||||
require(a.check in policy()['requiredEvidence'], 'Unknown evidence category.')
|
||||
require(a.check in ('image-provenance', 'change-safety') or r['state'] in ('deployed_pending_acceptance', 'accepted_with_evidence'),
|
||||
'Business evidence can be attached only after infrastructure verification.')
|
||||
path = Path(a.file).resolve()
|
||||
require(path.is_file() and path.stat().st_size > 0, 'Evidence must be a nonempty local file.')
|
||||
r['evidence'][a.check] = {'result': a.result, 'path': str(path),
|
||||
'sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
'at': now(), 'planHash': r['planHash'], 'note': a.note,
|
||||
'kind': 'operator-attested; content not automatically evaluated'}
|
||||
if a.result != 'passed' and r['state'] == 'accepted_with_evidence':
|
||||
r['state'] = 'deployed_pending_acceptance'
|
||||
event(r, 'evidence recorded: ' + a.check)
|
||||
|
||||
|
||||
def evidence_valid(e, r):
|
||||
path = Path(e['path'])
|
||||
return (e['result'] == 'passed' and e['planHash'] == r['planHash'] and path.is_file()
|
||||
and hashlib.sha256(path.read_bytes()).hexdigest() == e['sha256'])
|
||||
|
||||
|
||||
def verify(r):
|
||||
require(bool(r['operations']), 'No deployment attempt in this record.')
|
||||
# Failure cannot retain a stale accepted state.
|
||||
r['state'] = 'deployed_pending_verification'
|
||||
save(r)
|
||||
require(digest(policy()) == r['policyHash'], 'Policy changed; verify with the original policy revision.')
|
||||
s = snapshot(r)
|
||||
r['lastObservation'] = s
|
||||
expected = r['baseline']
|
||||
require(all(s[k] == expected[k] for k in ('configs', 'pvc', 'publicService')), 'Configuration drift detected.')
|
||||
for n, d in s['deployments'].items():
|
||||
require(d['uid'] == expected['deployments'][n]['uid'], 'Deployment was replaced.')
|
||||
require(d['nonReleaseHash'] == expected['deployments'][n]['nonReleaseHash'], 'Deployment settings drifted beyond the approved image/build stamp.')
|
||||
require(d['image'] == r['image'] and d['buildCommit'] == r['commit'] and d['ready'], 'Target image rollout incomplete; outcome remains pending.')
|
||||
r['state'] = 'deployed_pending_acceptance'
|
||||
missing = [k for k in policy()['requiredEvidence'] if k not in r['evidence'] or not evidence_valid(r['evidence'][k], r)]
|
||||
if not missing:
|
||||
r['state'] = 'accepted_with_evidence'
|
||||
event(r, 'infrastructure verified; remaining evidence: ' + ', '.join(missing))
|
||||
|
||||
|
||||
def status(r):
|
||||
p = policy()
|
||||
missing = [k for k in p['requiredEvidence'] if k not in r['evidence'] or not evidence_valid(r['evidence'][k], r)]
|
||||
state = 'acceptance_evidence_invalid' if r['state'] == 'accepted_with_evidence' and missing else r['state']
|
||||
print(json.dumps({'id': r['id'], 'state': state, 'recordedState': r['state'], 'commit': r['commit'], 'image': r['image'],
|
||||
'policyMatches': digest(p) == r['policyHash'], 'checkout': source(),
|
||||
'planHash': r['planHash'], 'missingEvidence': missing,
|
||||
'operations': r['operations'], 'lastEvent': r['events'][-1],
|
||||
'next': 'Resume from recorded state; fresh approval is required for any new deployment or rollback.'},
|
||||
indent=2, ensure_ascii=False))
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sub = parser.add_subparsers(dest='command', required=True)
|
||||
prep = sub.add_parser('prepare', help='Offline: create a release record from a clean committed checkout.')
|
||||
prep.add_argument('--id')
|
||||
for name in ('image', 'kubeconfig', 'context', 'server', 'namespace-uid', 'purpose', 'rollback'):
|
||||
prep.add_argument('--' + name, required=True)
|
||||
for cmd in ('check', 'plan', 'deploy', 'verify', 'status', 'evidence'):
|
||||
c = sub.add_parser(cmd)
|
||||
if cmd == 'status':
|
||||
c.add_argument('id', nargs='?')
|
||||
else:
|
||||
c.add_argument('id')
|
||||
if cmd == 'check':
|
||||
c.add_argument('--local', action='store_true')
|
||||
c.add_argument('--cluster', action='store_true')
|
||||
elif cmd == 'deploy':
|
||||
c.add_argument('--confirm-plan', required=True)
|
||||
c.add_argument('--approval-ref', required=True)
|
||||
elif cmd == 'evidence':
|
||||
c.add_argument('--check', required=True)
|
||||
c.add_argument('--result', choices=['passed', 'failed', 'unverified'], required=True)
|
||||
c.add_argument('--file', required=True)
|
||||
c.add_argument('--note', required=True)
|
||||
a = parser.parse_args()
|
||||
try:
|
||||
if a.command == 'prepare':
|
||||
prepare(a)
|
||||
return
|
||||
if a.command == 'status' and not a.id:
|
||||
summaries = []
|
||||
for path in sorted(RECORDS.glob('*/release.json')):
|
||||
item = json.loads(path.read_text())
|
||||
summaries.append({k: item[k] for k in ('id', 'state', 'commit', 'createdAt')})
|
||||
print(json.dumps(summaries, indent=2, ensure_ascii=False))
|
||||
return
|
||||
with locked(a.id) as r:
|
||||
if a.command == 'check':
|
||||
require(a.local or a.cluster, 'Choose --local and/or --cluster.')
|
||||
check(r, a)
|
||||
elif a.command == 'plan':
|
||||
make_plan(r)
|
||||
elif a.command == 'deploy':
|
||||
try:
|
||||
deploy(r, a)
|
||||
except Blocked:
|
||||
if r['state'] == 'deploying':
|
||||
r['state'] = 'outcome_unknown'
|
||||
event(r, 'interrupted; inspect partial outcome before further changes')
|
||||
raise
|
||||
elif a.command == 'verify':
|
||||
verify(r)
|
||||
elif a.command == 'evidence':
|
||||
record_evidence(r, a)
|
||||
status(r)
|
||||
except (Blocked, KeyError, ValueError, OSError) as e:
|
||||
# Validation errors are authored here; generic errors omit possibly sensitive values.
|
||||
print('BLOCKED: ' + (str(e) if isinstance(e, Blocked) else type(e).__name__), file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
332
tests/release/test_release.py
Normal file
332
tests/release/test_release.py
Normal file
@ -0,0 +1,332 @@
|
||||
"""No cluster or customer credentials: exercise release transitions with fake Kubernetes."""
|
||||
import argparse
|
||||
import copy
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
spec = importlib.util.spec_from_file_location('release', ROOT / 'scripts/release.py')
|
||||
m = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(m)
|
||||
IMAGE = 'registry.example/api@sha256:' + 'a' * 64
|
||||
|
||||
|
||||
def fixture(name, container):
|
||||
config = {'data': {'MONGO_DB': 'drobodata-fanyu', 'DROBOTICFLOW_REQUIRE_SESSION_AUTH': 'true',
|
||||
'ANNOTATION_PROCESSOR_MODE': 'external', 'ANNOTATION_WORKSPACE_BUCKET': 'drobodata-fanyu',
|
||||
'ANNOTATION_WORKSPACE_CONNECTION_ID': 'storage-example', 'WORKSPACE_MOUNT_PATH': '/workspace'},
|
||||
'metadata': {'uid': 'config-uid'}}
|
||||
env = [{'name': 'MONGO_URI', 'valueFrom': {'secretKeyRef': {'name': 'droboticflow-mongo-secret', 'key': 'MONGO_URI'}}}]
|
||||
if container == 'processor':
|
||||
env += [{'name': k, 'value': v} for k, v in {
|
||||
'ANNOTATION_PROCESSOR_MODE': 'processor', 'ANNOTATION_PROCESSOR_CONCURRENCY': '1',
|
||||
'ANNOTATION_INDEX_BACKFILL': '0'}.items()]
|
||||
d = {'metadata': {'name': name, 'uid': name + '-uid', 'resourceVersion': '1', 'generation': 1},
|
||||
'spec': {'replicas': 1, 'strategy': {'type': 'Recreate'}, 'template': {'spec': {
|
||||
'containers': [{'name': container, 'image': 'registry.example/api@sha256:' + 'b' * 64,
|
||||
'envFrom': [{'configMapRef': {'name': 'config'}}], 'env': env,
|
||||
'resources': {'limits': {'memory': '8Gi'}},
|
||||
'volumeMounts': [{'name': 'workspace', 'mountPath': '/workspace'}]}],
|
||||
'volumes': [{'name': 'workspace', 'persistentVolumeClaim': {'claimName': 'droboticflow-workspace'}}]}}},
|
||||
'status': {'observedGeneration': 1, 'replicas': 1, 'updatedReplicas': 1, 'readyReplicas': 1}}
|
||||
return d, config
|
||||
|
||||
|
||||
class ReleaseTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
self.records = Path(self.tmp.name) / 'records'
|
||||
self.p = m.policy()
|
||||
self.p['localChecks'] = [['fake-check']]
|
||||
self.start_patch('RECORDS', self.records)
|
||||
self.start_patch('policy', lambda: copy.deepcopy(self.p))
|
||||
self.start_patch('source', lambda: {'commit': 'c' * 40, 'dirty': False})
|
||||
self.start_patch('run', lambda *args, **kwargs: 'feature/release')
|
||||
self.ds = {}
|
||||
for n, c in self.p['deployments'].items():
|
||||
self.ds[n], config = fixture(n, c)
|
||||
self.configs = {'config': config}
|
||||
self.writes = []
|
||||
self.fail_patch = None
|
||||
self.before_snapshot = None
|
||||
self.start_patch('snapshot', self.snapshot)
|
||||
self.start_patch('kubectl', self.kubectl)
|
||||
m.prepare(argparse.Namespace(id='test', image=IMAGE, context='fanyu', server='https://cluster.example',
|
||||
namespace_uid='namespace-uid', kubeconfig='/example/config', purpose='test', rollback='review old image'))
|
||||
self.r = self.load()
|
||||
|
||||
def start_patch(self, name, value):
|
||||
p = patch.object(m, name, value)
|
||||
p.start()
|
||||
self.addCleanup(p.stop)
|
||||
|
||||
def load(self):
|
||||
return json.loads(m.record_path('test').read_text())
|
||||
|
||||
def snapshot(self, r):
|
||||
if self.before_snapshot:
|
||||
self.before_snapshot()
|
||||
return {'at': m.now(), 'deployments': {
|
||||
n: m.inspect_deployment(d, self.p['deployments'][n], self.configs, self.p) for n, d in self.ds.items()},
|
||||
'configs': {'config': {'uid': 'config-uid', 'hash': m.digest(self.configs)}},
|
||||
'pvc': {'uid': 'pvc-uid'}, 'publicService': {'bandwidthMbps': '5'}}
|
||||
|
||||
def kubectl(self, r, args, *, stdin=None):
|
||||
self.assertEqual(args[:2], ['patch', 'deployment'])
|
||||
name = args[2]
|
||||
self.writes.append(name)
|
||||
if self.fail_patch == len(self.writes):
|
||||
raise m.Blocked('simulated timeout')
|
||||
ops = json.loads(stdin)
|
||||
self.assertEqual(ops[0]['path'], '/metadata/resourceVersion')
|
||||
d = self.ds[name]
|
||||
self.assertEqual(ops[0]['value'], d['metadata']['resourceVersion'])
|
||||
c = d['spec']['template']['spec']['containers'][0]
|
||||
c['image'] = ops[3]['value']
|
||||
if ops[4]['op'] == 'add':
|
||||
c['env'].append(ops[4]['value'])
|
||||
else:
|
||||
c['env'][int(ops[4]['path'].split('/')[-1])] = ops[4]['value']
|
||||
d['metadata']['resourceVersion'] = str(int(d['metadata']['resourceVersion']) + 1)
|
||||
return '{}'
|
||||
|
||||
def plan(self):
|
||||
m.check(self.r, argparse.Namespace(local=True, cluster=True))
|
||||
m.make_plan(self.r)
|
||||
|
||||
def evidence(self, key, result='passed'):
|
||||
f = Path(self.tmp.name) / (key + '.txt')
|
||||
f.write_text('synthetic test evidence for current release')
|
||||
m.record_evidence(self.r, argparse.Namespace(check=key, file=str(f), result=result, note='test evidence only'))
|
||||
return f
|
||||
|
||||
def approve(self):
|
||||
self.evidence('image-provenance')
|
||||
self.evidence('change-safety')
|
||||
return argparse.Namespace(confirm_plan=self.r['planHash'], approval_ref='user-message:test')
|
||||
|
||||
def test_prepare_is_offline_and_private(self):
|
||||
self.assertEqual(self.r['state'], 'prepared')
|
||||
self.assertEqual(m.record_path('test').stat().st_mode & 0o777, 0o600)
|
||||
self.assertFalse(self.writes)
|
||||
|
||||
def test_cannot_plan_without_both_checks(self):
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.make_plan(self.r)
|
||||
m.check(self.r, argparse.Namespace(local=True, cluster=False))
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.make_plan(self.r)
|
||||
|
||||
def test_local_failure_cannot_pass(self):
|
||||
with patch.object(m, 'run', side_effect=m.Blocked('failure')):
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.check(self.r, argparse.Namespace(local=True, cluster=False))
|
||||
self.assertEqual(self.load()['state'], 'blocked')
|
||||
|
||||
def test_recheck_invalidates_previous_plan_and_failed_snapshot(self):
|
||||
self.plan()
|
||||
with patch.object(m, 'snapshot', side_effect=m.Blocked('offline')):
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.check(self.r, argparse.Namespace(local=False, cluster=True))
|
||||
saved = self.load()
|
||||
self.assertIsNone(saved['baseline'])
|
||||
self.assertIsNone(saved['planHash'])
|
||||
|
||||
def test_source_and_policy_change_block_deployment(self):
|
||||
self.plan()
|
||||
a = self.approve()
|
||||
for s in [{'commit': 'd' * 40, 'dirty': False}, {'commit': 'c' * 40, 'dirty': True}]:
|
||||
with patch.object(m, 'source', return_value=s), self.assertRaises(m.Blocked):
|
||||
m.deploy(self.r, a)
|
||||
self.p['database'] = 'another'
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.deploy(self.r, a)
|
||||
self.assertFalse(self.writes)
|
||||
|
||||
def test_exact_plan_and_explicit_reference_required(self):
|
||||
self.plan()
|
||||
a = self.approve()
|
||||
a.confirm_plan = 'wrong'
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.deploy(self.r, a)
|
||||
a.confirm_plan = self.r['planHash']
|
||||
a.approval_ref = ''
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.deploy(self.r, a)
|
||||
self.assertFalse(self.writes)
|
||||
|
||||
def test_changed_evidence_rejected(self):
|
||||
self.plan()
|
||||
a = self.approve()
|
||||
Path(self.r['evidence']['image-provenance']['path']).write_text('changed')
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.deploy(self.r, a)
|
||||
self.assertFalse(self.writes)
|
||||
|
||||
def test_colleague_live_change_blocks_before_patch(self):
|
||||
self.plan()
|
||||
a = self.approve()
|
||||
self.ds['droboticflow-api']['spec']['replicas'] = 3
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.deploy(self.r, a)
|
||||
self.assertFalse(self.writes)
|
||||
|
||||
def test_config_change_between_patches_stops_second(self):
|
||||
self.plan()
|
||||
a = self.approve()
|
||||
def change():
|
||||
if len(self.writes) == 1:
|
||||
self.configs['config']['data']['unrelated'] = 'changed'
|
||||
self.before_snapshot = change
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.deploy(self.r, a)
|
||||
self.assertEqual(len(self.writes), 1)
|
||||
self.assertEqual(len(self.load()['operations']), 1)
|
||||
|
||||
def test_timeout_preserves_unknown_intent_and_cannot_redeploy(self):
|
||||
self.plan()
|
||||
a = self.approve()
|
||||
self.fail_patch = 2
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.deploy(self.r, a)
|
||||
saved = self.load()
|
||||
self.assertEqual(saved['state'], 'outcome_unknown')
|
||||
self.assertEqual([o['result'] for o in saved['operations']], ['acknowledged', 'unknown'])
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.deploy(saved, a)
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.verify(saved)
|
||||
self.assertNotEqual(self.load()['state'], 'accepted_with_evidence')
|
||||
|
||||
def test_deploy_and_ready_are_not_business_acceptance(self):
|
||||
self.plan()
|
||||
m.deploy(self.r, self.approve())
|
||||
self.assertEqual(self.r['state'], 'deployed_pending_verification')
|
||||
m.verify(self.r)
|
||||
self.assertEqual(self.r['state'], 'deployed_pending_acceptance')
|
||||
for key in self.p['requiredEvidence']:
|
||||
if key not in self.r['evidence']:
|
||||
self.evidence(key)
|
||||
m.verify(self.r)
|
||||
self.assertEqual(self.r['state'], 'accepted_with_evidence')
|
||||
Path(self.r['evidence']['video-playback']['path']).unlink()
|
||||
m.verify(self.r)
|
||||
self.assertEqual(self.r['state'], 'deployed_pending_acceptance')
|
||||
|
||||
def test_no_predeployment_business_evidence(self):
|
||||
self.plan()
|
||||
with self.assertRaises(m.Blocked):
|
||||
self.evidence('video-playback')
|
||||
|
||||
def test_verify_rejects_unplanned_settings(self):
|
||||
self.plan()
|
||||
m.deploy(self.r, self.approve())
|
||||
self.ds['droboticflow-api']['spec']['template']['spec']['containers'][0]['resources'] = {'limits': {'memory': '1Gi'}}
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.verify(self.r)
|
||||
self.assertEqual(self.load()['state'], 'deployed_pending_verification')
|
||||
|
||||
def test_configuration_regressions_block(self):
|
||||
for key, value in [('MONGO_DB', 'droboticflow'), ('DROBOTICFLOW_REQUIRE_SESSION_AUTH', 'false'),
|
||||
('ANNOTATION_PROCESSOR_MODE', 'inprocess'), ('ANNOTATION_WORKSPACE_BUCKET', ''),
|
||||
('WORKSPACE_MOUNT_PATH', '/wrong')]:
|
||||
configs = copy.deepcopy(self.configs)
|
||||
configs['config']['data'][key] = value
|
||||
with self.subTest(key=key), self.assertRaises(m.Blocked):
|
||||
m.inspect_deployment(self.ds['droboticflow-api'], 'api', configs, self.p)
|
||||
|
||||
def test_unknown_explicit_env_does_not_fall_back_to_envfrom(self):
|
||||
d = copy.deepcopy(self.ds['droboticflow-api'])
|
||||
d['spec']['template']['spec']['containers'][0]['env'].append({
|
||||
'name': 'MONGO_DB', 'valueFrom': {'secretKeyRef': {'name': 'other', 'key': 'db'}}})
|
||||
with self.assertRaises(m.Blocked):
|
||||
m.inspect_deployment(d, 'api', self.configs, self.p)
|
||||
|
||||
def test_snapshot_identity_and_no_secret_reads(self):
|
||||
r = self.r
|
||||
replies = {'namespace': {'metadata': {'uid': 'namespace-uid'}},
|
||||
'pvc': {'metadata': {'uid': 'pvc'}, 'spec': {}, 'status': {'phase': 'Bound'}},
|
||||
'service': {'metadata': {'uid': 'svc'}, 'spec': {}}}
|
||||
reads = []
|
||||
def fake_get(_, kind, name):
|
||||
reads.append(kind)
|
||||
if kind == 'deployment': return self.ds[name]
|
||||
if kind == 'configmap': return self.configs[name]
|
||||
return replies[kind]
|
||||
# setUp replaced snapshot; recover implementation through a second module import.
|
||||
real = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(real)
|
||||
with patch.object(real, 'policy', return_value=self.p), patch.object(real, 'get', side_effect=fake_get), patch.object(real, 'kubectl', return_value=json.dumps({'clusters': [{'cluster': {'server': r['target']['server']}}]})):
|
||||
real.snapshot(r)
|
||||
self.assertNotIn('secret', reads)
|
||||
replies['namespace']['metadata']['uid'] = 'wrong'
|
||||
with self.assertRaises(real.Blocked): real.snapshot(r)
|
||||
|
||||
def test_stale_cluster_check_blocks_deploy(self):
|
||||
self.plan()
|
||||
self.r['baseline']['at'] = '2000-01-01T00:00:00+00:00'
|
||||
m.make_plan(self.r)
|
||||
with self.assertRaises(m.Blocked): m.deploy(self.r, self.approve())
|
||||
self.assertFalse(self.writes)
|
||||
|
||||
def test_record_path_traversal_rejected(self):
|
||||
with self.assertRaises(m.Blocked): m.record_path('../other')
|
||||
|
||||
def test_record_lock_prevents_two_writers(self):
|
||||
with m.locked('test'):
|
||||
with self.assertRaises(m.Blocked):
|
||||
with m.locked('test'): pass
|
||||
|
||||
|
||||
class EntrypointTests(unittest.TestCase):
|
||||
def test_cli_persists_and_resumes_offline(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
(root / 'scripts').mkdir()
|
||||
(root / 'docs/release').mkdir(parents=True)
|
||||
shutil.copy(ROOT / 'scripts/release.py', root / 'scripts/release.py')
|
||||
shutil.copy(ROOT / 'docs/release/fanyu-policy.json', root / 'docs/release/fanyu-policy.json')
|
||||
(root / '.gitignore').write_text('local-data/\n')
|
||||
for args in [ ['git', 'init', '-q'], ['git', 'add', '.'],
|
||||
['git', '-c', 'user.name=ReleaseTest', '-c', 'user.email=test@example.invalid',
|
||||
'commit', '-qm', 'fixture'] ]:
|
||||
subprocess.run(args, cwd=root, check=True, capture_output=True)
|
||||
def cli(*args):
|
||||
return subprocess.run([sys.executable, 'scripts/release.py', *args], cwd=root,
|
||||
text=True, capture_output=True)
|
||||
r = cli('prepare', '--id', 'offline-demo', '--image', IMAGE, '--context', 'example',
|
||||
'--kubeconfig', '/does-not-exist', '--server', 'https://cluster.example',
|
||||
'--namespace-uid', 'example-uid', '--purpose', 'offline smoke', '--rollback', 'not deployed')
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
resumed = cli('status', 'offline-demo')
|
||||
self.assertEqual(resumed.returncode, 0, resumed.stderr)
|
||||
self.assertEqual(json.loads(resumed.stdout)['state'], 'prepared')
|
||||
blocked = cli('deploy', 'offline-demo', '--confirm-plan', 'unprepared', '--approval-ref', 'none')
|
||||
self.assertEqual(blocked.returncode, 2)
|
||||
self.assertIn('No fresh deployable plan', blocked.stderr)
|
||||
|
||||
def test_legacy_fanyu_deploy_stops_before_kubectl(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fake = Path(tmp) / 'kubectl'
|
||||
called = Path(tmp) / 'called'
|
||||
fake.write_text('#!/bin/sh\ntouch "' + str(called) + '"\nexit 99\n')
|
||||
fake.chmod(0o755)
|
||||
env = dict(os.environ, PATH=tmp + ':' + os.environ['PATH'], DEPLOY_ENV='fanyu',
|
||||
DROBOTICFLOW_DEPLOY_ENV_LOADED='test', K8S_NAMESPACE='droboticflow-fanyu')
|
||||
p = subprocess.run(['bash', 'scripts/deploy-k8s.sh'], cwd=ROOT, env=env, text=True, capture_output=True)
|
||||
self.assertEqual(p.returncode, 2)
|
||||
self.assertIn('scripts/release.py', p.stderr)
|
||||
self.assertFalse(called.exists())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Loading…
x
Reference in New Issue
Block a user