🛡️ Harden Qwen node cache rollout

This commit is contained in:
eust-w 2026-08-22 08:47:40 +08:00
parent d960fab4d0
commit f81e22cc70
10 changed files with 363 additions and 39 deletions

View File

@ -107,6 +107,9 @@ For the `poc` environment, use [`scripts/deploy-envs/poc.env`](../scripts/deploy
- `MONGO_DB=poc`
- `API_PUBLIC_LOAD_BALANCER=true`
- 已验收 Qwen 节点本地缓存:`DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED=true`、
`droboticflow-qwen-node-cache` PVC、`/qwen-node-cache` 和 GPU node hostname selector;
这只影响 allowlisted Qwen 粗切 Job,通用模板仍默认关闭。
- `BOS_ENDPOINT=https://bj.bcebos.com`
- the same namespace, AI defaults, and PVC settings as `wjzh`

View File

@ -28,7 +28,7 @@ python3 scripts/verify_qwen3vl_model_manifest.py --model-root /workspace/models/
独立预载 Job 必须先逐字节验证全部 4 个 shard/16 个 payload 文件,再原子发布模型目录。生产 workflow 将该 Job 产出的 manifest SHA-256 固定为 `MODEL_MANIFEST_SHA256`;每个短生命周期 Qwen Pod 重新验证 sidecar、固定 SHA、config/processor/index、四 shard 证据,以及完整本地路径和大小布局,但不重复从共享 PVC 读取 17.5 GB 做同一遍哈希。未提供固定 SHA 时,`TRUST_PREVERIFIED_MODEL=true` 会失败关闭;非预载模式仍执行全文件哈希。由于 POC CSI 使用 direct I/O,默认行为是以四 shard 并行大块读把固定快照复制到该 Pod 的本地 `/tmp`,再从本地路径加载。
可选的节点本地缓存默认关闭,且只允许 Qwen3-VL 手部行为语义粗切使用。启用后,缓存目录以固定 manifest SHA 为 key,源 manifest/provenance 每次重验,miss 按文件 SHA/size 复制并原子写 receipt,hit 会完整本地哈希复验;空间不足或损坏会回退 Pod-local staging 而不会使用不完整缓存。详情与 PV 安全边界见 [Qwen 节点本地模型缓存](qwen-node-local-cache.md)。Pod provenance 同时记录 PVC manifest 路径、本地 runtime 路径、复制字节数和 `nodeLocalCache` 状态。Qwen 节点仅从本地快照加载(`local_files_only=True`),使用固定的 `transformers==4.57.6` 和 `Qwen3VLForConditionalGeneration`,不会执行远程模型代码。
可选的节点本地缓存通用默认关闭,且只允许 Qwen3-VL 手部行为语义粗切使用;已验收的 `poc` profile 通过 Local PV 显式启用。缓存目录以固定 manifest SHA 为 key,源 manifest/provenance 每次重验,miss 按文件 SHA/size 复制并原子写 receipt,hit 会完整本地哈希复验;空间不足或损坏会回退 Pod-local staging 而不会使用不完整缓存。详情、PV 安全边界和 disabled/miss/hit 实测证据见 [Qwen 节点本地模型缓存](qwen-node-local-cache.md)。Pod provenance 同时记录 PVC manifest 路径、本地 runtime 路径、复制字节数和 `nodeLocalCache` 状态。Qwen 节点仅从本地快照加载(`local_files_only=True`),使用固定的 `transformers==4.57.6` 和 `Qwen3VLForConditionalGeneration`,不会执行远程模型代码。
## 人在环与交付

View File

@ -1,4 +1,4 @@
# Qwen 节点本地模型缓存(默认关闭)
# Qwen 节点本地模型缓存(通用默认关闭;POC 已验收启用)
这个优化只服务于 `operator-builtin-qwen-behavior-rough-cut`。它不会改变
Qwen3-VL 的固定 revision、模型 manifest、BOS/PVC 来源、工作流图或任何已经验收的
@ -17,9 +17,10 @@ DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR=kubernetes.io/hostname=<gpu-node>
DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS=86400
```
默认值是 `false`、空 PVC、`/qwen-node-cache`、空 selector 和 `86400` 秒。启用时 PVC
和 selector 缺失会 fail-closed;其它 builtin、普通 operator、Qwen 时间定位、训练和
姿态算子都不会得到该挂载或这些环境变量。工作流 hyperparameter 也不能覆盖这些
通用渲染/部署模板仍默认 `false`、空 PVC、`/qwen-node-cache`、空 selector 和 `86400`
秒;只有 `scripts/deploy-envs/poc.env` 为已验收的 POC Local PV 预置了上面的五项值。
启用时 PVC 和 selector 缺失会 fail-closed;其它 builtin、普通 operator、Qwen 时间定位、
训练和姿态算子都不会得到该挂载或这些环境变量。工作流 hyperparameter 也不能覆盖这些
平台保留键。
本 POC 已只读确认 GPU node `192.168.80.136` 上 `/home/cce` 是独立的
@ -51,13 +52,38 @@ manifest SHA、Qwen repository/revision、config/processor/index 和完整布局
## Local PV 准备顺序
1. 保持 feature disabled 部署并验证 Worker 健康。
2. 在 GPU node 上只创建 `/home/cce/droboticflow-qwen-cache`,权限 `10001:10001`、`0750`;
不执行递归 chmod,也不格式化或挂载任何盘。
使用 `scripts/render-qwen-node-cache-local-pv.sh --prepare-job` 只渲染受限的一次性
准备 Job;该脚本不会执行 apply。默认 PV/PVC 同样通过
2. 在 GPU node 上只创建 `/home/cce/droboticflow-qwen-cache`,初始权限为
`10001:10001`、`0750`;不执行递归 chmod,也不格式化或挂载任何盘。Local PVC 被
运行时 Pod 以 `fsGroup=10001` 挂载后,Linux 可能将根目录表现为 `2770`(setgid,
同一受限组可写);这不是 world-writable,也不应被递归改回。
在渲染 `--preflight-job` 或 `--prepare-job` 前,必须显式提供 digest-pinned
`QWEN_NODE_CACHE_PREPARE_IMAGE`,并运行
`scripts/verify-qwen-node-cache-control-image.sh`:它分别验证 registry manifest、API
Deployment spec 和所有 API Pod imageID,三者任一不一致即停止。`--preflight-job`
只读挂载父目录和候选目录,以 UID/GID `10001` 检查 realpath、权限和内容;
`--prepare-job` 只挂载精确候选目录,使用最小 `CHOWN`、`DAC_OVERRIDE`、`FOWNER`
capability 顺序执行 chown/chmod/UID 10001 可写探针,无 service-account token、GPU、
privileged 或 host namespace。两者均只渲染,不会 apply。默认 PV/PVC 同样通过
`scripts/render-qwen-node-cache-local-pv.sh` 仅渲染,固定为 80Gi、Retain 与 hostname
affinity,模板不属于 `deploy-k8s.sh`。
3. 创建一个带 hostname nodeAffinity 的 `ReadWriteOnce` Local PV/PVC,再在 Worker 配置中
填入相同 PVC、mount path 与 hostname selector。
4. 先用真实固定 manifest 预热,检查 receipt、17.5GB 级别文件和 SHA 证据;再启用开关。
5. 启动两个独立 Qwen 粗切 Pod,记录首次 miss、第二次 hit、输出 hash 和 GPU 证据。
## 已验收的 POC 证据
- operator image 为
`ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-qwen-behavior-rough-cut@sha256:f1c4889b664d8bc24bf61ec9abf230a0c95c479fe54716514e370620450c25c7`。
- disabled compatibility Job `fanyu-qwen-cache-disabled-compat-r2-20260822`、cache miss Job
`fanyu-qwen-cache-miss-r2-20260822` 与独立 Pod cache hit Job
`fanyu-qwen-cache-hit-r2-20260822` 都成功。三次经过删除运行/缓存特有 provenance 后的
canonical 输出 SHA-256 都是
`a7656873eef8fe4d59ff46e07ff5dce66f4379fd684ec2800d04f4c1e156a775`。
- 固定模型 manifest SHA-256 是
`3c00bb1b1ada0542de91785001395fc6d4252e3d8a0844ec5fdfe05b306a481a`;sealed receipt
SHA-256 是 `dde9e72fd1d72b5556f16a121967dfc62ed57217a8b6be030923af4bcda870bb`。
miss 复制 `17,545,915,883` bytes,hit `copiedBytes=0` 且 `modelStagedBytes=0`。
- Worker scope fixture contract 的 SHA-256 是
`0e799649aeb9bd41018f8ad831bbc7eee7380d363c2be3aab9a0e04d94411c09`;它证明只会把
PVC/env/selector 注入 exact Qwen 粗切 builtin,其他 operator 保持为空。

View File

@ -0,0 +1,67 @@
# Render-only, read-only host directory preflight. It never creates a path and
# is separate from the one-shot mutating prepare Job.
apiVersion: batch/v1
kind: Job
metadata:
name: __QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME__
namespace: __QWEN_NODE_CACHE_NAMESPACE__
labels:
app: droboticflow
component: qwen-node-cache-preflight
spec:
backoffLimit: 0
activeDeadlineSeconds: 120
ttlSecondsAfterFinished: 3600
template:
spec:
nodeName: __QWEN_NODE_CACHE_NODE_NAME__
automountServiceAccountToken: false
hostNetwork: false
hostPID: false
hostIPC: false
restartPolicy: Never
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: preflight
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
command: ["/bin/sh", "-lc"]
args:
- |
set -eu
stat -Lc 'parent-mode=%a owner=%u:%g inode=%i path=%n' /host-cache-parent
stat -Lc 'candidate-mode=%a owner=%u:%g inode=%i path=%n' /cache-root
realpath /cache-root
find /cache-root -mindepth 1 -maxdepth 1 -printf '%f\n' | wc -l
test ! -w /cache-root
securityContext:
runAsUser: 10001
runAsGroup: 10001
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: host-cache-parent
mountPath: /host-cache-parent
readOnly: true
- name: cache-root
mountPath: /cache-root
readOnly: true
volumes:
- name: host-cache-parent
hostPath:
path: /home/cce
type: Directory
- name: cache-root
hostPath:
path: __QWEN_NODE_CACHE_HOST_PATH__
type: Directory

View File

@ -1,6 +1,6 @@
# Render-only, one-shot directory preparation Job. It is never applied by the
# platform deployment. Its only mutation is creating/chowning the exact cache
# directory under the independently verified /home/cce NVMe mount.
# platform deployment. Its only mutation is chown/chmod of the exact already
# preflighted cache directory; it never mounts the /home/cce parent directory.
apiVersion: batch/v1
kind: Job
metadata:
@ -11,31 +11,130 @@ metadata:
component: qwen-node-cache-prepare
spec:
backoffLimit: 0
activeDeadlineSeconds: 120
ttlSecondsAfterFinished: 3600
template:
spec:
nodeName: __QWEN_NODE_CACHE_NODE_NAME__
automountServiceAccountToken: false
hostNetwork: false
hostPID: false
hostIPC: false
restartPolicy: Never
containers:
- name: prepare
securityContext:
seccompProfile:
type: RuntimeDefault
initContainers:
- name: chown-cache-root
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
command: ["/bin/sh", "-lc"]
args:
- |
set -eu
install -d -m 0750 -o 10001 -g 10001 /host-cache/__QWEN_NODE_CACHE_DIR_NAME__
stat -c 'mode=%a owner=%u:%g path=%n' /host-cache/__QWEN_NODE_CACHE_DIR_NAME__
command: ["/usr/bin/chown", "10001:10001", "/cache-root"]
securityContext:
runAsUser: 0
runAsGroup: 0
privileged: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["CHOWN", "DAC_OVERRIDE", "FOWNER"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: host-cache-parent
mountPath: /host-cache
- name: cache-root
mountPath: /cache-root
- name: chmod-cache-root
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
command: ["/usr/bin/chmod", "0750", "/cache-root"]
securityContext:
runAsUser: 0
runAsGroup: 0
privileged: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["CHOWN", "DAC_OVERRIDE", "FOWNER"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: cache-root
mountPath: /cache-root
- name: rw-touch-as-10001
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
command: ["/usr/bin/touch", "/cache-root/.qwen-cache-rw-probe"]
securityContext:
runAsUser: 10001
runAsGroup: 10001
privileged: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: cache-root
mountPath: /cache-root
- name: rw-remove-as-10001
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
command: ["/usr/bin/rm", "/cache-root/.qwen-cache-rw-probe"]
securityContext:
runAsUser: 10001
runAsGroup: 10001
privileged: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: cache-root
mountPath: /cache-root
containers:
- name: verify-as-10001
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
command: ["/usr/bin/stat", "-c", "mode=%a owner=%u:%g path=%n", "/cache-root"]
securityContext:
runAsUser: 10001
runAsGroup: 10001
privileged: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: cache-root
mountPath: /cache-root
volumes:
- name: host-cache-parent
- name: cache-root
hostPath:
path: /home/cce
type: Directory
path: __QWEN_NODE_CACHE_HOST_PATH__
type: DirectoryOrCreate

View File

@ -1,6 +1,41 @@
{
"acceptance": {
"artifactId": "fanyu-qwen-cache-disabled-compat-20260822-fanyu-v5-rough-cut-clips-manifest-clips-manifest.json",
"nodeLocalCache": {
"canonicalOutputSha256": "a7656873eef8fe4d59ff46e07ff5dce66f4379fd684ec2800d04f4c1e156a775",
"canonicalization": "delete run/cache-specific provenance: inferenceId, executedAt, modelRuntimePath, modelStagedBytes, nodeLocalCache",
"fixtureContractSha256": "0e799649aeb9bd41018f8ad831bbc7eee7380d363c2be3aab9a0e04d94411c09",
"modelManifestSha256": "3c00bb1b1ada0542de91785001395fc6d4252e3d8a0844ec5fdfe05b306a481a",
"operatorImageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-qwen-behavior-rough-cut@sha256:f1c4889b664d8bc24bf61ec9abf230a0c95c479fe54716514e370620450c25c7",
"receiptSha256": "dde9e72fd1d72b5556f16a121967dfc62ed57217a8b6be030923af4bcda870bb",
"runs": {
"disabledCompatibility": {
"cacheHit": false,
"copiedBytes": 17545915883,
"jobId": "fanyu-qwen-cache-disabled-compat-r2-20260822",
"modelStagedBytes": 17545915883,
"runId": "fanyu-qwen-cache-disabled-compat-r2-20260822",
"status": "disabled"
},
"hit": {
"cacheHit": true,
"copiedBytes": 0,
"jobId": "fanyu-qwen-cache-hit-r2-20260822",
"modelStagedBytes": 0,
"runId": "fanyu-qwen-cache-hit-r2-20260822",
"status": "hit"
},
"miss": {
"cacheHit": false,
"copiedBytes": 17545915883,
"jobId": "fanyu-qwen-cache-miss-r2-20260822",
"modelStagedBytes": 17545915883,
"runId": "fanyu-qwen-cache-miss-r2-20260822",
"status": "miss"
}
},
"scopeFixture": "go test ./internal/worker -run ^TestQwenNodeCache(AppliesOnlyToExactBuiltin|DisabledAndInvalidConfigurationFailSafe|RejectsPVCMountCollision)$ -count=1"
},
"projectId": "project-271c97276a31d97dd3eff93e3f1ff3b0",
"runId": "fanyu-qwen-cache-disabled-compat-r2-20260822",
"verifiedAt": "2026-08-21T23:26:03Z",

View File

@ -42,10 +42,10 @@ export WORKER_REPLICAS="${WORKER_REPLICAS:-2}"
export DROSYNC_COORDINATOR_REPLICAS="${DROSYNC_COORDINATOR_REPLICAS:-2}"
export DROBOTICFLOW_OBJECT_SYNC_ENABLED="${DROBOTICFLOW_OBJECT_SYNC_ENABLED:-false}"
export OPERATOR_JOB_TTL_SECONDS="${OPERATOR_JOB_TTL_SECONDS:-86400}"
export DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED="${DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED:-false}"
export DROBOTICFLOW_QWEN_NODE_CACHE_PVC="${DROBOTICFLOW_QWEN_NODE_CACHE_PVC:-}"
export DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED="${DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED:-true}"
export DROBOTICFLOW_QWEN_NODE_CACHE_PVC="${DROBOTICFLOW_QWEN_NODE_CACHE_PVC:-droboticflow-qwen-node-cache}"
export DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH="${DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH:-/qwen-node-cache}"
export DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR="${DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR:-}"
export DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR="${DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR:-kubernetes.io/hostname=192.168.80.136}"
export DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS="${DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS:-86400}"
export CUSTOM_OPERATOR_IMAGE_REGISTRY="${CUSTOM_OPERATOR_IMAGE_REGISTRY:-ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/custom-operators}"
export CUSTOM_OPERATOR_IMAGE_IMPORTER_IMAGE="${CUSTOM_OPERATOR_IMAGE_IMPORTER_IMAGE:-quay.io/skopeo/stable:v1.16.1}"

View File

@ -8,9 +8,12 @@ MODE="pv"
if [[ "${1:-}" == "--prepare-job" ]]; then
MODE="prepare"
shift
elif [[ "${1:-}" == "--preflight-job" ]]; then
MODE="preflight"
shift
fi
if [[ $# -gt 0 ]]; then
echo "usage: $0 [--prepare-job]" >&2
echo "usage: $0 [--preflight-job|--prepare-job]" >&2
exit 2
fi
@ -21,9 +24,10 @@ QWEN_NODE_CACHE_SIZE="${QWEN_NODE_CACHE_SIZE:-80Gi}"
QWEN_NODE_CACHE_PV_NAME="${QWEN_NODE_CACHE_PV_NAME:-droboticflow-qwen-node-cache-pv}"
QWEN_NODE_CACHE_PVC_NAME="${QWEN_NODE_CACHE_PVC_NAME:-droboticflow-qwen-node-cache}"
QWEN_NODE_CACHE_PREPARE_JOB_NAME="${QWEN_NODE_CACHE_PREPARE_JOB_NAME:-droboticflow-qwen-node-cache-prepare}"
QWEN_NODE_CACHE_PREPARE_IMAGE="${QWEN_NODE_CACHE_PREPARE_IMAGE:-ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-api@sha256:23fa92a39016d7c31f88b63974453d67a6da9bcc02783d4b24bff68566f86828}"
QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME="${QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME:-droboticflow-qwen-node-cache-preflight}"
QWEN_NODE_CACHE_PREPARE_IMAGE="${QWEN_NODE_CACHE_PREPARE_IMAGE:-}"
export QWEN_NODE_CACHE_NAMESPACE QWEN_NODE_CACHE_NODE_NAME QWEN_NODE_CACHE_HOST_PATH QWEN_NODE_CACHE_SIZE \
QWEN_NODE_CACHE_PV_NAME QWEN_NODE_CACHE_PVC_NAME QWEN_NODE_CACHE_PREPARE_JOB_NAME QWEN_NODE_CACHE_PREPARE_IMAGE
QWEN_NODE_CACHE_PV_NAME QWEN_NODE_CACHE_PVC_NAME QWEN_NODE_CACHE_PREPARE_JOB_NAME QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME QWEN_NODE_CACHE_PREPARE_IMAGE
python3 - "$ROOT_DIR" "$MODE" <<'PY'
import os
@ -36,10 +40,10 @@ mode = sys.argv[2]
values = {key: os.environ[key] for key in (
"QWEN_NODE_CACHE_NAMESPACE", "QWEN_NODE_CACHE_NODE_NAME", "QWEN_NODE_CACHE_HOST_PATH",
"QWEN_NODE_CACHE_SIZE", "QWEN_NODE_CACHE_PV_NAME", "QWEN_NODE_CACHE_PVC_NAME",
"QWEN_NODE_CACHE_PREPARE_JOB_NAME", "QWEN_NODE_CACHE_PREPARE_IMAGE",
"QWEN_NODE_CACHE_PREPARE_JOB_NAME", "QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME", "QWEN_NODE_CACHE_PREPARE_IMAGE",
)}
label = re.compile(r"^[a-z0-9]([-a-z0-9]*[a-z0-9])?$")
for key in ("QWEN_NODE_CACHE_NAMESPACE", "QWEN_NODE_CACHE_PV_NAME", "QWEN_NODE_CACHE_PVC_NAME", "QWEN_NODE_CACHE_PREPARE_JOB_NAME"):
for key in ("QWEN_NODE_CACHE_NAMESPACE", "QWEN_NODE_CACHE_PV_NAME", "QWEN_NODE_CACHE_PVC_NAME", "QWEN_NODE_CACHE_PREPARE_JOB_NAME", "QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME"):
if len(values[key]) > 63 or not label.fullmatch(values[key]):
raise SystemExit(f"{key} must be a lowercase DNS label")
if not re.fullmatch(r"[a-z0-9]([a-z0-9.-]*[a-z0-9])?", values["QWEN_NODE_CACHE_NODE_NAME"]):
@ -49,9 +53,14 @@ if str(path) != values["QWEN_NODE_CACHE_HOST_PATH"] or not str(path).startswith(
raise SystemExit("QWEN_NODE_CACHE_HOST_PATH must be a clean unique directory under /home/cce")
if not re.fullmatch(r"[1-9][0-9]*(Mi|Gi|Ti)", values["QWEN_NODE_CACHE_SIZE"]):
raise SystemExit("QWEN_NODE_CACHE_SIZE must be a positive Mi/Gi/Ti quantity")
if "@sha256:" not in values["QWEN_NODE_CACHE_PREPARE_IMAGE"]:
raise SystemExit("QWEN_NODE_CACHE_PREPARE_IMAGE must be digest-pinned")
template = root / "k8s" / ("qwen-node-cache.prepare-dir.template.yaml" if mode == "prepare" else "qwen-node-cache.local-pv.template.yaml")
if mode != "pv" and not re.fullmatch(r".+@sha256:[a-f0-9]{64}", values["QWEN_NODE_CACHE_PREPARE_IMAGE"]):
raise SystemExit("QWEN_NODE_CACHE_PREPARE_IMAGE must be an exact digest-pinned image for prepare/preflight rendering")
template_name = {
"pv": "qwen-node-cache.local-pv.template.yaml",
"prepare": "qwen-node-cache.prepare-dir.template.yaml",
"preflight": "qwen-node-cache.preflight-dir.template.yaml",
}[mode]
template = root / "k8s" / template_name
rendered = template.read_text(encoding="utf-8")
replacements = {"__" + key + "__": value for key, value in values.items()}
replacements["__QWEN_NODE_CACHE_DIR_NAME__"] = path.name

View File

@ -0,0 +1,36 @@
#!/usr/bin/env bash
set -euo pipefail
# Verify the control image before rendering either node-cache control Job. This
# is read-only and intentionally checks the exact registry digest, the API
# Deployment spec, and the running API pod imageID independently.
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
K8S_NAMESPACE="${K8S_NAMESPACE:-droboticflow}"
K8S_IMAGE_PULL_SECRET_NAME="${K8S_IMAGE_PULL_SECRET_NAME:-droboticflow-registry}"
QWEN_NODE_CACHE_PREPARE_IMAGE="${QWEN_NODE_CACHE_PREPARE_IMAGE:-}"
EXPECTED_IMAGE="$QWEN_NODE_CACHE_PREPARE_IMAGE" python3 - <<'PY'
import os
import re
value = os.environ["EXPECTED_IMAGE"].strip()
if not re.fullmatch(r".+@sha256:[a-f0-9]{64}", value):
raise SystemExit("QWEN_NODE_CACHE_PREPARE_IMAGE must be an exact digest-pinned image")
PY
kubectl -n "$K8S_NAMESPACE" get secret "$K8S_IMAGE_PULL_SECRET_NAME" -o jsonpath='{.data.\.dockerconfigjson}' \
| base64 --decode \
| python3 "$ROOT_DIR/scripts/verify_registry_manifest.py" --image "$QWEN_NODE_CACHE_PREPARE_IMAGE" >&2
kubectl -n "$K8S_NAMESPACE" get deployment droboticflow-api -o json \
| EXPECTED_IMAGE="$QWEN_NODE_CACHE_PREPARE_IMAGE" python3 -c 'import json,os,sys; p=json.load(sys.stdin); image=next(c.get("image") for c in p["spec"]["template"]["spec"]["containers"] if c.get("name")=="api"); expected=os.environ["EXPECTED_IMAGE"]; assert image==expected, f"API deployment image mismatch: {image}"; print({"apiDeploymentImage":image})'
kubectl -n "$K8S_NAMESPACE" get pods -l app=droboticflow-api -o json \
| EXPECTED_IMAGE="$QWEN_NODE_CACHE_PREPARE_IMAGE" python3 -c 'import json,os,sys; p=json.load(sys.stdin); expected=os.environ["EXPECTED_IMAGE"]; rows=[]
for pod in p.get("items",[]):
status=next((c for c in pod.get("status",{}).get("containerStatuses",[]) if c.get("name")=="api"),{})
image_id=status.get("imageID","")
if image_id != expected: raise SystemExit("API pod imageID mismatch: {}={}".format(pod["metadata"]["name"], image_id))
rows.append({"pod":pod["metadata"]["name"],"imageID":image_id,"ready":status.get("ready")})
if not rows: raise SystemExit("no API pods found for control-image verification")
print({"apiPods":rows})'

View File

@ -8,6 +8,9 @@ import unittest
ROOT = Path(__file__).resolve().parents[2]
SCRIPT = ROOT / "scripts" / "render-qwen-node-cache-local-pv.sh"
CONTROL_VERIFY_SCRIPT = ROOT / "scripts" / "verify-qwen-node-cache-control-image.sh"
POC_ENV = ROOT / "scripts" / "deploy-envs" / "poc.env"
CONTROL_IMAGE = "registry.example/control@sha256:" + "a" * 64
class TestQwenNodeLocalPVTemplate(unittest.TestCase):
@ -28,12 +31,31 @@ class TestQwenNodeLocalPVTemplate(unittest.TestCase):
self.assertNotIn("kubectl apply", result.stdout)
def test_prepare_job_is_strictly_scoped_to_single_cache_directory(self) -> None:
result = self.render("--prepare-job")
result = self.render("--prepare-job", QWEN_NODE_CACHE_PREPARE_IMAGE=CONTROL_IMAGE)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("path: /home/cce", result.stdout)
self.assertIn("install -d -m 0750 -o 10001 -g 10001 /host-cache/droboticflow-qwen-cache", result.stdout)
self.assertIn("path: /home/cce/droboticflow-qwen-cache", result.stdout)
self.assertIn("type: DirectoryOrCreate", result.stdout)
self.assertIn('command: ["/usr/bin/chown", "10001:10001", "/cache-root"]', result.stdout)
self.assertIn('command: ["/usr/bin/chmod", "0750", "/cache-root"]', result.stdout)
self.assertIn('add: ["CHOWN", "DAC_OVERRIDE", "FOWNER"]', result.stdout)
self.assertIn("automountServiceAccountToken: false", result.stdout)
self.assertNotIn("chmod -R", result.stdout)
self.assertNotIn("path: /\n", result.stdout)
self.assertNotIn("privileged: true", result.stdout)
self.assertNotIn("host-cache-parent", result.stdout)
self.assertIn("activeDeadlineSeconds: 120", result.stdout)
self.assertNotIn("nvidia.com/gpu", result.stdout)
def test_preflight_job_is_read_only_and_requires_exact_control_image(self) -> None:
missing = self.render("--preflight-job")
self.assertNotEqual(missing.returncode, 0)
self.assertIn("must be an exact digest-pinned image", missing.stderr)
result = self.render("--preflight-job", QWEN_NODE_CACHE_PREPARE_IMAGE=CONTROL_IMAGE)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("readOnly: true", result.stdout)
self.assertIn("realpath /cache-root", result.stdout)
self.assertIn("host-cache-parent", result.stdout)
self.assertIn("runAsUser: 10001", result.stdout)
self.assertNotIn("install -d", result.stdout)
def test_rejects_root_and_outside_nvme_paths(self) -> None:
for path in ("/", "/var/lib/droboticflow-qwen-cache", "/home/cce", "/home/cce/../root"):
@ -42,6 +64,33 @@ class TestQwenNodeLocalPVTemplate(unittest.TestCase):
self.assertNotEqual(result.returncode, 0)
self.assertIn("must be a clean unique directory under /home/cce", result.stderr)
def test_control_image_verifier_requires_three_read_only_proofs(self) -> None:
script = CONTROL_VERIFY_SCRIPT.read_text(encoding="utf-8")
self.assertIn("verify_registry_manifest.py", script)
self.assertIn("get deployment droboticflow-api", script)
self.assertIn("get pods -l app=droboticflow-api", script)
self.assertIn("must be an exact digest-pinned image", script)
self.assertNotIn("kubectl apply", script)
def test_poc_profile_enables_only_the_verified_cache_configuration(self) -> None:
command = """
unset DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED DROBOTICFLOW_QWEN_NODE_CACHE_PVC
unset DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR
unset DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS
source \"$1\"
printf '%s\\n' \"$DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED\" \\
\"$DROBOTICFLOW_QWEN_NODE_CACHE_PVC\" \\
\"$DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH\" \\
\"$DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR\" \\
\"$DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS\"
"""
result = subprocess.run(["bash", "-c", command, "--", str(POC_ENV)], cwd=ROOT, text=True, capture_output=True, check=False)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(
result.stdout.splitlines(),
["true", "droboticflow-qwen-node-cache", "/qwen-node-cache", "kubernetes.io/hostname=192.168.80.136", "86400"],
)
if __name__ == "__main__":
unittest.main()