diff --git a/docs/deployment.md b/docs/deployment.md index b583769f..9b1a989d 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -107,6 +107,9 @@ For the `poc` environment, use [`scripts/deploy-envs/poc.env`](../scripts/deploy - `MONGO_DB=poc` - `API_PUBLIC_LOAD_BALANCER=true` +- 已验收 Qwen 节点本地缓存:`DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED=true`、 + `droboticflow-qwen-node-cache` PVC、`/qwen-node-cache` 和 GPU node hostname selector; + 这只影响 allowlisted Qwen 粗切 Job,通用模板仍默认关闭。 - `BOS_ENDPOINT=https://bj.bcebos.com` - the same namespace, AI defaults, and PVC settings as `wjzh` diff --git a/docs/fanyu-v5-workflow.md b/docs/fanyu-v5-workflow.md index e3ee90f9..df1f981e 100644 --- a/docs/fanyu-v5-workflow.md +++ b/docs/fanyu-v5-workflow.md @@ -28,7 +28,7 @@ python3 scripts/verify_qwen3vl_model_manifest.py --model-root /workspace/models/ 独立预载 Job 必须先逐字节验证全部 4 个 shard/16 个 payload 文件,再原子发布模型目录。生产 workflow 将该 Job 产出的 manifest SHA-256 固定为 `MODEL_MANIFEST_SHA256`;每个短生命周期 Qwen Pod 重新验证 sidecar、固定 SHA、config/processor/index、四 shard 证据,以及完整本地路径和大小布局,但不重复从共享 PVC 读取 17.5 GB 做同一遍哈希。未提供固定 SHA 时,`TRUST_PREVERIFIED_MODEL=true` 会失败关闭;非预载模式仍执行全文件哈希。由于 POC CSI 使用 direct I/O,默认行为是以四 shard 并行大块读把固定快照复制到该 Pod 的本地 `/tmp`,再从本地路径加载。 -可选的节点本地缓存默认关闭,且只允许 Qwen3-VL 手部行为语义粗切使用。启用后,缓存目录以固定 manifest SHA 为 key,源 manifest/provenance 每次重验,miss 按文件 SHA/size 复制并原子写 receipt,hit 会完整本地哈希复验;空间不足或损坏会回退 Pod-local staging 而不会使用不完整缓存。详情与 PV 安全边界见 [Qwen 节点本地模型缓存](qwen-node-local-cache.md)。Pod provenance 同时记录 PVC manifest 路径、本地 runtime 路径、复制字节数和 `nodeLocalCache` 状态。Qwen 节点仅从本地快照加载(`local_files_only=True`),使用固定的 `transformers==4.57.6` 和 `Qwen3VLForConditionalGeneration`,不会执行远程模型代码。 +可选的节点本地缓存通用默认关闭,且只允许 Qwen3-VL 手部行为语义粗切使用;已验收的 `poc` profile 通过 Local PV 显式启用。缓存目录以固定 manifest SHA 为 key,源 manifest/provenance 每次重验,miss 按文件 SHA/size 复制并原子写 receipt,hit 会完整本地哈希复验;空间不足或损坏会回退 Pod-local staging 而不会使用不完整缓存。详情、PV 安全边界和 disabled/miss/hit 实测证据见 [Qwen 节点本地模型缓存](qwen-node-local-cache.md)。Pod provenance 同时记录 PVC manifest 路径、本地 runtime 路径、复制字节数和 `nodeLocalCache` 状态。Qwen 节点仅从本地快照加载(`local_files_only=True`),使用固定的 `transformers==4.57.6` 和 `Qwen3VLForConditionalGeneration`,不会执行远程模型代码。 ## 人在环与交付 diff --git a/docs/qwen-node-local-cache.md b/docs/qwen-node-local-cache.md index b59679e6..6e1051a3 100644 --- a/docs/qwen-node-local-cache.md +++ b/docs/qwen-node-local-cache.md @@ -1,4 +1,4 @@ -# Qwen 节点本地模型缓存(默认关闭) +# Qwen 节点本地模型缓存(通用默认关闭;POC 已验收启用) 这个优化只服务于 `operator-builtin-qwen-behavior-rough-cut`。它不会改变 Qwen3-VL 的固定 revision、模型 manifest、BOS/PVC 来源、工作流图或任何已经验收的 @@ -17,9 +17,10 @@ DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR=kubernetes.io/hostname= DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS=86400 ``` -默认值是 `false`、空 PVC、`/qwen-node-cache`、空 selector 和 `86400` 秒。启用时 PVC -和 selector 缺失会 fail-closed;其它 builtin、普通 operator、Qwen 时间定位、训练和 -姿态算子都不会得到该挂载或这些环境变量。工作流 hyperparameter 也不能覆盖这些 +通用渲染/部署模板仍默认 `false`、空 PVC、`/qwen-node-cache`、空 selector 和 `86400` +秒;只有 `scripts/deploy-envs/poc.env` 为已验收的 POC Local PV 预置了上面的五项值。 +启用时 PVC 和 selector 缺失会 fail-closed;其它 builtin、普通 operator、Qwen 时间定位、 +训练和姿态算子都不会得到该挂载或这些环境变量。工作流 hyperparameter 也不能覆盖这些 平台保留键。 本 POC 已只读确认 GPU node `192.168.80.136` 上 `/home/cce` 是独立的 @@ -51,13 +52,38 @@ manifest SHA、Qwen repository/revision、config/processor/index 和完整布局 ## Local PV 准备顺序 1. 保持 feature disabled 部署并验证 Worker 健康。 -2. 在 GPU node 上只创建 `/home/cce/droboticflow-qwen-cache`,权限 `10001:10001`、`0750`; - 不执行递归 chmod,也不格式化或挂载任何盘。 - 使用 `scripts/render-qwen-node-cache-local-pv.sh --prepare-job` 只渲染受限的一次性 - 准备 Job;该脚本不会执行 apply。默认 PV/PVC 同样通过 +2. 在 GPU node 上只创建 `/home/cce/droboticflow-qwen-cache`,初始权限为 + `10001:10001`、`0750`;不执行递归 chmod,也不格式化或挂载任何盘。Local PVC 被 + 运行时 Pod 以 `fsGroup=10001` 挂载后,Linux 可能将根目录表现为 `2770`(setgid, + 同一受限组可写);这不是 world-writable,也不应被递归改回。 + 在渲染 `--preflight-job` 或 `--prepare-job` 前,必须显式提供 digest-pinned + `QWEN_NODE_CACHE_PREPARE_IMAGE`,并运行 + `scripts/verify-qwen-node-cache-control-image.sh`:它分别验证 registry manifest、API + Deployment spec 和所有 API Pod imageID,三者任一不一致即停止。`--preflight-job` + 只读挂载父目录和候选目录,以 UID/GID `10001` 检查 realpath、权限和内容; + `--prepare-job` 只挂载精确候选目录,使用最小 `CHOWN`、`DAC_OVERRIDE`、`FOWNER` + capability 顺序执行 chown/chmod/UID 10001 可写探针,无 service-account token、GPU、 + privileged 或 host namespace。两者均只渲染,不会 apply。默认 PV/PVC 同样通过 `scripts/render-qwen-node-cache-local-pv.sh` 仅渲染,固定为 80Gi、Retain 与 hostname affinity,模板不属于 `deploy-k8s.sh`。 3. 创建一个带 hostname nodeAffinity 的 `ReadWriteOnce` Local PV/PVC,再在 Worker 配置中 填入相同 PVC、mount path 与 hostname selector。 4. 先用真实固定 manifest 预热,检查 receipt、17.5GB 级别文件和 SHA 证据;再启用开关。 5. 启动两个独立 Qwen 粗切 Pod,记录首次 miss、第二次 hit、输出 hash 和 GPU 证据。 + +## 已验收的 POC 证据 + +- operator image 为 + `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-qwen-behavior-rough-cut@sha256:f1c4889b664d8bc24bf61ec9abf230a0c95c479fe54716514e370620450c25c7`。 +- disabled compatibility Job `fanyu-qwen-cache-disabled-compat-r2-20260822`、cache miss Job + `fanyu-qwen-cache-miss-r2-20260822` 与独立 Pod cache hit Job + `fanyu-qwen-cache-hit-r2-20260822` 都成功。三次经过删除运行/缓存特有 provenance 后的 + canonical 输出 SHA-256 都是 + `a7656873eef8fe4d59ff46e07ff5dce66f4379fd684ec2800d04f4c1e156a775`。 +- 固定模型 manifest SHA-256 是 + `3c00bb1b1ada0542de91785001395fc6d4252e3d8a0844ec5fdfe05b306a481a`;sealed receipt + SHA-256 是 `dde9e72fd1d72b5556f16a121967dfc62ed57217a8b6be030923af4bcda870bb`。 + miss 复制 `17,545,915,883` bytes,hit `copiedBytes=0` 且 `modelStagedBytes=0`。 +- Worker scope fixture contract 的 SHA-256 是 + `0e799649aeb9bd41018f8ad831bbc7eee7380d363c2be3aab9a0e04d94411c09`;它证明只会把 + PVC/env/selector 注入 exact Qwen 粗切 builtin,其他 operator 保持为空。 diff --git a/k8s/qwen-node-cache.preflight-dir.template.yaml b/k8s/qwen-node-cache.preflight-dir.template.yaml new file mode 100644 index 00000000..8369b388 --- /dev/null +++ b/k8s/qwen-node-cache.preflight-dir.template.yaml @@ -0,0 +1,67 @@ +# Render-only, read-only host directory preflight. It never creates a path and +# is separate from the one-shot mutating prepare Job. +apiVersion: batch/v1 +kind: Job +metadata: + name: __QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME__ + namespace: __QWEN_NODE_CACHE_NAMESPACE__ + labels: + app: droboticflow + component: qwen-node-cache-preflight +spec: + backoffLimit: 0 + activeDeadlineSeconds: 120 + ttlSecondsAfterFinished: 3600 + template: + spec: + nodeName: __QWEN_NODE_CACHE_NODE_NAME__ + automountServiceAccountToken: false + hostNetwork: false + hostPID: false + hostIPC: false + restartPolicy: Never + securityContext: + seccompProfile: + type: RuntimeDefault + containers: + - name: preflight + image: __QWEN_NODE_CACHE_PREPARE_IMAGE__ + command: ["/bin/sh", "-lc"] + args: + - | + set -eu + stat -Lc 'parent-mode=%a owner=%u:%g inode=%i path=%n' /host-cache-parent + stat -Lc 'candidate-mode=%a owner=%u:%g inode=%i path=%n' /cache-root + realpath /cache-root + find /cache-root -mindepth 1 -maxdepth 1 -printf '%f\n' | wc -l + test ! -w /cache-root + securityContext: + runAsUser: 10001 + runAsGroup: 10001 + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 100m + memory: 128Mi + volumeMounts: + - name: host-cache-parent + mountPath: /host-cache-parent + readOnly: true + - name: cache-root + mountPath: /cache-root + readOnly: true + volumes: + - name: host-cache-parent + hostPath: + path: /home/cce + type: Directory + - name: cache-root + hostPath: + path: __QWEN_NODE_CACHE_HOST_PATH__ + type: Directory diff --git a/k8s/qwen-node-cache.prepare-dir.template.yaml b/k8s/qwen-node-cache.prepare-dir.template.yaml index cffa6c6b..6e829eae 100644 --- a/k8s/qwen-node-cache.prepare-dir.template.yaml +++ b/k8s/qwen-node-cache.prepare-dir.template.yaml @@ -1,6 +1,6 @@ # Render-only, one-shot directory preparation Job. It is never applied by the -# platform deployment. Its only mutation is creating/chowning the exact cache -# directory under the independently verified /home/cce NVMe mount. +# platform deployment. Its only mutation is chown/chmod of the exact already +# preflighted cache directory; it never mounts the /home/cce parent directory. apiVersion: batch/v1 kind: Job metadata: @@ -11,31 +11,130 @@ metadata: component: qwen-node-cache-prepare spec: backoffLimit: 0 + activeDeadlineSeconds: 120 ttlSecondsAfterFinished: 3600 template: spec: nodeName: __QWEN_NODE_CACHE_NODE_NAME__ + automountServiceAccountToken: false + hostNetwork: false + hostPID: false + hostIPC: false restartPolicy: Never - containers: - - name: prepare + securityContext: + seccompProfile: + type: RuntimeDefault + initContainers: + - name: chown-cache-root image: __QWEN_NODE_CACHE_PREPARE_IMAGE__ - command: ["/bin/sh", "-lc"] - args: - - | - set -eu - install -d -m 0750 -o 10001 -g 10001 /host-cache/__QWEN_NODE_CACHE_DIR_NAME__ - stat -c 'mode=%a owner=%u:%g path=%n' /host-cache/__QWEN_NODE_CACHE_DIR_NAME__ + command: ["/usr/bin/chown", "10001:10001", "/cache-root"] securityContext: runAsUser: 0 runAsGroup: 0 + privileged: false allowPrivilegeEscalation: false + readOnlyRootFilesystem: true capabilities: drop: ["ALL"] + add: ["CHOWN", "DAC_OVERRIDE", "FOWNER"] + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 100m + memory: 128Mi volumeMounts: - - name: host-cache-parent - mountPath: /host-cache + - name: cache-root + mountPath: /cache-root + - name: chmod-cache-root + image: __QWEN_NODE_CACHE_PREPARE_IMAGE__ + command: ["/usr/bin/chmod", "0750", "/cache-root"] + securityContext: + runAsUser: 0 + runAsGroup: 0 + privileged: false + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + add: ["CHOWN", "DAC_OVERRIDE", "FOWNER"] + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 100m + memory: 128Mi + volumeMounts: + - name: cache-root + mountPath: /cache-root + - name: rw-touch-as-10001 + image: __QWEN_NODE_CACHE_PREPARE_IMAGE__ + command: ["/usr/bin/touch", "/cache-root/.qwen-cache-rw-probe"] + securityContext: + runAsUser: 10001 + runAsGroup: 10001 + privileged: false + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 100m + memory: 128Mi + volumeMounts: + - name: cache-root + mountPath: /cache-root + - name: rw-remove-as-10001 + image: __QWEN_NODE_CACHE_PREPARE_IMAGE__ + command: ["/usr/bin/rm", "/cache-root/.qwen-cache-rw-probe"] + securityContext: + runAsUser: 10001 + runAsGroup: 10001 + privileged: false + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 100m + memory: 128Mi + volumeMounts: + - name: cache-root + mountPath: /cache-root + containers: + - name: verify-as-10001 + image: __QWEN_NODE_CACHE_PREPARE_IMAGE__ + command: ["/usr/bin/stat", "-c", "mode=%a owner=%u:%g path=%n", "/cache-root"] + securityContext: + runAsUser: 10001 + runAsGroup: 10001 + privileged: false + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 100m + memory: 128Mi + volumeMounts: + - name: cache-root + mountPath: /cache-root volumes: - - name: host-cache-parent + - name: cache-root hostPath: - path: /home/cce - type: Directory + path: __QWEN_NODE_CACHE_HOST_PATH__ + type: DirectoryOrCreate diff --git a/operator-repos/builtin/qwen_behavior_rough_cut/release.json b/operator-repos/builtin/qwen_behavior_rough_cut/release.json index 33f1da1f..f1a2315c 100644 --- a/operator-repos/builtin/qwen_behavior_rough_cut/release.json +++ b/operator-repos/builtin/qwen_behavior_rough_cut/release.json @@ -1,6 +1,41 @@ { "acceptance": { "artifactId": "fanyu-qwen-cache-disabled-compat-20260822-fanyu-v5-rough-cut-clips-manifest-clips-manifest.json", + "nodeLocalCache": { + "canonicalOutputSha256": "a7656873eef8fe4d59ff46e07ff5dce66f4379fd684ec2800d04f4c1e156a775", + "canonicalization": "delete run/cache-specific provenance: inferenceId, executedAt, modelRuntimePath, modelStagedBytes, nodeLocalCache", + "fixtureContractSha256": "0e799649aeb9bd41018f8ad831bbc7eee7380d363c2be3aab9a0e04d94411c09", + "modelManifestSha256": "3c00bb1b1ada0542de91785001395fc6d4252e3d8a0844ec5fdfe05b306a481a", + "operatorImageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-qwen-behavior-rough-cut@sha256:f1c4889b664d8bc24bf61ec9abf230a0c95c479fe54716514e370620450c25c7", + "receiptSha256": "dde9e72fd1d72b5556f16a121967dfc62ed57217a8b6be030923af4bcda870bb", + "runs": { + "disabledCompatibility": { + "cacheHit": false, + "copiedBytes": 17545915883, + "jobId": "fanyu-qwen-cache-disabled-compat-r2-20260822", + "modelStagedBytes": 17545915883, + "runId": "fanyu-qwen-cache-disabled-compat-r2-20260822", + "status": "disabled" + }, + "hit": { + "cacheHit": true, + "copiedBytes": 0, + "jobId": "fanyu-qwen-cache-hit-r2-20260822", + "modelStagedBytes": 0, + "runId": "fanyu-qwen-cache-hit-r2-20260822", + "status": "hit" + }, + "miss": { + "cacheHit": false, + "copiedBytes": 17545915883, + "jobId": "fanyu-qwen-cache-miss-r2-20260822", + "modelStagedBytes": 17545915883, + "runId": "fanyu-qwen-cache-miss-r2-20260822", + "status": "miss" + } + }, + "scopeFixture": "go test ./internal/worker -run ^TestQwenNodeCache(AppliesOnlyToExactBuiltin|DisabledAndInvalidConfigurationFailSafe|RejectsPVCMountCollision)$ -count=1" + }, "projectId": "project-271c97276a31d97dd3eff93e3f1ff3b0", "runId": "fanyu-qwen-cache-disabled-compat-r2-20260822", "verifiedAt": "2026-08-21T23:26:03Z", diff --git a/scripts/deploy-envs/poc.env b/scripts/deploy-envs/poc.env index a9a59146..68cd3205 100644 --- a/scripts/deploy-envs/poc.env +++ b/scripts/deploy-envs/poc.env @@ -42,10 +42,10 @@ export WORKER_REPLICAS="${WORKER_REPLICAS:-2}" export DROSYNC_COORDINATOR_REPLICAS="${DROSYNC_COORDINATOR_REPLICAS:-2}" export DROBOTICFLOW_OBJECT_SYNC_ENABLED="${DROBOTICFLOW_OBJECT_SYNC_ENABLED:-false}" export OPERATOR_JOB_TTL_SECONDS="${OPERATOR_JOB_TTL_SECONDS:-86400}" -export DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED="${DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED:-false}" -export DROBOTICFLOW_QWEN_NODE_CACHE_PVC="${DROBOTICFLOW_QWEN_NODE_CACHE_PVC:-}" +export DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED="${DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED:-true}" +export DROBOTICFLOW_QWEN_NODE_CACHE_PVC="${DROBOTICFLOW_QWEN_NODE_CACHE_PVC:-droboticflow-qwen-node-cache}" export DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH="${DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH:-/qwen-node-cache}" -export DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR="${DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR:-}" +export DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR="${DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR:-kubernetes.io/hostname=192.168.80.136}" export DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS="${DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS:-86400}" export CUSTOM_OPERATOR_IMAGE_REGISTRY="${CUSTOM_OPERATOR_IMAGE_REGISTRY:-ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/custom-operators}" export CUSTOM_OPERATOR_IMAGE_IMPORTER_IMAGE="${CUSTOM_OPERATOR_IMAGE_IMPORTER_IMAGE:-quay.io/skopeo/stable:v1.16.1}" diff --git a/scripts/render-qwen-node-cache-local-pv.sh b/scripts/render-qwen-node-cache-local-pv.sh index 0f340b5f..09ca60c2 100755 --- a/scripts/render-qwen-node-cache-local-pv.sh +++ b/scripts/render-qwen-node-cache-local-pv.sh @@ -8,9 +8,12 @@ MODE="pv" if [[ "${1:-}" == "--prepare-job" ]]; then MODE="prepare" shift +elif [[ "${1:-}" == "--preflight-job" ]]; then + MODE="preflight" + shift fi if [[ $# -gt 0 ]]; then - echo "usage: $0 [--prepare-job]" >&2 + echo "usage: $0 [--preflight-job|--prepare-job]" >&2 exit 2 fi @@ -21,9 +24,10 @@ QWEN_NODE_CACHE_SIZE="${QWEN_NODE_CACHE_SIZE:-80Gi}" QWEN_NODE_CACHE_PV_NAME="${QWEN_NODE_CACHE_PV_NAME:-droboticflow-qwen-node-cache-pv}" QWEN_NODE_CACHE_PVC_NAME="${QWEN_NODE_CACHE_PVC_NAME:-droboticflow-qwen-node-cache}" QWEN_NODE_CACHE_PREPARE_JOB_NAME="${QWEN_NODE_CACHE_PREPARE_JOB_NAME:-droboticflow-qwen-node-cache-prepare}" -QWEN_NODE_CACHE_PREPARE_IMAGE="${QWEN_NODE_CACHE_PREPARE_IMAGE:-ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-api@sha256:23fa92a39016d7c31f88b63974453d67a6da9bcc02783d4b24bff68566f86828}" +QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME="${QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME:-droboticflow-qwen-node-cache-preflight}" +QWEN_NODE_CACHE_PREPARE_IMAGE="${QWEN_NODE_CACHE_PREPARE_IMAGE:-}" export QWEN_NODE_CACHE_NAMESPACE QWEN_NODE_CACHE_NODE_NAME QWEN_NODE_CACHE_HOST_PATH QWEN_NODE_CACHE_SIZE \ - QWEN_NODE_CACHE_PV_NAME QWEN_NODE_CACHE_PVC_NAME QWEN_NODE_CACHE_PREPARE_JOB_NAME QWEN_NODE_CACHE_PREPARE_IMAGE + QWEN_NODE_CACHE_PV_NAME QWEN_NODE_CACHE_PVC_NAME QWEN_NODE_CACHE_PREPARE_JOB_NAME QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME QWEN_NODE_CACHE_PREPARE_IMAGE python3 - "$ROOT_DIR" "$MODE" <<'PY' import os @@ -36,10 +40,10 @@ mode = sys.argv[2] values = {key: os.environ[key] for key in ( "QWEN_NODE_CACHE_NAMESPACE", "QWEN_NODE_CACHE_NODE_NAME", "QWEN_NODE_CACHE_HOST_PATH", "QWEN_NODE_CACHE_SIZE", "QWEN_NODE_CACHE_PV_NAME", "QWEN_NODE_CACHE_PVC_NAME", - "QWEN_NODE_CACHE_PREPARE_JOB_NAME", "QWEN_NODE_CACHE_PREPARE_IMAGE", + "QWEN_NODE_CACHE_PREPARE_JOB_NAME", "QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME", "QWEN_NODE_CACHE_PREPARE_IMAGE", )} label = re.compile(r"^[a-z0-9]([-a-z0-9]*[a-z0-9])?$") -for key in ("QWEN_NODE_CACHE_NAMESPACE", "QWEN_NODE_CACHE_PV_NAME", "QWEN_NODE_CACHE_PVC_NAME", "QWEN_NODE_CACHE_PREPARE_JOB_NAME"): +for key in ("QWEN_NODE_CACHE_NAMESPACE", "QWEN_NODE_CACHE_PV_NAME", "QWEN_NODE_CACHE_PVC_NAME", "QWEN_NODE_CACHE_PREPARE_JOB_NAME", "QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME"): if len(values[key]) > 63 or not label.fullmatch(values[key]): raise SystemExit(f"{key} must be a lowercase DNS label") if not re.fullmatch(r"[a-z0-9]([a-z0-9.-]*[a-z0-9])?", values["QWEN_NODE_CACHE_NODE_NAME"]): @@ -49,9 +53,14 @@ if str(path) != values["QWEN_NODE_CACHE_HOST_PATH"] or not str(path).startswith( raise SystemExit("QWEN_NODE_CACHE_HOST_PATH must be a clean unique directory under /home/cce") if not re.fullmatch(r"[1-9][0-9]*(Mi|Gi|Ti)", values["QWEN_NODE_CACHE_SIZE"]): raise SystemExit("QWEN_NODE_CACHE_SIZE must be a positive Mi/Gi/Ti quantity") -if "@sha256:" not in values["QWEN_NODE_CACHE_PREPARE_IMAGE"]: - raise SystemExit("QWEN_NODE_CACHE_PREPARE_IMAGE must be digest-pinned") -template = root / "k8s" / ("qwen-node-cache.prepare-dir.template.yaml" if mode == "prepare" else "qwen-node-cache.local-pv.template.yaml") +if mode != "pv" and not re.fullmatch(r".+@sha256:[a-f0-9]{64}", values["QWEN_NODE_CACHE_PREPARE_IMAGE"]): + raise SystemExit("QWEN_NODE_CACHE_PREPARE_IMAGE must be an exact digest-pinned image for prepare/preflight rendering") +template_name = { + "pv": "qwen-node-cache.local-pv.template.yaml", + "prepare": "qwen-node-cache.prepare-dir.template.yaml", + "preflight": "qwen-node-cache.preflight-dir.template.yaml", +}[mode] +template = root / "k8s" / template_name rendered = template.read_text(encoding="utf-8") replacements = {"__" + key + "__": value for key, value in values.items()} replacements["__QWEN_NODE_CACHE_DIR_NAME__"] = path.name diff --git a/scripts/verify-qwen-node-cache-control-image.sh b/scripts/verify-qwen-node-cache-control-image.sh new file mode 100755 index 00000000..4416c1c4 --- /dev/null +++ b/scripts/verify-qwen-node-cache-control-image.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Verify the control image before rendering either node-cache control Job. This +# is read-only and intentionally checks the exact registry digest, the API +# Deployment spec, and the running API pod imageID independently. +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +K8S_NAMESPACE="${K8S_NAMESPACE:-droboticflow}" +K8S_IMAGE_PULL_SECRET_NAME="${K8S_IMAGE_PULL_SECRET_NAME:-droboticflow-registry}" +QWEN_NODE_CACHE_PREPARE_IMAGE="${QWEN_NODE_CACHE_PREPARE_IMAGE:-}" + +EXPECTED_IMAGE="$QWEN_NODE_CACHE_PREPARE_IMAGE" python3 - <<'PY' +import os +import re + +value = os.environ["EXPECTED_IMAGE"].strip() +if not re.fullmatch(r".+@sha256:[a-f0-9]{64}", value): + raise SystemExit("QWEN_NODE_CACHE_PREPARE_IMAGE must be an exact digest-pinned image") +PY + +kubectl -n "$K8S_NAMESPACE" get secret "$K8S_IMAGE_PULL_SECRET_NAME" -o jsonpath='{.data.\.dockerconfigjson}' \ + | base64 --decode \ + | python3 "$ROOT_DIR/scripts/verify_registry_manifest.py" --image "$QWEN_NODE_CACHE_PREPARE_IMAGE" >&2 + +kubectl -n "$K8S_NAMESPACE" get deployment droboticflow-api -o json \ + | EXPECTED_IMAGE="$QWEN_NODE_CACHE_PREPARE_IMAGE" python3 -c 'import json,os,sys; p=json.load(sys.stdin); image=next(c.get("image") for c in p["spec"]["template"]["spec"]["containers"] if c.get("name")=="api"); expected=os.environ["EXPECTED_IMAGE"]; assert image==expected, f"API deployment image mismatch: {image}"; print({"apiDeploymentImage":image})' + +kubectl -n "$K8S_NAMESPACE" get pods -l app=droboticflow-api -o json \ + | EXPECTED_IMAGE="$QWEN_NODE_CACHE_PREPARE_IMAGE" python3 -c 'import json,os,sys; p=json.load(sys.stdin); expected=os.environ["EXPECTED_IMAGE"]; rows=[] +for pod in p.get("items",[]): + status=next((c for c in pod.get("status",{}).get("containerStatuses",[]) if c.get("name")=="api"),{}) + image_id=status.get("imageID","") + if image_id != expected: raise SystemExit("API pod imageID mismatch: {}={}".format(pod["metadata"]["name"], image_id)) + rows.append({"pod":pod["metadata"]["name"],"imageID":image_id,"ready":status.get("ready")}) +if not rows: raise SystemExit("no API pods found for control-image verification") +print({"apiPods":rows})' diff --git a/tests/backend/qwen_node_local_pv_template_test.py b/tests/backend/qwen_node_local_pv_template_test.py index 2cf5d023..5bd51e7b 100644 --- a/tests/backend/qwen_node_local_pv_template_test.py +++ b/tests/backend/qwen_node_local_pv_template_test.py @@ -8,6 +8,9 @@ import unittest ROOT = Path(__file__).resolve().parents[2] SCRIPT = ROOT / "scripts" / "render-qwen-node-cache-local-pv.sh" +CONTROL_VERIFY_SCRIPT = ROOT / "scripts" / "verify-qwen-node-cache-control-image.sh" +POC_ENV = ROOT / "scripts" / "deploy-envs" / "poc.env" +CONTROL_IMAGE = "registry.example/control@sha256:" + "a" * 64 class TestQwenNodeLocalPVTemplate(unittest.TestCase): @@ -28,12 +31,31 @@ class TestQwenNodeLocalPVTemplate(unittest.TestCase): self.assertNotIn("kubectl apply", result.stdout) def test_prepare_job_is_strictly_scoped_to_single_cache_directory(self) -> None: - result = self.render("--prepare-job") + result = self.render("--prepare-job", QWEN_NODE_CACHE_PREPARE_IMAGE=CONTROL_IMAGE) self.assertEqual(result.returncode, 0, result.stderr) - self.assertIn("path: /home/cce", result.stdout) - self.assertIn("install -d -m 0750 -o 10001 -g 10001 /host-cache/droboticflow-qwen-cache", result.stdout) + self.assertIn("path: /home/cce/droboticflow-qwen-cache", result.stdout) + self.assertIn("type: DirectoryOrCreate", result.stdout) + self.assertIn('command: ["/usr/bin/chown", "10001:10001", "/cache-root"]', result.stdout) + self.assertIn('command: ["/usr/bin/chmod", "0750", "/cache-root"]', result.stdout) + self.assertIn('add: ["CHOWN", "DAC_OVERRIDE", "FOWNER"]', result.stdout) + self.assertIn("automountServiceAccountToken: false", result.stdout) self.assertNotIn("chmod -R", result.stdout) - self.assertNotIn("path: /\n", result.stdout) + self.assertNotIn("privileged: true", result.stdout) + self.assertNotIn("host-cache-parent", result.stdout) + self.assertIn("activeDeadlineSeconds: 120", result.stdout) + self.assertNotIn("nvidia.com/gpu", result.stdout) + + def test_preflight_job_is_read_only_and_requires_exact_control_image(self) -> None: + missing = self.render("--preflight-job") + self.assertNotEqual(missing.returncode, 0) + self.assertIn("must be an exact digest-pinned image", missing.stderr) + result = self.render("--preflight-job", QWEN_NODE_CACHE_PREPARE_IMAGE=CONTROL_IMAGE) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("readOnly: true", result.stdout) + self.assertIn("realpath /cache-root", result.stdout) + self.assertIn("host-cache-parent", result.stdout) + self.assertIn("runAsUser: 10001", result.stdout) + self.assertNotIn("install -d", result.stdout) def test_rejects_root_and_outside_nvme_paths(self) -> None: for path in ("/", "/var/lib/droboticflow-qwen-cache", "/home/cce", "/home/cce/../root"): @@ -42,6 +64,33 @@ class TestQwenNodeLocalPVTemplate(unittest.TestCase): self.assertNotEqual(result.returncode, 0) self.assertIn("must be a clean unique directory under /home/cce", result.stderr) + def test_control_image_verifier_requires_three_read_only_proofs(self) -> None: + script = CONTROL_VERIFY_SCRIPT.read_text(encoding="utf-8") + self.assertIn("verify_registry_manifest.py", script) + self.assertIn("get deployment droboticflow-api", script) + self.assertIn("get pods -l app=droboticflow-api", script) + self.assertIn("must be an exact digest-pinned image", script) + self.assertNotIn("kubectl apply", script) + + def test_poc_profile_enables_only_the_verified_cache_configuration(self) -> None: + command = """ + unset DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED DROBOTICFLOW_QWEN_NODE_CACHE_PVC + unset DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR + unset DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS + source \"$1\" + printf '%s\\n' \"$DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED\" \\ + \"$DROBOTICFLOW_QWEN_NODE_CACHE_PVC\" \\ + \"$DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH\" \\ + \"$DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR\" \\ + \"$DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS\" + """ + result = subprocess.run(["bash", "-c", command, "--", str(POC_ENV)], cwd=ROOT, text=True, capture_output=True, check=False) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual( + result.stdout.splitlines(), + ["true", "droboticflow-qwen-node-cache", "/qwen-node-cache", "kubernetes.io/hostname=192.168.80.136", "86400"], + ) + if __name__ == "__main__": unittest.main()