🛡️ Harden Qwen node cache rollout
This commit is contained in:
parent
d960fab4d0
commit
f81e22cc70
@ -107,6 +107,9 @@ For the `poc` environment, use [`scripts/deploy-envs/poc.env`](../scripts/deploy
|
||||
|
||||
- `MONGO_DB=poc`
|
||||
- `API_PUBLIC_LOAD_BALANCER=true`
|
||||
- 已验收 Qwen 节点本地缓存:`DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED=true`、
|
||||
`droboticflow-qwen-node-cache` PVC、`/qwen-node-cache` 和 GPU node hostname selector;
|
||||
这只影响 allowlisted Qwen 粗切 Job,通用模板仍默认关闭。
|
||||
- `BOS_ENDPOINT=https://bj.bcebos.com`
|
||||
- the same namespace, AI defaults, and PVC settings as `wjzh`
|
||||
|
||||
|
||||
@ -28,7 +28,7 @@ python3 scripts/verify_qwen3vl_model_manifest.py --model-root /workspace/models/
|
||||
|
||||
独立预载 Job 必须先逐字节验证全部 4 个 shard/16 个 payload 文件,再原子发布模型目录。生产 workflow 将该 Job 产出的 manifest SHA-256 固定为 `MODEL_MANIFEST_SHA256`;每个短生命周期 Qwen Pod 重新验证 sidecar、固定 SHA、config/processor/index、四 shard 证据,以及完整本地路径和大小布局,但不重复从共享 PVC 读取 17.5 GB 做同一遍哈希。未提供固定 SHA 时,`TRUST_PREVERIFIED_MODEL=true` 会失败关闭;非预载模式仍执行全文件哈希。由于 POC CSI 使用 direct I/O,默认行为是以四 shard 并行大块读把固定快照复制到该 Pod 的本地 `/tmp`,再从本地路径加载。
|
||||
|
||||
可选的节点本地缓存默认关闭,且只允许 Qwen3-VL 手部行为语义粗切使用。启用后,缓存目录以固定 manifest SHA 为 key,源 manifest/provenance 每次重验,miss 按文件 SHA/size 复制并原子写 receipt,hit 会完整本地哈希复验;空间不足或损坏会回退 Pod-local staging 而不会使用不完整缓存。详情与 PV 安全边界见 [Qwen 节点本地模型缓存](qwen-node-local-cache.md)。Pod provenance 同时记录 PVC manifest 路径、本地 runtime 路径、复制字节数和 `nodeLocalCache` 状态。Qwen 节点仅从本地快照加载(`local_files_only=True`),使用固定的 `transformers==4.57.6` 和 `Qwen3VLForConditionalGeneration`,不会执行远程模型代码。
|
||||
可选的节点本地缓存通用默认关闭,且只允许 Qwen3-VL 手部行为语义粗切使用;已验收的 `poc` profile 通过 Local PV 显式启用。缓存目录以固定 manifest SHA 为 key,源 manifest/provenance 每次重验,miss 按文件 SHA/size 复制并原子写 receipt,hit 会完整本地哈希复验;空间不足或损坏会回退 Pod-local staging 而不会使用不完整缓存。详情、PV 安全边界和 disabled/miss/hit 实测证据见 [Qwen 节点本地模型缓存](qwen-node-local-cache.md)。Pod provenance 同时记录 PVC manifest 路径、本地 runtime 路径、复制字节数和 `nodeLocalCache` 状态。Qwen 节点仅从本地快照加载(`local_files_only=True`),使用固定的 `transformers==4.57.6` 和 `Qwen3VLForConditionalGeneration`,不会执行远程模型代码。
|
||||
|
||||
## 人在环与交付
|
||||
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
# Qwen 节点本地模型缓存(默认关闭)
|
||||
# Qwen 节点本地模型缓存(通用默认关闭;POC 已验收启用)
|
||||
|
||||
这个优化只服务于 `operator-builtin-qwen-behavior-rough-cut`。它不会改变
|
||||
Qwen3-VL 的固定 revision、模型 manifest、BOS/PVC 来源、工作流图或任何已经验收的
|
||||
@ -17,9 +17,10 @@ DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR=kubernetes.io/hostname=<gpu-node>
|
||||
DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS=86400
|
||||
```
|
||||
|
||||
默认值是 `false`、空 PVC、`/qwen-node-cache`、空 selector 和 `86400` 秒。启用时 PVC
|
||||
和 selector 缺失会 fail-closed;其它 builtin、普通 operator、Qwen 时间定位、训练和
|
||||
姿态算子都不会得到该挂载或这些环境变量。工作流 hyperparameter 也不能覆盖这些
|
||||
通用渲染/部署模板仍默认 `false`、空 PVC、`/qwen-node-cache`、空 selector 和 `86400`
|
||||
秒;只有 `scripts/deploy-envs/poc.env` 为已验收的 POC Local PV 预置了上面的五项值。
|
||||
启用时 PVC 和 selector 缺失会 fail-closed;其它 builtin、普通 operator、Qwen 时间定位、
|
||||
训练和姿态算子都不会得到该挂载或这些环境变量。工作流 hyperparameter 也不能覆盖这些
|
||||
平台保留键。
|
||||
|
||||
本 POC 已只读确认 GPU node `192.168.80.136` 上 `/home/cce` 是独立的
|
||||
@ -51,13 +52,38 @@ manifest SHA、Qwen repository/revision、config/processor/index 和完整布局
|
||||
## Local PV 准备顺序
|
||||
|
||||
1. 保持 feature disabled 部署并验证 Worker 健康。
|
||||
2. 在 GPU node 上只创建 `/home/cce/droboticflow-qwen-cache`,权限 `10001:10001`、`0750`;
|
||||
不执行递归 chmod,也不格式化或挂载任何盘。
|
||||
使用 `scripts/render-qwen-node-cache-local-pv.sh --prepare-job` 只渲染受限的一次性
|
||||
准备 Job;该脚本不会执行 apply。默认 PV/PVC 同样通过
|
||||
2. 在 GPU node 上只创建 `/home/cce/droboticflow-qwen-cache`,初始权限为
|
||||
`10001:10001`、`0750`;不执行递归 chmod,也不格式化或挂载任何盘。Local PVC 被
|
||||
运行时 Pod 以 `fsGroup=10001` 挂载后,Linux 可能将根目录表现为 `2770`(setgid,
|
||||
同一受限组可写);这不是 world-writable,也不应被递归改回。
|
||||
在渲染 `--preflight-job` 或 `--prepare-job` 前,必须显式提供 digest-pinned
|
||||
`QWEN_NODE_CACHE_PREPARE_IMAGE`,并运行
|
||||
`scripts/verify-qwen-node-cache-control-image.sh`:它分别验证 registry manifest、API
|
||||
Deployment spec 和所有 API Pod imageID,三者任一不一致即停止。`--preflight-job`
|
||||
只读挂载父目录和候选目录,以 UID/GID `10001` 检查 realpath、权限和内容;
|
||||
`--prepare-job` 只挂载精确候选目录,使用最小 `CHOWN`、`DAC_OVERRIDE`、`FOWNER`
|
||||
capability 顺序执行 chown/chmod/UID 10001 可写探针,无 service-account token、GPU、
|
||||
privileged 或 host namespace。两者均只渲染,不会 apply。默认 PV/PVC 同样通过
|
||||
`scripts/render-qwen-node-cache-local-pv.sh` 仅渲染,固定为 80Gi、Retain 与 hostname
|
||||
affinity,模板不属于 `deploy-k8s.sh`。
|
||||
3. 创建一个带 hostname nodeAffinity 的 `ReadWriteOnce` Local PV/PVC,再在 Worker 配置中
|
||||
填入相同 PVC、mount path 与 hostname selector。
|
||||
4. 先用真实固定 manifest 预热,检查 receipt、17.5GB 级别文件和 SHA 证据;再启用开关。
|
||||
5. 启动两个独立 Qwen 粗切 Pod,记录首次 miss、第二次 hit、输出 hash 和 GPU 证据。
|
||||
|
||||
## 已验收的 POC 证据
|
||||
|
||||
- operator image 为
|
||||
`ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-qwen-behavior-rough-cut@sha256:f1c4889b664d8bc24bf61ec9abf230a0c95c479fe54716514e370620450c25c7`。
|
||||
- disabled compatibility Job `fanyu-qwen-cache-disabled-compat-r2-20260822`、cache miss Job
|
||||
`fanyu-qwen-cache-miss-r2-20260822` 与独立 Pod cache hit Job
|
||||
`fanyu-qwen-cache-hit-r2-20260822` 都成功。三次经过删除运行/缓存特有 provenance 后的
|
||||
canonical 输出 SHA-256 都是
|
||||
`a7656873eef8fe4d59ff46e07ff5dce66f4379fd684ec2800d04f4c1e156a775`。
|
||||
- 固定模型 manifest SHA-256 是
|
||||
`3c00bb1b1ada0542de91785001395fc6d4252e3d8a0844ec5fdfe05b306a481a`;sealed receipt
|
||||
SHA-256 是 `dde9e72fd1d72b5556f16a121967dfc62ed57217a8b6be030923af4bcda870bb`。
|
||||
miss 复制 `17,545,915,883` bytes,hit `copiedBytes=0` 且 `modelStagedBytes=0`。
|
||||
- Worker scope fixture contract 的 SHA-256 是
|
||||
`0e799649aeb9bd41018f8ad831bbc7eee7380d363c2be3aab9a0e04d94411c09`;它证明只会把
|
||||
PVC/env/selector 注入 exact Qwen 粗切 builtin,其他 operator 保持为空。
|
||||
|
||||
67
k8s/qwen-node-cache.preflight-dir.template.yaml
Normal file
67
k8s/qwen-node-cache.preflight-dir.template.yaml
Normal file
@ -0,0 +1,67 @@
|
||||
# Render-only, read-only host directory preflight. It never creates a path and
|
||||
# is separate from the one-shot mutating prepare Job.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: __QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME__
|
||||
namespace: __QWEN_NODE_CACHE_NAMESPACE__
|
||||
labels:
|
||||
app: droboticflow
|
||||
component: qwen-node-cache-preflight
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
activeDeadlineSeconds: 120
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
spec:
|
||||
nodeName: __QWEN_NODE_CACHE_NODE_NAME__
|
||||
automountServiceAccountToken: false
|
||||
hostNetwork: false
|
||||
hostPID: false
|
||||
hostIPC: false
|
||||
restartPolicy: Never
|
||||
securityContext:
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: preflight
|
||||
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
|
||||
command: ["/bin/sh", "-lc"]
|
||||
args:
|
||||
- |
|
||||
set -eu
|
||||
stat -Lc 'parent-mode=%a owner=%u:%g inode=%i path=%n' /host-cache-parent
|
||||
stat -Lc 'candidate-mode=%a owner=%u:%g inode=%i path=%n' /cache-root
|
||||
realpath /cache-root
|
||||
find /cache-root -mindepth 1 -maxdepth 1 -printf '%f\n' | wc -l
|
||||
test ! -w /cache-root
|
||||
securityContext:
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
volumeMounts:
|
||||
- name: host-cache-parent
|
||||
mountPath: /host-cache-parent
|
||||
readOnly: true
|
||||
- name: cache-root
|
||||
mountPath: /cache-root
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: host-cache-parent
|
||||
hostPath:
|
||||
path: /home/cce
|
||||
type: Directory
|
||||
- name: cache-root
|
||||
hostPath:
|
||||
path: __QWEN_NODE_CACHE_HOST_PATH__
|
||||
type: Directory
|
||||
@ -1,6 +1,6 @@
|
||||
# Render-only, one-shot directory preparation Job. It is never applied by the
|
||||
# platform deployment. Its only mutation is creating/chowning the exact cache
|
||||
# directory under the independently verified /home/cce NVMe mount.
|
||||
# platform deployment. Its only mutation is chown/chmod of the exact already
|
||||
# preflighted cache directory; it never mounts the /home/cce parent directory.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
@ -11,31 +11,130 @@ metadata:
|
||||
component: qwen-node-cache-prepare
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
activeDeadlineSeconds: 120
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
spec:
|
||||
nodeName: __QWEN_NODE_CACHE_NODE_NAME__
|
||||
automountServiceAccountToken: false
|
||||
hostNetwork: false
|
||||
hostPID: false
|
||||
hostIPC: false
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: prepare
|
||||
securityContext:
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
initContainers:
|
||||
- name: chown-cache-root
|
||||
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
|
||||
command: ["/bin/sh", "-lc"]
|
||||
args:
|
||||
- |
|
||||
set -eu
|
||||
install -d -m 0750 -o 10001 -g 10001 /host-cache/__QWEN_NODE_CACHE_DIR_NAME__
|
||||
stat -c 'mode=%a owner=%u:%g path=%n' /host-cache/__QWEN_NODE_CACHE_DIR_NAME__
|
||||
command: ["/usr/bin/chown", "10001:10001", "/cache-root"]
|
||||
securityContext:
|
||||
runAsUser: 0
|
||||
runAsGroup: 0
|
||||
privileged: false
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
add: ["CHOWN", "DAC_OVERRIDE", "FOWNER"]
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
volumeMounts:
|
||||
- name: host-cache-parent
|
||||
mountPath: /host-cache
|
||||
- name: cache-root
|
||||
mountPath: /cache-root
|
||||
- name: chmod-cache-root
|
||||
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
|
||||
command: ["/usr/bin/chmod", "0750", "/cache-root"]
|
||||
securityContext:
|
||||
runAsUser: 0
|
||||
runAsGroup: 0
|
||||
privileged: false
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
add: ["CHOWN", "DAC_OVERRIDE", "FOWNER"]
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
volumeMounts:
|
||||
- name: cache-root
|
||||
mountPath: /cache-root
|
||||
- name: rw-touch-as-10001
|
||||
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
|
||||
command: ["/usr/bin/touch", "/cache-root/.qwen-cache-rw-probe"]
|
||||
securityContext:
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
privileged: false
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
volumeMounts:
|
||||
- name: cache-root
|
||||
mountPath: /cache-root
|
||||
- name: rw-remove-as-10001
|
||||
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
|
||||
command: ["/usr/bin/rm", "/cache-root/.qwen-cache-rw-probe"]
|
||||
securityContext:
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
privileged: false
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
volumeMounts:
|
||||
- name: cache-root
|
||||
mountPath: /cache-root
|
||||
containers:
|
||||
- name: verify-as-10001
|
||||
image: __QWEN_NODE_CACHE_PREPARE_IMAGE__
|
||||
command: ["/usr/bin/stat", "-c", "mode=%a owner=%u:%g path=%n", "/cache-root"]
|
||||
securityContext:
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
privileged: false
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
volumeMounts:
|
||||
- name: cache-root
|
||||
mountPath: /cache-root
|
||||
volumes:
|
||||
- name: host-cache-parent
|
||||
- name: cache-root
|
||||
hostPath:
|
||||
path: /home/cce
|
||||
type: Directory
|
||||
path: __QWEN_NODE_CACHE_HOST_PATH__
|
||||
type: DirectoryOrCreate
|
||||
|
||||
@ -1,6 +1,41 @@
|
||||
{
|
||||
"acceptance": {
|
||||
"artifactId": "fanyu-qwen-cache-disabled-compat-20260822-fanyu-v5-rough-cut-clips-manifest-clips-manifest.json",
|
||||
"nodeLocalCache": {
|
||||
"canonicalOutputSha256": "a7656873eef8fe4d59ff46e07ff5dce66f4379fd684ec2800d04f4c1e156a775",
|
||||
"canonicalization": "delete run/cache-specific provenance: inferenceId, executedAt, modelRuntimePath, modelStagedBytes, nodeLocalCache",
|
||||
"fixtureContractSha256": "0e799649aeb9bd41018f8ad831bbc7eee7380d363c2be3aab9a0e04d94411c09",
|
||||
"modelManifestSha256": "3c00bb1b1ada0542de91785001395fc6d4252e3d8a0844ec5fdfe05b306a481a",
|
||||
"operatorImageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-qwen-behavior-rough-cut@sha256:f1c4889b664d8bc24bf61ec9abf230a0c95c479fe54716514e370620450c25c7",
|
||||
"receiptSha256": "dde9e72fd1d72b5556f16a121967dfc62ed57217a8b6be030923af4bcda870bb",
|
||||
"runs": {
|
||||
"disabledCompatibility": {
|
||||
"cacheHit": false,
|
||||
"copiedBytes": 17545915883,
|
||||
"jobId": "fanyu-qwen-cache-disabled-compat-r2-20260822",
|
||||
"modelStagedBytes": 17545915883,
|
||||
"runId": "fanyu-qwen-cache-disabled-compat-r2-20260822",
|
||||
"status": "disabled"
|
||||
},
|
||||
"hit": {
|
||||
"cacheHit": true,
|
||||
"copiedBytes": 0,
|
||||
"jobId": "fanyu-qwen-cache-hit-r2-20260822",
|
||||
"modelStagedBytes": 0,
|
||||
"runId": "fanyu-qwen-cache-hit-r2-20260822",
|
||||
"status": "hit"
|
||||
},
|
||||
"miss": {
|
||||
"cacheHit": false,
|
||||
"copiedBytes": 17545915883,
|
||||
"jobId": "fanyu-qwen-cache-miss-r2-20260822",
|
||||
"modelStagedBytes": 17545915883,
|
||||
"runId": "fanyu-qwen-cache-miss-r2-20260822",
|
||||
"status": "miss"
|
||||
}
|
||||
},
|
||||
"scopeFixture": "go test ./internal/worker -run ^TestQwenNodeCache(AppliesOnlyToExactBuiltin|DisabledAndInvalidConfigurationFailSafe|RejectsPVCMountCollision)$ -count=1"
|
||||
},
|
||||
"projectId": "project-271c97276a31d97dd3eff93e3f1ff3b0",
|
||||
"runId": "fanyu-qwen-cache-disabled-compat-r2-20260822",
|
||||
"verifiedAt": "2026-08-21T23:26:03Z",
|
||||
|
||||
@ -42,10 +42,10 @@ export WORKER_REPLICAS="${WORKER_REPLICAS:-2}"
|
||||
export DROSYNC_COORDINATOR_REPLICAS="${DROSYNC_COORDINATOR_REPLICAS:-2}"
|
||||
export DROBOTICFLOW_OBJECT_SYNC_ENABLED="${DROBOTICFLOW_OBJECT_SYNC_ENABLED:-false}"
|
||||
export OPERATOR_JOB_TTL_SECONDS="${OPERATOR_JOB_TTL_SECONDS:-86400}"
|
||||
export DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED="${DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED:-false}"
|
||||
export DROBOTICFLOW_QWEN_NODE_CACHE_PVC="${DROBOTICFLOW_QWEN_NODE_CACHE_PVC:-}"
|
||||
export DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED="${DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED:-true}"
|
||||
export DROBOTICFLOW_QWEN_NODE_CACHE_PVC="${DROBOTICFLOW_QWEN_NODE_CACHE_PVC:-droboticflow-qwen-node-cache}"
|
||||
export DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH="${DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH:-/qwen-node-cache}"
|
||||
export DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR="${DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR:-}"
|
||||
export DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR="${DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR:-kubernetes.io/hostname=192.168.80.136}"
|
||||
export DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS="${DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS:-86400}"
|
||||
export CUSTOM_OPERATOR_IMAGE_REGISTRY="${CUSTOM_OPERATOR_IMAGE_REGISTRY:-ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/custom-operators}"
|
||||
export CUSTOM_OPERATOR_IMAGE_IMPORTER_IMAGE="${CUSTOM_OPERATOR_IMAGE_IMPORTER_IMAGE:-quay.io/skopeo/stable:v1.16.1}"
|
||||
|
||||
@ -8,9 +8,12 @@ MODE="pv"
|
||||
if [[ "${1:-}" == "--prepare-job" ]]; then
|
||||
MODE="prepare"
|
||||
shift
|
||||
elif [[ "${1:-}" == "--preflight-job" ]]; then
|
||||
MODE="preflight"
|
||||
shift
|
||||
fi
|
||||
if [[ $# -gt 0 ]]; then
|
||||
echo "usage: $0 [--prepare-job]" >&2
|
||||
echo "usage: $0 [--preflight-job|--prepare-job]" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
@ -21,9 +24,10 @@ QWEN_NODE_CACHE_SIZE="${QWEN_NODE_CACHE_SIZE:-80Gi}"
|
||||
QWEN_NODE_CACHE_PV_NAME="${QWEN_NODE_CACHE_PV_NAME:-droboticflow-qwen-node-cache-pv}"
|
||||
QWEN_NODE_CACHE_PVC_NAME="${QWEN_NODE_CACHE_PVC_NAME:-droboticflow-qwen-node-cache}"
|
||||
QWEN_NODE_CACHE_PREPARE_JOB_NAME="${QWEN_NODE_CACHE_PREPARE_JOB_NAME:-droboticflow-qwen-node-cache-prepare}"
|
||||
QWEN_NODE_CACHE_PREPARE_IMAGE="${QWEN_NODE_CACHE_PREPARE_IMAGE:-ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-api@sha256:23fa92a39016d7c31f88b63974453d67a6da9bcc02783d4b24bff68566f86828}"
|
||||
QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME="${QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME:-droboticflow-qwen-node-cache-preflight}"
|
||||
QWEN_NODE_CACHE_PREPARE_IMAGE="${QWEN_NODE_CACHE_PREPARE_IMAGE:-}"
|
||||
export QWEN_NODE_CACHE_NAMESPACE QWEN_NODE_CACHE_NODE_NAME QWEN_NODE_CACHE_HOST_PATH QWEN_NODE_CACHE_SIZE \
|
||||
QWEN_NODE_CACHE_PV_NAME QWEN_NODE_CACHE_PVC_NAME QWEN_NODE_CACHE_PREPARE_JOB_NAME QWEN_NODE_CACHE_PREPARE_IMAGE
|
||||
QWEN_NODE_CACHE_PV_NAME QWEN_NODE_CACHE_PVC_NAME QWEN_NODE_CACHE_PREPARE_JOB_NAME QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME QWEN_NODE_CACHE_PREPARE_IMAGE
|
||||
|
||||
python3 - "$ROOT_DIR" "$MODE" <<'PY'
|
||||
import os
|
||||
@ -36,10 +40,10 @@ mode = sys.argv[2]
|
||||
values = {key: os.environ[key] for key in (
|
||||
"QWEN_NODE_CACHE_NAMESPACE", "QWEN_NODE_CACHE_NODE_NAME", "QWEN_NODE_CACHE_HOST_PATH",
|
||||
"QWEN_NODE_CACHE_SIZE", "QWEN_NODE_CACHE_PV_NAME", "QWEN_NODE_CACHE_PVC_NAME",
|
||||
"QWEN_NODE_CACHE_PREPARE_JOB_NAME", "QWEN_NODE_CACHE_PREPARE_IMAGE",
|
||||
"QWEN_NODE_CACHE_PREPARE_JOB_NAME", "QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME", "QWEN_NODE_CACHE_PREPARE_IMAGE",
|
||||
)}
|
||||
label = re.compile(r"^[a-z0-9]([-a-z0-9]*[a-z0-9])?$")
|
||||
for key in ("QWEN_NODE_CACHE_NAMESPACE", "QWEN_NODE_CACHE_PV_NAME", "QWEN_NODE_CACHE_PVC_NAME", "QWEN_NODE_CACHE_PREPARE_JOB_NAME"):
|
||||
for key in ("QWEN_NODE_CACHE_NAMESPACE", "QWEN_NODE_CACHE_PV_NAME", "QWEN_NODE_CACHE_PVC_NAME", "QWEN_NODE_CACHE_PREPARE_JOB_NAME", "QWEN_NODE_CACHE_PREFLIGHT_JOB_NAME"):
|
||||
if len(values[key]) > 63 or not label.fullmatch(values[key]):
|
||||
raise SystemExit(f"{key} must be a lowercase DNS label")
|
||||
if not re.fullmatch(r"[a-z0-9]([a-z0-9.-]*[a-z0-9])?", values["QWEN_NODE_CACHE_NODE_NAME"]):
|
||||
@ -49,9 +53,14 @@ if str(path) != values["QWEN_NODE_CACHE_HOST_PATH"] or not str(path).startswith(
|
||||
raise SystemExit("QWEN_NODE_CACHE_HOST_PATH must be a clean unique directory under /home/cce")
|
||||
if not re.fullmatch(r"[1-9][0-9]*(Mi|Gi|Ti)", values["QWEN_NODE_CACHE_SIZE"]):
|
||||
raise SystemExit("QWEN_NODE_CACHE_SIZE must be a positive Mi/Gi/Ti quantity")
|
||||
if "@sha256:" not in values["QWEN_NODE_CACHE_PREPARE_IMAGE"]:
|
||||
raise SystemExit("QWEN_NODE_CACHE_PREPARE_IMAGE must be digest-pinned")
|
||||
template = root / "k8s" / ("qwen-node-cache.prepare-dir.template.yaml" if mode == "prepare" else "qwen-node-cache.local-pv.template.yaml")
|
||||
if mode != "pv" and not re.fullmatch(r".+@sha256:[a-f0-9]{64}", values["QWEN_NODE_CACHE_PREPARE_IMAGE"]):
|
||||
raise SystemExit("QWEN_NODE_CACHE_PREPARE_IMAGE must be an exact digest-pinned image for prepare/preflight rendering")
|
||||
template_name = {
|
||||
"pv": "qwen-node-cache.local-pv.template.yaml",
|
||||
"prepare": "qwen-node-cache.prepare-dir.template.yaml",
|
||||
"preflight": "qwen-node-cache.preflight-dir.template.yaml",
|
||||
}[mode]
|
||||
template = root / "k8s" / template_name
|
||||
rendered = template.read_text(encoding="utf-8")
|
||||
replacements = {"__" + key + "__": value for key, value in values.items()}
|
||||
replacements["__QWEN_NODE_CACHE_DIR_NAME__"] = path.name
|
||||
|
||||
36
scripts/verify-qwen-node-cache-control-image.sh
Executable file
36
scripts/verify-qwen-node-cache-control-image.sh
Executable file
@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Verify the control image before rendering either node-cache control Job. This
|
||||
# is read-only and intentionally checks the exact registry digest, the API
|
||||
# Deployment spec, and the running API pod imageID independently.
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
K8S_NAMESPACE="${K8S_NAMESPACE:-droboticflow}"
|
||||
K8S_IMAGE_PULL_SECRET_NAME="${K8S_IMAGE_PULL_SECRET_NAME:-droboticflow-registry}"
|
||||
QWEN_NODE_CACHE_PREPARE_IMAGE="${QWEN_NODE_CACHE_PREPARE_IMAGE:-}"
|
||||
|
||||
EXPECTED_IMAGE="$QWEN_NODE_CACHE_PREPARE_IMAGE" python3 - <<'PY'
|
||||
import os
|
||||
import re
|
||||
|
||||
value = os.environ["EXPECTED_IMAGE"].strip()
|
||||
if not re.fullmatch(r".+@sha256:[a-f0-9]{64}", value):
|
||||
raise SystemExit("QWEN_NODE_CACHE_PREPARE_IMAGE must be an exact digest-pinned image")
|
||||
PY
|
||||
|
||||
kubectl -n "$K8S_NAMESPACE" get secret "$K8S_IMAGE_PULL_SECRET_NAME" -o jsonpath='{.data.\.dockerconfigjson}' \
|
||||
| base64 --decode \
|
||||
| python3 "$ROOT_DIR/scripts/verify_registry_manifest.py" --image "$QWEN_NODE_CACHE_PREPARE_IMAGE" >&2
|
||||
|
||||
kubectl -n "$K8S_NAMESPACE" get deployment droboticflow-api -o json \
|
||||
| EXPECTED_IMAGE="$QWEN_NODE_CACHE_PREPARE_IMAGE" python3 -c 'import json,os,sys; p=json.load(sys.stdin); image=next(c.get("image") for c in p["spec"]["template"]["spec"]["containers"] if c.get("name")=="api"); expected=os.environ["EXPECTED_IMAGE"]; assert image==expected, f"API deployment image mismatch: {image}"; print({"apiDeploymentImage":image})'
|
||||
|
||||
kubectl -n "$K8S_NAMESPACE" get pods -l app=droboticflow-api -o json \
|
||||
| EXPECTED_IMAGE="$QWEN_NODE_CACHE_PREPARE_IMAGE" python3 -c 'import json,os,sys; p=json.load(sys.stdin); expected=os.environ["EXPECTED_IMAGE"]; rows=[]
|
||||
for pod in p.get("items",[]):
|
||||
status=next((c for c in pod.get("status",{}).get("containerStatuses",[]) if c.get("name")=="api"),{})
|
||||
image_id=status.get("imageID","")
|
||||
if image_id != expected: raise SystemExit("API pod imageID mismatch: {}={}".format(pod["metadata"]["name"], image_id))
|
||||
rows.append({"pod":pod["metadata"]["name"],"imageID":image_id,"ready":status.get("ready")})
|
||||
if not rows: raise SystemExit("no API pods found for control-image verification")
|
||||
print({"apiPods":rows})'
|
||||
@ -8,6 +8,9 @@ import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
SCRIPT = ROOT / "scripts" / "render-qwen-node-cache-local-pv.sh"
|
||||
CONTROL_VERIFY_SCRIPT = ROOT / "scripts" / "verify-qwen-node-cache-control-image.sh"
|
||||
POC_ENV = ROOT / "scripts" / "deploy-envs" / "poc.env"
|
||||
CONTROL_IMAGE = "registry.example/control@sha256:" + "a" * 64
|
||||
|
||||
|
||||
class TestQwenNodeLocalPVTemplate(unittest.TestCase):
|
||||
@ -28,12 +31,31 @@ class TestQwenNodeLocalPVTemplate(unittest.TestCase):
|
||||
self.assertNotIn("kubectl apply", result.stdout)
|
||||
|
||||
def test_prepare_job_is_strictly_scoped_to_single_cache_directory(self) -> None:
|
||||
result = self.render("--prepare-job")
|
||||
result = self.render("--prepare-job", QWEN_NODE_CACHE_PREPARE_IMAGE=CONTROL_IMAGE)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn("path: /home/cce", result.stdout)
|
||||
self.assertIn("install -d -m 0750 -o 10001 -g 10001 /host-cache/droboticflow-qwen-cache", result.stdout)
|
||||
self.assertIn("path: /home/cce/droboticflow-qwen-cache", result.stdout)
|
||||
self.assertIn("type: DirectoryOrCreate", result.stdout)
|
||||
self.assertIn('command: ["/usr/bin/chown", "10001:10001", "/cache-root"]', result.stdout)
|
||||
self.assertIn('command: ["/usr/bin/chmod", "0750", "/cache-root"]', result.stdout)
|
||||
self.assertIn('add: ["CHOWN", "DAC_OVERRIDE", "FOWNER"]', result.stdout)
|
||||
self.assertIn("automountServiceAccountToken: false", result.stdout)
|
||||
self.assertNotIn("chmod -R", result.stdout)
|
||||
self.assertNotIn("path: /\n", result.stdout)
|
||||
self.assertNotIn("privileged: true", result.stdout)
|
||||
self.assertNotIn("host-cache-parent", result.stdout)
|
||||
self.assertIn("activeDeadlineSeconds: 120", result.stdout)
|
||||
self.assertNotIn("nvidia.com/gpu", result.stdout)
|
||||
|
||||
def test_preflight_job_is_read_only_and_requires_exact_control_image(self) -> None:
|
||||
missing = self.render("--preflight-job")
|
||||
self.assertNotEqual(missing.returncode, 0)
|
||||
self.assertIn("must be an exact digest-pinned image", missing.stderr)
|
||||
result = self.render("--preflight-job", QWEN_NODE_CACHE_PREPARE_IMAGE=CONTROL_IMAGE)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn("readOnly: true", result.stdout)
|
||||
self.assertIn("realpath /cache-root", result.stdout)
|
||||
self.assertIn("host-cache-parent", result.stdout)
|
||||
self.assertIn("runAsUser: 10001", result.stdout)
|
||||
self.assertNotIn("install -d", result.stdout)
|
||||
|
||||
def test_rejects_root_and_outside_nvme_paths(self) -> None:
|
||||
for path in ("/", "/var/lib/droboticflow-qwen-cache", "/home/cce", "/home/cce/../root"):
|
||||
@ -42,6 +64,33 @@ class TestQwenNodeLocalPVTemplate(unittest.TestCase):
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("must be a clean unique directory under /home/cce", result.stderr)
|
||||
|
||||
def test_control_image_verifier_requires_three_read_only_proofs(self) -> None:
|
||||
script = CONTROL_VERIFY_SCRIPT.read_text(encoding="utf-8")
|
||||
self.assertIn("verify_registry_manifest.py", script)
|
||||
self.assertIn("get deployment droboticflow-api", script)
|
||||
self.assertIn("get pods -l app=droboticflow-api", script)
|
||||
self.assertIn("must be an exact digest-pinned image", script)
|
||||
self.assertNotIn("kubectl apply", script)
|
||||
|
||||
def test_poc_profile_enables_only_the_verified_cache_configuration(self) -> None:
|
||||
command = """
|
||||
unset DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED DROBOTICFLOW_QWEN_NODE_CACHE_PVC
|
||||
unset DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR
|
||||
unset DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS
|
||||
source \"$1\"
|
||||
printf '%s\\n' \"$DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED\" \\
|
||||
\"$DROBOTICFLOW_QWEN_NODE_CACHE_PVC\" \\
|
||||
\"$DROBOTICFLOW_QWEN_NODE_CACHE_MOUNT_PATH\" \\
|
||||
\"$DROBOTICFLOW_QWEN_NODE_CACHE_NODE_SELECTOR\" \\
|
||||
\"$DROBOTICFLOW_QWEN_NODE_CACHE_STALE_STAGING_SECONDS\"
|
||||
"""
|
||||
result = subprocess.run(["bash", "-c", command, "--", str(POC_ENV)], cwd=ROOT, text=True, capture_output=True, check=False)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(
|
||||
result.stdout.splitlines(),
|
||||
["true", "droboticflow-qwen-node-cache", "/qwen-node-cache", "kubernetes.io/hostname=192.168.80.136", "86400"],
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user