diff --git a/.scratch/mongo-cpu/work.md b/.scratch/mongo-cpu/work.md index c362b858..aad65696 100644 --- a/.scratch/mongo-cpu/work.md +++ b/.scratch/mongo-cpu/work.md @@ -26,3 +26,5 @@ The earlier new-cloud-disk selection is paused by the latest user instruction: i User selected node .4 and authorized a synthetic 20–30Gi pilot only after isolation is confirmed. Read-only follow-up confirms kernel 5.15/cgroup v2 and io delegation under kubepods.slice, but no configured hard capacity or validated per-pilot IO enforcement. Drafted docs/operations/query-replica-node4-isolation-plan.md with a 25Gi fixed-filesystem candidate and scoped cgroup limits, explicit loop/writeback proof and restart requirements. Local-only tests passed: 256Mi EXT4/tmpfs ENOSPC with unchanged backing-file size; 16Mi direct and buffered+fdatasync writes through loop at a configured 1MiB/s took 16.06/16.07 seconds; remount retained file size. Local kernel is 6.8, production 5.15, runtime management differs. No cluster writes or database started. Startup/recreation enforcement, IOPS/read caps, peak baseline, guarded node smoke and Mongo recovery remain pending; do not treat local proof as production isolation acceptance. Subsequent user authorization explicitly covers node .4 small synthetic tests. Created only the dedicated query-lab namespace/quota/deny-all policy/ConfigMap and512Mi test filesystem. No production data/application deployment changed. The original host path match was fixed for Pod mount-namespace changes. DirectIO did not satisfy buffered write throttling; stopped and retained failures. Global PSI also included throttled test waiting; protection additionally measures existing Pod PSI and physical latency rather than treating global PSI as business impact. Dual loop+physical-disk controls also FAILED buffered write enforcement (run-6):4Mi direct4.011s, buffered+fdatasync0.008s, buffered bytes not charged to target Pod. All test Pods stopped;512Mi image/mount retained.25Gi uncreated. Mongo cross-Pod data and full-disk recovery deliberately not executed after IO failure. Existing readiness/Pod-pressure sampling showed no deterioration within this short window. See docs/operations/query-replica-node4-smoke-results.md. All source/query feature switches remain OFF; do not deploy this loop approach or describe the failed suite as capacity acceptance. + +Latest follow-up supersedes the failed-configuration conclusion above. Synced live dev5c6023ec into this branch (merge5e4de799), preserving both verification mappings. A1Mi random-data differential proved old inode writeback escaped the new Pod, while new loop inodes and native EXT4 controls were limited. Normal unmount/rebind of only the exact-owned idle filesystem restored all four cases (~1s/Mi); no reformat/forced unmount/global cache changes. Full512Mi suite passed BPS/IOPS before/after Pod replacement, preserved journal-acknowledged data, and recovered from filesystem ENOSPC without deleting Mongo files. Then, under the user's conditional authorization, created a distinct25Gi synthetic single-member replica set on .4,1CPU/1Gi, loopback only/no Service, dual IO caps and75Gi free-space guard. A majority+journaled marker survived another Pod replacement. Running Pod node4-query25 (UID19f5f918-9bb1-4e6a-a362-bb70afacee26) has a6h deadline ending2026-11-13T09:47:06Z; file remains after stop. Production Pod identities/restarts and protected IO pressure unchanged in sampled window. Source/read feature flags still OFF; no original Mongo projection/sync or capacity acceptance claimed. Evidence: node4-smoke/attribution-\*, recovery-remount/, query25/. No purchase or customer-data writes. diff --git a/docs/operations/query-replica-node4-isolation-plan.md b/docs/operations/query-replica-node4-isolation-plan.md index 05dc7653..502bc30b 100644 --- a/docs/operations/query-replica-node4-isolation-plan.md +++ b/docs/operations/query-replica-node4-isolation-plan.md @@ -1,6 +1,6 @@ # .4 节点查询库:容量与IO隔离准入方案 -状态:用户后续明确授权在 `.4` 实施小规模验证;**512Mi实机测试的缓冲写限速未通过,现已停测,25Gi库未创建**。最新结果和失败记录见 [实机验证记录](query-replica-node4-smoke-results.md)。以下本机检查与拟议参数保留其时间点含义,不代表实机全部通过;不买新盘、不读写生产数据。 +状态:后续诊断已定位跨Pod保留挂载导致的旧inode写回归属问题,并通过专属测试文件系统的正常卸载/重新挂载修复。**512Mi限速、Pod重建和Mongo满盘恢复回归已通过,25Gi合成验证库已创建且重建后确认数据保留。** 最新结果和原失败记录见 [实机验证记录](query-replica-node4-smoke-results.md)。以下早期核查与拟议参数保留其时间点含义;本轮不买新盘、不读写生产数据。 ## 本轮核实的事实 diff --git a/docs/operations/query-replica-node4-smoke-results.md b/docs/operations/query-replica-node4-smoke-results.md index c3dd7b50..17c3eb83 100644 --- a/docs/operations/query-replica-node4-smoke-results.md +++ b/docs/operations/query-replica-node4-smoke-results.md @@ -1,6 +1,6 @@ # .4 节点隔离查询库实机验证记录 -状态:**实机IO隔离未通过,已停止测试,25Gi查询库未创建**。用户授权在 `.4` 做小规模合成数据验证,完成实际IO限速、Pod重建、满盘恢复及业务影响检查后再决定建25Gi库。没有部署查询应用代码、启用生产读库开关或接入生产Mongo。 +状态:**原失败已定位并修复;512Mi完整回归通过,25Gi合成验证库已创建并完成Pod重建数据校验**。用户授权在 `.4` 做小规模合成数据验证,通过后再建25Gi库。没有部署查询应用代码、启用生产读库开关或接入生产Mongo。下方原失败记录保留,不代表当前修复后状态。 ## 实际范围 @@ -10,7 +10,7 @@ - 唯一测试目录 `/var/lib/mongo-cpu-node4-smoke-20261113`,固定536870912字节文件上的独立EXT4,标签 `mongo-cpu-smoke`。从不重新格式化已有文件,不格式化现有块设备。目录与挂载保留用于核验。 - 只有初始化容器具有挂载与设置专属Pod cgroup的权限;主测试容器UID999、无capabilities、只读根文件系统、只读宿主cgroup映射。启动先核实限额与loop挂载,失败不能进入测试。 - IO预算:读2MiB/s、写1MiB/s、读64 IOPS、写32 IOPS。初始仅限制底层vda;后续候选增加专属loop的同等限额。每次更换脚本均在测试Pod停止后CAS更新自己的ConfigMap。 -- 测试脚本见 [README](../../scripts/local/node4-query-pilot/README.md)。不支持从该脚本直接扩展到25Gi。 +- 测试脚本见 [README](../../scripts/local/node4-query-pilot/README.md)。默认512Mi模式不会自动扩盘;新增25Gi模式必须绑定小盘通过证据和独立计划。 ## 保留的尝试与发现 @@ -39,7 +39,7 @@ run-5是实机隔离不达标证据,不能以配置 `io.max` 存在或本机6. 本地3项安全回归通过:启动校验失败不能标记ready、非法节点/Pod身份不能触达宿主、重建使用动态Pod UID且主容器无特权。脚本语法检查通过;`verify:task --base f56c8faf` 因新诊断脚本缺少自动行为映射返回incomplete,不冒充正式发布门禁。 -## 最终判断与保留状态 +## 上一阶段失败判断与保留状态 run-6实际观察到目标Pod的loop计数仅增加4Mi直接写,后续4Mi缓冲写没有被计入该Pod;底层盘在目标Pod下也未累计对应写入。由此可确认**当前5.15内核/loop/写回组合未满足预期的IO归属与限速**,但未定位到某个确切上游内核缺陷,不能把推断写成已确认的内核bug。配置存在不能代替执行证据。 @@ -50,3 +50,50 @@ run-6的7个短窗口采样:API健康均成功,最大约50.3ms;既有业 所有测试Pod已停止,保留命名空间、脚本、策略、配额及512Mi合成承载文件/专属挂载供复核。停测后只读检查见 `postcheck.json`。没有购买磁盘、修改生产记录、重启运行时、调整业务Pod或改系统根挂载。 下一步需要先更换或修复存储/IO实现,再重跑同一门槛;不能仅增大测试盘、放宽判定阈值或用 `ionice` 代替硬限速。在不改系统盘配额/挂载、不调整全局运行时、不购买新盘的边界内,当前loop方案仍不具备建25Gi库的依据。 + +## 后续根因对照与修复 + +当前工作树先合入 `origin/dev` 的 `5c6023ec`,合并提交 `5e4de799` 保留独立查询分支和主线双方的验证范围。未部署应用代码。 + +内核配置确认 `CONFIG_MEMCG`、`CONFIG_BLK_CGROUP`、`CONFIG_CGROUP_WRITEBACK`、`CONFIG_BLK_DEV_THROTTLING`、`CONFIG_BLK_DEV_LOOP` 均启用。随后在同一Pod、同一内核、同一限额下,以随机数据对照旧文件、新文件、只驱逐承载文件干净缓存和原生EXT4控制文件,每笔仅1Mi,均带fdatasync: + +| 条件 | 保留旧挂载 | 正常卸载/重新挂载专属文件系统后 | +| --------------------------- | ----------------: | ------------------------------: | +| 旧inode | 0.010秒,未受限 | 1.016秒,受限 | +| 新inode | 1.013秒,受限 | 0.995秒,受限 | +| 旧inode,先驱逐承载文件缓存 | 0.010秒,仍未受限 | 0.995秒,受限 | +| 原生EXT4新文件控制 | 1.014秒,受限 | 1.014秒,受限 | + +旧inode写回计数出现在根层级,而新inode写回计入新Pod。证据指向**跨Pod保留挂载时,旧inode的写回归属没有立即切换到新Pod**;不是所有loop写入都无法限速,也不是写零被优化或控制器未启用。内核按inode管理写回归属,旧归属与新页归属可以暂时不同,不能把重新设置 `io.max` 当作归属已更新。[官方写回说明](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html) + +修复只改变测试存储生命周期:在原Pod完全停止后,校验专属承载文件和loop的身份,正常卸载该文件系统并重新挂载,再让新Pod打开数据。它重建inode写回上下文,**保留文件内容**。不使用强制/懒卸载,不重新格式化、不清全局缓存、不卸载其他设备。繁忙或身份不匹配即拒绝启动。保留双层限额和非特权主进程。 + +可行性判断:这一局部生命周期修复不买盘、不改系统根挂载或全局运行时,优先采用。原生目录配额方案在当前未启用配额的系统盘上还涉及宿主变更;用户态块设备方案引入新的驱动/恢复边界;两者没有必要在本次并行部署。原生EXT4控制文件只用来诊断,没有作为无硬容量上限的Mongo回退目录。 + +对照证据位于 `attribution-1/` 和 `attribution-remount/`;后者以同一阈值断言全部受限。前一阶段“现有路径未满足隔离”的判断保留为历史,不能扩大解释为所有loop方案都不可行。 + +## 修复后的完整小盘回归 + +`recovery-remount/result.json` 为 `small-smoke-passed`。首次及重建后的Pod UID分别为 `1e9586d1-80bd-4521-9b2c-6bef8126d6ee`、`4342b25e-df6f-4291-86bf-0751a6ab75a2`。 + +- 4Mi直接/缓冲写首次4.01/8.10秒,重建后7.07/8.12秒,均未超过配置上限。双层控制可使实际吞吐更低,不能据此承诺1Mi/s的最低吞吐。 +- 4Mi冷读首次4.11秒、重建后3.96秒;256次4Ki直接写约8.6秒、读取约4.2秒,符合32/64 IOPS限额。冷读仅驱逐该合成承载文件的干净缓存,没有清全局缓存。 +- Mongo先以单机 `w:1,j:true` 确认一个合成标记,Pod替换后标记仍在。 +- 512Mi文件系统产生ENOSPC,Mongo实际退出、连接被拒绝;只移除合成预留/填充文件后重新启动,原标记仍在,新journal确认写入成功。未删除WiredTiger/journal或用快照覆盖。 +- 完整前/中/后观察窗口内现有API健康成功、既有业务/系统Pod独立IO压力未升高;这是该受限测试窗口的证据,不是高峰容量或五角色写验收。 + +执行时的runner快照和原始输出保存在该目录;后续增加25Gi模式不追溯替换这轮执行证据。离线安全回归已增至5项,并加入专用验证范围;自动检查不访问集群,也不代替上述实机证据。 + +## 25Gi验证库阶段 + +计划绑定小盘通过结果的SHA256;单独创建 `/var/lib/mongo-cpu-node4-query25-20261113` 和 `node4-query25-scripts`,不覆盖512Mi故障现场。固定承载文件26843545600字节,EXT4采用128Ki/inode、32Mi文件系统journal,减少初始化写入;原Mongo数据journal保留。1核/1Gi、WiredTiger缓存0.25Gi、64Mi oplog、相同双层IO上限,均为隔离验证预算。 + +仅使用 `query25_synthetic` 合成库,单成员 `query-lab-rs`,监听127.0.0.1:27129,无Service、无生产凭证。初始化后按majority+journal确认标记,再替换Pod核对标记保留。主进程持续检查挂载/限额与宿主至少75Gi可用空间;Pod6小时到期退出但数据盘保留。此实例不是生产HA,不承载原库投影,不代表原Mongo减负、81/200/500人或查询延迟验收已完成。 + +阶段结果为 `query25/result.json` 的 `query25-created-synthetic-only`,计划 `7711333dd4e7…`;重建后Pod为 `19f5f918-9bb1-4e6a-a362-bb70afacee26`,Running/Ready,primary=true,合成标记数1。承载文件实际长度26843545600字节,目标Pod读回loop和vda两层限额均为2/1MiB/s、64/32IOPS。 + +该阶段25个采样中:现有API健康均成功,最大约56ms;既有业务/系统Pod IO `full avg10` 最大0;物理盘平均完成耗时最高1ms,忙碌比例最高约5.9%;生产Pod UID和重启数未变。宿主仍约136Gi可用。该文件为固定长度稀疏承载文件,当前实际分配约38Mi;**不是已经预留了25Gi物理空闲容量**。即使增长到其容量上限,按当前余量仍有约111Gi;运行时另外保留75Gi停止线。 + +最后复核时间为2026-11-13 03:50 UTC。Pod创建于03:47:06 UTC,六小时期限为09:47:06 UTC,届时退出但保留承载文件;没有将该临时实例描述成持续可用或生产HA。证据位于 `query25/postcheck.json` 和原始样本。 + +本轮工具验证新增专用离线范围,不会在CI自动写公司集群。首次受限本地工具回归在HTTP生命周期夹具中停滞,已仅终止本任务的本地Node进程组并保留失败报告;在允许本机端口的环境重跑,同一代码/夹具结果以本轮 `verify-*` 报告为准。5项离线安全回归独立通过;正式发布和大规模性能证据仍没有由此替代。 diff --git a/scripts/local/node4-query-pilot/README.md b/scripts/local/node4-query-pilot/README.md index 2c1adbe3..a67d8255 100644 --- a/scripts/local/node4-query-pilot/README.md +++ b/scripts/local/node4-query-pilot/README.md @@ -2,7 +2,7 @@ This is a diagnostic workload, not a deployment of the query application or a connection to production Mongo. Company-cluster execution requires explicit authorization for the synthetic node test. Rendering a plan is read-only. -**Current result: node4 buffered-write throttling FAILED, including dual loop/backing-device limits. Do not use this workload to provision a25Gi database.** Failures and unexecuted Mongo recovery checks are recorded in [the node report](../../../docs/operations/query-replica-node4-smoke-results.md). +The original retained-mount implementation FAILED across Pod lifetimes. The corrected initializer normally unmounts/rebinds only the exact-owned, idle synthetic filesystem before admitting a new Pod; it never reformats retained data or force-unmounts. The small recovery suite has since passed. Historical failures, contrasting inode measurements and current results are recorded in [the node report](../../../docs/operations/query-replica-node4-smoke-results.md). ```sh python3 scripts/local/node4-query-pilot/pilot.py \ @@ -11,7 +11,7 @@ python3 scripts/local/node4-query-pilot/pilot.py \ Review the rendered objects and SHA256 before supplying `--execute --confirm-plan `. This runner is fixed to the verified node and production namespace identities, an already cached immutable Mongo image, a 512Mi backing file and a new dedicated namespace. It creates no Service/PVC and exposes no port. The Mongo process binds loopback and uses only `node4_synthetic_pilot`. -The privileged initializer sets limits on the exact Pod UID's Guaranteed cgroup before creating or mounting the backing file. It does not change root or other Pod limits. The running probe has UID999, no capabilities, a read-only container root and read-only host cgroup access. It rejects missing mounts or limits instead of writing to an unbounded fallback directory. +The privileged initializer sets limits on the exact Pod UID's Guaranteed cgroup before creating or mounting the backing file. It does not change root or other Pod limits. Retaining the old mounted filesystem can retain inode writeback ownership from a deleted cgroup; merely setting limits on the new Pod does not repair it. Each bootstrap verifies the backing inode, normally unmounts/rebinds only that filesystem and sets both loop/backing-device limits. EBUSY fails closed. The running probe has UID999, no capabilities, a read-only container root and read-only host cgroup access. It rejects missing mounts or limits instead of writing to an unbounded fallback directory. The test performs direct/buffered/fsync IO probes, journals a synthetic marker, deletes and recreates only its own Pod, checks the new UID/limits and preserved marker, then exhausts its own filesystem with a filler file. Recovery removes only the filler/reserve files, preserves database and journal files, restarts Mongo and verifies the previously acknowledged marker plus a new journaled write. A filesystem ENOSPC alone does not prove that Mongo rejected a write: inspect the captured `MONGO_FULL_WRITE_EXIT` and output. @@ -19,7 +19,9 @@ During the test, read-only monitoring samples the existing API readiness endpoin Creation errors and timeouts are not retried automatically. Resume requires inspection of recorded state, an empty test namespace, an exact namespace UID, unchanged scripts/network policy/quota and a freshly confirmed plan. The runner does not overwrite existing objects. Preserve admission failures and partial results. -On exit it stops only the test Pod and retains the backing file/mount and namespace for review; it does not delete host directories, unmount arbitrary loop devices or delete Mongo files. Cleanup or expansion to25Gi requires a separately scoped plan based on the observed result. The25Gi database is not created by this script. +The normal smoke run stops only the test Pod on exit and retains the backing file/mount and namespace for review. It does not delete host directories, unmount arbitrary loop devices or delete Mongo files. + +The explicit `--query25 --acceptance-file ` mode requires a passing small recovery result, binds its hash into a new plan, and uses a different25Gi backing file and ConfigMap. Rendering is still offline; execution still requires a matching plan and user authorization. This mode creates only a loopback synthetic single-member replica set, verifies a majority+journaled marker through another Pod replacement, and retains a successful test Pod. It uses1CPU/1Gi, dual IO limits, a continuous75Gi host-free-space guard and a six-hour Pod deadline. Data remains after the deadline; it is not production HA, a production sync consumer or a capacity acceptance result. There is no public Service or production credential. Local safety checks: diff --git a/scripts/local/node4-query-pilot/attribution.sh b/scripts/local/node4-query-pilot/attribution.sh new file mode 100644 index 00000000..9b3b3457 --- /dev/null +++ b/scripts/local/node4-query-pilot/attribution.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -euo pipefail +bash /scripts/guard.sh +group="/hostcg/kubepods.slice/kubepods-pod${POD_UID//-/_}.slice" +cd /lab/data +require_bound=${1:-observe} +[[ $require_bound == observe || $require_bound == assert ]] +snapshot() { + printf '[DEBUG-node4-attribution] %s\n' "$1" + cat "$group/io.stat" + printf 'ROOT_CGROUP\n' + cat /hostcg/io.stat + printf 'PHYSICAL_VDA\n' + cat /sys/class/block/vda/stat +} +write_probe() { + local label=$1 target=$2 start end + snapshot "before-$label" + start=$(date +%s%N) + (ulimit -f 1024; dd if=/dev/urandom of="$target" bs=1048576 count=1 conv=fdatasync status=none) + end=$(date +%s%N) + awk -v label="$label" -v start="$start" -v end="$end" -v require_bound="$require_bound" 'BEGIN{s=(end-start)/1e9;printf "ATTRIBUTION %s seconds=%.6f\n",label,s;if(require_bound=="assert"&&s<0.75)exit 1}' + snapshot "after-$label" +} +write_probe reused-inode io-buffered +write_probe new-inode "attribution-new-$POD_UID" +dd if=/lab/disk.img of=/dev/null count=0 iflag=nocache status=none +write_probe reused-after-backing-eviction io-buffered +write_probe plain-ext4-control "/lab/control/$POD_UID" +rm -- "attribution-new-$POD_UID" "/lab/control/$POD_UID" +printf 'ATTRIBUTION_COMPLETE\n' diff --git a/scripts/local/node4-query-pilot/init.sh b/scripts/local/node4-query-pilot/init.sh index 6f4b5038..1d19888e 100644 --- a/scripts/local/node4-query-pilot/init.sh +++ b/scripts/local/node4-query-pilot/init.sh @@ -28,9 +28,17 @@ fi [[ $(blkid -p -s TYPE -o value /lab/disk.img) == ext4 ]] [[ $(blkid -p -s LABEL -o value /lab/disk.img) == mongo-cpu-smoke ]] mkdir -p /lab/data -if ! mountpoint -q /lab/data; then - mount -o loop,nodev,nosuid,noexec /lab/disk.img /lab/data +if mountpoint -q /lab/data; then + previous_source=$(findmnt -n -o SOURCE --target /lab/data) + [[ $previous_source == /dev/loop* ]] + previous_associated=$(losetup --associated /lab/disk.img --noheadings --output NAME | awk '{$1=$1;print}') + [[ $previous_associated == "$previous_source" ]] + # Retained inodes may still own a writeback domain belonging to the deleted + # Pod. Only an idle, exact-owned filesystem is unmounted; EBUSY fails closed. + # Never use lazy/forced unmount, drop global caches, or reformat existing data. + umount /lab/data fi +mount -o loop,nodev,nosuid,noexec /lab/disk.img /lab/data source=$(findmnt -n -o SOURCE --target /lab/data) [[ $source == /dev/loop* ]] # Match the backing inode, not a stale pathname from the previous Pod's mount namespace. @@ -48,4 +56,6 @@ chgrp 999 /lab/disk.img chmod 0440 /lab/disk.img chmod 0755 /lab chown 999:999 /lab/data +mkdir -p /lab/control +chown 999:999 /lab/control printf 'INIT_OK pod=%s group=%s image_bytes=536870912\n' "$POD_UID" "$group" diff --git a/scripts/local/node4-query-pilot/pilot.py b/scripts/local/node4-query-pilot/pilot.py index 53231b46..c94080ea 100644 --- a/scripts/local/node4-query-pilot/pilot.py +++ b/scripts/local/node4-query-pilot/pilot.py @@ -41,7 +41,7 @@ def pod(): {"name": "cg", "mountPath": "/hostcg"}, {"name": "scripts", "mountPath": "/scripts", "readOnly": True}]}], "containers": [{"name": "probe", "image": IMAGE, "imagePullPolicy": "Never", "env": env, - "command": ["bash", "-c", "set -e; bash /scripts/guard.sh; touch /tmp/ready; exec sleep 1700"], + "command": ["bash", "-c", "set -e; trap 'exit 0' TERM INT; bash /scripts/guard.sh; touch /tmp/ready; for ((i=0;i<340;i++)); do sleep 5 & wait $!; done"], "resources": resources(), "securityContext": {"runAsUser": 999, "runAsGroup": 999, "runAsNonRoot": True, "allowPrivilegeEscalation": False, "readOnlyRootFilesystem": True, @@ -57,17 +57,36 @@ def pod(): {"name": "tmp", "emptyDir": {"medium": "Memory", "sizeLimit": "64Mi"}}]}} -def manifests(): - return [ +def manifests(query25=False): + objects = [ {"apiVersion": "v1", "kind": "Namespace", "metadata": {"name": NS, "labels": {"purpose": "mongo-cpu-synthetic"}}}, {"apiVersion": "v1", "kind": "ResourceQuota", "metadata": {"name": "smoke-budget", "namespace": NS}, "spec": {"hard": {"pods": "1", "requests.cpu": "1", "limits.cpu": "1", "requests.memory": "1Gi", "limits.memory": "1Gi", "persistentvolumeclaims": "0", "services": "0"}}}, {"apiVersion": "networking.k8s.io/v1", "kind": "NetworkPolicy", "metadata": {"name": "deny-all", "namespace": NS}, "spec": {"podSelector": {}, "policyTypes": ["Ingress", "Egress"], "ingress": [], "egress": []}}, {"apiVersion": "v1", "kind": "ConfigMap", "metadata": {"name": "node4-smoke-scripts", "namespace": NS}, - "data": {name: (HERE / name).read_text() for name in ["init.sh", "guard.sh", "io.sh", "mongo.sh"]}}, + "data": {name: (HERE / name).read_text() for name in ["init.sh", "guard.sh", "io.sh", "mongo.sh", "attribution.sh"]}}, pod(), ] + if query25: + init = (HERE / "init.sh").read_text().replace("536870912", "26843545600").replace("mongo-cpu-smoke", "mongo-cpu-query").replace("node4-synthetic-pilot-v1", "node4-synthetic-query25-v1") + init = init.replace("mkfs.ext4 -F -q -m 0", "mkfs.ext4 -F -q -m 0 -i 131072 -J size=32") + guard = (HERE / "guard.sh").read_text().replace("131072", "6553600").replace("node4-synthetic-pilot-v1", "node4-synthetic-query25-v1") + objects[-2]["metadata"]["name"] = "node4-query25-scripts" + objects[-2]["data"] = {"init.sh": init, "guard.sh": guard, **{name: (HERE / name).read_text() for name in ["query25.sh", "seed25.sh", "verify25.sh"]}} + obj = objects[-1] + obj["metadata"]["name"] = "node4-query25" + obj["metadata"]["labels"] = {"app": "node4-query25"} + obj["spec"]["activeDeadlineSeconds"] = 21600 + for volume in obj["spec"]["volumes"]: + if volume["name"] == "lab": + volume["hostPath"]["path"] = "/var/lib/mongo-cpu-node4-query25-20261113" + if volume["name"] == "scripts": + volume["configMap"]["name"] = "node4-query25-scripts" + container = obj["spec"]["containers"][0] + container["command"] = ["bash", "/scripts/query25.sh"] + container["readinessProbe"] = {"exec": {"command": ["mongosh", "--quiet", "--host", "127.0.0.1", "--port", "27129", "--eval", "quit(db.adminCommand({ping:1}).ok?0:1)"]}, "periodSeconds": 15, "timeoutSeconds": 5} + return objects def main(): @@ -77,16 +96,30 @@ def main(): ap.add_argument("--execute", action="store_true") ap.add_argument("--confirm-plan") ap.add_argument("--resume-namespace-uid") + ap.add_argument("--diagnose-only", action="store_true") + ap.add_argument("--query25", action="store_true") + ap.add_argument("--acceptance-file") args = ap.parse_args() out = pathlib.Path(args.output).resolve() out.mkdir(parents=True, exist_ok=True) - objects = manifests() + assert not (args.query25 and args.diagnose_only) + acceptance_sha = None + if args.query25: + assert args.acceptance_file, "passing small recovery evidence required" + evidence_bytes = pathlib.Path(args.acceptance_file).read_bytes() + evidence = json.loads(evidence_bytes) + assert evidence["result"] == "small-smoke-passed" + assert evidence["firstPodUID"] != evidence["secondPodUID"] + acceptance_sha = hashlib.sha256(evidence_bytes).hexdigest() + objects = manifests(args.query25) + target_pod = objects[-1]["metadata"]["name"] runner_sha = hashlib.sha256(pathlib.Path(__file__).read_bytes()).hexdigest() - digest = hashlib.sha256(json.dumps({"objects": objects, "runner": runner_sha}, sort_keys=True).encode()).hexdigest() + mode = "query25-synthetic-only" if args.query25 else "attribution-only" if args.diagnose_only else "full-synthetic-recovery" + digest = hashlib.sha256(json.dumps({"objects": objects, "runner": runner_sha, "mode": mode, "acceptance": acceptance_sha}, sort_keys=True).encode()).hexdigest() (out / "objects.json").write_text(json.dumps(objects, indent=2)) (out / "plan.json").write_text(json.dumps({"sha256": digest, "node": NODE, "nodeUID": NODE_UID, - "namespace": NS, "image": IMAGE, "hostPath": PATH, - "bytes": 536870912, "productionWrites": False}, indent=2)) + "namespace": NS, "image": IMAGE, "hostPath": objects[-1]["spec"]["volumes"][0]["hostPath"]["path"], + "bytes": 26843545600 if args.query25 else 536870912, "productionWrites": False, "mode": mode, "acceptanceSha256": acceptance_sha}, indent=2)) print("PLAN", digest, flush=True) if not args.execute: return @@ -112,12 +145,19 @@ def main(): if c["type"] in ("DiskPressure", "MemoryPressure", "PIDPressure"): assert c["status"] == "False" existing = subprocess.run(k + ["get", "namespace", NS, "--ignore-not-found", "-o", "name"], capture_output=True, text=True, check=True) + config_missing = False if existing.stdout.strip(): existing_ns = get(["get", "namespace", NS]) assert args.resume_namespace_uid == existing_ns["metadata"]["uid"], "explicit recorded namespace UID required" assert existing_ns["metadata"].get("labels", {}).get("purpose") == "mongo-cpu-synthetic" assert not get(["get", "pods", "-n", NS])["items"], "stop and inspect existing pilot Pod before resuming" - assert get(["get", "configmap", "node4-smoke-scripts", "-n", NS])["data"] == objects[-2]["data"] + config_name = objects[-2]["metadata"]["name"] + config_raw = call(["get", "configmap", config_name, "-n", NS, "--ignore-not-found", "-o", "json"]) + if config_raw.strip(): + assert json.loads(config_raw)["data"] == objects[-2]["data"] + else: + assert args.query25 + config_missing = True policy = get(["get", "networkpolicy", "deny-all", "-n", NS])["spec"] assert policy["podSelector"] == {} and set(policy["policyTypes"]) == {"Ingress", "Egress"} assert not policy.get("ingress") and not policy.get("egress") @@ -217,7 +257,7 @@ def main(): except Exception as e: monitor_failed.append(str(e)) (out / "monitor-failure.json").write_text(json.dumps(monitor_failed)) - subprocess.run(k + ["-n", NS, "delete", "pod", POD, "--ignore-not-found", "--wait=false"], capture_output=True, timeout=15) + subprocess.run(k + ["-n", NS, "delete", "pod", target_pod, "--ignore-not-found", "--wait=false"], capture_output=True, timeout=15) return def create(obj): @@ -226,11 +266,11 @@ def main(): print(call(["create", "-f", "-"], payload=json.dumps(obj)).strip(), flush=True) def ready(): - deadline = time.monotonic() + 180 + deadline = time.monotonic() + (900 if args.query25 else 180) while time.monotonic() < deadline: if monitor_failed: raise RuntimeError(str(monitor_failed)) - p = get(["get", "pod", POD, "-n", NS]) + p = get(["get", "pod", target_pod, "-n", NS]) (out / "last-pod.json").write_text(json.dumps(p, indent=2)) if any(c["type"] == "Ready" and c["status"] == "True" for c in p["status"].get("conditions", [])): return p["metadata"]["uid"] @@ -242,7 +282,7 @@ def main(): def probe(name, script, *extra, timeout=180): print("PROBE", name, flush=True) try: - result = call(["-n", NS, "exec", POD, "-c", "probe", "--", "bash", "/scripts/" + script, *extra], timeout=timeout) + result = call(["-n", NS, "exec", target_pod, "-c", "probe", "--", "bash", "/scripts/" + script, *extra], timeout=timeout) except Exception: # Preserve bounded logs/status; never retry an unknown write result. raise @@ -251,10 +291,13 @@ def main(): assert not monitor_failed worker = None + retain_query = False try: if not existing.stdout.strip(): for obj in objects[:-1]: create(obj) + elif config_missing: + create(objects[-2]) for _ in range(20): quota = get(["get", "resourcequota", "smoke-budget", "-n", NS]) if quota.get("status", {}).get("hard"): @@ -266,9 +309,28 @@ def main(): worker.start() create(objects[-1]) first_uid = ready() + if args.query25: + probe("query25-seed", "seed25.sh", timeout=180) + call(["-n", NS, "delete", "pod", target_pod, "--wait=true", "--timeout=90s"], timeout=100) + create(objects[-1]) + second_uid = ready() + assert first_uid != second_uid + probe("query25-after-rebuild", "verify25.sh", timeout=180) + for _ in range(6): + sample() + time.sleep(5) + assert not monitor_failed + (out / "result.json").write_text(json.dumps({"result": "query25-created-synthetic-only", "plan": digest, "acceptanceSha256": acceptance_sha, "firstPodUID": first_uid, "secondPodUID": second_uid, "activeDeadlineSeconds": 21600, "network": "loopback-only-no-Service", "limitations": ["single-node lab", "no production sync", "not capacity acceptance"]}, indent=2)) + retain_query = True + print("QUERY25_CREATED_SYNTHETIC_ONLY", flush=True) + return + if args.diagnose_only: + probe("attribution", "attribution.sh", "assert", timeout=90) + (out / "result.json").write_text(json.dumps({"result": "attribution-captured-not-acceptance", "plan": digest, "podUID": first_uid}, indent=2)) + return probe("io-first", "io.sh", timeout=240) probe("mongo-seed", "mongo.sh", "seed", timeout=180) - call(["-n", NS, "delete", "pod", POD, "--wait=true", "--timeout=90s"], timeout=100) + call(["-n", NS, "delete", "pod", target_pod, "--wait=true", "--timeout=90s"], timeout=100) create(pod()) second_uid = ready() assert second_uid != first_uid @@ -292,12 +354,13 @@ def main(): if worker: worker.join(timeout=30) try: - logs = call(["-n", NS, "logs", POD, "-c", "isolate", "--tail=60"]) + logs = call(["-n", NS, "logs", target_pod, "-c", "isolate", "--tail=60"]) (out / "init.log").write_text(logs) except Exception: pass # Stop only this synthetic Pod; retain its backing image and exact mount for review. - subprocess.run(k + ["-n", NS, "delete", "pod", POD, "--ignore-not-found", "--wait=true", "--timeout=90s"], capture_output=True, timeout=100) + if not retain_query: + subprocess.run(k + ["-n", NS, "delete", "pod", target_pod, "--ignore-not-found", "--wait=true", "--timeout=90s"], capture_output=True, timeout=100) if __name__ == "__main__": diff --git a/scripts/local/node4-query-pilot/query25.sh b/scripts/local/node4-query-pilot/query25.sh new file mode 100644 index 00000000..3baf807b --- /dev/null +++ b/scripts/local/node4-query-pilot/query25.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash +set -euo pipefail +bash /scripts/guard.sh +mkdir -p /lab/data/db +mongod --dbpath /lab/data/db --port 27129 --bind_ip 127.0.0.1 --replSet query-lab-rs --oplogSize 64 --wiredTigerCacheSizeGB 0.25 --logpath /lab/data/mongo.log --logappend --setParameter diagnosticDataCollectionEnabled=false & +pid=$! +stop() { kill -TERM "$pid" 2>/dev/null || true; wait "$pid" || true; } +trap stop EXIT +trap 'exit 0' TERM INT +while kill -0 "$pid" 2>/dev/null; do + bash /scripts/guard.sh + available=$(stat -f -c '%a %S' /lab | awk '{printf "%.0f",$1*$2}') + (( available >= 80530636800 )) + sleep 15 & wait $! +done +wait "$pid" diff --git a/scripts/local/node4-query-pilot/seed25.sh b/scripts/local/node4-query-pilot/seed25.sh new file mode 100644 index 00000000..2916636a --- /dev/null +++ b/scripts/local/node4-query-pilot/seed25.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +set -euo pipefail +bash /scripts/guard.sh +mongosh --quiet --host 127.0.0.1 --port 27129 --eval 'try {rs.status()}catch(e){if(e.code!==94)throw e;rs.initiate({_id:"query-lab-rs",members:[{_id:0,host:"127.0.0.1:27129"}]})}' +for ((i=0;i<60;i++)); do + if mongosh --quiet --host 127.0.0.1 --port 27129 --eval 'quit(db.hello().isWritablePrimary?0:1)' >/dev/null; then + mongosh --quiet --host 127.0.0.1 --port 27129 --eval 'const d=db.getSiblingDB("query25_synthetic");d.marker.insertOne({_id:"query25-confirmed",value:"preserve-query25-v1"},{writeConcern:{w:"majority",j:true}});print("QUERY25_JOURNALED");' + exit 0 + fi + sleep 1 +done +exit 1 diff --git a/scripts/local/node4-query-pilot/test_pilot.py b/scripts/local/node4-query-pilot/test_pilot.py index b1412cf0..33ff6a3e 100644 --- a/scripts/local/node4-query-pilot/test_pilot.py +++ b/scripts/local/node4-query-pilot/test_pilot.py @@ -1,9 +1,14 @@ import importlib.util +import contextlib +import hashlib +import io +import json import os import pathlib import subprocess import tempfile import unittest +from unittest.mock import patch ROOT = pathlib.Path(__file__).resolve().parent @@ -13,6 +18,37 @@ spec.loader.exec_module(pilot) class SafetyTests(unittest.TestCase): + def test_query25_rejects_failed_small_recovery(self): + with tempfile.TemporaryDirectory() as directory: + evidence = pathlib.Path(directory) / "failed.json" + evidence.write_text(json.dumps({"result": "failed"})) + argv = ["pilot.py", "--kubeconfig", "not-used", "--output", str(pathlib.Path(directory) / "out"), "--query25", "--acceptance-file", str(evidence)] + with patch("sys.argv", argv), patch.object(pilot.subprocess, "run") as call: + with self.assertRaises(AssertionError): + pilot.main() + call.assert_not_called() + + def test_query25_render_is_offline_and_uses_distinct_bounded_storage(self): + with tempfile.TemporaryDirectory() as directory: + evidence = pathlib.Path(directory) / "passed.json" + evidence.write_text(json.dumps({"result": "small-smoke-passed", "firstPodUID": "first", "secondPodUID": "second"})) + output = pathlib.Path(directory) / "out" + argv = ["pilot.py", "--kubeconfig", "not-used", "--output", str(output), "--query25", "--acceptance-file", str(evidence)] + with patch("sys.argv", argv), patch.object(pilot.subprocess, "run") as call, contextlib.redirect_stdout(io.StringIO()): + pilot.main() + call.assert_not_called() + plan = json.loads((output / "plan.json").read_text()) + self.assertEqual(plan["bytes"], 25 * 1024 ** 3) + self.assertEqual(plan["acceptanceSha256"], hashlib.sha256(evidence.read_bytes()).hexdigest()) + self.assertNotEqual(plan["hostPath"], pilot.PATH) + objects = json.loads((output / "objects.json").read_text()) + self.assertFalse(any(obj["kind"] in ("Service", "Secret", "PersistentVolumeClaim") for obj in objects)) + config = objects[-2]["data"] + self.assertIn("truncate -s 26843545600", config["init.sh"]) + self.assertIn("umount /lab/data", config["init.sh"]) + self.assertNotIn("umount -", config["init.sh"]) + self.assertIn("--bind_ip 127.0.0.1", config["query25.sh"]) + def test_boot_does_not_mark_ready_after_guard_failure(self): command = pilot.pod()["spec"]["containers"][0]["command"][2] with tempfile.TemporaryDirectory() as directory: diff --git a/scripts/local/node4-query-pilot/verify25.sh b/scripts/local/node4-query-pilot/verify25.sh new file mode 100644 index 00000000..24ac8c7c --- /dev/null +++ b/scripts/local/node4-query-pilot/verify25.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +set -euo pipefail +bash /scripts/guard.sh +for ((i=0;i<60;i++)); do + if mongosh --quiet --host 127.0.0.1 --port 27129 --eval 'quit(db.hello().isWritablePrimary?0:1)' >/dev/null; then + mongosh --quiet --host 127.0.0.1 --port 27129 --eval 'const d=db.getSiblingDB("query25_synthetic");if(d.marker.countDocuments({_id:"query25-confirmed",value:"preserve-query25-v1"})!==1)throw Error("acknowledged marker lost");print("QUERY25_MARKER_PRESERVED");' + exit 0 + fi + sleep 1 +done +exit 1 diff --git a/scripts/verification-scopes.json b/scripts/verification-scopes.json index 21131013..be3522a2 100644 --- a/scripts/verification-scopes.json +++ b/scripts/verification-scopes.json @@ -966,6 +966,32 @@ ], "warnings": ["真实三组件仅使用固定本机合成夹具;完整规模三轮、故障恢复容量及生产部署独立验收。"] }, + { + "name": "node4-query-pilot-guardrails", + "paths": [ + "^scripts/local/node4-query-pilot/", + "^docs/operations/query-replica-node4-(isolation-plan|smoke-results)\\.md$" + ], + "frontend": [], + "go": {}, + "task": [ + { + "argv": [ + "python3", + "-m", + "unittest", + "discover", + "-s", + "scripts/local/node4-query-pilot", + "-p", + "test_*.py", + "-v" + ], + "tests": true + } + ], + "warnings": ["仅执行离线安全回归;节点写回归属、Pod重建、满盘及容量仍须已授权的独立合成实测,不能以该检查替代。"] + }, { "name": "independent-query-replica", "paths": [