diff --git a/docs/operations/query-replica-query25-comparison.md b/docs/operations/query-replica-query25-comparison.md index 4c048256..274257ac 100644 --- a/docs/operations/query-replica-query25-comparison.md +++ b/docs/operations/query-replica-query25-comparison.md @@ -238,3 +238,36 @@ IO保护新增严格的FUSE归因分支,并有先失败后通过的回归: 证据:`scale-81-12-grouped-slices/stopped.json`、`production-state-readonly.json`、`manifest.json`和原始 `test.log`。测试源Pod由保护器按UID删除;随后桥接/两条策略按身份清理,配额CAS恢复原1Pod/1CPU/1Gi。仅删除本轮精确随机合成查询数据库,保留25Gi文件系统、持久标记与查询Pod原3600秒deadline,见 `fixture-9/cleanup.json` 与 `synthetic-db-cleanup.json`。没有修改生产数据、线上读路由、资源、超时或持久性。 下一步在无并发构建/发布的稳定窗口重跑原规模两轮,先确认冷构建在原期限内完成,再判断HTTP正确性、p95与源MongoCPU/扫描下降。未完成这一证据前不准备上线;也不通过扩大IO、延长超时、关闭journal或将未确认写入当成功来取得通过。 + +## 授权的资源分档对照(2026-11-13) + +本次用户明确授权调整 `.4` 节点25Gi合成查询库的CPU、内存及有限IO,替代此前维持原预算的实验限制。生产数据、生产入口、API业务实现、请求超时、租约、majority+j及25Gi容量均不变;不购买新盘。 + +预检确认节点和命名空间UID、原查询Pod已按3600秒deadline退出,未发现侧对话遗留变更。原配置、期望配置、回退说明和每次重建UID保存在 `local-data/diagnostics/mongo-cpu/resource-stages-20261113/`。节点初始CPU约1%,MemAvailable约28.6GiB,未出现Memory/Disk/PIDPressure。共享虚拟设备为252:0/vda;内核报告rotational=1,不能据此声称底层SSD或确定云盘物理介质。未做全局cache drop。 + +| 档位 | CPU/内存 | WT缓存 | 两层读/写MiB/s | 两层读/写IOPS | +| --- | --- | --- | --- | --- | +| baseline | 1核/1GiB | 0.25GiB | 2/1 | 64/32 | +| compute | 2核/4GiB | 1GiB | 2/1 | 64/32 | +| balanced | 2核/4GiB | 1GiB | 4/2 | 128/64 | +| read-headroom | 2核/4GiB | 1GiB | 8/2 | 256/64 | + +统一来源为 `scripts/local/query_lab_profiles.py`,同步生成Pod资源/配额、初始化器、loop动态设备号和底层252:0的io.max、运行时CPU/内存/IO检查及WT启动与实际值核验。变更先停止自有夹具,保存原对象,按Pod UID删除并正常重挂已存在文件系统;ConfigMap/配额使用UID及resourceVersion/值前置检查。没有取消IO上限、强制卸载或重建数据文件。75Gi宿主空闲保护保留。回退脚本恢复确切旧配置,不因生产不健康拒绝执行资源回退。 + +所有档位保持26,218条任务、81账号(60标注/12质检/8验收/1管理员)、两波各81个并发请求、源Mongo4核/4Gi/tmpfs及请求顺序。旧/新结果逐项匹配,标注、状态及版本摘要严格校验。`compute`复用历史基线的相同API/bridge测试二进制;`balanced`起仅补充测试计时字段,并使用当前已有的API命令监测接线,未改变生产API算法、请求序列或调用次数。两个新IO档位使用相同增强二进制。首次读取是应用缓存未热的已有等价请求,不声称清除了操作系统页缓存;两波并发单独记录,不能把第二波称为每个用户每条路由都已热。 + +`compute`:初始同步239.168秒(原1核/1Gi历史基线372.203秒),搜索32.369秒,事实确认93.183秒,完整HTTP测试459.84秒。新旧各162请求零错误、162项业务匹配;源CPU21.40→3.95秒、扫描10,480,573→1,450,972。查询路径计数p95为5,730.505ms,个人看板282.956ms、作业记录188.618ms。测试因计数超过2秒失败,不能记为性能通过;`coverageComplete=true`只表示覆盖完整,`returncode=1`原样保留。 + +`balanced`:初始同步112.483秒,搜索29.525秒,事实确认78.049秒;物化就绪137.501秒,仍使用原4分钟期限,整体测试302.72秒通过。查询路径162请求零错误、162项匹配,计数p95为1,038.164ms、生产总览8.952ms、个人看板272.390ms、作业记录180.871ms。合成源CPU21.23→3.89秒(下降81.68%),扫描10,480,573→1,450,972(下降86.16%),减少的是合成源负载而非已测得生产Mongo CPU下降。首次个人看板2,674.617ms,仍是明确未达理想冷读目标的样本;首次计数18.644ms、生产总览182.888ms、作业记录41.236ms。两波计数p95分别1,031.751/1,070.903ms。`exit.json`为0、源码未变、覆盖完整、最终健康错误为null;`capacityAccepted`仍为false。 + +balanced全程物理盘等待最高3.78ms、忙碌13.54%,节点CPU最高36%,受保护生产Pod的IO PSI最高0,readiness最长约65ms,未发生生产Pod/重启集合变化。保护采样不能证明所有客户业务延迟零影响。资源重建前以majority+j、3秒确认向既有合成marker集合写入512项测试对象,重建后数量及SHA-256完全相同,原持久标记亦保留;未触碰客户数据。 + +后续read-headroom结果、最终资源和清理状态记录于本节后续条目。短两波对照不替代30分钟三轮、200/500人、HA或生产容量验收。当前仅证明合成读路径的降载和正确性,不准备切换生产入口。 + +`read-headroom`:同一增强二进制、同样规模/顺序、写上限保持2MiB/s及64 IOPS。初始同步107.566秒,发生1次原有检查点租约恢复;搜索26.380秒、事实确认71.590秒,物化就绪123.502秒,总测试285.17秒通过。查询路径162项全部匹配、零错误,计数p95为1,086.154ms、个人看板317.153ms、作业记录153.605ms、生产总览9.221ms。首次个人看板2,653.925ms,与balanced的2,674.617ms近似,未观察到增加读取余量的明确冷读收益。源CPU20.28→3.99秒(下降80.33%),扫描下降86.16%。受保护Pod IO PSI仍为0、物理盘等待最高3.35ms、忙碌15.08%、节点CPU最高35%,最终健康无错误。 + +因此最终选择较保守的 **balanced:2核/4GiB、WT1GiB、双层读4/写2MiB/s、128/64 IOPS**,不继续扩大IO。已按保存的balanced配置恢复并再次重建Pod,实际资源/两层限额/WT缓存及512项合成内容摘要再次核对。最终精确Pod UID、25Gi容量、3600秒deadline、配额、原持久标记和生产状态见 `final-state.json`。各轮合成源Pod和两条策略均按UID删除、桥接按PID/可执行文件核对后停止,单Pod配额恢复;只保留查询库和deny-all网络策略。原始1核/1GiB/WT0.25GiB/原IO的完整回退对象位于 `compute/original.json`,不要误用后续档位的original覆盖最初回退点。 + +三个档位的查询Pod内存峰值约0.87~1.01GiB,未出现OOM或memory.max事件;CPU限流仅2~4个周期,不是持续限流。`stages-summary.json`保存逐档CPU/内存/磁盘等待、完整性、退出码和源库降载。单次扩容对照不能证明三轮稳态内存增长或长期容量。提高读取上限没有改善首次看板,下一步应分解首次事实装载/聚合和后台预热,而不是继续增加共享盘上限。所有新旧对照保留原超时、确认与结果规则;首次个人看板约2.6秒仍待优化,81人短测通过不表示全部性能验收完成。 + +本批代码只改实验配置、隔离/回退校验及测试计时;没有修改生产业务处理器、源Mongo配置、客户正文/归属/版本、源文件或生产读取入口。工具回归包含配置不一致、单侧IO漂移、未知档位及回退目标/脚本偏差拒绝。首次沙箱工具检查因本机端口环境超时,原记录保留;同一命令在允许本机端口的环境通过。查询读库/缓存/权限race回归亦已通过,最终检查报告随本批记录保留。未执行生产发布或生产写验收。 diff --git a/internal/api/query_replica_isolated_test.go b/internal/api/query_replica_isolated_test.go index 14e411b0..ca620d9b 100644 --- a/internal/api/query_replica_isolated_test.go +++ b/internal/api/query_replica_isolated_test.go @@ -303,12 +303,14 @@ func TestQueryReplicaIsolatedHTTPComparisonLocal(t *testing.T) { } }() defer func() { stopWorkers(); workers.Wait(); builder.Close() }() + materializeStarted := time.Now() readyCtx, cancel := context.WithTimeout(ctx, 4*time.Minute) err = builder.WaitReady(readyCtx, fixture.org, fixture.projects) cancel() if err != nil { t.Fatal("query materialization", err) } + materializeSeconds := time.Since(materializeStarted).Seconds() // The projector is intentionally paused for the authority measurement. A // long initial build must not leave that fixture's watermark expired before // the first paired HTTP read, even though its source has had no changes. @@ -366,10 +368,16 @@ func TestQueryReplicaIsolatedHTTPComparisonLocal(t *testing.T) { "/api/dashboard/workforce?from=2026-11-01T00:00:00Z&to=2026-11-13T00:00:00Z&groupBy=member", "/api/workforce-returns?from=2026-11-01T00:00:00Z&to=2026-11-13T00:00:00Z&limit=20&offset=0", } + // First reads use the existing parity requests, with no extra query load. // Exact business JSON equality; only generated report timestamps are volatile. + firstReads := map[string]any{} for i, path := range paths { + start := time.Now() a, ab, ae := request(before.URL, path, 0) + oldMs := float64(time.Since(start).Microseconds()) / 1000 + start = time.Now() b, bb, be := request(after.URL, path, 0) + firstReads[strings.Split(path, "?")[0]] = map[string]any{"oldMs": oldMs, "queryMs": float64(time.Since(start).Microseconds()) / 1000} if ae != nil || be != nil || a != 200 || b != 200 { t.Fatalf("route %d: old=%d/%v %.300s new=%d/%v %.300s", i, a, ae, ab, b, be, bb) } @@ -430,7 +438,12 @@ func TestQueryReplicaIsolatedHTTPComparisonLocal(t *testing.T) { if maxInflight > 0 { slots = make(chan struct{}, maxInflight) } + waveReports := []any{} for wave := 0; wave < waves; wave++ { + offsets := map[string]int{} + for path, samples := range byRoute { + offsets[path] = len(samples) + } var wg sync.WaitGroup var results sync.Mutex barrier := make(chan struct{}) @@ -480,6 +493,16 @@ func TestQueryReplicaIsolatedHTTPComparisonLocal(t *testing.T) { } close(barrier) wg.Wait() + waveRoutes := map[string]any{} + for path, samples := range byRoute { + values := append([]float64(nil), samples[offsets[path]:]...) + if len(values) == 0 { + continue + } + sort.Float64s(values) + waveRoutes[path] = map[string]any{"requests": len(values), "p95Ms": values[(len(values)*95+99)/100-1], "maxMs": values[len(values)-1]} + } + waveReports = append(waveReports, map[string]any{"wave": wave + 1, "routes": waveRoutes}) } mu.Lock() phase = "" @@ -508,6 +531,7 @@ func TestQueryReplicaIsolatedHTTPComparisonLocal(t *testing.T) { } } result := map[string]any{"name": name, "seconds": time.Since(started).Seconds(), "requests": len(latency), "p50Ms": latency[(len(latency)-1)/2], "p95Ms": latency[(len(latency)-1)*95/100], "errors": errors, "firstError": firstError, "routes": routes, "responseBytes": responses, "sourceCPUSeconds": cpu, "sourceScans": scans, "source": m} + result["waves"] = waveReports result["matchedBusinessResults"], result["businessMismatches"] = matchedResults, mismatches if mode == "query" && mismatches > 0 { t.Errorf("%d per-actor business results differed", mismatches) @@ -526,6 +550,8 @@ func TestQueryReplicaIsolatedHTTPComparisonLocal(t *testing.T) { } } } + report["materializationSeconds"] = materializeSeconds + report["firstReads"] = firstReads report["cases"] = cases report["errorFree"] = accepted report["capacityAccepted"] = false diff --git a/scripts/local/node4-query-pilot/README.md b/scripts/local/node4-query-pilot/README.md index a67d8255..cd7fd393 100644 --- a/scripts/local/node4-query-pilot/README.md +++ b/scripts/local/node4-query-pilot/README.md @@ -28,3 +28,41 @@ Local safety checks: ```sh python3 -m unittest discover -s scripts/local/node4-query-pilot -p 'test_*.py' -v ``` + +## Authorized query25 resource stages + +For an existing query25 filesystem, use `scripts/local/query-lab-resource-stage.py`, +not the initial namespace provisioning flow. The authoritative finite profiles +are in `scripts/local/query_lab_profiles.py`: `baseline`, `compute`, `balanced` +and `read-headroom`. Rendering updates the Guaranteed Pod resources, quota, +WiredTiger cache, initializer and runtime checks for **both** loop and backing +block devices together. The verifier checks the running WiredTiger cache as +well as the actual cgroup caps. No profile changes disk capacity or durability. + +The resource-stage runner first saves exact original objects, desired objects +and rollback instructions in a new evidence directory. `--execute` is allowed +only within separately authorized lab changes. It requires the exact current +query Pod UID and an otherwise empty test namespace; it never replaces a +concurrent fixture. It uses UID deletion and resourceVersion/value preconditions +for ConfigMap/quota updates. A recreated query Pod retains its data and has a +3600-second deadline; ordinary bootstrap provisions the older six-hour pilot. +The runtime host-free-space floor remains75Gi. The original smoke profiles and +IO probe thresholds remain unchanged. + +Example (the current UID must come from a fresh read-only preflight): + +```sh +python3 scripts/local/query-lab-resource-stage.py \ + --kubeconfig --output \ + --profile compute --expected-pod-uid --execute +``` + +Before any resource change, check node/production health and current occupation. +Before increasing the next stage, UID-clean the owned source fixture/policies +and PID/executable-check the bridge, restoring its recorded query-only quota. +Use the comparison monitor throughout synthetic load. On protection failure, +stop load first and restore the original stage using `--restore ` +with the current UID and a new output directory. Restoration is not blocked by +an unhealthy readiness check; it only restores saved lab objects. A CAS conflict +requires inspection, never a force overwrite. Neither rebuild nor rollback +reformats data or performs a global cache drop. diff --git a/scripts/local/node4-query-pilot/pilot.py b/scripts/local/node4-query-pilot/pilot.py index c94080ea..592601f9 100644 --- a/scripts/local/node4-query-pilot/pilot.py +++ b/scripts/local/node4-query-pilot/pilot.py @@ -10,6 +10,9 @@ import subprocess import threading import time import urllib.request +import sys +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent)) +import query_lab_profiles NODE = "10.189.16.4" @@ -57,7 +60,7 @@ def pod(): {"name": "tmp", "emptyDir": {"medium": "Memory", "sizeLimit": "64Mi"}}]}} -def manifests(query25=False): +def manifests(query25=False, query_profile="baseline"): objects = [ {"apiVersion": "v1", "kind": "Namespace", "metadata": {"name": NS, "labels": {"purpose": "mongo-cpu-synthetic"}}}, {"apiVersion": "v1", "kind": "ResourceQuota", "metadata": {"name": "smoke-budget", "namespace": NS}, @@ -86,7 +89,7 @@ def manifests(query25=False): container = obj["spec"]["containers"][0] container["command"] = ["bash", "/scripts/query25.sh"] container["readinessProbe"] = {"exec": {"command": ["mongosh", "--quiet", "--host", "127.0.0.1", "--port", "27129", "--eval", "quit(db.adminCommand({ping:1}).ok?0:1)"]}, "periodSeconds": 15, "timeoutSeconds": 5} - return objects + return query_lab_profiles.render(objects, query_profile) if query25 else objects def main(): @@ -99,6 +102,7 @@ def main(): ap.add_argument("--diagnose-only", action="store_true") ap.add_argument("--query25", action="store_true") ap.add_argument("--acceptance-file") + ap.add_argument("--query-profile", choices=query_lab_profiles.PROFILES, default="baseline") args = ap.parse_args() out = pathlib.Path(args.output).resolve() out.mkdir(parents=True, exist_ok=True) @@ -111,7 +115,8 @@ def main(): assert evidence["result"] == "small-smoke-passed" assert evidence["firstPodUID"] != evidence["secondPodUID"] acceptance_sha = hashlib.sha256(evidence_bytes).hexdigest() - objects = manifests(args.query25) + assert args.query25 or args.query_profile == "baseline" + objects = manifests(args.query25, args.query_profile) target_pod = objects[-1]["metadata"]["name"] runner_sha = hashlib.sha256(pathlib.Path(__file__).read_bytes()).hexdigest() mode = "query25-synthetic-only" if args.query25 else "attribution-only" if args.diagnose_only else "full-synthetic-recovery" diff --git a/scripts/local/node4-query-pilot/test_pilot.py b/scripts/local/node4-query-pilot/test_pilot.py index 33ff6a3e..54676fcc 100644 --- a/scripts/local/node4-query-pilot/test_pilot.py +++ b/scripts/local/node4-query-pilot/test_pilot.py @@ -86,3 +86,56 @@ class SafetyTests(unittest.TestCase): if __name__ == "__main__": unittest.main() + +class ProfileTests(unittest.TestCase): + def test_profiles_preserve_disk_security_and_both_io_layers(self): + profiles = pilot.query_lab_profiles + for name in profiles.PROFILES: + objects = pilot.manifests(True, name) + config, pod = objects[-2:] + p = profiles.profile(name) + profiles.verify_resources(pod, name) + profiles.verify_scripts(config['data'], name) + expected = ' '.join(f'{k}={p[k]}' for k in ('rbps', 'wbps', 'riops', 'wiops')) + for filename in ('init.sh', 'guard.sh'): + self.assertIn('252:0 ' + expected, config['data'][filename]) + self.assertIn('$loop_device ' + expected, config['data'][filename]) + self.assertIn('--wiredTigerCacheSizeGB ' + p['cacheGi'], config['data']['query25.sh']) + self.assertIn('WT cache drift', config['data']['verify25.sh']) + self.assertIn('truncate -s 26843545600', config['data']['init.sh']) + self.assertIn('80530636800', config['data']['query25.sh']) + self.assertIn('memory.max', config['data']['guard.sh']) + pod['spec']['containers'][0]['resources']['limits']['memory'] = '8Gi' + with self.assertRaises(AssertionError): + profiles.verify_resources(pod, name) + profiles.verify_scripts(config['data'], name) + + def test_unknown_profile_fails_closed(self): + with self.assertRaises(KeyError): + pilot.manifests(True, 'unlimited') + + def test_one_device_io_drift_is_rejected(self): + for filename in ('init.sh', 'guard.sh'): + data = pilot.manifests(True, 'balanced')[-2]['data'] + data[filename] = data[filename].replace('252:0 rbps=4194304', '252:0 rbps=2097152') + with self.assertRaises(AssertionError): + pilot.query_lab_profiles.verify_scripts(data, 'balanced') + +class RollbackTests(unittest.TestCase): + def test_rollback_rejects_storage_or_script_drift(self): + import copy + spec = importlib.util.spec_from_file_location('resource_stage', ROOT.parent / 'query-lab-resource-stage.py') + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + for name in pilot.query_lab_profiles.PROFILES: + objects = pilot.manifests(True, name) + saved = {'pod': objects[-1], 'config': objects[-2], 'quota': objects[1]} + module.validate_saved(saved, pilot) + for drift in ('node', 'path', 'script', 'resource'): + bad = copy.deepcopy(saved) + if drift == 'node': bad['pod']['spec']['nodeName'] = 'other-node' + elif drift == 'path': bad['pod']['spec']['volumes'][0]['hostPath']['path'] = '/var/lib/production' + elif drift == 'script': bad['config']['data']['init.sh'] += '\necho unreviewed\n' + else: bad['pod']['spec']['containers'][0]['resources']['limits']['cpu'] = '8' + with self.assertRaises(AssertionError, msg=f'{name}/{drift}'): + module.validate_saved(bad, pilot) diff --git a/scripts/local/query-lab-resource-stage.py b/scripts/local/query-lab-resource-stage.py new file mode 100644 index 00000000..bcaf3551 --- /dev/null +++ b/scripts/local/query-lab-resource-stage.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +"""Rebuild only the owned query25 lab Pod with an exact finite resource profile. + +Original objects are saved before mutation. --restore uses that snapshot and +preserves the backing filesystem; neither path writes customer data. +""" +import argparse +import importlib.util +import hashlib +import json +from pathlib import Path +import subprocess +import urllib.request +import time +import query_lab_profiles as profiles + +HERE = Path(__file__).resolve().parent +NS = 'droboticflow-query-lab' +NS_UID = '07d647bb-943c-472c-be0a-f6b1380d7105' +NODE = '10.189.16.4' +NODE_UID = 'ca3f254a-8452-4769-8929-6ec303905dbe' + + +def validate_saved(saved, pilot): + """Rollback accepts only this lab's fixed storage and reviewed scripts.""" + pod = saved['pod'] + assert pod['metadata']['name'] == 'node4-query25' and pod['metadata']['namespace'] == NS + spec = pod['spec'] + assert spec['nodeName'] == NODE and not any(spec.get(k, False) for k in ('hostNetwork', 'hostPID', 'hostIPC')) + assert len(spec['containers']) == len(spec['initContainers']) == 1 + for c in spec['containers'] + spec['initContainers']: + assert c['image'] == pilot.IMAGE and not c.get('envFrom') + assert {e['name'] for e in c['env']} == {'POD_UID', 'NODE_NAME', 'HOME', 'MONGOSH_DISABLE_TELEMETRY'} + assert spec['containers'][0]['command'] == ['bash', '/scripts/query25.sh'] + assert spec['initContainers'][0]['command'] == ['bash', '/scripts/init.sh'] + volumes = {v['name']: v for v in spec['volumes']} + assert set(volumes) == {'lab', 'cg', 'scripts', 'tmp'} + assert volumes['lab']['hostPath']['path'] == '/var/lib/mongo-cpu-node4-query25-20261113' + assert volumes['cg']['hostPath']['path'] == '/sys/fs/cgroup' + assert volumes['scripts']['configMap']['name'] == 'node4-query25-scripts' + name = pod['metadata'].get('annotations', {}).get('query-lab/profile', 'baseline') + profiles.verify_resources(pod, name) + assert saved['quota']['spec']['hard'] == profiles.quota(name) + expected = pilot.manifests(True, name)[-2]['data'] + actual = saved['config']['data'] + legacy = name == 'baseline' and hashlib.sha256(json.dumps(actual, sort_keys=True).encode()).hexdigest() == 'e5eebd9dacae29e817d90909fa97dd31e648ef8eaf4aa81f6bcb31ac58ee8eed' + assert actual == expected or legacy, 'rollback scripts are not an approved profile' + + +def main(): + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument('--kubeconfig', required=True) + ap.add_argument('--output', required=True) + ap.add_argument('--profile', choices=profiles.PROFILES, default='baseline') + ap.add_argument('--restore', help='original.json from the first stage; requires owned current UID') + ap.add_argument('--expected-pod-uid', required=True) + ap.add_argument('--execute', action='store_true') + args = ap.parse_args() + out = Path(args.output).resolve() + out.mkdir(parents=True, exist_ok=True) + assert not (out / 'original.json').exists(), 'never overwrite rollback evidence' + k = ['kubectl', '--kubeconfig', args.kubeconfig] + + def call(*cmd, data=None, timeout=30): + p = subprocess.run(k + list(cmd), input=data, text=True, capture_output=True, timeout=timeout) + if p.returncode: + raise RuntimeError(p.stderr[:2000]) + return p.stdout + + def get(*cmd): + return json.loads(call(*cmd, '-o', 'json')) + + assert get('get', 'namespace', NS)['metadata']['uid'] == NS_UID + node = get('get', 'node', NODE) + assert node['metadata']['uid'] == NODE_UID + assert args.restore or all(c['status'] == 'False' for c in node['status']['conditions'] if c['type'] in ('MemoryPressure', 'DiskPressure', 'PIDPressure')) + top = call('top', 'node', NODE, '--no-headers').split() + assert args.restore or (float(top[2].rstrip('%')) <= 25 and float(top[4].rstrip('%')) <= 60) + def prod_states(): + return {p['metadata']['uid']: [s.get('restartCount', 0) for s in p['status'].get('containerStatuses', [])] for p in get('-n', 'droboticflow-fanyu', 'get', 'pods')['items']} + + initial_production = prod_states() + + def health(): + with urllib.request.urlopen('http://120.48.180.80/api/ready', timeout=5) as r: + assert json.load(r)['ready'] is True + assert prod_states() == initial_production, 'production Pod/restart set changed' + + if not args.restore: + health() + (out / 'production-before.json').write_text(json.dumps(initial_production)) + pods = get('-n', NS, 'get', 'pods')['items'] + assert len(pods) == 1 and pods[0]['metadata']['name'] == 'node4-query25', 'stop/clean owned fixture before rebuild' + pod = pods[0] + assert pod['metadata']['uid'] == args.expected_pod_uid + assert pod['spec']['nodeName'] == NODE + quota = get('-n', NS, 'get', 'resourcequota', 'smoke-budget') + config = get('-n', NS, 'get', 'configmap', 'node4-query25-scripts') + original = {'pod': pod, 'quota': quota, 'config': config} + (out / 'original.json').write_text(json.dumps(original, indent=2)) + spec = importlib.util.spec_from_file_location('pilot', HERE / 'node4-query-pilot/pilot.py') + pilot = importlib.util.module_from_spec(spec) + spec.loader.exec_module(pilot) + objects = pilot.manifests(True, args.profile) + desired_pod = objects[-1] + desired_pod['spec']['activeDeadlineSeconds'] = 3600 + desired_config, desired_hard = objects[-2]['data'], profiles.quota(args.profile) + if args.restore: + saved = json.loads(Path(args.restore).read_text()) + validate_saved(saved, pilot) + desired_pod = {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {key: saved['pod']['metadata'][key] for key in ('name', 'namespace', 'labels')}, 'spec': saved['pod']['spec']} + if 'annotations' in saved['pod']['metadata']: + desired_pod['metadata']['annotations'] = saved['pod']['metadata']['annotations'] + desired_config, desired_hard = saved['config']['data'], saved['quota']['spec']['hard'] + (out / 'desired.json').write_text(json.dumps({'pod': desired_pod, 'data': desired_config, 'hard': desired_hard}, indent=2)) + (out / 'rollback.txt').write_text('Stop and UID-clean only the owned source fixture/policies/bridge first. Then run query-lab-resource-stage.py --restore ' + str(out / 'original.json') + ' --expected-pod-uid --execute with a NEW output directory. This restores exact scripts/quota/resources, remounts retained 25Gi data and preserves majority+j. No reformat, forced unmount, global cache drop, or disk resize.\n') + if not args.execute: + print('PLAN_ONLY', out) + return + delete = {'apiVersion': 'v1', 'kind': 'DeleteOptions', 'preconditions': {'uid': args.expected_pod_uid}} + call('delete', '--raw', f'/api/v1/namespaces/{NS}/pods/node4-query25', '-f', '-', data=json.dumps(delete)) + call('-n', NS, 'wait', '--for=delete', 'pod/node4-query25', '--timeout=90s', timeout=95) + # Preconditions prevent overwriting another operator's config or quota. + for kind, old, field, value in [('configmap', config, 'data', desired_config), ('resourcequota', quota, 'spec/hard', desired_hard)]: + before = old['data'] if field == 'data' else old['spec']['hard'] + fresh = get('-n', NS, 'get', kind, old['metadata']['name']) + assert fresh['metadata']['uid'] == old['metadata']['uid'] + assert (fresh['data'] if field == 'data' else fresh['spec']['hard']) == before + old = fresh # Quota status resourceVersion changes as the old Pod exits. + patch = [{'op': 'test', 'path': '/metadata/resourceVersion', 'value': old['metadata']['resourceVersion']}, {'op': 'test', 'path': '/' + field, 'value': before}, {'op': 'replace', 'path': '/' + field, 'value': value}] + call('-n', NS, 'patch', kind, old['metadata']['name'], '--type=json', '-p', json.dumps(patch)) + call('create', '-f', '-', data=json.dumps(desired_pod)) + current = get('-n', NS, 'get', 'pod', 'node4-query25') + (out / 'created.json').write_text(json.dumps({'uid': current['metadata']['uid'], 'profile': desired_pod['metadata'].get('annotations', {}).get('query-lab/profile', 'baseline'), 'restored': bool(args.restore)})) + started = time.monotonic() + while True: + current = get('-n', NS, 'get', 'pod', 'node4-query25') + assert current['metadata']['uid'] == json.loads((out / 'created.json').read_text())['uid'] + if not args.restore: + health() + if any(c['type'] == 'Ready' and c['status'] == 'True' for c in current['status'].get('conditions', [])): + break + assert current['status']['phase'] not in ('Failed', 'Succeeded'), 'lab startup failed' + assert time.monotonic() - started < 180, 'lab readiness deadline exceeded' + time.sleep(5) + verification = call('-n', NS, 'exec', 'node4-query25', '-c', 'probe', '--', 'bash', '/scripts/verify25.sh', timeout=90) + (out / 'verification.log').write_text(verification) + (out / 'after.json').write_text(json.dumps(get('-n', NS, 'get', 'pod', 'node4-query25'), indent=2)) + print('VERIFIED', current['metadata']['uid'], verification.strip()) + + +if __name__ == '__main__': + main() diff --git a/scripts/local/query-replica-cluster-fixture.py b/scripts/local/query-replica-cluster-fixture.py index 30205702..d24e485c 100644 --- a/scripts/local/query-replica-cluster-fixture.py +++ b/scripts/local/query-replica-cluster-fixture.py @@ -9,6 +9,9 @@ import json from pathlib import Path import subprocess import uuid +import sys +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import query_lab_profiles NODE = "10.189.16.4" NODE_UID = "ca3f254a-8452-4769-8929-6ec303905dbe" @@ -39,7 +42,7 @@ wait "$pid" return [ingress, egress, pod] -def verify_query_pod(query, expected_uid): +def verify_query_pod(query, expected_uid, query_profile="baseline"): # Rebuilds change UID. Bind that explicitly into the reviewed plan, while # preserving fixed namespace/node/image/storage/resource identity checks. assert str(uuid.UUID(expected_uid)) == expected_uid @@ -52,10 +55,11 @@ def verify_query_pod(query, expected_uid): assert container["image"] == IMAGE for kind in ("requests", "limits"): resources = container["resources"][kind] - assert resources["cpu"] == "1" and resources["memory"] == "1Gi" + assert all(resources.get(k) == v for k, v in query_lab_profiles.resources(query_profile).items()) # CCE admission injects one private-IP resource into both maps. assert set(resources) <= {"cpu", "memory", "cce.baidubce.com/ip"} assert resources.get("cce.baidubce.com/ip", "1") == "1" + query_lab_profiles.verify_resources(query, query_profile) volumes = {v["name"]: v for v in query["spec"]["volumes"]} assert volumes["lab"]["hostPath"]["path"] == "/var/lib/mongo-cpu-node4-query25-20261113" assert volumes["scripts"]["configMap"]["name"] == "node4-query25-scripts" @@ -72,7 +76,10 @@ def main(): ap.add_argument("--update-api-test", action="store_true", help="stage a new content-addressed test binary after its previous run ended") ap.add_argument("--confirm-plan") ap.add_argument("--query-pod-uid", default=QUERY_UID, help="exact inspected UID after an authorized synthetic Pod rebuild") + ap.add_argument("--query-profile", choices=query_lab_profiles.PROFILES, default="baseline") args = ap.parse_args() + global HARD + HARD = query_lab_profiles.quota(args.query_profile, fixture=True) query_uid = args.query_pod_uid out = Path(args.output).resolve() out.mkdir(parents=True, exist_ok=True) @@ -94,11 +101,13 @@ def main(): assert float(top[2].rstrip("%")) <= 25 and float(top[4].rstrip("%")) <= 60, "insufficient idle headroom for synthetic fixture" assert get("get", "namespace", NS)["metadata"]["uid"] == NS_UID query = get("-n", NS, "get", "pod", "node4-query25") - verify_query_pod(query, query_uid) + verify_query_pod(query, query_uid, args.query_profile) + query_lab_profiles.verify_scripts(get("-n", NS, "get", "configmap", "node4-query25-scripts")["data"], args.query_profile) + call("-n", NS, "exec", "node4-query25", "-c", "probe", "--", "bash", "/scripts/verify25.sh", timeout=90) ip = query["status"]["podIP"] assert ipaddress.IPv4Address(ip) in ipaddress.IPv4Network("10.189.16.0/24") quota = get("-n", NS, "get", "resourcequota", "smoke-budget") - original_quota = {"pods": "1", "requests.cpu": "1", "limits.cpu": "1", "requests.memory": "1Gi", "limits.memory": "1Gi", "services": "0", "persistentvolumeclaims": "0"} + original_quota = query_lab_profiles.quota(args.query_profile) recorded = None if args.finish_existing: recorded = json.loads((out / "created.json").read_text()) diff --git a/scripts/local/query-replica-isolated-compare.py b/scripts/local/query-replica-isolated-compare.py index c0afe164..ac5ecfbf 100644 --- a/scripts/local/query-replica-isolated-compare.py +++ b/scripts/local/query-replica-isolated-compare.py @@ -11,6 +11,9 @@ import signal import subprocess import time import urllib.request +import sys +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import query_lab_profiles NODE = "10.189.16.4" NODE_UID = "ca3f254a-8452-4769-8929-6ec303905dbe" @@ -132,6 +135,7 @@ def main(): ap.add_argument("--cold-write-probe", action="store_true", help="bounded 8MiB synthetic diagnostic; never HTTP capacity evidence") ap.add_argument("--periodic-refresh", action="store_true", help="explicit periodic display policy; same load and safety limits") ap.add_argument("--cold-pace-kib", type=int, choices=(0, 128), default=0) + ap.add_argument("--query-profile", choices=query_lab_profiles.PROFILES, default="baseline") args = ap.parse_args() assert args.cold_write_probe or args.cold_pace_kib == 0 assert not args.cold_write_probe or args.fixture_plan, "cold diagnostic requires the owned in-cluster fixture" @@ -150,7 +154,10 @@ def main(): query = get("-n", "droboticflow-query-lab", "get", "pod", "node4-query25") assert query["status"]["phase"] == "Running" assert query["spec"]["nodeName"] == NODE - assert all(query["spec"]["containers"][0]["resources"]["limits"].get(k) == v for k, v in {"cpu": "1", "memory": "1Gi"}.items()) + query_lab_profiles.verify_resources(query, args.query_profile) + query_scripts = get("-n", "droboticflow-query-lab", "get", "configmap", "node4-query25-scripts")["data"] + query_lab_profiles.verify_scripts(query_scripts, args.query_profile) + run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query25", "-c", "probe", "--", "bash", "/scripts/verify25.sh"]) fixture = None owned = None if args.fixture_plan: @@ -243,7 +250,7 @@ def main(): baseline = collect_baseline(sample) limits = {"ready": max(1, max(s["readySeconds"] for s in baseline) * 2), "pressure": max(0.5, max(s["protectedFullAvg10"] for s in baseline) + 0.5), "await": max(10, max(s["awaitMs"] for s in baseline) * 3)} - manifest = {"sourceCommit": run(["git", "-C", str(ROOT), "rev-parse", "HEAD"]), "sourceFingerprint": source_fingerprint(), "sourceContainer": source["Id"], "sourceImage": source["Image"], "sourceCPU": 4, "sourceMemory": 4_294_967_296, "queryPodUID": query["metadata"]["uid"], "queryImage": query["spec"]["containers"][0]["image"], "queryCPU": 1, "queryMemory": "1Gi", "tasks": args.tasks, "waves": args.waves, "maxActors": args.max_actors, "wireCompressionBothVariants": os.getenv("QUERY_COMPARE_WIRE_COMPRESSION", ""), "minimumPoolBothVariants": os.getenv("QUERY_COMPARE_MIN_POOL", "0"), "limits": limits, "limitations": ["source Docker VM shares CPU with existing local containers", "target has 2MiB read / 1MiB write and 64/32 IOPS hard limits", "short comparison; not 30-minute three-round production capacity acceptance", "health probes do not establish customer workflow latency"]} + manifest = {"sourceCommit": run(["git", "-C", str(ROOT), "rev-parse", "HEAD"]), "sourceFingerprint": source_fingerprint(), "sourceContainer": source["Id"], "sourceImage": source["Image"], "sourceCPU": 4, "sourceMemory": 4_294_967_296, "queryPodUID": query["metadata"]["uid"], "queryImage": query["spec"]["containers"][0]["image"], "queryCPU": int(query_lab_profiles.profile(args.query_profile)["cpu"]), "queryMemory": query_lab_profiles.resources(args.query_profile)["memory"], "queryProfile": query_lab_profiles.profile(args.query_profile), "queryScriptSHA256": hashlib.sha256(json.dumps(query_scripts, sort_keys=True).encode()).hexdigest(), "tasks": args.tasks, "waves": args.waves, "maxActors": args.max_actors, "wireCompressionBothVariants": os.getenv("QUERY_COMPARE_WIRE_COMPRESSION", ""), "minimumPoolBothVariants": os.getenv("QUERY_COMPARE_MIN_POOL", "0"), "limits": limits, "limitations": ["source Docker VM shares CPU with existing local containers", "target loop and backing device use the recorded finite queryProfile IO caps", "short comparison; not 30-minute three-round production capacity acceptance", "health probes do not establish customer workflow latency"]} manifest["baselineSamples"] = len(baseline) manifest["refreshMode"] = "periodic" if args.periodic_refresh else "continuous" manifest["baselineSeconds"] = baseline[-1]["time"] - baseline[0]["time"] diff --git a/scripts/local/query_lab_profiles.py b/scripts/local/query_lab_profiles.py new file mode 100644 index 00000000..4e6fc323 --- /dev/null +++ b/scripts/local/query_lab_profiles.py @@ -0,0 +1,71 @@ +"""Exact, bounded profiles for the synthetic node4 query25 experiment only.""" +from copy import deepcopy + +PROFILES = { + "baseline": {"cpu": "1", "memoryGi": 1, "cacheGi": "0.25", "rbps": 2097152, "wbps": 1048576, "riops": 64, "wiops": 32}, + "compute": {"cpu": "2", "memoryGi": 4, "cacheGi": "1", "rbps": 2097152, "wbps": 1048576, "riops": 64, "wiops": 32}, + "balanced": {"cpu": "2", "memoryGi": 4, "cacheGi": "1", "rbps": 4194304, "wbps": 2097152, "riops": 128, "wiops": 64}, + "read-headroom": {"cpu": "2", "memoryGi": 4, "cacheGi": "1", "rbps": 8388608, "wbps": 2097152, "riops": 256, "wiops": 64}, +} + + +def profile(name): + return deepcopy(PROFILES[name]) + + +def resources(name): + p = profile(name) + return {"cpu": p["cpu"], "memory": f'{p["memoryGi"]}Gi'} + + +def quota(name, fixture=False): + p = profile(name) + return {"pods": "2" if fixture else "1", "requests.cpu": str(int(p["cpu"]) + int(fixture)), + "limits.cpu": str(int(p["cpu"]) + 4 * int(fixture)), + "requests.memory": f'{p["memoryGi"] + 2 * int(fixture)}Gi', + "limits.memory": f'{p["memoryGi"] + 4 * int(fixture)}Gi', + "services": "0", "persistentvolumeclaims": "0"} + + +def render(objects, name): + """Update one complete query25 manifest; never changes disk or durability.""" + p = profile(name) + objects = deepcopy(objects) + config, pod = objects[-2:] + assert config["metadata"]["name"] == "node4-query25-scripts" + assert pod["metadata"]["name"] == "node4-query25" + original = 'rbps=2097152 wbps=1048576 riops=64 wiops=32' + io = ' '.join(f'{k}={p[k]}' for k in ('rbps', 'wbps', 'riops', 'wiops')) + for filename in ("init.sh", "guard.sh"): + assert original in config["data"][filename] + config["data"][filename] = config["data"][filename].replace(original, io) + config["data"]["guard.sh"] += f'\n[[ $(cat "$group/cpu.max") == "{int(p["cpu"]) * 100000} 100000" ]]\n[[ $(cat "$group/memory.max") == {p["memoryGi"] * 1024 ** 3} ]]\n' + config["data"]["query25.sh"] = config["data"]["query25.sh"].replace('--wiredTigerCacheSizeGB 0.25', '--wiredTigerCacheSizeGB ' + p['cacheGi']) + # Verify actual server settings, not only command text or admission requests. + config["data"]["verify25.sh"] = config["data"]["verify25.sh"].replace('const d=db.getSiblingDB', f'if(db.serverStatus().wiredTiger.cache["maximum bytes configured"]!=={int(float(p["cacheGi"]) * 1024 ** 3)})throw Error("WT cache drift");const d=db.getSiblingDB') + config["data"]["profile.json"] = __import__('json').dumps({"name": name, **p}, sort_keys=True) + pod["metadata"].setdefault("annotations", {})["query-lab/profile"] = name + for container in pod["spec"]["containers"] + pod["spec"]["initContainers"]: + container["resources"] = {kind: resources(name) for kind in ("requests", "limits")} + objects[1]["spec"]["hard"] = quota(name) + return objects + + +def verify_resources(pod, name): + assert pod["metadata"].get("annotations", {}).get("query-lab/profile", "baseline") == name, "profile identity drift" + for c in pod["spec"]["containers"] + pod["spec"]["initContainers"]: + for kind in ("requests", "limits"): + actual = c["resources"][kind] + assert all(actual.get(k) == v for k, v in resources(name).items()), "resource drift" + assert set(actual) <= {"cpu", "memory", "cce.baidubce.com/ip"} + + +def verify_scripts(data, name): + import json + p = profile(name) + assert json.loads(data['profile.json']) == {'name': name, **p}, 'script profile drift' + io = ' '.join(f'{k}={p[k]}' for k in ('rbps', 'wbps', 'riops', 'wiops')) + for filename in ('init.sh', 'guard.sh'): + assert '252:0 ' + io in data[filename], 'backing IO drift' + assert '$loop_device ' + io in data[filename], 'loop IO drift' + assert '--wiredTigerCacheSizeGB ' + p['cacheGi'] + ' ' in data['query25.sh'], 'cache command drift' diff --git a/scripts/local/test_query_replica_cluster_fixture.py b/scripts/local/test_query_replica_cluster_fixture.py index e9144701..97eee27d 100644 --- a/scripts/local/test_query_replica_cluster_fixture.py +++ b/scripts/local/test_query_replica_cluster_fixture.py @@ -17,6 +17,7 @@ class FixtureSafety(unittest.TestCase): import copy uid = "e1210ce7-16ee-4bda-9f8f-e3a02efefa01" pod = {"metadata": {"name": "node4-query25", "namespace": fixture.NS, "uid": uid, "labels": {"app": "node4-query25"}}, "status": {"phase": "Running"}, "spec": {"nodeName": fixture.NODE, "containers": [{"name": "probe", "image": fixture.IMAGE, "resources": {"requests": {"cpu": "1", "memory": "1Gi"}, "limits": {"cpu": "1", "memory": "1Gi"}}}], "volumes": [{"name": "lab", "hostPath": {"path": "/var/lib/mongo-cpu-node4-query25-20261113"}}, {"name": "scripts", "configMap": {"name": "node4-query25-scripts"}}]}} + pod["spec"]["initContainers"] = [copy.deepcopy(pod["spec"]["containers"][0])] fixture.verify_query_pod(pod, uid) admitted = copy.deepcopy(pod) for kind in ("requests", "limits"): diff --git a/scripts/verification-scopes.json b/scripts/verification-scopes.json index 40a65b9b..85b2f01a 100644 --- a/scripts/verification-scopes.json +++ b/scripts/verification-scopes.json @@ -975,6 +975,7 @@ "name": "node4-query-pilot-guardrails", "paths": [ "^scripts/local/node4-query-pilot/", + "^scripts/local/(query-lab-resource-stage|query_lab_profiles)\\.py$", "^docs/operations/query-replica-node4-(isolation-plan|smoke-results)\\.md$", "^scripts/local/(query-replica-(cluster-fixture|isolated-compare)|test_query_replica_cluster_fixture)\\.py$", "^docs/operations/query-replica-query25-comparison\\.md$"