From e226b1d1559e66d4889418850397669a06ec4c95 Mon Sep 17 00:00:00 2001 From: Codex Date: Thu, 12 Nov 2026 17:04:34 -0800 Subject: [PATCH] =?UTF-8?q?=F0=9F=93=9D=20Plan=20isolated=20CDS=20storage?= =?UTF-8?q?=20for=20query=20verification?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .scratch/mongo-cpu/work.md | 4 ++ docs/operations/query-replica-cds-plan.md | 49 +++++++++++++++ k8s/proposals/mongo-cpu-cds-storage.yaml | 72 +++++++++++++++++++++++ 3 files changed, 125 insertions(+) create mode 100644 docs/operations/query-replica-cds-plan.md create mode 100644 k8s/proposals/mongo-cpu-cds-storage.yaml diff --git a/.scratch/mongo-cpu/work.md b/.scratch/mongo-cpu/work.md index 1b6d2e03..67dc41a9 100644 --- a/.scratch/mongo-cpu/work.md +++ b/.scratch/mongo-cpu/work.md @@ -18,3 +18,7 @@ Worktree: local-data/worktrees/mongo-cpu ## Current evidence Local read-replica implementation and scoped regressions complete; capacity verification remains pending. Keep this worktree and isolated source/query lab for follow-up. Source and query feature switches stay OFF in production. Merge-back only after remaining acceptance and explicit integration request; never release this unmerged branch directly. See [verification](verification.md) for passing, failed and pending evidence. + +## Storage selection follow-up + +User selected new independent cloud disks + CSI PVs. Read-only discovery confirms only BOS CSI exists; managed CDS driver is a prerequisite. Prepared a disabled storage-only proposal (50Gi synthetic source +100Gi query, Retain, WaitForFirstConsumer) and operations plan. Driver compatibility/version, actual disk/node quotes and isolated compute allocation are pending; no install, purchase, cluster mutation or production data write authorized/performed. See docs/operations/query-replica-cds-plan.md. diff --git a/docs/operations/query-replica-cds-plan.md b/docs/operations/query-replica-cds-plan.md new file mode 100644 index 00000000..1b49441f --- /dev/null +++ b/docs/operations/query-replica-cds-plan.md @@ -0,0 +1,49 @@ +# mongo-cpu 独立云盘与 CSI 验证方案 + +状态:选型已确认,未执行。用户确认采用“新建独立云盘及 CSI 持久卷”;这不是安装插件、购盘、创建公司资源或切换生产的授权。工作仍在 `codex/mongo-cpu`。 + +## 已只读核实 + +- 当前 Kubernetes 为1.35.3,三节点均在北京F可用区;仅安装并注册了 `csi-bosplugin`,没有CDS CSI。 +- 既有BOS卷和80Gi本地卷均已占用,保留原用途。现有BOS插件使用的kubelet路径为 `/var/lib/kubelet`,注册目录为其 `plugins_registry`。 +- CPU allocatable约54.6/7.8/7.8核,CPU requests约29.325/2.675/2.325核;两台BCC现有CPU limits已超过节点核数。瞬时空闲或requests可调度,不等于压测不会争用生产的突发资源。 +- 原始元数据及采样时间保存在忽略目录 `local-data/diagnostics/mongo-cpu/storage-*.json`,不包含Secret正文。 + +## 驱动和磁盘选择 + +采用百度CCE托管 **CSI CDS Plugin**,保留BOS插件。官方说明该组件涉及 `kube-system` 控制器、节点DaemonSet及集群RBAC;因此它是独立的集群变更,不能当成只创建一个PVC。以控制台当前提供且明确适配1.35的版本为准,先记录版本、镜像和资源配置,不直接安装未核验的上游master或仅凭“1.22+”推断完整兼容。[官方组件说明](https://intl.cloud.baidu.com/en/doc/CCE/s/Llc7917cx-intl-en) + +首轮使用 `cloud_hp1` 通用型SSD,源库50Gi、查询库100Gi,合计150Gi、按量计费;之前20Gi源库草案调整为50Gi以符合官方SSD起售规格。`hp1`是另一种高性能云盘,不能仅因名称含“高性能”就替代已选择的SSD类型。实际可售SKU、最小容量及报价仍须以本账号本可用区确认为准。[官方磁盘类型](https://cloud.baidu.com/doc/BCC/s/6jwvyo0q2-en)、[创建磁盘规格](https://cloud.baidu.com/doc/BCC/s/gjwvyojxn-en) + +存储提案为 [mongo-cpu-cds-storage.yaml](../../k8s/proposals/mongo-cpu-cds-storage.yaml),不被现有render/deploy入口引用: + +| 项目 | 提案 | +| --- | --- | +| 命名空间 | `droboticflow-query-lab`,独立配额与Pod安全策略 | +| StorageClass | `mongo-cpu-lab-cds-ssd`,不设为默认、不修改既有类 | +| 创建时机 | `WaitForFirstConsumer`,消费者调度后才按所在可用区建盘 | +| 文件系统 | 首轮两侧同为EXT4,检查实际挂载及目录fsync;XFS作为后续同条件验证项 | +| 回收 | `Retain`;删除PVC不等于删除云盘或停止费用 | +| 配额 | 初期2个PVC、150Gi;CPU/内存上限22核/38Gi,仅是上限,不预留节点资源 | +| 应用 | 当前无Pod、Mongo、Secret、Service或生产连接配置 | + +CDS动态供给会产生按量费用,且受可用区和节点挂载能力限制;先核对CSI节点注册、拓扑键和可用挂载槽位,再启动消费者。暂不猜测并写死驱动尚未报告的拓扑键。[官方CDS使用说明](https://intl.cloud.baidu.com/en/doc/CCE/s/Tjxppp24p-intl-en) + +MongoDB支持合适的EXT4/XFS并要求目录fsync;官方推荐WiredTiger使用XFS。本轮显式记录EXT4,不能把盘类型或文件系统差异造成的收益全部记到代码优化。[MongoDB生产说明](https://www.mongodb.com/docs/manual/administration/production-notes/) + +## 分两步执行,均需具体授权后开始 + +1. **存储冒烟**:确定托管驱动版本与云盘报价后,单列安装计划及上述新对象。只在新命名空间用不超过1核/1Gi的小型消费者,验证建盘、实际文件系统、目录fsync、重挂载及校验和;不得在未知设备运行格式化或裸盘写入命令。消费者镜像摘要与操作脚本须先入具体计划。本阶段不复制客户数据,不启用业务查询开关。 +2. **独立容量环境**:完整压测默认使用专用CPU节点池,起步预算4台8核/32Gi(计入4核/2核Pod装箱及节点预留),需另核SKU、节点费用、可用区和调度余量;不把现有CPU空闲当作可无干扰压测的承诺。固定源库4核/4Gi、查询库4核/8Gi,全部测试Pod合计起步21核/36Gi,包含两库、API、处理器、投影器、聚合器、Valkey及负载生成器。正式节点/Pod清单按实际可分配资源核算,不在本提案创建。 + +全链路候选仍须按项目主线规则先集成、验证、同步dev并审查镜像身份;本次不直接部署未合并分支。第一轮查询库为单成员副本集,仅用于性能和恢复验证,不算生产HA。三副本阶段各用独立100Gi卷,总容量和配额另审;现有单可用区只能验证节点故障,跨可用区容灾需要另行节点/卷布局。 + +## 验收与回退 + +存储先测fsync延迟、随机IO、吞吐、重挂载和重启一致性,再做81/200/500人完整HTTP及作业流、5分钟预热/30分钟稳态三轮,包含增量写放大、积压、缓存丢失和重启。原Mongo总CPU、扫描量和读取字节下降,结果/权限正确、延迟达标后才准备切读。 + +故障时先停测试消费者和压测,保留PV/PVC及证据;Retain盘继续收费,删除指定云盘必须单独核准。驱动回退通过托管组件流程,并先确认没有其他使用者和挂载;不得按名字批量删除CSI/RBAC或改动BOS。生产API、Worker、源Mongo记录及公网带宽本轮保持现状。 + +## 本地方案校验记录 + +本轮只修改方案和未接入部署入口的清单。`kubectl create --dry-run=client --validate=false` 已解析5个对象,未提交任何创建请求;新增文件格式检查通过。`verify:quick --base 8e225d11` 因新提案尚无行为映射返回incomplete,未将其记为CSI/挂载验收通过。实际CSI供给、磁盘IO和恢复仍须在独立授权后的环境验证。 diff --git a/k8s/proposals/mongo-cpu-cds-storage.yaml b/k8s/proposals/mongo-cpu-cds-storage.yaml new file mode 100644 index 00000000..efb5dea4 --- /dev/null +++ b/k8s/proposals/mongo-cpu-cds-storage.yaml @@ -0,0 +1,72 @@ +# PROPOSAL ONLY. No deployment/render script consumes this file. +# Requires approved managed CDS driver/version, disk quote and change plan. +# Scheduling a consumer can create billable disks; Retain preserves disks on PVC removal. +apiVersion: v1 +kind: Namespace +metadata: + name: droboticflow-query-lab + labels: + app.kubernetes.io/part-of: mongo-cpu-lab + pod-security.kubernetes.io/enforce: restricted + pod-security.kubernetes.io/enforce-version: v1.35 +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: mongo-cpu-lab-cds-ssd + labels: + app.kubernetes.io/part-of: mongo-cpu-lab +provisioner: csi-cdsplugin +reclaimPolicy: Retain +volumeBindingMode: WaitForFirstConsumer +allowVolumeExpansion: true +parameters: + paymentTiming: Postpaid + storageType: cloud_hp1 + csi.storage.k8s.io/fstype: ext4 +--- +apiVersion: v1 +kind: ResourceQuota +metadata: + name: mongo-cpu-lab-budget + namespace: droboticflow-query-lab +spec: + hard: + requests.cpu: "22" + limits.cpu: "22" + requests.memory: 38Gi + limits.memory: 38Gi + requests.storage: 150Gi + persistentvolumeclaims: "2" + mongo-cpu-lab-cds-ssd.storageclass.storage.k8s.io/requests.storage: 150Gi + mongo-cpu-lab-cds-ssd.storageclass.storage.k8s.io/persistentvolumeclaims: "2" +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: synthetic-source-data + namespace: droboticflow-query-lab + labels: + app.kubernetes.io/part-of: mongo-cpu-lab +spec: + accessModes: [ReadWriteOnce] + volumeMode: Filesystem + storageClassName: mongo-cpu-lab-cds-ssd + resources: + requests: + storage: 50Gi +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: query-data + namespace: droboticflow-query-lab + labels: + app.kubernetes.io/part-of: mongo-cpu-lab +spec: + accessModes: [ReadWriteOnce] + volumeMode: Filesystem + storageClassName: mongo-cpu-lab-cds-ssd + resources: + requests: + storage: 100Gi