📝 Plan isolated CDS storage for query verification
This commit is contained in:
parent
8e225d1158
commit
e226b1d155
@ -18,3 +18,7 @@ Worktree: local-data/worktrees/mongo-cpu
|
||||
## Current evidence
|
||||
|
||||
Local read-replica implementation and scoped regressions complete; capacity verification remains pending. Keep this worktree and isolated source/query lab for follow-up. Source and query feature switches stay OFF in production. Merge-back only after remaining acceptance and explicit integration request; never release this unmerged branch directly. See [verification](verification.md) for passing, failed and pending evidence.
|
||||
|
||||
## Storage selection follow-up
|
||||
|
||||
User selected new independent cloud disks + CSI PVs. Read-only discovery confirms only BOS CSI exists; managed CDS driver is a prerequisite. Prepared a disabled storage-only proposal (50Gi synthetic source +100Gi query, Retain, WaitForFirstConsumer) and operations plan. Driver compatibility/version, actual disk/node quotes and isolated compute allocation are pending; no install, purchase, cluster mutation or production data write authorized/performed. See docs/operations/query-replica-cds-plan.md.
|
||||
|
||||
49
docs/operations/query-replica-cds-plan.md
Normal file
49
docs/operations/query-replica-cds-plan.md
Normal file
@ -0,0 +1,49 @@
|
||||
# mongo-cpu 独立云盘与 CSI 验证方案
|
||||
|
||||
状态:选型已确认,未执行。用户确认采用“新建独立云盘及 CSI 持久卷”;这不是安装插件、购盘、创建公司资源或切换生产的授权。工作仍在 `codex/mongo-cpu`。
|
||||
|
||||
## 已只读核实
|
||||
|
||||
- 当前 Kubernetes 为1.35.3,三节点均在北京F可用区;仅安装并注册了 `csi-bosplugin`,没有CDS CSI。
|
||||
- 既有BOS卷和80Gi本地卷均已占用,保留原用途。现有BOS插件使用的kubelet路径为 `/var/lib/kubelet`,注册目录为其 `plugins_registry`。
|
||||
- CPU allocatable约54.6/7.8/7.8核,CPU requests约29.325/2.675/2.325核;两台BCC现有CPU limits已超过节点核数。瞬时空闲或requests可调度,不等于压测不会争用生产的突发资源。
|
||||
- 原始元数据及采样时间保存在忽略目录 `local-data/diagnostics/mongo-cpu/storage-*.json`,不包含Secret正文。
|
||||
|
||||
## 驱动和磁盘选择
|
||||
|
||||
采用百度CCE托管 **CSI CDS Plugin**,保留BOS插件。官方说明该组件涉及 `kube-system` 控制器、节点DaemonSet及集群RBAC;因此它是独立的集群变更,不能当成只创建一个PVC。以控制台当前提供且明确适配1.35的版本为准,先记录版本、镜像和资源配置,不直接安装未核验的上游master或仅凭“1.22+”推断完整兼容。[官方组件说明](https://intl.cloud.baidu.com/en/doc/CCE/s/Llc7917cx-intl-en)
|
||||
|
||||
首轮使用 `cloud_hp1` 通用型SSD,源库50Gi、查询库100Gi,合计150Gi、按量计费;之前20Gi源库草案调整为50Gi以符合官方SSD起售规格。`hp1`是另一种高性能云盘,不能仅因名称含“高性能”就替代已选择的SSD类型。实际可售SKU、最小容量及报价仍须以本账号本可用区确认为准。[官方磁盘类型](https://cloud.baidu.com/doc/BCC/s/6jwvyo0q2-en)、[创建磁盘规格](https://cloud.baidu.com/doc/BCC/s/gjwvyojxn-en)
|
||||
|
||||
存储提案为 [mongo-cpu-cds-storage.yaml](../../k8s/proposals/mongo-cpu-cds-storage.yaml),不被现有render/deploy入口引用:
|
||||
|
||||
| 项目 | 提案 |
|
||||
| --- | --- |
|
||||
| 命名空间 | `droboticflow-query-lab`,独立配额与Pod安全策略 |
|
||||
| StorageClass | `mongo-cpu-lab-cds-ssd`,不设为默认、不修改既有类 |
|
||||
| 创建时机 | `WaitForFirstConsumer`,消费者调度后才按所在可用区建盘 |
|
||||
| 文件系统 | 首轮两侧同为EXT4,检查实际挂载及目录fsync;XFS作为后续同条件验证项 |
|
||||
| 回收 | `Retain`;删除PVC不等于删除云盘或停止费用 |
|
||||
| 配额 | 初期2个PVC、150Gi;CPU/内存上限22核/38Gi,仅是上限,不预留节点资源 |
|
||||
| 应用 | 当前无Pod、Mongo、Secret、Service或生产连接配置 |
|
||||
|
||||
CDS动态供给会产生按量费用,且受可用区和节点挂载能力限制;先核对CSI节点注册、拓扑键和可用挂载槽位,再启动消费者。暂不猜测并写死驱动尚未报告的拓扑键。[官方CDS使用说明](https://intl.cloud.baidu.com/en/doc/CCE/s/Tjxppp24p-intl-en)
|
||||
|
||||
MongoDB支持合适的EXT4/XFS并要求目录fsync;官方推荐WiredTiger使用XFS。本轮显式记录EXT4,不能把盘类型或文件系统差异造成的收益全部记到代码优化。[MongoDB生产说明](https://www.mongodb.com/docs/manual/administration/production-notes/)
|
||||
|
||||
## 分两步执行,均需具体授权后开始
|
||||
|
||||
1. **存储冒烟**:确定托管驱动版本与云盘报价后,单列安装计划及上述新对象。只在新命名空间用不超过1核/1Gi的小型消费者,验证建盘、实际文件系统、目录fsync、重挂载及校验和;不得在未知设备运行格式化或裸盘写入命令。消费者镜像摘要与操作脚本须先入具体计划。本阶段不复制客户数据,不启用业务查询开关。
|
||||
2. **独立容量环境**:完整压测默认使用专用CPU节点池,起步预算4台8核/32Gi(计入4核/2核Pod装箱及节点预留),需另核SKU、节点费用、可用区和调度余量;不把现有CPU空闲当作可无干扰压测的承诺。固定源库4核/4Gi、查询库4核/8Gi,全部测试Pod合计起步21核/36Gi,包含两库、API、处理器、投影器、聚合器、Valkey及负载生成器。正式节点/Pod清单按实际可分配资源核算,不在本提案创建。
|
||||
|
||||
全链路候选仍须按项目主线规则先集成、验证、同步dev并审查镜像身份;本次不直接部署未合并分支。第一轮查询库为单成员副本集,仅用于性能和恢复验证,不算生产HA。三副本阶段各用独立100Gi卷,总容量和配额另审;现有单可用区只能验证节点故障,跨可用区容灾需要另行节点/卷布局。
|
||||
|
||||
## 验收与回退
|
||||
|
||||
存储先测fsync延迟、随机IO、吞吐、重挂载和重启一致性,再做81/200/500人完整HTTP及作业流、5分钟预热/30分钟稳态三轮,包含增量写放大、积压、缓存丢失和重启。原Mongo总CPU、扫描量和读取字节下降,结果/权限正确、延迟达标后才准备切读。
|
||||
|
||||
故障时先停测试消费者和压测,保留PV/PVC及证据;Retain盘继续收费,删除指定云盘必须单独核准。驱动回退通过托管组件流程,并先确认没有其他使用者和挂载;不得按名字批量删除CSI/RBAC或改动BOS。生产API、Worker、源Mongo记录及公网带宽本轮保持现状。
|
||||
|
||||
## 本地方案校验记录
|
||||
|
||||
本轮只修改方案和未接入部署入口的清单。`kubectl create --dry-run=client --validate=false` 已解析5个对象,未提交任何创建请求;新增文件格式检查通过。`verify:quick --base 8e225d11` 因新提案尚无行为映射返回incomplete,未将其记为CSI/挂载验收通过。实际CSI供给、磁盘IO和恢复仍须在独立授权后的环境验证。
|
||||
72
k8s/proposals/mongo-cpu-cds-storage.yaml
Normal file
72
k8s/proposals/mongo-cpu-cds-storage.yaml
Normal file
@ -0,0 +1,72 @@
|
||||
# PROPOSAL ONLY. No deployment/render script consumes this file.
|
||||
# Requires approved managed CDS driver/version, disk quote and change plan.
|
||||
# Scheduling a consumer can create billable disks; Retain preserves disks on PVC removal.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: droboticflow-query-lab
|
||||
labels:
|
||||
app.kubernetes.io/part-of: mongo-cpu-lab
|
||||
pod-security.kubernetes.io/enforce: restricted
|
||||
pod-security.kubernetes.io/enforce-version: v1.35
|
||||
---
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: mongo-cpu-lab-cds-ssd
|
||||
labels:
|
||||
app.kubernetes.io/part-of: mongo-cpu-lab
|
||||
provisioner: csi-cdsplugin
|
||||
reclaimPolicy: Retain
|
||||
volumeBindingMode: WaitForFirstConsumer
|
||||
allowVolumeExpansion: true
|
||||
parameters:
|
||||
paymentTiming: Postpaid
|
||||
storageType: cloud_hp1
|
||||
csi.storage.k8s.io/fstype: ext4
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ResourceQuota
|
||||
metadata:
|
||||
name: mongo-cpu-lab-budget
|
||||
namespace: droboticflow-query-lab
|
||||
spec:
|
||||
hard:
|
||||
requests.cpu: "22"
|
||||
limits.cpu: "22"
|
||||
requests.memory: 38Gi
|
||||
limits.memory: 38Gi
|
||||
requests.storage: 150Gi
|
||||
persistentvolumeclaims: "2"
|
||||
mongo-cpu-lab-cds-ssd.storageclass.storage.k8s.io/requests.storage: 150Gi
|
||||
mongo-cpu-lab-cds-ssd.storageclass.storage.k8s.io/persistentvolumeclaims: "2"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: synthetic-source-data
|
||||
namespace: droboticflow-query-lab
|
||||
labels:
|
||||
app.kubernetes.io/part-of: mongo-cpu-lab
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
volumeMode: Filesystem
|
||||
storageClassName: mongo-cpu-lab-cds-ssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 50Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: query-data
|
||||
namespace: droboticflow-query-lab
|
||||
labels:
|
||||
app.kubernetes.io/part-of: mongo-cpu-lab
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
volumeMode: Filesystem
|
||||
storageClassName: mongo-cpu-lab-cds-ssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 100Gi
|
||||
Loading…
x
Reference in New Issue
Block a user