feat: 新增 GPU 节点画像与 3D 拓扑工具
- 新增只读 GPU 节点采集器,以统一 JSON 描述系统、CPU/NUMA、GPU、NVLink、PCIe、RDMA、网络与功耗信息 - 统一 Bond、独立 RoCE 和 InfiniBand 端点模型,支持 GPU 到 RDMA 端点的全局最优唯一匹配 - 亲和关系严格记录 nvidia-smi topo -m 证据,未暴露的 IB 设备不推测 GPU 连线 - 新增原生 HTML 运维画像和 Canvas 3D CT,可按 NUMA、PCIe 分组及 Bond 逐层查看物理成员端口 - 展示 CPU 亲和、P-State、当前功耗与 Power Limit,并提供 RDMA、Bond、IB、管理网络及整机健康检查 - 排除 CNI/veth 等虚拟接口对管理网络画像的干扰,普通管理 Bond 不计入 RDMA Bond 验收 - 提供脱敏 B300 真机示例、离线自包含页面、使用文档及 17 项解析和渲染回归测试 - 修正生成物忽略规则,确保两个 HTML 源模板能够随工具提交
This commit is contained in:
parent
219065eea1
commit
6ea2acf2d6
27
README.md
27
README.md
@ -19,6 +19,7 @@
|
|||||||
- [配置说明](#配置说明)
|
- [配置说明](#配置说明)
|
||||||
- [测试 SOP(标准操作流程)](#测试-sop标准操作流程)
|
- [测试 SOP(标准操作流程)](#测试-sop标准操作流程)
|
||||||
- [报告输出](#报告输出)
|
- [报告输出](#报告输出)
|
||||||
|
- [GPU 节点画像工具](#gpu-节点画像工具)
|
||||||
- [故障排除](#故障排除)
|
- [故障排除](#故障排除)
|
||||||
|
|
||||||
---
|
---
|
||||||
@ -42,6 +43,11 @@ servertest/
|
|||||||
│ ├── training_sim.py # 训练模拟
|
│ ├── training_sim.py # 训练模拟
|
||||||
│ └── report.py # 报告生成
|
│ └── report.py # 报告生成
|
||||||
├── scripts/
|
├── scripts/
|
||||||
|
│ ├── gpu_node_profile/ # JSON 驱动的运维画像与 3D CT
|
||||||
|
│ │ ├── gpu_node_profile.py # 只读采集与渲染入口
|
||||||
|
│ │ ├── templates/ # 两个独立 HTML 源模板
|
||||||
|
│ │ ├── examples/ # 脱敏真机 JSON 与自包含示例页面
|
||||||
|
│ │ └── tests/ # 解析与渲染回归测试
|
||||||
│ └── multinode_nccl/ # 独立 shell 脚本版 (历史备用)
|
│ └── multinode_nccl/ # 独立 shell 脚本版 (历史备用)
|
||||||
│ ├── nccl_env.sh # 环境变量 (MPI路径/IB网卡/bond0)
|
│ ├── nccl_env.sh # 环境变量 (MPI路径/IB网卡/bond0)
|
||||||
│ ├── hosts_all # 全量 34 台节点列表
|
│ ├── hosts_all # 全量 34 台节点列表
|
||||||
@ -677,6 +683,27 @@ python3 gpu_tester.py --test all --format html --output report.html
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## GPU 节点画像工具
|
||||||
|
|
||||||
|
`scripts/gpu_node_profile/` 提供独立、只读的 GPU 节点画像工具。采集器可在
|
||||||
|
一次执行中生成结构化 JSON,并由同一份 JSON 分别渲染运维画像和可逐层钻取
|
||||||
|
的 3D CT 页面。两个 HTML 源模板位于 `templates/`;`examples/` 包含脱敏
|
||||||
|
真机 JSON 和由其生成的两张自包含示例页面。支持 NUMA、PCIe、GPU、
|
||||||
|
NVLink/NVSwitch、RoCE Bond、无 Bond 的独立 RoCE/InfiniBand 端口、
|
||||||
|
物理网卡、CPU 亲和性和功耗信息。GPU 到 RDMA 端点的亲和关系来自
|
||||||
|
`nvidia-smi topo -m`,页面不会为矩阵未暴露的设备推测连线;详细命令与
|
||||||
|
采集边界见该目录的 `README.md`。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd scripts/gpu_node_profile
|
||||||
|
python3 gpu_node_profile.py \
|
||||||
|
--snapshot-out gpu_node_snapshot.json \
|
||||||
|
--output gpu_node_profile.html \
|
||||||
|
--topology-output gpu_topology_3d.html
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 故障排除
|
## 故障排除
|
||||||
|
|
||||||
| 问题 | 原因 | 解决方案 |
|
| 问题 | 原因 | 解决方案 |
|
||||||
|
|||||||
5
scripts/gpu_node_profile/.gitignore
vendored
Normal file
5
scripts/gpu_node_profile/.gitignore
vendored
Normal file
@ -0,0 +1,5 @@
|
|||||||
|
/gpu_node_snapshot*.json
|
||||||
|
/gpu_node_profile*.html
|
||||||
|
/gpu_topology_3d*.html
|
||||||
|
/b300_*_live.json
|
||||||
|
/b300_*_live.html
|
||||||
155
scripts/gpu_node_profile/README.md
Normal file
155
scripts/gpu_node_profile/README.md
Normal file
@ -0,0 +1,155 @@
|
|||||||
|
# GPU Node Profile
|
||||||
|
|
||||||
|
`gpu_node_profile.py` 通过只读命令采集 GPU、NVLink、PCIe/NUMA、RoCE
|
||||||
|
Bond、RDMA 和 InfiniBand 信息,生成通用结构化 JSON,并使用 `templates/`
|
||||||
|
中的两个独立 HTML 源模板,从同一份 JSON 渲染完全离线、无 Mermaid、
|
||||||
|
无 CDN 依赖的运维画像与 3D 拓扑 CT。
|
||||||
|
|
||||||
|
页面顶部按“系统与软件 / GPU / CPU与内存”展示整机概览;中部的
|
||||||
|
GPU-RDMA 拓扑支持“总览 / NUMA 分览”即时切换,并继续展示 InfiniBand、
|
||||||
|
OS 可见的带内管理网卡和可读取的 BMC 带外信息;最底部横向展示健康检查
|
||||||
|
与验证结论。
|
||||||
|
|
||||||
|
3D CT 页面使用原生 Canvas 对结构化 JSON 动态布局,包含 NUMA 半透明地域、
|
||||||
|
CPU Socket、PCIe Root Complex、PCIe Switch 分组、GPU、NVSwitch、RoCE、
|
||||||
|
InfiniBand 和管理网卡。NUMA 与 PCIe 均来自数据,不包含 B300 专用坐标,
|
||||||
|
因此同一渲染器也可用于 A800 等其他 GPU 服务器。
|
||||||
|
|
||||||
|
页面提供递归内部镜头,可按“整机 → NUMA → PCIe 分组 → Bond → 物理网卡”
|
||||||
|
逐层钻取。进入任一层后会隐藏外层与无关的同级设备,只保留当前壳体和后代
|
||||||
|
节点;顶部面包屑说明当前位置,“上一层”按原路径返回。双击壳体继续深入,
|
||||||
|
双击普通设备时也会沿该设备的父级路径每次深入一层,例如连续双击 GPU 会
|
||||||
|
依次进入整机、NUMA 和对应 PCIe 分组;到达最深层后才执行设备聚焦。
|
||||||
|
`Shift + 拖动`可平移,`Esc`返回整机外观。
|
||||||
|
|
||||||
|
Canvas 标签使用高对比浅色底与黑色粗体。每个 RoCE Bond 会作为透明聚合
|
||||||
|
容器展示,并在内部继续拆出物理成员网卡、速率和 BDF。NVLink/NVSwitch
|
||||||
|
连线在设备盒子上层高亮绘制,GPU 端点带编号;该 Fabric 只连接 GPU,
|
||||||
|
不会连接网卡。
|
||||||
|
|
||||||
|
网络模型同时支持两种形态:存在 Bond 时渲染
|
||||||
|
`GPU → RoCE Bond → 物理成员端口`;不存在 Bond 时根据
|
||||||
|
`nvidia-smi topo -m`、RDMA 名称和 PCIe 路径渲染
|
||||||
|
`GPU → 独立 RoCE/InfiniBand 端点`。多张 GPU 与多个等距端点会进行
|
||||||
|
全局最优、确定性的唯一分配,避免顺序贪心导致所有 GPU 错误指向同一张
|
||||||
|
网卡;端点少于 GPU 时才保留可共享的最近路径语义。亲和关系只接受
|
||||||
|
`nvidia-smi topo -m` 暴露的 `PIX/PXB/PHB/NODE/SYS` 证据;若独立 IB
|
||||||
|
未出现在矩阵中,会保留设备和 PCIe/NUMA 信息并标记 `not-exposed`,但不会
|
||||||
|
推测一条 GPU 连线。
|
||||||
|
|
||||||
|
JSON 中的 `network_endpoints` 是两张页面共用的标准网络模型:
|
||||||
|
|
||||||
|
```text
|
||||||
|
network_endpoints[]
|
||||||
|
id / type(bond|standalone) / protocol(RoCE|InfiniBand)
|
||||||
|
rdma / numa / bdf / state / topology_visibility
|
||||||
|
members[] # Bond 为多个物理口,直连端点为一个物理口
|
||||||
|
gpu_links[] # GPU、PIX/PXB/...、用途及证据来源
|
||||||
|
|
||||||
|
gpus[].primary_network_id # 指向主要计算 RDMA 端点
|
||||||
|
gpus[].network_links[] # 保留主要端点和额外最近端点
|
||||||
|
```
|
||||||
|
|
||||||
|
CPU 与内存概览、NUMA 标尺和每张 GPU 卡片会同时展示 `nvidia-smi topo -m`
|
||||||
|
解析出的推荐 CPU 范围及内存 NUMA。该信息描述硬件拓扑的近端路径,不表示
|
||||||
|
训练或推理进程已经通过 `taskset`、cgroup、Slurm 或 Kubernetes 完成实际绑核。
|
||||||
|
|
||||||
|
GPU 概览和单卡画像会展示采集时刻的 `power.draw`、当前 `power.limit` 与
|
||||||
|
P-State。Power Limit 表示当前生效的 GPU 功耗上限,不等于整机额定功耗;
|
||||||
|
P0 表示最高性能状态,也不单独等价于 GPU 持续满载。
|
||||||
|
|
||||||
|
## 在 GPU 节点执行
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 gpu_node_profile.py \
|
||||||
|
--output gpu_node_profile.html \
|
||||||
|
--topology-output gpu_topology_3d.html \
|
||||||
|
--snapshot-out gpu_node_snapshot.json \
|
||||||
|
--expected-gpus 8 \
|
||||||
|
--expected-bonds 8 \
|
||||||
|
--expected-ib 4
|
||||||
|
```
|
||||||
|
|
||||||
|
脚本不会修改网卡、驱动、Bond 或系统配置。唯一写操作是生成用户指定的
|
||||||
|
HTML/JSON 文件。`--expected-bonds` 校验的是 RDMA/RoCE Bond;普通管理
|
||||||
|
Bond 会保留在原始网卡信息中,但不会计入加速网络端点数量。
|
||||||
|
|
||||||
|
三个产物的关系:
|
||||||
|
|
||||||
|
```text
|
||||||
|
只读采集 → gpu_node_snapshot.json
|
||||||
|
├─→ gpu_node_profile.html
|
||||||
|
└─→ gpu_topology_3d.html
|
||||||
|
```
|
||||||
|
|
||||||
|
源码与演示数据的关系:
|
||||||
|
|
||||||
|
```text
|
||||||
|
gpu_node_profile.py # 只读采集、校验和渲染入口
|
||||||
|
templates/gpu_node_profile.html # 运维画像源模板
|
||||||
|
templates/gpu_topology_3d.html # 3D CT 源模板
|
||||||
|
examples/b300_demo.json # 由真机快照脱敏得到的演示数据
|
||||||
|
examples/b300_profile_demo.html # demo.json 生成的自包含示例
|
||||||
|
examples/b300_topology_3d_demo.html
|
||||||
|
```
|
||||||
|
|
||||||
|
最终生成的两张 HTML 会把 JSON 嵌入页面,因此可以直接双击、复制到其他机器
|
||||||
|
或离线归档。它们不会在打开时读取旁边的 `demo.json`;修改或替换 JSON 后,
|
||||||
|
需要重新运行 Python 渲染:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 gpu_node_profile.py \
|
||||||
|
--snapshot-in examples/b300_demo.json \
|
||||||
|
--output examples/b300_profile_demo.html \
|
||||||
|
--topology-output examples/b300_topology_3d_demo.html \
|
||||||
|
--expected-gpus 8 \
|
||||||
|
--expected-bonds 8 \
|
||||||
|
--expected-ib 4
|
||||||
|
```
|
||||||
|
|
||||||
|
## 使用已有快照离线渲染
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 gpu_node_profile.py \
|
||||||
|
--snapshot-in examples/b300_demo.json \
|
||||||
|
--output b300_profile_demo.html \
|
||||||
|
--topology-output b300_topology_3d_demo.html \
|
||||||
|
--expected-gpus 8 \
|
||||||
|
--expected-bonds 8 \
|
||||||
|
--expected-ib 4
|
||||||
|
```
|
||||||
|
|
||||||
|
只采集 JSON 并写到标准输出,适合通过 SSH 在远端只读采集、在本地渲染:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 gpu_node_profile.py --snapshot-stdout
|
||||||
|
```
|
||||||
|
|
||||||
|
## 测试
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m unittest discover -s tests -v
|
||||||
|
```
|
||||||
|
|
||||||
|
## 采集边界
|
||||||
|
|
||||||
|
- `nvcc` 表示当前 PATH 指向的 CUDA Toolkit,不代表容器或应用实际加载的
|
||||||
|
CUDA Runtime。
|
||||||
|
- LACP Bond 展示的是成员链路聚合容量,不承诺单连接可达到聚合带宽。
|
||||||
|
- 虚拟机中的 PCIe/NUMA 数据表示 Guest 可见拓扑,不能代替宿主机物理布线
|
||||||
|
和交换机侧验证。
|
||||||
|
- `pci_fabric` 保存 Guest 可见的 PCIe 祖先路径、桥和端点。3D 页面中的空间
|
||||||
|
坐标由这些逻辑关系动态生成,不代表设备在机箱中的厘米级真实位置。
|
||||||
|
- `PIX/PXB/PHB/NODE/SYS` 表示拓扑距离。`PIX` 可作为 GPUDirect RDMA 的
|
||||||
|
良好候选路径,但只有完成驱动能力和实际 RDMA 传输验证后才能标记为
|
||||||
|
`GDR Verified`。
|
||||||
|
- CPU 亲和表示 NVIDIA 拓扑给出的推荐 CPU/内存范围,不是运行中进程的
|
||||||
|
`Cpus_allowed_list`。评估实际 H2D 性能还需结合进程绑核、内存落点、
|
||||||
|
pinned memory 和带宽实测。
|
||||||
|
- GPU 功耗为报告生成时的一次瞬时采样;周期性训练负载的平均功耗和峰值
|
||||||
|
需要连续采样。整机功耗还需结合 BMC/DCMI 或机柜 PDU,不能简单等同于
|
||||||
|
所有 GPU Power Limit 之和。
|
||||||
|
- 独立BMC口通常不进入宿主OS网络命名空间。脚本会尝试从常见LAN通道只读
|
||||||
|
执行 `ipmitool lan print`;读取不到时会在画像中明确标记“OS不可见”。
|
||||||
|
- 管理网络区域只渲染物理 PCI 网卡、Bond 或默认路由接口;CNI、veth 等
|
||||||
|
容器虚拟接口仍可保留在原始 JSON 的 `nics` 中,但不会污染整机画像。
|
||||||
3458
scripts/gpu_node_profile/examples/b300_demo.json
Normal file
3458
scripts/gpu_node_profile/examples/b300_demo.json
Normal file
File diff suppressed because it is too large
Load Diff
4370
scripts/gpu_node_profile/examples/b300_profile_demo.html
Normal file
4370
scripts/gpu_node_profile/examples/b300_profile_demo.html
Normal file
File diff suppressed because one or more lines are too long
980
scripts/gpu_node_profile/examples/b300_topology_3d_demo.html
Normal file
980
scripts/gpu_node_profile/examples/b300_topology_3d_demo.html
Normal file
File diff suppressed because one or more lines are too long
2155
scripts/gpu_node_profile/gpu_node_profile.py
Executable file
2155
scripts/gpu_node_profile/gpu_node_profile.py
Executable file
File diff suppressed because it is too large
Load Diff
246
scripts/gpu_node_profile/templates/gpu_node_profile.html
Normal file
246
scripts/gpu_node_profile/templates/gpu_node_profile.html
Normal file
@ -0,0 +1,246 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||||
|
<meta name="color-scheme" content="dark">
|
||||||
|
<title>@@TITLE@@</title>
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
--bg:#07100e; --surface:#0d1916; --surface-2:#12231e; --line:#284238;
|
||||||
|
--text:#edf8f3; --muted:#8da79b; --green:#76b900; --green-2:#a5e34b;
|
||||||
|
--cyan:#39d5d0; --amber:#ffbd4a; --red:#ff6b6b; --shadow:0 24px 60px rgba(0,0,0,.28);
|
||||||
|
}
|
||||||
|
* { box-sizing:border-box; }
|
||||||
|
html { background:var(--bg); }
|
||||||
|
body {
|
||||||
|
margin:0; color:var(--text); background:
|
||||||
|
radial-gradient(circle at 18% -5%,rgba(118,185,0,.14),transparent 29rem),
|
||||||
|
radial-gradient(circle at 92% 8%,rgba(57,213,208,.08),transparent 24rem), var(--bg);
|
||||||
|
font:14px/1.5 Inter,ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;
|
||||||
|
}
|
||||||
|
body::before {
|
||||||
|
content:""; position:fixed; inset:0; pointer-events:none; opacity:.26;
|
||||||
|
background-image:linear-gradient(rgba(255,255,255,.025) 1px,transparent 1px),
|
||||||
|
linear-gradient(90deg,rgba(255,255,255,.025) 1px,transparent 1px);
|
||||||
|
background-size:32px 32px;
|
||||||
|
}
|
||||||
|
.hero,main,footer { width:min(1680px,calc(100% - 48px)); margin-inline:auto; position:relative; }
|
||||||
|
.hero { padding:44px 0 28px; display:flex; align-items:flex-end; justify-content:space-between; gap:24px; }
|
||||||
|
.kicker,.eyebrow { color:var(--green-2); font-size:11px; font-weight:800; letter-spacing:.18em; }
|
||||||
|
h1 { margin:6px 0 2px; font-size:clamp(34px,4vw,58px); letter-spacing:-.045em; line-height:1; }
|
||||||
|
.hero p,.gpu-card p,.nic-card p { color:var(--muted); margin:8px 0 0; }
|
||||||
|
.hero-actions { display:flex; align-items:center; gap:12px; }
|
||||||
|
button {
|
||||||
|
border:1px solid var(--line); background:var(--surface-2); color:var(--text);
|
||||||
|
border-radius:12px; padding:10px 14px; font-weight:750; cursor:pointer;
|
||||||
|
}
|
||||||
|
button:hover { border-color:var(--green); }
|
||||||
|
.overall { display:flex; gap:9px; align-items:center; font-weight:800; padding:10px 14px; border-radius:999px; border:1px solid var(--line); }
|
||||||
|
.overall i { width:9px; height:9px; border-radius:50%; box-shadow:0 0 18px currentColor; }
|
||||||
|
.overall.healthy { color:var(--green-2); }.overall.healthy i{background:var(--green-2)}
|
||||||
|
.overall.attention { color:var(--amber); }.overall.attention i{background:var(--amber)}
|
||||||
|
.overall.critical { color:var(--red); }.overall.critical i{background:var(--red)}
|
||||||
|
.overview-grid { display:grid; grid-template-columns:1.15fr .9fr 1fr; gap:12px; margin-bottom:14px; }
|
||||||
|
.overview-card,.panel {
|
||||||
|
background:linear-gradient(145deg,rgba(18,35,30,.94),rgba(10,21,18,.96));
|
||||||
|
border:1px solid var(--line); box-shadow:var(--shadow);
|
||||||
|
}
|
||||||
|
.overview-card { min-height:290px; border-radius:18px; padding:20px; overflow:hidden; }
|
||||||
|
.overview-head { display:flex; align-items:flex-start; justify-content:space-between; gap:14px; }
|
||||||
|
.overview-lead { color:var(--muted); margin:12px 0 0; min-height:42px; }
|
||||||
|
.overview-primary { display:flex; align-items:baseline; gap:9px; margin:8px 0 17px; }
|
||||||
|
.overview-primary b { font-size:52px; line-height:1; letter-spacing:-.055em; }
|
||||||
|
.overview-primary span { color:var(--muted); font-weight:750; }
|
||||||
|
.gpu-power-summary { margin:-5px 0 12px; }
|
||||||
|
.gpu-power-summary>div:first-child { display:flex; justify-content:space-between; align-items:baseline; gap:10px; font-size:10px; color:var(--muted); }
|
||||||
|
.gpu-power-summary>div:first-child b { color:var(--text); font-size:12px; }
|
||||||
|
.gpu-power-summary>small { display:block; margin-top:4px; color:var(--muted); font-size:8px; }
|
||||||
|
.overview-metrics { display:grid; grid-template-columns:repeat(2,1fr); gap:1px; background:var(--line); border:1px solid var(--line); border-radius:10px; overflow:hidden; }
|
||||||
|
.overview-metrics span { background:rgba(7,16,14,.72); padding:10px; min-width:0; }
|
||||||
|
.overview-metrics b,.overview-metrics small { display:block; }
|
||||||
|
.overview-metrics b { font-size:17px; }.overview-metrics small { color:var(--muted); margin-top:2px; font-size:9px; }
|
||||||
|
.overview-facts { display:grid; grid-template-columns:repeat(2,1fr); gap:1px; background:var(--line); border:1px solid var(--line); border-radius:10px; overflow:hidden; margin-top:12px; }
|
||||||
|
.overview-facts div { background:rgba(7,16,14,.72); padding:9px 10px; min-width:0; }
|
||||||
|
.overview-facts dt { font-size:9px; letter-spacing:.08em; text-transform:uppercase; }
|
||||||
|
.overview-facts dd { text-align:left; margin-top:3px; font-size:11px; font-weight:700; overflow-wrap:anywhere; }
|
||||||
|
.overview-facts small { color:var(--muted); }
|
||||||
|
.compact { padding:7px 9px; font-size:11px; }
|
||||||
|
.memory-row { display:flex; justify-content:space-between; gap:10px; color:var(--muted); font-size:10px; }
|
||||||
|
.memory-row b { color:var(--text); }
|
||||||
|
.memory-meter { height:6px; background:#172821; border-radius:10px; margin:7px 0 14px; overflow:hidden; }
|
||||||
|
.memory-meter i { display:block; height:100%; background:linear-gradient(90deg,var(--cyan),var(--green-2)); }
|
||||||
|
.cpu-metrics { grid-template-columns:repeat(3,1fr); }
|
||||||
|
.cpu-affinity-summary { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:6px; margin-top:9px; }
|
||||||
|
.cpu-affinity-summary span { border:1px solid var(--line); background:rgba(7,16,14,.55); border-radius:8px; padding:7px 8px; min-width:0; }
|
||||||
|
.cpu-affinity-summary b,.cpu-affinity-summary small { display:block; }
|
||||||
|
.cpu-affinity-summary b { color:var(--green-2); font-size:10px; }.cpu-affinity-summary small { color:var(--muted); font-size:8px; overflow-wrap:anywhere; }
|
||||||
|
.panel { border-radius:20px; padding:22px; margin-bottom:14px; overflow:hidden; }
|
||||||
|
.section-head { display:flex; justify-content:space-between; align-items:flex-start; gap:20px; margin-bottom:20px; }
|
||||||
|
h2 { margin:3px 0 0; font-size:24px; letter-spacing:-.025em; }
|
||||||
|
.legend { display:flex; flex-wrap:wrap; gap:7px; justify-content:flex-end; }
|
||||||
|
.badge { display:inline-flex; align-items:center; width:max-content; border:1px solid var(--line); color:var(--muted); border-radius:999px; padding:3px 8px; font-size:10px; font-weight:850; letter-spacing:.05em; white-space:nowrap; }
|
||||||
|
.badge.ok { color:var(--green-2); border-color:rgba(165,227,75,.34); background:rgba(118,185,0,.09); }
|
||||||
|
.badge.warn { color:var(--amber); border-color:rgba(255,189,74,.35); background:rgba(255,189,74,.08); }
|
||||||
|
.badge.bad { color:var(--red); border-color:rgba(255,107,107,.35); background:rgba(255,107,107,.08); }
|
||||||
|
.topology-viewbar { display:flex; align-items:center; justify-content:space-between; gap:14px; margin:-5px 0 14px; padding:10px 12px; border:1px solid var(--line); border-radius:12px; background:rgba(7,16,14,.48); }
|
||||||
|
.topology-viewbar b,.topology-viewbar small { display:block; }
|
||||||
|
.topology-viewbar b { font-size:11px; }.topology-viewbar small { color:var(--muted); font-size:9px; margin-top:2px; }
|
||||||
|
.numa-switcher { display:flex; align-items:center; gap:4px; padding:3px; border:1px solid var(--line); border-radius:10px; background:#07100e; }
|
||||||
|
.numa-switch { min-width:72px; padding:6px 10px; border:0; border-radius:7px; background:transparent; color:var(--muted); font-size:10px; }
|
||||||
|
.numa-switch:hover { border-color:transparent; color:var(--text); }
|
||||||
|
.numa-switch.active { color:#07100e; background:var(--green-2); box-shadow:0 0 18px rgba(165,227,75,.12); }
|
||||||
|
.numa-ruler { display:grid; grid-template-columns:1fr 1fr; margin:0 0 8px; gap:2px; }
|
||||||
|
.numa-ruler span { text-align:center; padding:5px; color:var(--muted); font-size:11px; font-weight:800; letter-spacing:.14em; border-bottom:1px solid var(--line); }
|
||||||
|
.numa-ruler small { display:block; margin-top:2px; color:var(--green-2); font-size:9px; letter-spacing:.04em; }
|
||||||
|
.affinity-note { color:var(--muted); text-align:center; font-size:9px; margin:3px 0 10px; }
|
||||||
|
.nvlink-fabric {
|
||||||
|
border:1px solid rgba(118,185,0,.5); background:linear-gradient(90deg,rgba(118,185,0,.18),rgba(57,213,208,.08),rgba(118,185,0,.18));
|
||||||
|
border-radius:13px; padding:11px 16px; display:flex; align-items:center; justify-content:center; gap:18px; color:var(--green-2); margin-bottom:16px;
|
||||||
|
}
|
||||||
|
.nvlink-fabric span { font-size:11px; letter-spacing:.12em; font-weight:900; }
|
||||||
|
.nvlink-fabric small { color:var(--muted); }
|
||||||
|
.rails { display:grid; gap:10px; }
|
||||||
|
.topology-panel { overflow-x:auto; }
|
||||||
|
.rail { position:relative; display:grid; grid-template-rows:auto 54px 1fr; }
|
||||||
|
.rail[hidden],.numa-ruler [hidden] { display:none!important; }
|
||||||
|
.rail::before { content:""; position:absolute; z-index:0; left:50%; top:150px; bottom:150px; width:2px; background:linear-gradient(var(--green),var(--cyan)); opacity:.6; }
|
||||||
|
.gpu-card,.nic-card,.ib-card { position:relative; z-index:1; border:1px solid var(--line); border-radius:14px; background:rgba(7,16,14,.92); }
|
||||||
|
.gpu-card { padding:13px; border-top:3px solid var(--green); min-height:278px; }
|
||||||
|
.card-head { display:flex; justify-content:space-between; align-items:center; gap:8px; }
|
||||||
|
.gpu-card h3,.nic-card h4,.ib-card h4 { margin:10px 0 0; line-height:1.14; }
|
||||||
|
.gpu-card h3 { min-height:33px; font-size:15px; }
|
||||||
|
.mono { font:10px/1.4 ui-monospace,SFMono-Regular,Menlo,monospace; }
|
||||||
|
.gpu-affinity { display:grid; grid-template-columns:1.4fr .6fr; gap:5px; margin-top:8px; }
|
||||||
|
.gpu-affinity span { min-width:0; background:var(--surface); border:1px solid var(--line); border-radius:7px; padding:5px 6px; }
|
||||||
|
.gpu-affinity small,.gpu-affinity b { display:block; }
|
||||||
|
.gpu-affinity small { color:var(--muted); font-size:7px; letter-spacing:.08em; }
|
||||||
|
.gpu-affinity b { color:var(--green-2); margin-top:1px; font:9px/1.25 ui-monospace,monospace; overflow-wrap:anywhere; }
|
||||||
|
.gpu-stats { display:grid; grid-template-columns:1.05fr .75fr .9fr; gap:3px; margin-top:15px; }
|
||||||
|
.gpu-stats span { min-width:0; }.gpu-stats b,.gpu-stats small { display:block; }
|
||||||
|
.gpu-stats b { font-size:12px; }.gpu-stats small { color:var(--muted); font-size:9px; }
|
||||||
|
.meter { height:4px; background:#172821; border-radius:10px; margin-top:11px; overflow:hidden; }
|
||||||
|
.meter i { display:block; height:100%; background:linear-gradient(90deg,var(--green),var(--green-2)); }
|
||||||
|
.gpu-power { margin-top:10px; padding-top:8px; border-top:1px solid var(--line); }
|
||||||
|
.gpu-power>div:first-child { display:flex; align-items:flex-start; justify-content:space-between; gap:5px; }
|
||||||
|
.gpu-power span,.gpu-power b,.gpu-power small { display:block; min-width:0; }
|
||||||
|
.gpu-power b { font-size:10px; }.gpu-power small { color:var(--muted); font-size:7px; margin-top:1px; }
|
||||||
|
.gpu-power em { flex:0 0 auto; color:var(--cyan); border:1px solid rgba(57,213,208,.3); border-radius:999px; padding:2px 5px; font:normal 800 8px/1.2 ui-monospace,monospace; }
|
||||||
|
.power-meter { height:5px; margin-top:6px; overflow:hidden; border-radius:999px; background:#172821; }
|
||||||
|
.power-meter i { display:block; height:100%; background:linear-gradient(90deg,var(--cyan),var(--green-2),var(--amber)); }
|
||||||
|
.relation { position:relative; z-index:2; align-self:center; justify-self:center; display:flex; align-items:center; flex-direction:column; gap:3px; color:var(--muted); font-size:9px; }
|
||||||
|
.nic-card { padding:13px; border-top:3px solid var(--cyan); min-height:234px; }
|
||||||
|
.nic-card h4,.ib-card h4 { font-size:18px; }.nic-card p { font:10px ui-monospace,monospace; }
|
||||||
|
.bond-capacity { display:flex; align-items:baseline; gap:6px; margin:14px 0 10px; }
|
||||||
|
.bond-capacity b { font-size:24px; }.bond-capacity span { color:var(--muted); font-size:9px; }
|
||||||
|
.ports { display:grid; gap:6px; }
|
||||||
|
.port { display:flex; align-items:center; gap:8px; background:var(--surface); border:1px solid var(--line); border-radius:8px; padding:7px; }
|
||||||
|
.port-dot { width:7px; height:7px; flex:0 0 auto; border-radius:50%; background:var(--muted); }
|
||||||
|
.port.up .port-dot { background:var(--green-2); box-shadow:0 0 9px rgba(165,227,75,.6); }
|
||||||
|
.port.down .port-dot { background:var(--red); }
|
||||||
|
.port b,.port small { display:block; }.port b { font-size:11px; }.port small { color:var(--muted); font:8px ui-monospace,monospace; margin-top:2px; }
|
||||||
|
.capacity-note { color:var(--muted); margin:16px 0 0; font-size:11px; }
|
||||||
|
.network-grid { display:grid; grid-template-columns:1fr 1fr; gap:14px; }
|
||||||
|
.ib-grid { display:grid; grid-template-columns:repeat(4,minmax(0,1fr)); gap:10px; }
|
||||||
|
.ib-card { min-width:0; padding:14px; border-top:3px solid var(--amber); overflow:hidden; }
|
||||||
|
.ib-card .card-head { align-items:flex-start; flex-wrap:wrap; row-gap:7px; }
|
||||||
|
.ib-card .card-head .eyebrow { min-width:0; overflow-wrap:anywhere; font-size:9px; letter-spacing:.13em; }
|
||||||
|
.ib-card .card-head .badge { flex:0 0 auto; margin-left:auto; max-width:100%; padding:3px 6px; font-size:8px; letter-spacing:.03em; }
|
||||||
|
.ib-rate { color:var(--amber); font-size:30px; font-weight:850; margin:13px 0; }
|
||||||
|
.ib-rate small { font-size:11px; margin-left:3px; color:var(--muted); }
|
||||||
|
dl { margin:0; }.ib-card dl { display:grid; gap:5px; }
|
||||||
|
.ib-card dl div { display:flex; justify-content:space-between; gap:8px; font-size:10px; }
|
||||||
|
.ib-card dd { min-width:0; overflow-wrap:anywhere; }
|
||||||
|
dt { color:var(--muted); } dd { margin:0; text-align:right; }
|
||||||
|
.management-grid { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:10px; }
|
||||||
|
.management-card { border:1px solid var(--line); border-top:3px solid var(--cyan); border-radius:14px; background:rgba(7,16,14,.92); padding:14px; min-height:205px; }
|
||||||
|
.management-card.bmc-card { border-top-color:#8da79b; }
|
||||||
|
.management-card.unavailable { border-top-color:var(--line); }
|
||||||
|
.management-title { display:flex; justify-content:space-between; align-items:baseline; gap:10px; margin-top:12px; }
|
||||||
|
.management-title h4 { margin:0; font-size:19px; }.management-title b { color:var(--cyan); font-size:18px; }
|
||||||
|
.management-card .address { color:var(--text); background:var(--surface); border:1px solid var(--line); border-radius:8px; padding:8px; min-height:38px; font:10px/1.5 ui-monospace,monospace; }
|
||||||
|
.management-card dl { display:grid; gap:4px; }
|
||||||
|
.management-card dl div { display:flex; justify-content:space-between; gap:8px; font-size:9px; }
|
||||||
|
.management-card .card-note { color:var(--muted); font-size:10px; margin-top:12px; }
|
||||||
|
.health-panel { margin-bottom:14px; }
|
||||||
|
.checks { list-style:none; padding:0; margin:0; display:grid; grid-template-columns:repeat(auto-fit,minmax(175px,1fr)); gap:8px; }
|
||||||
|
.check { display:flex; align-items:center; gap:11px; padding:12px; min-height:68px; border:1px solid var(--line); border-radius:10px; background:rgba(7,16,14,.55); }
|
||||||
|
.check-icon { width:25px; height:25px; display:grid; place-items:center; border-radius:50%; font-weight:900; flex:0 0 auto; }
|
||||||
|
.check.ok .check-icon { color:var(--green-2); background:rgba(118,185,0,.13); }
|
||||||
|
.check.warning .check-icon { color:var(--amber); background:rgba(255,189,74,.12); }
|
||||||
|
.check.critical .check-icon { color:var(--red); background:rgba(255,107,107,.12); }
|
||||||
|
.check b,.check small { display:block; }.check small { color:var(--muted); }
|
||||||
|
.ghost { background:transparent; }
|
||||||
|
pre { white-space:pre-wrap; word-break:break-word; background:#050b09; color:#b8cdc3; border:1px solid var(--line); border-radius:12px; padding:16px; max-height:480px; overflow:auto; font:11px/1.6 ui-monospace,monospace; }
|
||||||
|
dialog { width:min(1100px,calc(100% - 32px)); max-height:86vh; color:var(--text); background:var(--surface); border:1px solid var(--line); border-radius:18px; padding:20px; box-shadow:0 35px 100px rgba(0,0,0,.65); }
|
||||||
|
dialog::backdrop { background:rgba(0,0,0,.72); backdrop-filter:blur(5px); }
|
||||||
|
.dialog-head { display:flex; align-items:flex-start; justify-content:space-between; gap:20px; }
|
||||||
|
dialog pre { max-height:68vh; }
|
||||||
|
.empty-state { color:var(--muted); border:1px dashed var(--line); border-radius:12px; padding:30px; text-align:center; grid-column:1/-1; }
|
||||||
|
footer { padding:8px 0 30px; color:var(--muted); display:flex; justify-content:space-between; font-size:11px; }
|
||||||
|
@media (max-width:1100px) {
|
||||||
|
.overview-grid { grid-template-columns:1fr 1fr; }
|
||||||
|
.system-overview { grid-column:1/-1; }
|
||||||
|
.network-grid { grid-template-columns:1fr; }
|
||||||
|
}
|
||||||
|
@media (max-width:640px) {
|
||||||
|
.hero,main,footer { width:min(100% - 24px,1680px); }
|
||||||
|
.hero { align-items:flex-start; flex-direction:column; padding-top:28px; }
|
||||||
|
.overview-grid { grid-template-columns:1fr; }.system-overview { grid-column:auto; }
|
||||||
|
.panel { padding:15px; }.section-head { flex-direction:column; }
|
||||||
|
.ib-grid,.management-grid { grid-template-columns:1fr; }
|
||||||
|
.topology-viewbar { align-items:flex-start; flex-direction:column; }
|
||||||
|
.numa-switcher { width:100%; }.numa-switch { flex:1; min-width:0; }
|
||||||
|
.nvlink-fabric { align-items:flex-start; flex-direction:column; gap:2px; }
|
||||||
|
}
|
||||||
|
@media print {
|
||||||
|
:root { --bg:#fff; --surface:#fff; --surface-2:#f3f6f4; --line:#cbd6d0; --text:#102018; --muted:#52675d; --shadow:none; }
|
||||||
|
body { background:#fff; }.hero,main,footer { width:100%; }.hero { padding-top:10px; }
|
||||||
|
button,dialog { display:none!important; }.panel,.overview-card { break-inside:avoid; }
|
||||||
|
.topology-panel { overflow:visible; zoom:.72; }.rails { min-width:0; }
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
@@BODY@@
|
||||||
|
<script>
|
||||||
|
const rawDialog=document.getElementById('raw-dialog');
|
||||||
|
document.querySelector('[data-open-raw]')?.addEventListener('click',()=>rawDialog?.showModal());
|
||||||
|
document.querySelector('[data-close-raw]')?.addEventListener('click',()=>rawDialog?.close());
|
||||||
|
rawDialog?.addEventListener('click',event=>{if(event.target===rawDialog)rawDialog.close();});
|
||||||
|
const topologyRails=document.querySelector('.rails');
|
||||||
|
const numaRuler=document.querySelector('.numa-ruler');
|
||||||
|
const numaSwitches=[...document.querySelectorAll('[data-numa-filter]')];
|
||||||
|
function setNumaView(value){
|
||||||
|
const rails=[...document.querySelectorAll('.rail[data-numa]')];
|
||||||
|
const rulers=[...document.querySelectorAll('.numa-ruler [data-numa]')];
|
||||||
|
let visibleRails=0;
|
||||||
|
rails.forEach(rail=>{
|
||||||
|
const visible=value==='all'||rail.dataset.numa===value;
|
||||||
|
rail.hidden=!visible;
|
||||||
|
if(visible)visibleRails+=1;
|
||||||
|
});
|
||||||
|
const visibleRulers=[];
|
||||||
|
rulers.forEach(ruler=>{
|
||||||
|
const visible=value==='all'||ruler.dataset.numa===value;
|
||||||
|
ruler.hidden=!visible;
|
||||||
|
if(visible)visibleRulers.push(ruler);
|
||||||
|
});
|
||||||
|
if(topologyRails){
|
||||||
|
const count=Math.max(1,visibleRails);
|
||||||
|
topologyRails.style.gridTemplateColumns=`repeat(${count},minmax(132px,1fr))`;
|
||||||
|
topologyRails.style.minWidth=`${count*132+Math.max(0,count-1)*10}px`;
|
||||||
|
}
|
||||||
|
if(numaRuler){
|
||||||
|
numaRuler.style.gridTemplateColumns=visibleRulers.map(item=>`${item.dataset.count||1}fr`).join(' ')||'1fr';
|
||||||
|
}
|
||||||
|
numaSwitches.forEach(button=>{
|
||||||
|
const active=button.dataset.numaFilter===value;
|
||||||
|
button.classList.toggle('active',active);
|
||||||
|
button.setAttribute('aria-pressed',String(active));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
numaSwitches.forEach(button=>button.addEventListener('click',()=>setNumaView(button.dataset.numaFilter)));
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
980
scripts/gpu_node_profile/templates/gpu_topology_3d.html
Normal file
980
scripts/gpu_node_profile/templates/gpu_topology_3d.html
Normal file
@ -0,0 +1,980 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||||
|
<meta name="color-scheme" content="dark">
|
||||||
|
<title>@@TITLE@@</title>
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
--bg:#050c0a; --surface:#0b1713; --surface-2:#10241d; --line:#28493d;
|
||||||
|
--text:#eef9f4; --muted:#8fa99e; --green:#a5e34b; --cyan:#51d9d2;
|
||||||
|
--amber:#ffbd4a; --purple:#b592ff; --red:#ff6b6b;
|
||||||
|
}
|
||||||
|
* { box-sizing:border-box; }
|
||||||
|
html,body { margin:0; min-height:100%; background:var(--bg); color:var(--text); }
|
||||||
|
body {
|
||||||
|
font:14px/1.45 Inter,ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;
|
||||||
|
background:
|
||||||
|
radial-gradient(circle at 16% 0,rgba(118,185,0,.15),transparent 30rem),
|
||||||
|
radial-gradient(circle at 90% 8%,rgba(81,217,210,.08),transparent 28rem),
|
||||||
|
var(--bg);
|
||||||
|
}
|
||||||
|
button { font:inherit; }
|
||||||
|
.shell { min-height:100vh; display:grid; grid-template-rows:auto 1fr auto; }
|
||||||
|
.topbar {
|
||||||
|
min-width:0; display:flex; align-items:flex-end; justify-content:space-between;
|
||||||
|
gap:24px; padding:22px 28px 16px; border-bottom:1px solid var(--line);
|
||||||
|
}
|
||||||
|
.kicker { color:var(--green); font-size:10px; font-weight:850; letter-spacing:.19em; }
|
||||||
|
h1 { margin:4px 0 0; font-size:clamp(25px,3vw,42px); letter-spacing:-.045em; line-height:1; }
|
||||||
|
.subtitle { margin:7px 0 0; color:var(--muted); }
|
||||||
|
.top-stats { display:flex; flex-wrap:wrap; justify-content:flex-end; gap:8px; }
|
||||||
|
.stat {
|
||||||
|
min-width:96px; padding:8px 11px; border:1px solid var(--line); border-radius:10px;
|
||||||
|
background:rgba(11,23,19,.76);
|
||||||
|
}
|
||||||
|
.stat b,.stat small { display:block; }.stat b { font-size:15px; }.stat small { color:var(--muted); font-size:9px; }
|
||||||
|
.workspace { min-height:0; display:grid; grid-template-columns:minmax(0,1fr) 330px; gap:0; }
|
||||||
|
.stage { min-width:0; min-height:720px; position:relative; overflow:hidden; }
|
||||||
|
#topology-canvas { display:block; width:100%; height:100%; min-height:720px; outline:none; cursor:grab; }
|
||||||
|
#topology-canvas:active { cursor:grabbing; }
|
||||||
|
#topology-canvas:focus-visible { box-shadow:inset 0 0 0 2px var(--green); }
|
||||||
|
.toolbar {
|
||||||
|
position:absolute; z-index:4; left:18px; right:18px; top:16px;
|
||||||
|
display:flex; align-items:flex-start; justify-content:space-between; gap:12px;
|
||||||
|
pointer-events:none;
|
||||||
|
}
|
||||||
|
.toolbar-group {
|
||||||
|
pointer-events:auto; display:flex; flex-wrap:wrap; align-items:center; gap:5px;
|
||||||
|
padding:5px; border:1px solid var(--line); border-radius:11px;
|
||||||
|
background:rgba(5,12,10,.86); backdrop-filter:blur(10px);
|
||||||
|
}
|
||||||
|
.toolbar-label { padding:0 7px; color:var(--muted); font-size:9px; font-weight:800; letter-spacing:.12em; }
|
||||||
|
.tool-button {
|
||||||
|
border:0; border-radius:8px; padding:7px 10px; color:var(--muted);
|
||||||
|
background:transparent; cursor:pointer; font-size:10px; font-weight:800;
|
||||||
|
}
|
||||||
|
.tool-button:hover { color:var(--text); background:rgba(255,255,255,.05); }
|
||||||
|
.tool-button[aria-pressed="true"] { color:#07100e; background:var(--green); }
|
||||||
|
.tool-button:disabled { opacity:.35; cursor:not-allowed; }
|
||||||
|
.tool-button.reset { color:var(--text); border:1px solid var(--line); }
|
||||||
|
.camera-status {
|
||||||
|
position:absolute; z-index:3; left:50%; top:72px; transform:translateX(-50%);
|
||||||
|
padding:6px 10px; border:1px solid rgba(165,227,75,.32); border-radius:999px;
|
||||||
|
color:var(--green); background:rgba(5,12,10,.76); backdrop-filter:blur(8px);
|
||||||
|
font:800 9px ui-monospace,monospace; letter-spacing:.1em; pointer-events:none;
|
||||||
|
}
|
||||||
|
.canvas-note {
|
||||||
|
position:absolute; z-index:3; left:22px; bottom:18px; max-width:min(630px,calc(100% - 44px));
|
||||||
|
padding:8px 11px; color:var(--muted); font-size:10px; border:1px solid var(--line);
|
||||||
|
border-radius:9px; background:rgba(5,12,10,.8); backdrop-filter:blur(8px);
|
||||||
|
}
|
||||||
|
.legend {
|
||||||
|
position:absolute; z-index:3; right:20px; bottom:18px; display:flex; flex-wrap:wrap;
|
||||||
|
justify-content:flex-end; gap:9px 12px; max-width:52%; padding:8px 10px;
|
||||||
|
border:1px solid var(--line); border-radius:9px; background:rgba(5,12,10,.8);
|
||||||
|
}
|
||||||
|
.legend span { display:flex; align-items:center; gap:5px; color:var(--muted); font-size:9px; }
|
||||||
|
.legend i { width:18px; height:3px; border-radius:9px; background:var(--line-color); }
|
||||||
|
.inspector {
|
||||||
|
min-width:0; padding:20px; border-left:1px solid var(--line);
|
||||||
|
background:linear-gradient(180deg,rgba(13,28,23,.97),rgba(7,16,13,.97));
|
||||||
|
overflow:auto;
|
||||||
|
}
|
||||||
|
.inspector h2 { margin:4px 0 5px; font-size:21px; }.inspector>p { color:var(--muted); margin:0 0 18px; }
|
||||||
|
.scope {
|
||||||
|
margin-bottom:16px; padding:10px; color:var(--muted); border:1px solid var(--line);
|
||||||
|
border-left:3px solid var(--amber); border-radius:9px; background:rgba(255,189,74,.04); font-size:10px;
|
||||||
|
}
|
||||||
|
.mini-grid { display:grid; grid-template-columns:1fr 1fr; gap:7px; margin-bottom:18px; }
|
||||||
|
.mini-grid div { min-width:0; padding:9px; border:1px solid var(--line); border-radius:8px; background:rgba(5,12,10,.5); }
|
||||||
|
.mini-grid b,.mini-grid small { display:block; }.mini-grid b { font-size:16px; }.mini-grid small { color:var(--muted); font-size:9px; }
|
||||||
|
.detail {
|
||||||
|
padding-top:15px; border-top:1px solid var(--line);
|
||||||
|
}
|
||||||
|
.detail-head { display:flex; justify-content:space-between; gap:12px; align-items:flex-start; }
|
||||||
|
.detail h3 { margin:3px 0 0; font-size:19px; }.detail-kind { color:var(--green); font-size:9px; font-weight:850; letter-spacing:.15em; }
|
||||||
|
.detail-status { color:var(--cyan); font:800 10px ui-monospace,monospace; }
|
||||||
|
.detail dl { display:grid; gap:1px; margin:14px 0 0; border:1px solid var(--line); border-radius:9px; overflow:hidden; }
|
||||||
|
.detail dl div { display:grid; grid-template-columns:92px minmax(0,1fr); gap:8px; padding:8px 9px; background:rgba(5,12,10,.52); border-bottom:1px solid var(--line); }
|
||||||
|
.detail dl div:last-child { border-bottom:0; }
|
||||||
|
.detail dt { color:var(--muted); font-size:9px; }.detail dd { margin:0; text-align:right; overflow-wrap:anywhere; font:10px/1.4 ui-monospace,monospace; }
|
||||||
|
.path-guide { margin-top:18px; }.path-guide h3 { margin:0 0 9px; font-size:13px; }
|
||||||
|
.path-guide ul { margin:0; padding-left:17px; color:var(--muted); font-size:10px; }
|
||||||
|
.path-guide li+li { margin-top:5px; }
|
||||||
|
footer { display:flex; justify-content:space-between; gap:16px; padding:10px 28px 16px; color:var(--muted); font-size:9px; border-top:1px solid var(--line); }
|
||||||
|
@media (max-width:980px) {
|
||||||
|
.workspace { grid-template-columns:1fr; }
|
||||||
|
.inspector { border-left:0; border-top:1px solid var(--line); }
|
||||||
|
.stage,#topology-canvas { min-height:620px; }
|
||||||
|
.mini-grid { grid-template-columns:repeat(4,minmax(0,1fr)); }
|
||||||
|
}
|
||||||
|
@media (max-width:640px) {
|
||||||
|
.topbar { align-items:flex-start; flex-direction:column; padding-inline:16px; }
|
||||||
|
.top-stats { justify-content:flex-start; }.workspace { display:block; }
|
||||||
|
.stage,#topology-canvas { min-height:560px; }
|
||||||
|
.toolbar { left:10px; right:10px; flex-direction:column; }
|
||||||
|
.toolbar-group { width:100%; }.tool-button { flex:1; }
|
||||||
|
.legend { display:none; }.canvas-note { left:10px; bottom:10px; max-width:calc(100% - 20px); }
|
||||||
|
.mini-grid { grid-template-columns:1fr 1fr; }
|
||||||
|
footer { padding-inline:16px; flex-direction:column; }
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<header class="topbar">
|
||||||
|
<div>
|
||||||
|
<span class="kicker">MACHINE CT · GUEST-VISIBLE TOPOLOGY</span>
|
||||||
|
<h1 id="machine-title">GPU Machine 3D CT</h1>
|
||||||
|
<p class="subtitle" id="machine-subtitle"></p>
|
||||||
|
</div>
|
||||||
|
<div class="top-stats" id="top-stats"></div>
|
||||||
|
</header>
|
||||||
|
<main class="workspace">
|
||||||
|
<section class="stage" aria-label="交互式 3D 硬件拓扑">
|
||||||
|
<div class="toolbar">
|
||||||
|
<div class="toolbar-group" id="view-controls"><span class="toolbar-label">视图</span></div>
|
||||||
|
<div class="toolbar-group" id="camera-controls">
|
||||||
|
<span class="toolbar-label">镜头</span>
|
||||||
|
<button class="tool-button" type="button" data-camera="outside" aria-pressed="true">外观</button>
|
||||||
|
<button class="tool-button" type="button" data-camera="inside" aria-pressed="false">进入内部</button>
|
||||||
|
<button class="tool-button" type="button" id="camera-back" disabled>上一层</button>
|
||||||
|
<button class="tool-button" type="button" id="focus-selected" disabled>聚焦所选</button>
|
||||||
|
</div>
|
||||||
|
<div class="toolbar-group" id="layer-controls">
|
||||||
|
<span class="toolbar-label">图层</span>
|
||||||
|
<button class="tool-button" type="button" data-layer="cpu" aria-pressed="true">CPU</button>
|
||||||
|
<button class="tool-button" type="button" data-layer="pcie" aria-pressed="true">PCIe</button>
|
||||||
|
<button class="tool-button" type="button" data-layer="gpu" aria-pressed="true">GPU</button>
|
||||||
|
<button class="tool-button" type="button" data-layer="fabric" aria-pressed="true">NVSwitch</button>
|
||||||
|
<button class="tool-button" type="button" data-layer="network" aria-pressed="true">网络</button>
|
||||||
|
<button class="tool-button reset" type="button" id="reset-view">复位</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<canvas id="topology-canvas" tabindex="0" aria-label="可旋转的 GPU、CPU、PCIe、NVSwitch 和网卡三维拓扑"></canvas>
|
||||||
|
<div class="camera-status" id="camera-status">整机外观 · 双击壳体逐层进入</div>
|
||||||
|
<div class="canvas-note">双击壳体逐层进入 · 上一层返回 · 拖动旋转 · Shift+拖动平移 · Esc 返回整机</div>
|
||||||
|
<div class="legend" aria-label="连线图例">
|
||||||
|
<span><i style="--line-color:#a5e34b"></i>NVLink(仅连 GPU)</span>
|
||||||
|
<span><i style="--line-color:#51d9d2"></i>GPUDirect 候选</span>
|
||||||
|
<span><i style="--line-color:#ffbd4a"></i>PCIe</span>
|
||||||
|
<span><i style="--line-color:#b592ff"></i>CPU 亲和</span>
|
||||||
|
<span><i style="--line-color:#ff6b6b"></i>跨 NUMA / SYS</span>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
<aside class="inspector">
|
||||||
|
<span class="kicker">TOPOLOGY INSPECTOR</span>
|
||||||
|
<h2>逻辑拓扑切片</h2>
|
||||||
|
<p>NUMA 是地域,PCIe 是道路,GPUDirect 是设备间数据路径。</p>
|
||||||
|
<div class="scope">当前视图来自 Guest 可见的 PCIe/NUMA 拓扑。直通或虚拟 PCIe 可能隐藏宿主机真实插槽几何位置。</div>
|
||||||
|
<div class="mini-grid" id="machine-metrics"></div>
|
||||||
|
<section class="detail" id="selection-detail" aria-live="polite"></section>
|
||||||
|
<section class="path-guide">
|
||||||
|
<h3>距离语义</h3>
|
||||||
|
<ul>
|
||||||
|
<li>PIX:最多经过一个 PCIe Switch,通常是最优 GPU–NIC 路径。</li>
|
||||||
|
<li>PXB/PHB:经过更多 Switch 或 Host Bridge。</li>
|
||||||
|
<li>NODE:同 NUMA 内跨 Host Bridge。</li>
|
||||||
|
<li>SYS:跨 NUMA/CPU 互联,路径最长。</li>
|
||||||
|
</ul>
|
||||||
|
</section>
|
||||||
|
</aside>
|
||||||
|
</main>
|
||||||
|
<footer><span>GPU Machine CT v@@VERSION@@ · 原生 Canvas · 无外部依赖</span><span id="generated-at"></span></footer>
|
||||||
|
</div>
|
||||||
|
<script type="application/json" id="machine-data">@@MACHINE_DATA@@</script>
|
||||||
|
<script>
|
||||||
|
(() => {
|
||||||
|
'use strict';
|
||||||
|
const data = JSON.parse(document.getElementById('machine-data').textContent);
|
||||||
|
const canvas = document.getElementById('topology-canvas');
|
||||||
|
const ctx = canvas.getContext('2d');
|
||||||
|
const system = data.system || {};
|
||||||
|
const gpus = data.gpus || [];
|
||||||
|
const networkEndpoints = data.network_endpoints || [];
|
||||||
|
const allNics = data.nics || [];
|
||||||
|
const endpointById = new Map(networkEndpoints.map(endpoint => [endpoint.id,endpoint]));
|
||||||
|
const networkByGpu = new Map(gpus.map(gpu => [Number(gpu.index),endpointById.get(gpu.primary_network_id)]).filter(([,endpoint]) => endpoint));
|
||||||
|
const numaIds = [...new Set(gpus.map(gpu => gpu.numa).filter(value => value !== null && value !== undefined))];
|
||||||
|
if (!numaIds.length) numaIds.push(0);
|
||||||
|
|
||||||
|
document.getElementById('machine-title').textContent = `${system.hostname || 'GPU Node'} · 3D Hardware CT`;
|
||||||
|
document.getElementById('machine-subtitle').textContent = `${system.manufacturer || ''} ${system.product || ''} · ${system.os || ''}`.trim();
|
||||||
|
document.getElementById('generated-at').textContent = `采集时间 ${data.generated_at || '—'} · ${data.pci_fabric?.scope || 'guest-visible'}`;
|
||||||
|
|
||||||
|
const totalPower = gpus.reduce((sum,gpu) => sum + (Number(gpu.power_w) || 0),0);
|
||||||
|
const totalLimit = gpus.reduce((sum,gpu) => sum + (Number(gpu.power_limit_w) || 0),0);
|
||||||
|
const metrics = [
|
||||||
|
['GPU', gpus.length],
|
||||||
|
['CPU', system.cpu_physical_cores || system.cpu_logical || '—'],
|
||||||
|
['NUMA', numaIds.length],
|
||||||
|
['GPU Power', totalPower ? `${(totalPower/1000).toFixed(2)} kW` : '—']
|
||||||
|
];
|
||||||
|
document.getElementById('top-stats').innerHTML = metrics.map(([label,value]) => `<span class="stat"><b>${escapeHtml(value)}</b><small>${escapeHtml(label)}</small></span>`).join('');
|
||||||
|
document.getElementById('machine-metrics').innerHTML = [
|
||||||
|
['CPU Sockets', system.cpu_sockets || '—'],
|
||||||
|
['PCI Bridges', data.pci_fabric?.bridges?.length || '—'],
|
||||||
|
['NVLinks', data.nvlink?.active_links || '—'],
|
||||||
|
['Power Limit', totalLimit ? `${(totalLimit/1000).toFixed(2)} kW` : '—']
|
||||||
|
].map(([label,value]) => `<div><b>${escapeHtml(value)}</b><small>${escapeHtml(label)}</small></div>`).join('');
|
||||||
|
|
||||||
|
const state = {
|
||||||
|
yaw:-0.64, pitch:0.52, zoom:1, focus:'all', selected:null,
|
||||||
|
cameraMode:'outside', target:{x:0,y:3.4,z:0}, scopeStack:[],
|
||||||
|
layers:new Set(['cpu','pcie','gpu','fabric','network'])
|
||||||
|
};
|
||||||
|
let scene = {boxes:[],edges:[]};
|
||||||
|
let hitTargets = [];
|
||||||
|
let pointer = null;
|
||||||
|
let cameraAnimation = 0;
|
||||||
|
|
||||||
|
function escapeHtml(value) {
|
||||||
|
return String(value ?? '').replace(/[&<>"']/g, char => ({'&':'&','<':'<','>':'>','"':'"',"'":'''}[char]));
|
||||||
|
}
|
||||||
|
|
||||||
|
function deepestCommonPath(paths) {
|
||||||
|
const valid = paths.filter(path => Array.isArray(path) && path.length);
|
||||||
|
if (valid.length < 2) return '';
|
||||||
|
const shortest = Math.min(...valid.map(path => path.length));
|
||||||
|
let common = '';
|
||||||
|
for (let index=0; index<shortest; index+=1) {
|
||||||
|
const value = valid[0][index];
|
||||||
|
if (valid.every(path => path[index] === value)) common = value;
|
||||||
|
else break;
|
||||||
|
}
|
||||||
|
return common;
|
||||||
|
}
|
||||||
|
|
||||||
|
function addBox(box) {
|
||||||
|
scene.boxes.push({
|
||||||
|
selectable:true, opacity:1, ...box,
|
||||||
|
size:box.size || {x:2,y:1,z:1}
|
||||||
|
});
|
||||||
|
return box.id;
|
||||||
|
}
|
||||||
|
|
||||||
|
function addEdge(from,to,type,label,layer,meta={}) {
|
||||||
|
scene.edges.push({from,to,type,label,layer,meta});
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildScene() {
|
||||||
|
scene = {boxes:[],edges:[]};
|
||||||
|
const visibleNumas = state.focus === 'all' ? numaIds : numaIds.filter(id => String(id) === String(state.focus));
|
||||||
|
const domainGap = 18;
|
||||||
|
const totalWidth = Math.max(18,visibleNumas.length * domainGap);
|
||||||
|
addBox({
|
||||||
|
id:'chassis', kind:'chassis', layer:'pcie', selectable:true,
|
||||||
|
label:`${system.virtualization || 'HOST'} · GUEST VISIBLE`,
|
||||||
|
position:{x:0,y:3.4,z:0}, size:{x:totalWidth,y:10,z:17},
|
||||||
|
color:'#56756a', opacity:.08, wire:true
|
||||||
|
});
|
||||||
|
|
||||||
|
const gpuNodeIds = [];
|
||||||
|
visibleNumas.forEach((numa,domainIndex) => {
|
||||||
|
const centerX = (domainIndex - (visibleNumas.length-1)/2) * domainGap;
|
||||||
|
const domainGpus = gpus.filter(gpu => String(gpu.numa) === String(numa));
|
||||||
|
const cpuAffinity = domainGpus.find(gpu => gpu.cpu_affinity)?.cpu_affinity || '未解析';
|
||||||
|
const domainColor = numaIds.indexOf(numa) % 2 ? '#4fc6e8' : '#76b900';
|
||||||
|
addBox({
|
||||||
|
id:`numa-${numa}`, kind:'numa', layer:'cpu', selectable:true,
|
||||||
|
parentId:'chassis',
|
||||||
|
label:`NUMA ${numa}`, sublabel:`CPU ${cpuAffinity}`,
|
||||||
|
position:{x:centerX,y:3.2,z:0}, size:{x:16,y:8,z:15},
|
||||||
|
color:domainColor, opacity:.11, wire:true,
|
||||||
|
meta:{'CPU Affinity':cpuAffinity,'Memory NUMA':numa,'GPU Count':domainGpus.length}
|
||||||
|
});
|
||||||
|
addBox({
|
||||||
|
id:`cpu-${numa}`, kind:'cpu', layer:'cpu', label:`CPU / NUMA ${numa}`,
|
||||||
|
parentId:`numa-${numa}`,
|
||||||
|
sublabel:`Cores ${cpuAffinity}`, position:{x:centerX,y:1.4,z:4.7},
|
||||||
|
size:{x:5.8,y:1.3,z:2.3}, color:'#b592ff',
|
||||||
|
meta:{NUMA:numa,'CPU Affinity':cpuAffinity,Model:system.cpu_model || '—'}
|
||||||
|
});
|
||||||
|
addBox({
|
||||||
|
id:`root-${numa}`, kind:'pcie-root', layer:'pcie', label:'PCIe Root Complex',
|
||||||
|
parentId:`numa-${numa}`,
|
||||||
|
sublabel:`NUMA ${numa}`, position:{x:centerX,y:2.6,z:2.2},
|
||||||
|
size:{x:6.8,y:.8,z:1.6}, color:'#ffbd4a',
|
||||||
|
meta:{NUMA:numa,Scope:'Guest-visible host bridge'}
|
||||||
|
});
|
||||||
|
addEdge(`cpu-${numa}`,`root-${numa}`,'control','CPU / Host Bridge','pcie');
|
||||||
|
|
||||||
|
const columns = Math.max(1,Math.min(4,domainGpus.length));
|
||||||
|
domainGpus.forEach((gpu,index) => {
|
||||||
|
const row = Math.floor(index/columns);
|
||||||
|
const column = index%columns;
|
||||||
|
const x = centerX + (column-(columns-1)/2)*3.45;
|
||||||
|
const z = -.7-row*4.4;
|
||||||
|
const gpuId = `gpu-${gpu.index}`;
|
||||||
|
const endpoint = networkByGpu.get(Number(gpu.index));
|
||||||
|
const nicId = endpoint ? `network-${endpoint.id}` : '';
|
||||||
|
const primaryOwners = (endpoint?.gpu_links || [])
|
||||||
|
.filter(link => link.role === 'primary')
|
||||||
|
.map(link => Number(link.gpu_index))
|
||||||
|
.sort((left,right) => left-right);
|
||||||
|
const endpointOwner = primaryOwners.length > 0
|
||||||
|
&& primaryOwners[0] === Number(gpu.index)
|
||||||
|
&& (endpoint?.type === 'bond' || primaryOwners.length === 1);
|
||||||
|
const memberPaths = (endpoint?.members || []).map(member => member.pci_path);
|
||||||
|
const sharedBridge = deepestCommonPath([gpu.pci_path,...memberPaths]);
|
||||||
|
const relation = gpu.network_relation || '未解析';
|
||||||
|
|
||||||
|
addBox({
|
||||||
|
id:`pair-${gpu.index}`, kind:'pcie-group', layer:'pcie', selectable:true,
|
||||||
|
parentId:`numa-${numa}`,
|
||||||
|
label:sharedBridge ? `PCIe Switch ${sharedBridge}` : `PCIe Group · ${relation}`,
|
||||||
|
sublabel:`GPU${gpu.index} ↔ ${endpoint?.name || 'RDMA ?'}`,
|
||||||
|
position:{x,y:1.55,z:z-1.05}, size:{x:3.15,y:3.4,z:4.35},
|
||||||
|
color:relation === 'PIX' ? '#51d9d2' : '#ffbd4a', opacity:.08, wire:true,
|
||||||
|
meta:{Relation:relation,'Shared Bridge':sharedBridge || '未暴露','GPU BDF':gpu.bdf || '—'}
|
||||||
|
});
|
||||||
|
addBox({
|
||||||
|
id:gpuId, kind:'gpu', layer:'gpu', label:`GPU ${gpu.index}`,
|
||||||
|
parentId:`pair-${gpu.index}`,
|
||||||
|
sublabel:gpu.name || 'NVIDIA GPU', position:{x,y:2.35,z},
|
||||||
|
size:{x:2.55,y:1.45,z:1.7}, color:'#76b900',
|
||||||
|
meta:{
|
||||||
|
NUMA:gpu.numa,'CPU Affinity':gpu.cpu_affinity || '—',BDF:gpu.bdf || '—',
|
||||||
|
'PCI Path':(gpu.pci_path || []).join(' → ') || '未采集',
|
||||||
|
Power:gpu.power_w ? `${Number(gpu.power_w).toFixed(0)} W / ${Number(gpu.power_limit_w || 0).toFixed(0)} W` : '—',
|
||||||
|
PState:gpu.pstate || '—',Utilization:gpu.utilization_pct !== null ? `${gpu.utilization_pct}%` : '—'
|
||||||
|
}
|
||||||
|
});
|
||||||
|
gpuNodeIds.push(gpuId);
|
||||||
|
addEdge(`root-${numa}`,gpuId,'pcie',relation,'pcie',{relation});
|
||||||
|
addEdge(`cpu-${numa}`,gpuId,'affinity',`CPU ${cpuAffinity}`,'cpu');
|
||||||
|
|
||||||
|
if (endpoint?.type === 'bond' && endpointOwner) {
|
||||||
|
const slaves=endpoint.members || [];
|
||||||
|
const capacity=Number(endpoint.capacity_mbps)||0;
|
||||||
|
addBox({
|
||||||
|
id:nicId, kind:'bond', layer:'network', label:endpoint.name,
|
||||||
|
parentId:`pair-${gpu.index}`,
|
||||||
|
sublabel:`${slaves.length} ports · ${endpoint.rdma || 'RoCE / RDMA'}`,
|
||||||
|
position:{x,y:.62,z:z-2.15}, size:{x:2.85,y:1.45,z:1.65},
|
||||||
|
color:'#51d9d2',opacity:.1,wire:true,labelOffset:.85,
|
||||||
|
meta:{
|
||||||
|
NUMA:slaves[0]?.numa ?? numa,Relation:relation,
|
||||||
|
RDMA:endpoint.rdma || '—',Capacity:capacity ? `${capacity/1000} Gb/s` : '—',
|
||||||
|
Members:slaves.map(slave => `${slave.name} ${slave.bdf || ''}`).join(', ') || '—',
|
||||||
|
'Shared Bridge':sharedBridge || '未暴露'
|
||||||
|
}
|
||||||
|
});
|
||||||
|
slaves.forEach((slave,slaveIndex) => {
|
||||||
|
const portId=`nic-${endpoint.name}-${slave.name}`;
|
||||||
|
const offset=(slaveIndex-(slaves.length-1)/2)*1.18;
|
||||||
|
const speed=Number(slave.speed_mbps) ? `${Number(slave.speed_mbps)/1000}G` : '—';
|
||||||
|
addBox({
|
||||||
|
id:portId,kind:'roce-port',layer:'network',label:slave.name,
|
||||||
|
parentId:nicId,
|
||||||
|
sublabel:speed,position:{x:x+offset,y:.68,z:z-2.15},
|
||||||
|
size:{x:1.02,y:.78,z:1.08},color:'#79f4ee',
|
||||||
|
meta:{
|
||||||
|
Bond:endpoint.name,RDMA:endpoint.rdma || '—',NUMA:slave.numa ?? numa,
|
||||||
|
BDF:slave.bdf || '—',Speed:speed,State:slave.status || '—',
|
||||||
|
Driver:slave.driver || '—','PCI Path':(slave.pci_path || []).join(' → ') || '未采集'
|
||||||
|
}
|
||||||
|
});
|
||||||
|
addEdge(portId,nicId,'bond-member','Bond member','network');
|
||||||
|
addEdge(`root-${numa}`,portId,'pcie',relation,'pcie',{relation});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (endpoint?.type === 'bond') {
|
||||||
|
(endpoint.members || []).forEach(slave => {
|
||||||
|
addEdge(
|
||||||
|
gpuId,
|
||||||
|
`nic-${endpoint.name}-${slave.name}`,
|
||||||
|
'gdr',
|
||||||
|
`${relation} · GDR candidate`,
|
||||||
|
'network',
|
||||||
|
{relation,port:slave.name}
|
||||||
|
);
|
||||||
|
});
|
||||||
|
} else if (endpoint && endpointOwner) {
|
||||||
|
const member=(endpoint.members || [])[0] || endpoint;
|
||||||
|
const speed=Number(endpoint.capacity_mbps) ? `${Number(endpoint.capacity_mbps)/1000}G` : '—';
|
||||||
|
const kind=endpoint.protocol === 'InfiniBand' ? 'infiniband' : 'roce-port';
|
||||||
|
addBox({
|
||||||
|
id:nicId,kind,layer:'network',label:endpoint.name,
|
||||||
|
parentId:`pair-${gpu.index}`,
|
||||||
|
sublabel:`${endpoint.protocol || 'RDMA'} · ${speed}`,
|
||||||
|
position:{x,y:.68,z:z-2.15},size:{x:2.35,y:1.05,z:1.35},
|
||||||
|
color:endpoint.protocol === 'InfiniBand' ? '#ff9f43' : '#79f4ee',
|
||||||
|
meta:{
|
||||||
|
Mode:'Standalone RDMA',RDMA:endpoint.rdma || '—',
|
||||||
|
NUMA:endpoint.numa ?? member.numa ?? numa,Relation:relation,
|
||||||
|
Evidence:endpoint.topology_visibility === 'nvidia-smi-topo'
|
||||||
|
? 'nvidia-smi topo -m' : 'Topology relation not exposed',
|
||||||
|
BDF:endpoint.bdf || member.bdf || '—',Speed:speed,
|
||||||
|
State:endpoint.state || member.status || '—',
|
||||||
|
'PCI Path':(endpoint.pci_path || member.pci_path || []).join(' → ') || '未采集'
|
||||||
|
}
|
||||||
|
});
|
||||||
|
addEdge(`root-${numa}`,nicId,'pcie',relation,'pcie',{relation});
|
||||||
|
}
|
||||||
|
if (endpoint?.type !== 'bond' && endpoint) {
|
||||||
|
addEdge(gpuId,nicId,'gdr',`${relation} · GDR candidate`,'network',{relation,port:endpoint.name});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const unpairedEndpoints = networkEndpoints.filter(endpoint =>
|
||||||
|
endpoint.type === 'standalone'
|
||||||
|
&& String(endpoint.numa) === String(numa)
|
||||||
|
&& (endpoint.gpu_links || []).filter(link => link.role === 'primary').length !== 1
|
||||||
|
);
|
||||||
|
unpairedEndpoints.forEach((endpoint,index) => {
|
||||||
|
const x = centerX - 5.4 + index*2.8;
|
||||||
|
const id = `network-${endpoint.id}`;
|
||||||
|
const isIb = endpoint.protocol === 'InfiniBand';
|
||||||
|
const speed = Number(endpoint.capacity_mbps)
|
||||||
|
? `${Number(endpoint.capacity_mbps)/1000} Gb/s` : '—';
|
||||||
|
addBox({
|
||||||
|
id,kind:isIb ? 'infiniband' : 'roce-port',
|
||||||
|
layer:'network',label:endpoint.name,
|
||||||
|
parentId:`numa-${numa}`,
|
||||||
|
sublabel:`${endpoint.protocol || 'RDMA'} ${speed} · GPU relation ${endpoint.topology_visibility === 'nvidia-smi-topo' ? 'exposed' : 'not exposed'}`,
|
||||||
|
position:{x,y:.75,z:6.35},
|
||||||
|
size:{x:2.35,y:1.05,z:1.4},color:isIb ? '#ff9f43' : '#79f4ee',
|
||||||
|
meta:{
|
||||||
|
NUMA:endpoint.numa,BDF:endpoint.bdf || '—',
|
||||||
|
RDMA:endpoint.rdma || '—',State:endpoint.state || '—',
|
||||||
|
'GPU Topology':endpoint.topology_visibility === 'nvidia-smi-topo'
|
||||||
|
? 'nvidia-smi topo -m' : 'Not exposed; no affinity inferred',
|
||||||
|
'PCI Path':(endpoint.pci_path || []).join(' → ') || '未采集'
|
||||||
|
}
|
||||||
|
});
|
||||||
|
addEdge(`root-${numa}`,id,'pcie',isIb ? 'Storage / IB' : 'Standalone RoCE','pcie');
|
||||||
|
(endpoint?.gpu_links || []).filter(link => link.role==='nearest').forEach(link => {
|
||||||
|
const gpuId=`gpu-${link.gpu_index}`;
|
||||||
|
if (scene.boxes.some(box => box.id===gpuId)) {
|
||||||
|
addEdge(gpuId,id,'gdr',`${link.relation} · nearest ${endpoint.protocol}`,'network',{relation:link.relation,port:endpoint.name});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
const acceleratorMembers = new Set(networkEndpoints.flatMap(endpoint => (endpoint.members || []).map(member => member.name)));
|
||||||
|
const acceleratorGroups = new Set(networkEndpoints.filter(endpoint => endpoint.type==='bond').map(endpoint => endpoint.name));
|
||||||
|
const management = allNics.filter(nic =>
|
||||||
|
String(nic.numa) === String(numa)
|
||||||
|
&& !acceleratorMembers.has(nic.name)
|
||||||
|
&& !acceleratorGroups.has(nic.name)
|
||||||
|
&& (nic.bdf || nic.default_route || nic.is_bond)
|
||||||
|
);
|
||||||
|
management.slice(0,2).forEach((nic,index) => {
|
||||||
|
const id = `mgmt-${nic.name}`;
|
||||||
|
addBox({
|
||||||
|
id,kind:'management',layer:'network',label:nic.name,
|
||||||
|
parentId:`numa-${numa}`,
|
||||||
|
sublabel:nic.default_route ? 'In-band management' : 'OS network',
|
||||||
|
position:{x:centerX+5.4-index*2.7,y:.65,z:6.35},size:{x:2.25,y:1,z:1.35},color:'#7d9b90',
|
||||||
|
meta:{NUMA:nic.numa,BDF:nic.bdf || '—',Driver:nic.driver || '—',Address:(nic.addresses || []).join(', ') || '—'}
|
||||||
|
});
|
||||||
|
addEdge(`root-${numa}`,id,'pcie','Management','pcie');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
if (gpuNodeIds.length > 1 && data.nvlink?.active_links) {
|
||||||
|
addBox({
|
||||||
|
id:'nvswitch',kind:'nvswitch',layer:'fabric',label:data.nvlink.fabric_type || 'NVLink Fabric',
|
||||||
|
parentId:'chassis',
|
||||||
|
sublabel:`${gpuNodeIds.length}/${gpus.length} GPU · GPU endpoints only`,
|
||||||
|
position:{x:0,y:8.15,z:-1.2},size:{x:Math.max(9,totalWidth-3),y:.8,z:2.1},color:'#a5e34b',
|
||||||
|
meta:{
|
||||||
|
Connection:'GPU endpoints only · not NIC',
|
||||||
|
'Active Links':data.nvlink.active_links,
|
||||||
|
'Links / GPU':data.nvlink.links_per_gpu || '—',
|
||||||
|
'Speed / Link':`${data.nvlink.speed_gbs || '—'} GB/s`
|
||||||
|
}
|
||||||
|
});
|
||||||
|
gpuNodeIds.forEach(id => {
|
||||||
|
const gpuIndex=Number(id.replace('gpu-',''));
|
||||||
|
addEdge(id,'nvswitch','nvlink',`GPU ${gpuIndex} ↔ NVSwitch`,'fabric',{gpuIndex});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
renderDetail();
|
||||||
|
}
|
||||||
|
|
||||||
|
const viewControls = document.getElementById('view-controls');
|
||||||
|
function buildViewControls() {
|
||||||
|
const views = [['all','总览'],...numaIds.map(id => [String(id),`NUMA ${id}`])];
|
||||||
|
viewControls.querySelectorAll('button').forEach(button => button.remove());
|
||||||
|
views.forEach(([value,label]) => {
|
||||||
|
const button = document.createElement('button');
|
||||||
|
button.type='button'; button.className='tool-button'; button.textContent=label;
|
||||||
|
button.dataset.view=value; button.setAttribute('aria-pressed',String(state.focus === value));
|
||||||
|
button.addEventListener('click',() => {
|
||||||
|
state.focus=value; state.selected=null;
|
||||||
|
state.scopeStack=state.cameraMode==='inside' ? ['chassis'] : [];
|
||||||
|
buildViewControls(); buildScene();
|
||||||
|
setCameraMode(state.cameraMode,null,false);
|
||||||
|
});
|
||||||
|
viewControls.appendChild(button);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function cameraPreset(mode,box=null) {
|
||||||
|
if (mode === 'inside') {
|
||||||
|
const zoomByKind={chassis:1.48,numa:1.72,'pcie-group':2.5,bond:3.25};
|
||||||
|
return {
|
||||||
|
yaw:-.2,pitch:.27,zoom:box ? (zoomByKind[box.kind] || 2.15) : 1.48,
|
||||||
|
target:box ? {...box.position} : {x:0,y:2.25,z:-1.2}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return {yaw:-.64,pitch:.52,zoom:1,target:{x:0,y:3.4,z:0}};
|
||||||
|
}
|
||||||
|
|
||||||
|
function animateCamera(next,duration=380) {
|
||||||
|
const animation=++cameraAnimation;
|
||||||
|
const start={
|
||||||
|
yaw:state.yaw,pitch:state.pitch,zoom:state.zoom,target:{...state.target}
|
||||||
|
};
|
||||||
|
const started=performance.now();
|
||||||
|
function frame(now) {
|
||||||
|
if (animation!==cameraAnimation) return;
|
||||||
|
const raw=Math.min(1,(now-started)/duration);
|
||||||
|
const progress=1-Math.pow(1-raw,3);
|
||||||
|
state.yaw=start.yaw+(next.yaw-start.yaw)*progress;
|
||||||
|
state.pitch=start.pitch+(next.pitch-start.pitch)*progress;
|
||||||
|
state.zoom=start.zoom+(next.zoom-start.zoom)*progress;
|
||||||
|
state.target={
|
||||||
|
x:start.target.x+(next.target.x-start.target.x)*progress,
|
||||||
|
y:start.target.y+(next.target.y-start.target.y)*progress,
|
||||||
|
z:start.target.z+(next.target.z-start.target.z)*progress
|
||||||
|
};
|
||||||
|
draw();
|
||||||
|
if (raw<1) requestAnimationFrame(frame);
|
||||||
|
}
|
||||||
|
requestAnimationFrame(frame);
|
||||||
|
}
|
||||||
|
|
||||||
|
function updateCameraControls() {
|
||||||
|
document.querySelectorAll('[data-camera]').forEach(button => {
|
||||||
|
button.setAttribute('aria-pressed',String(button.dataset.camera===state.cameraMode));
|
||||||
|
});
|
||||||
|
const selectedBox=scene.boxes.find(box => box.id===state.selected && box.selectable);
|
||||||
|
const scopeBoxes=state.scopeStack.map(id => scene.boxes.find(box => box.id===id)).filter(Boolean);
|
||||||
|
const breadcrumb=['整机',...scopeBoxes.filter(box => box.kind!=='chassis').map(box => box.label)].join(' › ');
|
||||||
|
document.getElementById('focus-selected').disabled=!selectedBox;
|
||||||
|
document.getElementById('camera-back').disabled=state.cameraMode!=='inside';
|
||||||
|
document.getElementById('camera-status').textContent=state.cameraMode==='inside'
|
||||||
|
? `${breadcrumb} · 当前层之外已隐藏`
|
||||||
|
: '整机外观 · 双击壳体逐层进入';
|
||||||
|
}
|
||||||
|
|
||||||
|
function setCameraMode(mode,box=null,animate=true) {
|
||||||
|
state.cameraMode=mode;
|
||||||
|
if (mode==='outside') state.scopeStack=[];
|
||||||
|
const next=cameraPreset(mode,box);
|
||||||
|
updateCameraControls();
|
||||||
|
if (animate) animateCamera(next);
|
||||||
|
else {
|
||||||
|
cameraAnimation+=1;
|
||||||
|
state.yaw=next.yaw;state.pitch=next.pitch;state.zoom=next.zoom;
|
||||||
|
state.target={...next.target};draw();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const shellKinds=new Set(['chassis','numa','pcie-group','bond']);
|
||||||
|
function isShell(box) {
|
||||||
|
return Boolean(box && shellKinds.has(box.kind));
|
||||||
|
}
|
||||||
|
function activeScopeId() {
|
||||||
|
return state.scopeStack[state.scopeStack.length-1] || null;
|
||||||
|
}
|
||||||
|
function isDescendant(boxId,scopeId) {
|
||||||
|
let current=scene.boxes.find(box => box.id===boxId);
|
||||||
|
const visited=new Set();
|
||||||
|
while (current?.parentId && !visited.has(current.id)) {
|
||||||
|
if (current.parentId===scopeId) return true;
|
||||||
|
visited.add(current.id);
|
||||||
|
current=scene.boxes.find(box => box.id===current.parentId);
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
function scopePath(box) {
|
||||||
|
const path=[];
|
||||||
|
let current=box;
|
||||||
|
const visited=new Set();
|
||||||
|
while (current && !visited.has(current.id)) {
|
||||||
|
visited.add(current.id);
|
||||||
|
if (isShell(current)) path.unshift(current.id);
|
||||||
|
current=current.parentId ? scene.boxes.find(item => item.id===current.parentId) : null;
|
||||||
|
}
|
||||||
|
return path;
|
||||||
|
}
|
||||||
|
function nextScopeToward(box) {
|
||||||
|
const path=scopePath(box);
|
||||||
|
if (!path.length) return null;
|
||||||
|
const current=activeScopeId();
|
||||||
|
if (!current) return scene.boxes.find(item => item.id===path[0]) || null;
|
||||||
|
const currentIndex=path.indexOf(current);
|
||||||
|
const nextId=currentIndex>=0 ? path[currentIndex+1] : null;
|
||||||
|
return nextId ? scene.boxes.find(item => item.id===nextId) || null : null;
|
||||||
|
}
|
||||||
|
function enterScope(box,animate=true) {
|
||||||
|
const scope=isShell(box) ? box : scene.boxes.find(item => item.id==='chassis');
|
||||||
|
if (!scope) return;
|
||||||
|
const current=activeScopeId();
|
||||||
|
const targetPath=scopePath(scope);
|
||||||
|
if (state.cameraMode!=='inside' || !current) {
|
||||||
|
state.scopeStack=targetPath;
|
||||||
|
} else if (scope.id!==current && isDescendant(scope.id,current)) {
|
||||||
|
const currentIndex=targetPath.indexOf(current);
|
||||||
|
state.scopeStack=[
|
||||||
|
...state.scopeStack,
|
||||||
|
...targetPath.slice(Math.max(0,currentIndex+1)).filter(id => !state.scopeStack.includes(id))
|
||||||
|
];
|
||||||
|
}
|
||||||
|
state.cameraMode='inside';
|
||||||
|
state.selected=scope.kind==='chassis' ? null : scope.id;
|
||||||
|
renderDetail();setCameraMode('inside',scope,animate);
|
||||||
|
}
|
||||||
|
function leaveScope() {
|
||||||
|
if (state.cameraMode!=='inside') return;
|
||||||
|
state.scopeStack.pop();
|
||||||
|
const parentId=activeScopeId();
|
||||||
|
if (!parentId) {
|
||||||
|
state.selected=null;renderDetail();setCameraMode('outside');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const parent=scene.boxes.find(box => box.id===parentId);
|
||||||
|
state.selected=parent?.kind==='chassis' ? null : parent?.id || null;
|
||||||
|
renderDetail();setCameraMode('inside',parent);
|
||||||
|
}
|
||||||
|
|
||||||
|
document.querySelectorAll('[data-camera]').forEach(button => button.addEventListener('click',() => {
|
||||||
|
const selectedBox=scene.boxes.find(box => box.id===state.selected && box.selectable);
|
||||||
|
const nextScope=selectedBox ? nextScopeToward(selectedBox) : null;
|
||||||
|
if (button.dataset.camera==='outside') {
|
||||||
|
state.selected=null;renderDetail();setCameraMode('outside');
|
||||||
|
} else if (nextScope) {
|
||||||
|
enterScope(nextScope);
|
||||||
|
} else if (isShell(selectedBox)) {
|
||||||
|
enterScope(selectedBox);
|
||||||
|
} else if (state.cameraMode!=='inside') {
|
||||||
|
enterScope(scene.boxes.find(box => box.id==='chassis'));
|
||||||
|
} else if (selectedBox) {
|
||||||
|
setCameraMode('inside',selectedBox);
|
||||||
|
}
|
||||||
|
}));
|
||||||
|
document.getElementById('camera-back').addEventListener('click',leaveScope);
|
||||||
|
document.getElementById('focus-selected').addEventListener('click',() => {
|
||||||
|
const selectedBox=scene.boxes.find(box => box.id===state.selected && box.selectable);
|
||||||
|
if (selectedBox) setCameraMode('inside',selectedBox);
|
||||||
|
});
|
||||||
|
|
||||||
|
document.querySelectorAll('[data-layer]').forEach(button => button.addEventListener('click',() => {
|
||||||
|
const layer = button.dataset.layer;
|
||||||
|
if (state.layers.has(layer)) state.layers.delete(layer); else state.layers.add(layer);
|
||||||
|
button.setAttribute('aria-pressed',String(state.layers.has(layer)));
|
||||||
|
if (state.selected && !visibleBox(scene.boxes.find(box => box.id === state.selected))) state.selected=null;
|
||||||
|
renderDetail(); draw();
|
||||||
|
}));
|
||||||
|
document.getElementById('reset-view').addEventListener('click',() => {
|
||||||
|
state.selected=null;setCameraMode('outside');renderDetail();
|
||||||
|
});
|
||||||
|
|
||||||
|
function visibleBox(box) {
|
||||||
|
if (!box) return false;
|
||||||
|
if (state.cameraMode==='outside') {
|
||||||
|
return box.kind==='chassis' || box.kind==='numa' || state.layers.has(box.layer);
|
||||||
|
}
|
||||||
|
const scopeId=activeScopeId() || 'chassis';
|
||||||
|
if (box.id===scopeId) return box.kind!=='chassis';
|
||||||
|
if (!isDescendant(box.id,scopeId)) return false;
|
||||||
|
return box.kind==='numa' || state.layers.has(box.layer);
|
||||||
|
}
|
||||||
|
function visibleEdge(edge) {
|
||||||
|
const from=scene.boxes.find(box => box.id===edge.from);
|
||||||
|
const to=scene.boxes.find(box => box.id===edge.to);
|
||||||
|
return state.layers.has(edge.layer) && visibleBox(from) && visibleBox(to);
|
||||||
|
}
|
||||||
|
|
||||||
|
function cameraPoint(point) {
|
||||||
|
const cy=Math.cos(state.yaw), sy=Math.sin(state.yaw);
|
||||||
|
const cp=Math.cos(state.pitch), sp=Math.sin(state.pitch);
|
||||||
|
const translated={
|
||||||
|
x:point.x-state.target.x,y:point.y-state.target.y,z:point.z-state.target.z
|
||||||
|
};
|
||||||
|
const x=translated.x*cy-translated.z*sy;
|
||||||
|
const z=translated.x*sy+translated.z*cy;
|
||||||
|
return {x,y:translated.y*cp-z*sp,z:translated.y*sp+z*cp};
|
||||||
|
}
|
||||||
|
function project(point,scale) {
|
||||||
|
const p=cameraPoint(point);
|
||||||
|
return {x:canvas.clientWidth/2+p.x*scale,y:canvas.clientHeight*.58-p.y*scale,z:p.z};
|
||||||
|
}
|
||||||
|
function corners(box) {
|
||||||
|
const {position:p,size:s}=box;
|
||||||
|
const points=[];
|
||||||
|
[-1,1].forEach(dx => [-1,1].forEach(dy => [-1,1].forEach(dz => points.push({x:p.x+dx*s.x/2,y:p.y+dy*s.y/2,z:p.z+dz*s.z/2}))));
|
||||||
|
return points;
|
||||||
|
}
|
||||||
|
const faceIndexes=[[0,1,3,2],[4,6,7,5],[0,4,5,1],[2,3,7,6],[0,2,6,4],[1,5,7,3]];
|
||||||
|
function hexToRgba(hex,alpha) {
|
||||||
|
const value=hex.replace('#','');
|
||||||
|
const number=parseInt(value.length===3?value.split('').map(x=>x+x).join(''):value,16);
|
||||||
|
return `rgba(${(number>>16)&255},${(number>>8)&255},${number&255},${alpha})`;
|
||||||
|
}
|
||||||
|
function boxCenter(id) {
|
||||||
|
const box=scene.boxes.find(item => item.id===id);
|
||||||
|
return box?.position || null;
|
||||||
|
}
|
||||||
|
function relationColor(edge) {
|
||||||
|
if (edge.meta?.relation === 'SYS') return '#ff6b6b';
|
||||||
|
return {
|
||||||
|
nvlink:'#b8ff56',gdr:'#51d9d2',pcie:'#ffbd4a',affinity:'#b592ff',
|
||||||
|
control:'#8fa99e','bond-member':'#79f4ee'
|
||||||
|
}[edge.type] || '#8fa99e';
|
||||||
|
}
|
||||||
|
|
||||||
|
function roundedRectPath(x,y,width,height,radius) {
|
||||||
|
const r=Math.min(radius,width/2,height/2);
|
||||||
|
ctx.beginPath();
|
||||||
|
ctx.moveTo(x+r,y);ctx.lineTo(x+width-r,y);ctx.quadraticCurveTo(x+width,y,x+width,y+r);
|
||||||
|
ctx.lineTo(x+width,y+height-r);ctx.quadraticCurveTo(x+width,y+height,x+width-r,y+height);
|
||||||
|
ctx.lineTo(x+r,y+height);ctx.quadraticCurveTo(x,y+height,x,y+height-r);
|
||||||
|
ctx.lineTo(x,y+r);ctx.quadraticCurveTo(x,y,x+r,y);ctx.closePath();
|
||||||
|
}
|
||||||
|
|
||||||
|
function drawLabelChip(box,point,dimmed) {
|
||||||
|
const isDomain=box.kind==='numa';
|
||||||
|
const mainFont=isDomain?'900 12px ui-sans-serif,sans-serif':'900 10px ui-sans-serif,sans-serif';
|
||||||
|
const subFont='800 8px ui-monospace,monospace';
|
||||||
|
ctx.save();ctx.globalAlpha=dimmed ? .34 : 1;ctx.textAlign='center';
|
||||||
|
ctx.font=mainFont;
|
||||||
|
const mainWidth=ctx.measureText(box.label).width;
|
||||||
|
ctx.font=subFont;
|
||||||
|
const subWidth=box.sublabel ? ctx.measureText(box.sublabel).width : 0;
|
||||||
|
const width=Math.max(mainWidth,subWidth)+14;
|
||||||
|
const height=box.sublabel ? 29 : 18;
|
||||||
|
const left=point.x-width/2,top=point.y-13;
|
||||||
|
roundedRectPath(left,top,width,height,5);
|
||||||
|
ctx.fillStyle=isDomain ? hexToRgba(box.color,.96) : 'rgba(238,249,244,.96)';
|
||||||
|
ctx.fill();
|
||||||
|
ctx.strokeStyle=hexToRgba(box.color,.9);ctx.lineWidth=1.2;ctx.stroke();
|
||||||
|
ctx.fillStyle='#06100c';ctx.font=mainFont;
|
||||||
|
ctx.fillText(box.label,point.x,top+12);
|
||||||
|
if (box.sublabel) {
|
||||||
|
ctx.fillStyle='#17251f';ctx.font=subFont;
|
||||||
|
ctx.fillText(box.sublabel,point.x,top+23);
|
||||||
|
}
|
||||||
|
ctx.restore();
|
||||||
|
}
|
||||||
|
|
||||||
|
function drawEdgeTag(text,x,y,color) {
|
||||||
|
ctx.save();ctx.font='900 8px ui-monospace,monospace';ctx.textAlign='center';
|
||||||
|
const width=ctx.measureText(text).width+9;
|
||||||
|
roundedRectPath(x-width/2,y-8,width,14,4);
|
||||||
|
ctx.fillStyle=color;ctx.fill();
|
||||||
|
ctx.fillStyle='#06100c';ctx.fillText(text,x,y+2);
|
||||||
|
ctx.restore();
|
||||||
|
}
|
||||||
|
|
||||||
|
function drawEdge(edge,scale,selected,overlay=false) {
|
||||||
|
const from=boxCenter(edge.from),to=boxCenter(edge.to);
|
||||||
|
if (!from || !to) return;
|
||||||
|
const a=project(from,scale),b=project(to,scale);
|
||||||
|
const related=!selected || edge.from===selected || edge.to===selected;
|
||||||
|
const color=relationColor(edge);
|
||||||
|
ctx.save();
|
||||||
|
ctx.strokeStyle=hexToRgba(color,related ? (overlay ? 1 : .85) : (overlay ? .24 : .13));
|
||||||
|
ctx.lineWidth=related
|
||||||
|
? (overlay ? 4.8 : edge.type==='gdr' ? 3 : edge.type==='bond-member' ? 2 : 1.3)
|
||||||
|
: (overlay ? 2 : 1);
|
||||||
|
if (overlay) {
|
||||||
|
ctx.shadowColor=color;ctx.shadowBlur=related ? 12 : 3;
|
||||||
|
ctx.lineCap='round';
|
||||||
|
}
|
||||||
|
if (edge.type==='affinity') ctx.setLineDash([5,5]);
|
||||||
|
if (edge.meta?.relation==='SYS') ctx.setLineDash([8,4]);
|
||||||
|
ctx.beginPath();ctx.moveTo(a.x,a.y);ctx.lineTo(b.x,b.y);ctx.stroke();
|
||||||
|
ctx.shadowBlur=0;
|
||||||
|
if (overlay) {
|
||||||
|
[a,b].forEach(point => {
|
||||||
|
ctx.beginPath();ctx.arc(point.x,point.y,related ? 4.5 : 3,0,Math.PI*2);
|
||||||
|
ctx.fillStyle=hexToRgba(color,related ? 1 : .3);ctx.fill();
|
||||||
|
ctx.strokeStyle='#06100c';ctx.lineWidth=1.5;ctx.stroke();
|
||||||
|
});
|
||||||
|
if (related) {
|
||||||
|
const markerX=a.x+(b.x-a.x)*.18,markerY=a.y+(b.y-a.y)*.18;
|
||||||
|
drawEdgeTag(`GPU ${edge.meta?.gpuIndex ?? '?'}`,markerX,markerY,color);
|
||||||
|
}
|
||||||
|
} else if (related && selected && Math.hypot(a.x-b.x,a.y-b.y)>70) {
|
||||||
|
drawEdgeTag(edge.label,(a.x+b.x)/2,(a.y+b.y)/2-4,color);
|
||||||
|
}
|
||||||
|
ctx.restore();
|
||||||
|
}
|
||||||
|
|
||||||
|
function drawGrid(scale) {
|
||||||
|
ctx.save(); ctx.strokeStyle='rgba(122,164,147,.08)'; ctx.lineWidth=1;
|
||||||
|
for (let value=-24; value<=24; value+=2) {
|
||||||
|
const a=project({x:value,y:0,z:-12},scale),b=project({x:value,y:0,z:12},scale);
|
||||||
|
const c=project({x:-24,y:0,z:value},scale),d=project({x:24,y:0,z:value},scale);
|
||||||
|
ctx.beginPath();ctx.moveTo(a.x,a.y);ctx.lineTo(b.x,b.y);ctx.stroke();
|
||||||
|
ctx.beginPath();ctx.moveTo(c.x,c.y);ctx.lineTo(d.x,d.y);ctx.stroke();
|
||||||
|
}
|
||||||
|
ctx.restore();
|
||||||
|
}
|
||||||
|
|
||||||
|
function draw() {
|
||||||
|
const width=canvas.clientWidth,height=canvas.clientHeight;
|
||||||
|
const dpr=Math.min(2,window.devicePixelRatio || 1);
|
||||||
|
if (canvas.width!==Math.round(width*dpr) || canvas.height!==Math.round(height*dpr)) {
|
||||||
|
canvas.width=Math.round(width*dpr);canvas.height=Math.round(height*dpr);
|
||||||
|
}
|
||||||
|
ctx.setTransform(dpr,0,0,dpr,0,0);
|
||||||
|
ctx.clearRect(0,0,width,height);
|
||||||
|
const domainCount=state.focus==='all'?Math.max(1,numaIds.length):1;
|
||||||
|
const scale=Math.max(18,Math.min(width/(domainCount*22),height/23))*state.zoom;
|
||||||
|
drawGrid(scale);
|
||||||
|
|
||||||
|
const selected=state.selected;
|
||||||
|
scene.edges.filter(edge => visibleEdge(edge) && edge.layer!=='fabric').forEach(edge => {
|
||||||
|
drawEdge(edge,scale,selected);
|
||||||
|
});
|
||||||
|
|
||||||
|
const faces=[];
|
||||||
|
const labels=[];
|
||||||
|
hitTargets=[];
|
||||||
|
scene.boxes.filter(visibleBox).forEach(box => {
|
||||||
|
const projected=corners(box).map(point => project(point,scale));
|
||||||
|
const alpha = box.opacity ?? 1;
|
||||||
|
faceIndexes.forEach((indexes,faceIndex) => {
|
||||||
|
const points=indexes.map(index => projected[index]);
|
||||||
|
faces.push({box,points,depth:points.reduce((sum,p)=>sum+p.z,0)/points.length,faceIndex,alpha});
|
||||||
|
});
|
||||||
|
const xs=projected.map(p=>p.x),ys=projected.map(p=>p.y);
|
||||||
|
hitTargets.push({box,rect:{left:Math.min(...xs),right:Math.max(...xs),top:Math.min(...ys),bottom:Math.max(...ys)},depth:cameraPoint(box.position).z});
|
||||||
|
labels.push({
|
||||||
|
box,
|
||||||
|
point:project({
|
||||||
|
x:box.position.x,
|
||||||
|
y:box.position.y+box.size.y/2+(box.labelOffset ?? .18),
|
||||||
|
z:box.position.z
|
||||||
|
},scale)
|
||||||
|
});
|
||||||
|
});
|
||||||
|
faces.sort((a,b) => a.depth-b.depth).forEach(face => {
|
||||||
|
const isSelected=face.box.id===selected;
|
||||||
|
const dimmed=selected && !isSelected && face.box.kind!=='numa' && face.box.kind!=='chassis';
|
||||||
|
const internalBoundary=state.cameraMode==='inside' && (
|
||||||
|
face.box.kind==='numa' || face.box.kind==='pcie-group' || face.box.kind==='bond'
|
||||||
|
);
|
||||||
|
const baseAlpha=internalBoundary
|
||||||
|
? face.alpha*.22
|
||||||
|
: face.box.wire ? face.alpha : Math.min(.86,face.alpha*(face.faceIndex===4?1:.72));
|
||||||
|
ctx.save();
|
||||||
|
ctx.fillStyle=hexToRgba(face.box.color,dimmed?baseAlpha*.16:baseAlpha);
|
||||||
|
ctx.strokeStyle=hexToRgba(face.box.color,isSelected ? 1 : internalBoundary ? .26 : face.box.wire ? .55 : .75);
|
||||||
|
ctx.lineWidth=isSelected ? 2.8 : face.box.wire ? 1.1 : 1;
|
||||||
|
if (face.box.wire) ctx.setLineDash([7,5]);
|
||||||
|
ctx.beginPath();face.points.forEach((point,index) => index?ctx.lineTo(point.x,point.y):ctx.moveTo(point.x,point.y));ctx.closePath();
|
||||||
|
ctx.fill();ctx.stroke();ctx.restore();
|
||||||
|
});
|
||||||
|
scene.edges.filter(edge => visibleEdge(edge) && edge.layer==='fabric').forEach(edge => {
|
||||||
|
drawEdge(edge,scale,selected,true);
|
||||||
|
});
|
||||||
|
labels.sort((a,b)=>a.point.z-b.point.z).forEach(({box,point}) => {
|
||||||
|
if (box.kind==='chassis') return;
|
||||||
|
if (state.cameraMode==='inside' && box.kind==='numa') return;
|
||||||
|
const dimmed=selected && box.id!==selected && box.kind!=='numa';
|
||||||
|
drawLabelChip(box,point,dimmed);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderDetail() {
|
||||||
|
const root=document.getElementById('selection-detail');
|
||||||
|
const box=scene.boxes.find(item => item.id===state.selected);
|
||||||
|
if (!box) {
|
||||||
|
root.innerHTML=`<span class="detail-kind">MACHINE OVERVIEW</span><h3>${escapeHtml(system.hostname || 'GPU Node')}</h3><dl>
|
||||||
|
<div><dt>CPU Model</dt><dd>${escapeHtml(system.cpu_model || '—')}</dd></div>
|
||||||
|
<div><dt>CPU Topology</dt><dd>${escapeHtml(`${system.cpu_sockets || '—'} sockets · ${system.cpu_logical || '—'} logical`)}</dd></div>
|
||||||
|
<div><dt>GPU</dt><dd>${escapeHtml(`${gpus.length} × ${gpus[0]?.name || '—'}`)}</dd></div>
|
||||||
|
<div><dt>Virtualization</dt><dd>${escapeHtml(system.virtualization || 'bare metal / unknown')}</dd></div>
|
||||||
|
<div><dt>PCI Scope</dt><dd>${escapeHtml(data.pci_fabric?.scope || 'guest-visible')}</dd></div>
|
||||||
|
</dl>`;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const meta=box.meta || {};
|
||||||
|
const fields=Object.entries(meta).map(([key,value]) => `<div><dt>${escapeHtml(key)}</dt><dd>${escapeHtml(value)}</dd></div>`).join('');
|
||||||
|
root.innerHTML=`<div class="detail-head"><div><span class="detail-kind">${escapeHtml(box.kind.toUpperCase())}</span><h3>${escapeHtml(box.label)}</h3></div><span class="detail-status">${escapeHtml(box.sublabel || '')}</span></div><dl>${fields || '<div><dt>Info</dt><dd>无附加字段</dd></div>'}</dl>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function pickTarget(x,y) {
|
||||||
|
return hitTargets
|
||||||
|
.filter(hit => hit.box.selectable && x>=hit.rect.left && x<=hit.rect.right && y>=hit.rect.top && y<=hit.rect.bottom)
|
||||||
|
.sort((a,b) => {
|
||||||
|
const areaA=(a.rect.right-a.rect.left)*(a.rect.bottom-a.rect.top);
|
||||||
|
const areaB=(b.rect.right-b.rect.left)*(b.rect.bottom-b.rect.top);
|
||||||
|
return areaA-areaB || b.depth-a.depth;
|
||||||
|
})[0];
|
||||||
|
}
|
||||||
|
|
||||||
|
canvas.addEventListener('pointerdown',event => {
|
||||||
|
cameraAnimation+=1;
|
||||||
|
canvas.setPointerCapture(event.pointerId);
|
||||||
|
pointer={
|
||||||
|
id:event.pointerId,x:event.clientX,y:event.clientY,startX:event.clientX,startY:event.clientY,
|
||||||
|
yaw:state.yaw,pitch:state.pitch,target:{...state.target},pan:event.shiftKey || event.button===1
|
||||||
|
};
|
||||||
|
});
|
||||||
|
canvas.addEventListener('pointermove',event => {
|
||||||
|
if (!pointer || pointer.id!==event.pointerId) return;
|
||||||
|
if (pointer.pan) {
|
||||||
|
const factor=.022/state.zoom;
|
||||||
|
state.target.x=pointer.target.x-(event.clientX-pointer.x)*factor;
|
||||||
|
state.target.y=pointer.target.y+(event.clientY-pointer.y)*factor;
|
||||||
|
} else {
|
||||||
|
state.yaw=pointer.yaw+(event.clientX-pointer.x)*.008;
|
||||||
|
state.pitch=Math.max(-.15,Math.min(1.1,pointer.pitch+(event.clientY-pointer.y)*.006));
|
||||||
|
}
|
||||||
|
draw();
|
||||||
|
});
|
||||||
|
canvas.addEventListener('pointerup',event => {
|
||||||
|
if (!pointer || pointer.id!==event.pointerId) return;
|
||||||
|
const moved=Math.hypot(event.clientX-pointer.startX,event.clientY-pointer.startY);
|
||||||
|
if (moved<6) {
|
||||||
|
const rect=canvas.getBoundingClientRect(),x=event.clientX-rect.left,y=event.clientY-rect.top;
|
||||||
|
const target=pickTarget(x,y);
|
||||||
|
state.selected=target?.box.id || null;renderDetail();updateCameraControls();draw();
|
||||||
|
}
|
||||||
|
pointer=null;
|
||||||
|
});
|
||||||
|
canvas.addEventListener('dblclick',event => {
|
||||||
|
const rect=canvas.getBoundingClientRect(),x=event.clientX-rect.left,y=event.clientY-rect.top;
|
||||||
|
const target=pickTarget(x,y);
|
||||||
|
if (!target) return;
|
||||||
|
state.selected=target.box.id;
|
||||||
|
const nextScope=nextScopeToward(target.box);
|
||||||
|
if (nextScope) enterScope(nextScope);
|
||||||
|
else if (isShell(target.box)) enterScope(target.box);
|
||||||
|
else { renderDetail();setCameraMode('inside',target.box); }
|
||||||
|
});
|
||||||
|
canvas.addEventListener('wheel',event => {
|
||||||
|
event.preventDefault();
|
||||||
|
cameraAnimation+=1;
|
||||||
|
const limits=state.cameraMode==='inside' ? [.72,3.8] : [.55,2.3];
|
||||||
|
state.zoom=Math.max(limits[0],Math.min(limits[1],state.zoom*(event.deltaY>0 ? .92 : 1.08)));draw();
|
||||||
|
},{passive:false});
|
||||||
|
canvas.addEventListener('keydown',event => {
|
||||||
|
const key=event.key;
|
||||||
|
if (['ArrowLeft','ArrowRight','ArrowUp','ArrowDown'].includes(key)) {
|
||||||
|
event.preventDefault();
|
||||||
|
cameraAnimation+=1;
|
||||||
|
if (key==='ArrowLeft') state.yaw-=.08;if (key==='ArrowRight') state.yaw+=.08;
|
||||||
|
if (key==='ArrowUp') state.pitch=Math.min(1.1,state.pitch+.06);if (key==='ArrowDown') state.pitch=Math.max(-.15,state.pitch-.06);
|
||||||
|
draw();
|
||||||
|
}
|
||||||
|
if (key==='Escape') {
|
||||||
|
event.preventDefault();state.selected=null;renderDetail();setCameraMode('outside');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
new ResizeObserver(draw).observe(canvas);
|
||||||
|
buildViewControls();buildScene();updateCameraControls();draw();
|
||||||
|
})();
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
133
scripts/gpu_node_profile/tests/fixtures/b300_snapshot.json
vendored
Normal file
133
scripts/gpu_node_profile/tests/fixtures/b300_snapshot.json
vendored
Normal file
@ -0,0 +1,133 @@
|
|||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"generated_at": "2026-07-25T09:45:54+08:00",
|
||||||
|
"collector_version": "1.0.0",
|
||||||
|
"system": {
|
||||||
|
"hostname": "root-2",
|
||||||
|
"os": "Ubuntu 24.04.1 LTS",
|
||||||
|
"kernel": "6.8.0-71-generic",
|
||||||
|
"architecture": "x86_64",
|
||||||
|
"manufacturer": "Baidu Cloud",
|
||||||
|
"product": "Baidu Cloud BCC",
|
||||||
|
"product_version": "6.10.0",
|
||||||
|
"virtualization": "kvm",
|
||||||
|
"cpu_model": "Intel(R) Xeon(R) 6787P",
|
||||||
|
"numa_nodes": 2,
|
||||||
|
"cpu_logical": 332,
|
||||||
|
"cpu_sockets": 2,
|
||||||
|
"cpu_cores_per_socket": 83,
|
||||||
|
"cpu_physical_cores": 166,
|
||||||
|
"cpu_threads_per_core": 2,
|
||||||
|
"memory_total_kib": 4227858432,
|
||||||
|
"memory_available_kib": 3549581076,
|
||||||
|
"memory_used_kib": 678277356
|
||||||
|
},
|
||||||
|
"software": {
|
||||||
|
"nvidia_driver": "595.71.05",
|
||||||
|
"cuda_toolkit": "12.8",
|
||||||
|
"cuda_note": "Toolkit selected by nvcc; application/container runtime may differ."
|
||||||
|
},
|
||||||
|
"gpus": [
|
||||||
|
{"index": 0, "name": "NVIDIA B30Z", "uuid": "GPU-0", "bdf": "63:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 34720, "temperature_c": 48, "utilization_pct": 12, "pstate": "P0", "power_w": 312, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 0},
|
||||||
|
{"index": 1, "name": "NVIDIA B30Z", "uuid": "GPU-1", "bdf": "67:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 50812, "temperature_c": 51, "utilization_pct": 18, "pstate": "P0", "power_w": 326, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 0},
|
||||||
|
{"index": 2, "name": "NVIDIA B30Z", "uuid": "GPU-2", "bdf": "6b:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 19236, "temperature_c": 47, "utilization_pct": 8, "pstate": "P0", "power_w": 296, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 0},
|
||||||
|
{"index": 3, "name": "NVIDIA B30Z", "uuid": "GPU-3", "bdf": "6f:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 62840, "temperature_c": 54, "utilization_pct": 23, "pstate": "P0", "power_w": 341, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 0},
|
||||||
|
{"index": 4, "name": "NVIDIA B30Z", "uuid": "GPU-4", "bdf": "a3:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 41560, "temperature_c": 50, "utilization_pct": 15, "pstate": "P0", "power_w": 318, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 1},
|
||||||
|
{"index": 5, "name": "NVIDIA B30Z", "uuid": "GPU-5", "bdf": "a7:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 28410, "temperature_c": 49, "utilization_pct": 10, "pstate": "P0", "power_w": 307, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 1},
|
||||||
|
{"index": 6, "name": "NVIDIA B30Z", "uuid": "GPU-6", "bdf": "ab:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 73320, "temperature_c": 56, "utilization_pct": 29, "pstate": "P0", "power_w": 359, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 1},
|
||||||
|
{"index": 7, "name": "NVIDIA B30Z", "uuid": "GPU-7", "bdf": "af:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 22350, "temperature_c": 48, "utilization_pct": 9, "pstate": "P0", "power_w": 301, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 1}
|
||||||
|
],
|
||||||
|
"topology": {
|
||||||
|
"rows": {},
|
||||||
|
"nic_legend": {
|
||||||
|
"NIC1": "mlx5_bond_0", "NIC2": "mlx5_bond_1", "NIC3": "mlx5_bond_2", "NIC4": "mlx5_bond_3",
|
||||||
|
"NIC5": "mlx5_bond_4", "NIC6": "mlx5_bond_5", "NIC7": "mlx5_bond_6", "NIC8": "mlx5_bond_7"
|
||||||
|
},
|
||||||
|
"gpu_nic": {
|
||||||
|
"GPU0": [{"slot": "NIC1", "rdma": "mlx5_bond_0", "relation": "PIX"}],
|
||||||
|
"GPU1": [{"slot": "NIC2", "rdma": "mlx5_bond_1", "relation": "PIX"}],
|
||||||
|
"GPU2": [{"slot": "NIC3", "rdma": "mlx5_bond_2", "relation": "PIX"}],
|
||||||
|
"GPU3": [{"slot": "NIC4", "rdma": "mlx5_bond_3", "relation": "PIX"}],
|
||||||
|
"GPU4": [{"slot": "NIC5", "rdma": "mlx5_bond_4", "relation": "PIX"}],
|
||||||
|
"GPU5": [{"slot": "NIC6", "rdma": "mlx5_bond_5", "relation": "PIX"}],
|
||||||
|
"GPU6": [{"slot": "NIC7", "rdma": "mlx5_bond_6", "relation": "PIX"}],
|
||||||
|
"GPU7": [{"slot": "NIC8", "rdma": "mlx5_bond_7", "relation": "PIX"}]
|
||||||
|
},
|
||||||
|
"gpu_affinity": {
|
||||||
|
"GPU0": {"cpu_affinity": "0-165", "numa_affinity": "0", "gpu_numa_id": "N/A"},
|
||||||
|
"GPU1": {"cpu_affinity": "0-165", "numa_affinity": "0", "gpu_numa_id": "N/A"},
|
||||||
|
"GPU2": {"cpu_affinity": "0-165", "numa_affinity": "0", "gpu_numa_id": "N/A"},
|
||||||
|
"GPU3": {"cpu_affinity": "0-165", "numa_affinity": "0", "gpu_numa_id": "N/A"},
|
||||||
|
"GPU4": {"cpu_affinity": "166-331", "numa_affinity": "1", "gpu_numa_id": "N/A"},
|
||||||
|
"GPU5": {"cpu_affinity": "166-331", "numa_affinity": "1", "gpu_numa_id": "N/A"},
|
||||||
|
"GPU6": {"cpu_affinity": "166-331", "numa_affinity": "1", "gpu_numa_id": "N/A"},
|
||||||
|
"GPU7": {"cpu_affinity": "166-331", "numa_affinity": "1", "gpu_numa_id": "N/A"}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"pci_fabric": {
|
||||||
|
"scope": "guest-visible",
|
||||||
|
"virtualization": "passthrough-or-virtual-pci",
|
||||||
|
"bridges": [
|
||||||
|
{"bdf": "61:00.0", "parent_bdf": "60:00.0", "numa": 0, "description": "PCI bridge", "children": ["63:00.0", "64:00.0"]},
|
||||||
|
{"bdf": "a1:00.0", "parent_bdf": "a0:00.0", "numa": 1, "description": "PCI bridge", "children": ["a3:00.0", "a4:00.0"]}
|
||||||
|
],
|
||||||
|
"endpoints": [
|
||||||
|
{"kind": "gpu", "label": "GPU0", "bdf": "63:00.0", "numa": 0, "path": ["60:00.0", "61:00.0", "63:00.0"], "description": "NVIDIA GPU"},
|
||||||
|
{"kind": "nic", "label": "eth1", "bdf": "64:00.0", "numa": 0, "path": ["60:00.0", "61:00.0", "64:00.0"], "description": "Ethernet controller"},
|
||||||
|
{"kind": "gpu", "label": "GPU4", "bdf": "a3:00.0", "numa": 1, "path": ["a0:00.0", "a1:00.0", "a3:00.0"], "description": "NVIDIA GPU"},
|
||||||
|
{"kind": "nic", "label": "eth9", "bdf": "a4:00.0", "numa": 1, "path": ["a0:00.0", "a1:00.0", "a4:00.0"], "description": "Ethernet controller"}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"nvlink": {
|
||||||
|
"fabric_type": "NVSwitch Fabric",
|
||||||
|
"per_gpu": {
|
||||||
|
"GPU0": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
|
||||||
|
"GPU1": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
|
||||||
|
"GPU2": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
|
||||||
|
"GPU3": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
|
||||||
|
"GPU4": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
|
||||||
|
"GPU5": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
|
||||||
|
"GPU6": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
|
||||||
|
"GPU7": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125]
|
||||||
|
},
|
||||||
|
"active_links": 144,
|
||||||
|
"links_per_gpu": 18,
|
||||||
|
"speed_gbs": 53.125
|
||||||
|
},
|
||||||
|
"nics": [
|
||||||
|
{
|
||||||
|
"name": "eth0",
|
||||||
|
"driver": "mlx5_core",
|
||||||
|
"bdf": "02:00.0",
|
||||||
|
"numa": 0,
|
||||||
|
"speed_mbps": 200000,
|
||||||
|
"state": "up",
|
||||||
|
"carrier": 1,
|
||||||
|
"mac": "fa:20:20:2e:4a:3f",
|
||||||
|
"addresses": ["192.0.2.10/24"],
|
||||||
|
"default_route": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"bonds": [
|
||||||
|
{"name": "bond0", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_0", "slaves": [{"name": "eth1", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "64:00.0", "numa": 0}, {"name": "eth2", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "64:00.1", "numa": 0}]},
|
||||||
|
{"name": "bond1", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_1", "slaves": [{"name": "eth3", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "68:00.0", "numa": 0}, {"name": "eth4", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "68:00.1", "numa": 0}]},
|
||||||
|
{"name": "bond2", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_2", "slaves": [{"name": "eth5", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "6c:00.0", "numa": 0}, {"name": "eth6", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "6c:00.1", "numa": 0}]},
|
||||||
|
{"name": "bond3", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_3", "slaves": [{"name": "eth7", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "70:00.0", "numa": 0}, {"name": "eth8", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "70:00.1", "numa": 0}]},
|
||||||
|
{"name": "bond4", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_4", "slaves": [{"name": "eth9", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "a4:00.0", "numa": 1}, {"name": "eth10", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "a4:00.1", "numa": 1}]},
|
||||||
|
{"name": "bond5", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_5", "slaves": [{"name": "eth11", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "a8:00.0", "numa": 1}, {"name": "eth12", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "a8:00.1", "numa": 1}]},
|
||||||
|
{"name": "bond6", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_6", "slaves": [{"name": "eth13", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "ac:00.0", "numa": 1}, {"name": "eth14", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "ac:00.1", "numa": 1}]},
|
||||||
|
{"name": "bond7", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_7", "slaves": [{"name": "eth15", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "b0:00.0", "numa": 1}, {"name": "eth16", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "b0:00.1", "numa": 1}]}
|
||||||
|
],
|
||||||
|
"ib_devices": [
|
||||||
|
{"name": "mlx5_17", "ca_type": "MT4129", "firmware": "28.46.5500", "state": "Active", "physical_state": "LinkUp", "rate_gbps": 100, "lid": "1", "link_layer": "InfiniBand", "bdf": "b3:00.0", "numa": 1},
|
||||||
|
{"name": "mlx5_18", "ca_type": "MT4129", "firmware": "28.46.5500", "state": "Active", "physical_state": "LinkUp", "rate_gbps": 100, "lid": "4", "link_layer": "InfiniBand", "bdf": "b4:00.0", "numa": 1},
|
||||||
|
{"name": "mlx5_19", "ca_type": "MT4129", "firmware": "28.46.5500", "state": "Active", "physical_state": "LinkUp", "rate_gbps": 100, "lid": "5", "link_layer": "InfiniBand", "bdf": "b5:00.0", "numa": 1},
|
||||||
|
{"name": "mlx5_20", "ca_type": "MT4129", "firmware": "28.46.5500", "state": "Active", "physical_state": "LinkUp", "rate_gbps": 100, "lid": "6", "link_layer": "InfiniBand", "bdf": "b6:00.0", "numa": 1}
|
||||||
|
],
|
||||||
|
"bmc": {
|
||||||
|
"detected": false,
|
||||||
|
"note": "独立 BMC 带外口未向当前 OS/IPMI 通道暴露"
|
||||||
|
},
|
||||||
|
"rdma_map": {},
|
||||||
|
"notes": []
|
||||||
|
}
|
||||||
430
scripts/gpu_node_profile/tests/test_gpu_node_profile.py
Normal file
430
scripts/gpu_node_profile/tests/test_gpu_node_profile.py
Normal file
@ -0,0 +1,430 @@
|
|||||||
|
import io
|
||||||
|
import json
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import gpu_node_profile as profile
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
FIXTURE = ROOT / "tests" / "fixtures" / "b300_snapshot.json"
|
||||||
|
|
||||||
|
|
||||||
|
class ParserTests(unittest.TestCase):
|
||||||
|
def test_normalize_bdf_uses_endpoint_from_sysfs_path(self):
|
||||||
|
path = (
|
||||||
|
"/sys/devices/pci0000:a0/0000:a0:00.4/0000:b1:00.0/"
|
||||||
|
"0000:b2:00.0/0000:b3:00.0"
|
||||||
|
)
|
||||||
|
self.assertEqual(profile.normalize_bdf(path), "b3:00.0")
|
||||||
|
|
||||||
|
def test_cuda_build_version_is_concise(self):
|
||||||
|
self.assertEqual(profile.concise_version("12.8.20250303"), "12.8")
|
||||||
|
|
||||||
|
def test_parse_topology_and_mapping(self):
|
||||||
|
text = """
|
||||||
|
GPU0 GPU1 NIC0 NIC1 CPU Affinity NUMA Affinity
|
||||||
|
GPU0 X NV18 PIX SYS 0-165 0
|
||||||
|
GPU1 NV18 X SYS PIX 166-331 1
|
||||||
|
NIC Legend:
|
||||||
|
NIC0: mlx5_bond_0
|
||||||
|
NIC1: mlx5_bond_1
|
||||||
|
"""
|
||||||
|
parsed = profile.parse_topology(text)
|
||||||
|
self.assertEqual(parsed["nic_legend"]["NIC0"], "mlx5_bond_0")
|
||||||
|
self.assertEqual(parsed["gpu_nic"]["GPU0"][0]["relation"], "PIX")
|
||||||
|
self.assertEqual(
|
||||||
|
parsed["gpu_affinity"]["GPU0"]["cpu_affinity"], "0-165"
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
parsed["gpu_affinity"]["GPU1"]["numa_affinity"], "1"
|
||||||
|
)
|
||||||
|
self.assertNotIn("CPU", parsed["rows"]["GPU0"])
|
||||||
|
|
||||||
|
def test_parse_bond(self):
|
||||||
|
text = """
|
||||||
|
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
|
||||||
|
Transmit Hash Policy: layer3+4 (1)
|
||||||
|
MII Status: up
|
||||||
|
Slave Interface: eth1
|
||||||
|
MII Status: up
|
||||||
|
Speed: 400000 Mbps
|
||||||
|
Link Failure Count: 0
|
||||||
|
"""
|
||||||
|
bond = profile.parse_bond_text("bond0", text)
|
||||||
|
self.assertEqual(bond["status"], "up")
|
||||||
|
self.assertEqual(bond["slaves"][0]["speed_mbps"], 400000)
|
||||||
|
|
||||||
|
def test_nvlink_fabric_type_does_not_assume_nvswitch(self):
|
||||||
|
direct = {"rows": {"GPU0": {"GPU1": "NV4"}, "GPU1": {"GPU0": "NV4"}}}
|
||||||
|
self.assertEqual(
|
||||||
|
profile.infer_nvlink_fabric_type(direct, 2), "NVLink Fabric"
|
||||||
|
)
|
||||||
|
switched = {
|
||||||
|
"rows": {
|
||||||
|
f"GPU{source}": {
|
||||||
|
f"GPU{target}": "NV18"
|
||||||
|
for target in range(4)
|
||||||
|
if target != source
|
||||||
|
}
|
||||||
|
for source in range(4)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
self.assertEqual(
|
||||||
|
profile.infer_nvlink_fabric_type(switched, 4), "NVSwitch Fabric"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_render_is_offline_native_html(self):
|
||||||
|
snapshot = json.loads(FIXTURE.read_text(encoding="utf-8"))
|
||||||
|
profile.attach_gpu_bond_mapping(snapshot)
|
||||||
|
report = profile.render_html(
|
||||||
|
snapshot, {"gpus": 8, "bonds": 8, "ib": 4}
|
||||||
|
)
|
||||||
|
self.assertIn("GPU ↔ RDMA 亲和画像", report)
|
||||||
|
self.assertIn("系统与软件", report)
|
||||||
|
self.assertIn("CPU 与内存", report)
|
||||||
|
self.assertIn("带内与带外管理网络", report)
|
||||||
|
self.assertIn("eth0", report)
|
||||||
|
self.assertIn("带外 · BMC", report)
|
||||||
|
self.assertIn("HEALTH 与验证结果", report)
|
||||||
|
self.assertIn("CPU / NUMA 亲和", report)
|
||||||
|
self.assertIn("GPU 功耗遥测", report)
|
||||||
|
self.assertIn("GPU 瞬时功耗", report)
|
||||||
|
self.assertIn("2.56 kW / 8.80 kW", report)
|
||||||
|
self.assertIn("瞬时 / 1.10 kW 上限", report)
|
||||||
|
self.assertIn(">P0<", report)
|
||||||
|
self.assertIn("GPU0–3", report)
|
||||||
|
self.assertIn("推荐 CPU 0-165", report)
|
||||||
|
self.assertIn("不代表当前训练进程已经绑核", report)
|
||||||
|
self.assertIn('data-numa-filter="all"', report)
|
||||||
|
self.assertIn('data-numa-filter="0"', report)
|
||||||
|
self.assertIn('data-numa-filter="1"', report)
|
||||||
|
self.assertIn("function setNumaView(value)", report)
|
||||||
|
self.assertIn(".rail[hidden],.numa-ruler [hidden]", report)
|
||||||
|
self.assertIn(".ib-card .card-head { align-items:flex-start; flex-wrap:wrap;", report)
|
||||||
|
self.assertIn("eth0 承载默认路由且链路 Up", report)
|
||||||
|
self.assertIn("mlx5_bond_7", report)
|
||||||
|
self.assertIn("NVLINK / NVSWITCH FABRIC", report)
|
||||||
|
self.assertLess(report.index("系统与软件"), report.index("GPU ↔ RDMA 亲和画像"))
|
||||||
|
self.assertLess(report.index("带内与带外管理网络"), report.index("HEALTH 与验证结果"))
|
||||||
|
self.assertNotIn("mermaid", report.lower())
|
||||||
|
self.assertNotIn("https://", report.lower())
|
||||||
|
self.assertNotIn("http://", report.lower())
|
||||||
|
|
||||||
|
def test_render_3d_topology_is_offline_and_json_driven(self):
|
||||||
|
snapshot = json.loads(FIXTURE.read_text(encoding="utf-8"))
|
||||||
|
profile.attach_gpu_bond_mapping(snapshot)
|
||||||
|
report = profile.render_topology_3d_html(snapshot)
|
||||||
|
self.assertIn("GPU Machine 3D CT", report)
|
||||||
|
self.assertIn('id="machine-data"', report)
|
||||||
|
self.assertIn('id="topology-canvas"', report)
|
||||||
|
self.assertIn("function buildScene()", report)
|
||||||
|
self.assertIn("function deepestCommonPath(paths)", report)
|
||||||
|
self.assertIn('data-camera="inside"', report)
|
||||||
|
self.assertIn("function setCameraMode(mode,box=null,animate=true)", report)
|
||||||
|
self.assertIn("双击壳体逐层进入", report)
|
||||||
|
self.assertIn("state.cameraMode==='inside'", report)
|
||||||
|
self.assertIn("kind:'bond'", report)
|
||||||
|
self.assertIn("kind:'roce-port'", report)
|
||||||
|
self.assertIn("GPU endpoints only · not NIC", report)
|
||||||
|
self.assertIn("function drawEdge(edge,scale,selected,overlay=false)", report)
|
||||||
|
self.assertIn("function scopePath(box)", report)
|
||||||
|
self.assertIn("function nextScopeToward(box)", report)
|
||||||
|
self.assertIn("function enterScope(box,animate=true)", report)
|
||||||
|
self.assertIn('id="camera-back"', report)
|
||||||
|
self.assertIn("parentId:`pair-${gpu.index}`", report)
|
||||||
|
self.assertIn("PCIe Root Complex", report)
|
||||||
|
self.assertIn("GDR candidate", report)
|
||||||
|
self.assertIn('"scope": "guest-visible"', report)
|
||||||
|
self.assertNotIn("https://", report.lower())
|
||||||
|
self.assertNotIn("http://", report.lower())
|
||||||
|
|
||||||
|
def test_html_templates_are_separate_source_files(self):
|
||||||
|
operations = profile.load_html_template("gpu_node_profile.html")
|
||||||
|
topology = profile.load_html_template("gpu_topology_3d.html")
|
||||||
|
self.assertIn("@@BODY@@", operations)
|
||||||
|
self.assertIn("@@MACHINE_DATA@@", topology)
|
||||||
|
self.assertNotIn("https://", operations.lower())
|
||||||
|
self.assertNotIn("https://", topology.lower())
|
||||||
|
|
||||||
|
def test_build_pci_fabric_preserves_bridge_hierarchy(self):
|
||||||
|
gpus = [
|
||||||
|
{
|
||||||
|
"index": 0,
|
||||||
|
"bdf": "63:00.0",
|
||||||
|
"numa": 0,
|
||||||
|
"pci_path": ["60:00.0", "61:00.0", "63:00.0"],
|
||||||
|
}
|
||||||
|
]
|
||||||
|
nics = [
|
||||||
|
{
|
||||||
|
"name": "eth1",
|
||||||
|
"bdf": "64:00.0",
|
||||||
|
"numa": 0,
|
||||||
|
"pci_path": ["60:00.0", "61:00.0", "64:00.0"],
|
||||||
|
}
|
||||||
|
]
|
||||||
|
with patch.object(profile, "pci_numa", return_value=0):
|
||||||
|
fabric = profile.build_pci_fabric(gpus, nics, [], {})
|
||||||
|
self.assertEqual(fabric["scope"], "guest-visible")
|
||||||
|
bridge = next(item for item in fabric["bridges"] if item["bdf"] == "61:00.0")
|
||||||
|
self.assertEqual(bridge["children"], ["63:00.0", "64:00.0"])
|
||||||
|
|
||||||
|
def test_standalone_roce_endpoints_map_uniquely_to_gpus(self):
|
||||||
|
nics = [
|
||||||
|
{
|
||||||
|
"name": "eth0",
|
||||||
|
"bdf": "0b:00.0",
|
||||||
|
"numa": 0,
|
||||||
|
"pci_path": ["08:00.0", "0b:00.0"],
|
||||||
|
"speed_mbps": 100000,
|
||||||
|
"state": "up",
|
||||||
|
"carrier": 1,
|
||||||
|
"default_route": True,
|
||||||
|
"is_bond": False,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "eth1",
|
||||||
|
"bdf": "5a:00.0",
|
||||||
|
"numa": 0,
|
||||||
|
"pci_path": ["4c:00.0", "54:00.0", "5a:00.0"],
|
||||||
|
"speed_mbps": 100000,
|
||||||
|
"state": "up",
|
||||||
|
"carrier": 1,
|
||||||
|
"default_route": False,
|
||||||
|
"is_bond": False,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "eth2",
|
||||||
|
"bdf": "5a:00.1",
|
||||||
|
"numa": 0,
|
||||||
|
"pci_path": ["4c:00.0", "54:00.0", "5a:00.1"],
|
||||||
|
"speed_mbps": 100000,
|
||||||
|
"state": "up",
|
||||||
|
"carrier": 1,
|
||||||
|
"default_route": False,
|
||||||
|
"is_bond": False,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
rdma_map = {"mlx5_0": "eth0", "mlx5_1": "eth1", "mlx5_2": "eth2"}
|
||||||
|
endpoints = profile.build_network_endpoints(
|
||||||
|
nics,
|
||||||
|
[],
|
||||||
|
[],
|
||||||
|
rdma_map,
|
||||||
|
{
|
||||||
|
"mlx5_1": {"state": "ACTIVE"},
|
||||||
|
"mlx5_2": {"state": "ACTIVE"},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
self.assertEqual([item["name"] for item in endpoints], ["eth1", "eth2"])
|
||||||
|
self.assertTrue(all(item["type"] == "standalone" for item in endpoints))
|
||||||
|
|
||||||
|
snapshot = {
|
||||||
|
"gpus": [
|
||||||
|
{"index": 0, "pci_path": ["4c:00.0", "53:00.0"], "numa": 0},
|
||||||
|
{"index": 1, "pci_path": ["4c:00.0", "58:00.0"], "numa": 0},
|
||||||
|
],
|
||||||
|
"nics": nics,
|
||||||
|
"bonds": [],
|
||||||
|
"ib_devices": [],
|
||||||
|
"rdma_map": rdma_map,
|
||||||
|
"network_endpoints": endpoints,
|
||||||
|
"topology": {
|
||||||
|
"gpu_affinity": {},
|
||||||
|
"gpu_nic": {
|
||||||
|
"GPU0": [
|
||||||
|
{"rdma": "mlx5_1", "relation": "PXB"},
|
||||||
|
{"rdma": "mlx5_2", "relation": "PXB"},
|
||||||
|
],
|
||||||
|
"GPU1": [
|
||||||
|
{"rdma": "mlx5_1", "relation": "PXB"},
|
||||||
|
{"rdma": "mlx5_2", "relation": "PXB"},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
profile.attach_gpu_network_mapping(snapshot)
|
||||||
|
primary_ids = {
|
||||||
|
gpu.get("primary_network_id") for gpu in snapshot["gpus"]
|
||||||
|
}
|
||||||
|
self.assertEqual(primary_ids, {"rdma:mlx5_1", "rdma:mlx5_2"})
|
||||||
|
self.assertTrue(
|
||||||
|
all(gpu.get("network_relation") == "PXB" for gpu in snapshot["gpus"])
|
||||||
|
)
|
||||||
|
health = profile.evaluate(
|
||||||
|
snapshot, {"gpus": 2, "bonds": None, "ib": None}
|
||||||
|
)
|
||||||
|
affinity = next(
|
||||||
|
item for item in health["checks"] if item["title"] == "GPU ↔ RDMA 亲和"
|
||||||
|
)
|
||||||
|
self.assertEqual(affinity["level"], "ok")
|
||||||
|
self.assertIn("PXB 2", affinity["detail"])
|
||||||
|
operations = profile.render_html(
|
||||||
|
snapshot, {"gpus": 2, "bonds": None, "ib": None}
|
||||||
|
)
|
||||||
|
topology = profile.render_topology_3d_html(snapshot)
|
||||||
|
self.assertEqual(operations.count("ROCE · DIRECT"), 2)
|
||||||
|
self.assertNotIn("未发现对应网络路径", operations)
|
||||||
|
self.assertIn("Mode:'Standalone RDMA'", topology)
|
||||||
|
self.assertIn("endpoint.topology_visibility", topology)
|
||||||
|
|
||||||
|
def test_unique_mapping_uses_global_minimum_not_greedy_first_match(self):
|
||||||
|
endpoints = [
|
||||||
|
{
|
||||||
|
"id": "rdma:mlx5_1",
|
||||||
|
"type": "standalone",
|
||||||
|
"protocol": "RoCE",
|
||||||
|
"name": "eth1",
|
||||||
|
"rdma": "mlx5_1",
|
||||||
|
"role": "compute",
|
||||||
|
"members": [{"name": "eth1", "pci_path": []}],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "rdma:mlx5_2",
|
||||||
|
"type": "standalone",
|
||||||
|
"protocol": "RoCE",
|
||||||
|
"name": "eth2",
|
||||||
|
"rdma": "mlx5_2",
|
||||||
|
"role": "compute",
|
||||||
|
"members": [{"name": "eth2", "pci_path": []}],
|
||||||
|
},
|
||||||
|
]
|
||||||
|
snapshot = {
|
||||||
|
"gpus": [
|
||||||
|
{"index": 0, "pci_path": [], "numa": 0},
|
||||||
|
{"index": 1, "pci_path": [], "numa": 0},
|
||||||
|
],
|
||||||
|
"bonds": [],
|
||||||
|
"network_endpoints": endpoints,
|
||||||
|
"topology": {
|
||||||
|
"gpu_affinity": {},
|
||||||
|
"gpu_nic": {
|
||||||
|
"GPU0": [
|
||||||
|
{"rdma": "mlx5_1", "relation": "PIX"},
|
||||||
|
{"rdma": "mlx5_2", "relation": "PIX"},
|
||||||
|
],
|
||||||
|
"GPU1": [
|
||||||
|
{"rdma": "mlx5_1", "relation": "PIX"},
|
||||||
|
{"rdma": "mlx5_2", "relation": "PHB"},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
profile.attach_gpu_network_mapping(snapshot)
|
||||||
|
self.assertEqual(snapshot["gpus"][0]["network"], "eth2")
|
||||||
|
self.assertEqual(snapshot["gpus"][1]["network"], "eth1")
|
||||||
|
self.assertTrue(
|
||||||
|
all(gpu["network_relation"] == "PIX" for gpu in snapshot["gpus"])
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_non_rdma_bond_is_not_an_accelerator_endpoint(self):
|
||||||
|
management_bond = {
|
||||||
|
"name": "bond-mgmt",
|
||||||
|
"rdma": "",
|
||||||
|
"status": "up",
|
||||||
|
"slaves": [{"name": "eno1", "status": "up"}],
|
||||||
|
}
|
||||||
|
endpoints = profile.build_network_endpoints(
|
||||||
|
[],
|
||||||
|
[management_bond],
|
||||||
|
[],
|
||||||
|
{},
|
||||||
|
)
|
||||||
|
self.assertEqual(endpoints, [])
|
||||||
|
health = profile.evaluate(
|
||||||
|
{
|
||||||
|
"gpus": [],
|
||||||
|
"bonds": [management_bond],
|
||||||
|
"ib_devices": [],
|
||||||
|
"network_endpoints": [],
|
||||||
|
"topology": {},
|
||||||
|
},
|
||||||
|
{"gpus": None, "bonds": 0, "ib": None},
|
||||||
|
)
|
||||||
|
self.assertFalse(
|
||||||
|
any(item["title"] == "Bond 数量不符合预期" for item in health["checks"])
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_power_telemetry_health_does_not_flag_p0_as_an_error(self):
|
||||||
|
snapshot = json.loads(FIXTURE.read_text(encoding="utf-8"))
|
||||||
|
profile.attach_gpu_bond_mapping(snapshot)
|
||||||
|
health = profile.evaluate(snapshot, {"gpus": 8, "bonds": 8, "ib": 4})
|
||||||
|
power_check = next(
|
||||||
|
item for item in health["checks"] if item["title"] == "GPU 功耗遥测"
|
||||||
|
)
|
||||||
|
self.assertEqual(power_check["level"], "ok")
|
||||||
|
self.assertIn("P0 8/8", power_check["detail"])
|
||||||
|
|
||||||
|
def test_collect_gpus_keeps_power_when_optional_range_is_unsupported(self):
|
||||||
|
core = (
|
||||||
|
"0, NVIDIA Test, GPU-0, 00000000:63:00.0, 595.71.05, "
|
||||||
|
"275040, 1024, 42, 75"
|
||||||
|
)
|
||||||
|
power = "0, P0, 875.5, 1100"
|
||||||
|
with (
|
||||||
|
patch.object(profile, "run", side_effect=[core, power, ""]),
|
||||||
|
patch.object(profile, "pci_numa", return_value=0),
|
||||||
|
):
|
||||||
|
gpus = profile.collect_gpus()
|
||||||
|
self.assertEqual(len(gpus), 1)
|
||||||
|
self.assertEqual(gpus[0]["pstate"], "P0")
|
||||||
|
self.assertEqual(gpus[0]["power_w"], 875.5)
|
||||||
|
self.assertEqual(gpus[0]["power_limit_w"], 1100)
|
||||||
|
self.assertNotIn("power_max_limit_w", gpus[0])
|
||||||
|
|
||||||
|
def test_incomplete_power_telemetry_is_a_warning(self):
|
||||||
|
snapshot = json.loads(FIXTURE.read_text(encoding="utf-8"))
|
||||||
|
snapshot["gpus"][0].pop("power_limit_w")
|
||||||
|
profile.attach_gpu_bond_mapping(snapshot)
|
||||||
|
health = profile.evaluate(snapshot, {"gpus": 8, "bonds": 8, "ib": 4})
|
||||||
|
power_check = next(
|
||||||
|
item for item in health["checks"] if item["title"] == "GPU功耗信息不完整"
|
||||||
|
)
|
||||||
|
self.assertEqual(power_check["level"], "warning")
|
||||||
|
report = profile.render_html(snapshot, {"gpus": 8, "bonds": 8, "ib": 4})
|
||||||
|
self.assertIn("遥测不完整", report)
|
||||||
|
|
||||||
|
def test_cli_renders_snapshot(self):
|
||||||
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
|
output = Path(directory) / "profile.html"
|
||||||
|
topology_output = Path(directory) / "topology-3d.html"
|
||||||
|
result = profile.main(
|
||||||
|
[
|
||||||
|
"--snapshot-in",
|
||||||
|
str(FIXTURE),
|
||||||
|
"--output",
|
||||||
|
str(output),
|
||||||
|
"--topology-output",
|
||||||
|
str(topology_output),
|
||||||
|
"--expected-gpus",
|
||||||
|
"8",
|
||||||
|
"--expected-bonds",
|
||||||
|
"8",
|
||||||
|
"--expected-ib",
|
||||||
|
"4",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
self.assertEqual(result, 0)
|
||||||
|
self.assertTrue(output.exists())
|
||||||
|
self.assertGreater(output.stat().st_size, 30_000)
|
||||||
|
self.assertTrue(topology_output.exists())
|
||||||
|
self.assertIn(
|
||||||
|
'id="topology-canvas"',
|
||||||
|
topology_output.read_text(encoding="utf-8"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_cli_can_emit_structured_snapshot_only(self):
|
||||||
|
buffer = io.StringIO()
|
||||||
|
with patch.object(profile.sys, "stdout", buffer):
|
||||||
|
result = profile.main(["--snapshot-in", str(FIXTURE), "--snapshot-stdout"])
|
||||||
|
self.assertEqual(result, 0)
|
||||||
|
snapshot = json.loads(buffer.getvalue())
|
||||||
|
self.assertEqual(snapshot["schema_version"], 1)
|
||||||
|
self.assertEqual(snapshot["pci_fabric"]["scope"], "guest-visible")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Loading…
x
Reference in New Issue
Block a user