Compare commits

..

No commits in common. "main" and "hanks/test_gpu" have entirely different histories.

36 changed files with 354 additions and 16295 deletions

1
.gitignore vendored
View File

@ -14,4 +14,3 @@ reports/
.venv/ .venv/
venv/ venv/
.qoder/* .qoder/*
.claude/settings.local.json

148
README.md
View File

@ -19,7 +19,6 @@
- [配置说明](#配置说明) - [配置说明](#配置说明)
- [测试 SOP(标准操作流程)](#测试-sop标准操作流程) - [测试 SOP(标准操作流程)](#测试-sop标准操作流程)
- [报告输出](#报告输出) - [报告输出](#报告输出)
- [GPU 节点画像工具](#gpu-节点画像工具)
- [故障排除](#故障排除) - [故障排除](#故障排除)
--- ---
@ -37,27 +36,13 @@ servertest/
│ ├── gpu_info.py # GPU 检测 & 信息 │ ├── gpu_info.py # GPU 检测 & 信息
│ ├── health_check.py # 健康诊断 │ ├── health_check.py # 健康诊断
│ ├── benchmark.py # 内存带宽 + 计算吞吐 │ ├── benchmark.py # 内存带宽 + 计算吞吐
│ ├── nccl_test.py # NCCL 单节点多卡通信 (NCCLTest) + 多机 Scaling (MultiNodeNCCLTest) │ ├── nccl_test.py # NCCL 多卡通信
│ ├── stress_test.py # GPU 压力/稳定性 │ ├── stress_test.py # GPU 压力/稳定性
│ ├── rdma_test.py # RDMA/InfiniBand │ ├── rdma_test.py # RDMA/InfiniBand
│ ├── training_sim.py # 训练模拟 │ ├── training_sim.py # 训练模拟
│ └── report.py # 报告生成 │ └── report.py # 报告生成
├── scripts/
│ ├── gpu_node_profile/ # JSON 驱动的运维画像与 3D CT
│ │ ├── gpu_node_profile.py # 只读采集与渲染入口
│ │ ├── templates/ # 两个独立 HTML 源模板
│ │ ├── examples/ # 脱敏真机 JSON 与自包含示例页面
│ │ └── tests/ # 解析与渲染回归测试
│ └── multinode_nccl/ # 独立 shell 脚本版 (历史备用)
│ ├── nccl_env.sh # 环境变量 (MPI路径/IB网卡/bond0)
│ ├── hosts_all # 全量 34 台节点列表
│ ├── run_scaling.sh # Scaling 测试 (2→32节点, 同交换机)
│ └── run_cross_switch_test.sh # 跨交换机新机接入验证
├── reports/ # 测试报告
│ ├── multinode_nccl_scaling_20260602.md
│ └── multinode_nccl_cross_switch_20260606.md
├── requirements.txt ├── requirements.txt
└── doc/ # 验收标准文档 └── 调研.md # 行业框架调研
``` ```
--- ---
@ -297,7 +282,7 @@ python3 gpu_tester.py --config /path/to/config.yaml --test all
默认配置:4096×4096 矩阵,10 次 warmup,100 次迭代。 默认配置:4096×4096 矩阵,10 次 warmup,100 次迭代。
### 5. NCCL Multi-GPU Test(单节点多卡通信) ### 5. NCCL Multi-GPU Test(多卡通信)
优先使用官方 nccl-tests(通过 mpirun 调用),不可用时 torchrun fallback。 优先使用官方 nccl-tests(通过 mpirun 调用),不可用时 torchrun fallback。
@ -314,34 +299,6 @@ python3 gpu_tester.py --config /path/to/config.yaml --test all
**NVLink 参考带宽:** A100/A800 ≥ 240 GB/s | H100/H200 ≥ 360 GB/s | B200/B300 ≥ 720 GB/s(40% NVLink 峰值) **NVLink 参考带宽:** A100/A800 ≥ 240 GB/s | H100/H200 ≥ 360 GB/s | B200/B300 ≥ 720 GB/s(40% NVLink 峰值)
### 5m. Multi-node NCCL Scaling(多机跨节点通信)
通过 mpirun + hostfile 在集群多台节点上运行 NCCL Scaling 测试,验证 IB fabric 健康和带宽随节点数的线性扩展。**菜单选 `m`,CLI 用 `--test multinode-nccl`。**
```bash
# 交互式菜单选 m
python3 gpu_tester.py
# 或 CLI 模式
python3 gpu_tester.py --test multinode-nccl
# 自定义节点数和 hostfile
python3 gpu_tester.py --test multinode-nccl --config configs/h20_cluster.yaml
```
配置(`configs/default.yaml` 的 `multinode_nccl` 段):
| 参数 | 默认值 | 说明 |
|---|---|---|
| `hosts_file` | `~/nccl_kit/hosts_all` | OpenMPI hostfile 路径 |
| `mpi_home` | `/usr/mpi/gcc/openmpi-4.1.7rc1` | MPI 安装路径 |
| `nccl_ib_hca` | `mlx5_0~5,8,9` | GPU 直连 IB 卡(排除存储 NIC) |
| `nccl_socket_ifname` | `bond0` | NCCL bootstrap 网卡 |
| `scale_levels` | `[2, 4, 8]` | 要测试的节点数列表 |
| `allreduce_min_busbw_gbps` | `310` | AllReduce PASS 阈值(GB/s) |
**验收标准:** AllReduce busbw ≥ 310 GB/s(4 节点起稳态 ~317 GB/s,32 节点无衰减)
### 6. GPU Stress Test(压力测试) ### 6. GPU Stress Test(压力测试)
使用 gpu-burn 进行长时满载测试,验证热稳定性和内存正确性。 使用 gpu-burn 进行长时满载测试,验证热稳定性和内存正确性。
@ -522,10 +479,9 @@ report:
--- ---
### SOP-3: 多节点集群验收 (H20-3e, 34 台) ### SOP-3: 多节点集群验收
**适用场景:** 多台 GPU 服务器组成训练集群,验证节点间 NCCL 通信及 IB fabric 健康。 **适用场景:** 多台 GPU 服务器组成训练集群,验证节点间通信。
**工具脚本位置:** `scripts/multinode_nccl/`,主节点 `~/nccl_kit/`(gpu-20)
``` ```
前置条件: 每台单节点已通过 SOP-1 前置条件: 每台单节点已通过 SOP-1
@ -534,61 +490,32 @@ report:
├── 在每台节点上执行 SOP-1 ├── 在每台节点上执行 SOP-1
└── 确保所有单节点测试通过 └── 确保所有单节点测试通过
步骤 2: 确认 IB 网卡拓扑 步骤 2: RDMA 网络测试
├── nvidia-smi topo -m ├── python3 gpu_tester.py --test rdma
├── 确认 GPU 直连 NIC (PIX): mlx5_0/1/2/3/4/5/8/9 (每节点 8 张) ├── 确认: IB 设备被识别
├── mlx5_6/7 为存储 NIC (NODE 级别), 不参与 NCCL ├── 确认: 端口状态 Active
└── 全集群 34 台拓扑一致: md5 均为 65a5f1bd43922dc879adadc5cce523b2 ├── 确认: 写带宽 ≥ 50 GB/s
├── 确认: 延迟 ≤ 10 μs
└── 异常: 检查 IB 线缆、交换机配置、子网管理器
步骤 3: 准备主节点环境 步骤 3: 多节点 NCCL 测试
├── # 在 gpu-20 上: ├── 在每个节点上配置:
├── mkdir -p ~/nccl_kit && cd ~/nccl_kit │ export MASTER_ADDR=<主节点IP>
├── # 上传 scripts/multinode_nccl/ 下的所有文件到 ~/nccl_kit/ │ export MASTER_PORT=29500
├── # 若 nccl-tests 为 MPI=0 编译, 需重编: │ export NCCL_SOCKET_IFNAME=ib0 # IB 网卡名
│ cd /opt/nccl-tests && make clean │ export NCCL_DEBUG=INFO
│ make MPI=1 MPI_HOME=/usr/mpi/gcc/openmpi-4.1.7rc1 CUDA_HOME=/usr/local/cuda -j ├── 运行 nccl-tests 手动测试:
│ ldd build/all_reduce_perf | grep -i mpi # 确认 libmpi 链接 │ mpirun -np <总GPU数> -hostfile hosts \
└── # 确认各节点免密 SSH 已打通 (集群预置共享 key) │ /opt/gpu-test-tools/nccl-tests/build/all_reduce_perf \
│ -b 8 -e 256M -f 2 -g 1 -w 5 -n 20
└── 确认: 多节点 AllReduce 带宽正常
步骤 4: Scaling 测试 (同交换机, 2→32 节点) 步骤 4: 训练验证
├── # 在 gpu-20 上: ├── python3 gpu_tester.py --test training
├── source ~/nccl_kit/nccl_env.sh ├── 可选: 加载更大模型(如 llama 模型)
├── bash ~/nccl_kit/run_scaling.sh 2 4 8 16 32 └── 确认: 训练 loss 正常下降
├── 验收标准:
│ AllReduce busbw: 2节点 ~480 GB/s, 4节点起稳定 ~317 GB/s
│ AllGather / ReduceScatter: ~315-340 GB/s (随节点数平稳)
│ AllToAll (1G): ~48-70 GB/s (随节点数轻微下降属正常)
└── 结果: ~/nccl_kit/results_*/summary.csv
步骤 5: 新机 / 跨交换机验证 (按需)
├── # 新机接入时需额外操作:
│ 1. SSH 互信: 在新机和现有节点间互相追加 authorized_keys
│ 2. 新机重编 nccl-tests (预装版通常为 MPI=0):
│ cd /opt/nccl-tests && make clean
│ make MPI=1 MPI_HOME=/usr/mpi/gcc/openmpi-4.1.7rc1 CUDA_HOME=/usr/local/cuda -j
│ 3. 确认新机 nvidia-smi topo -m 拓扑与集群一致
├── bash ~/nccl_kit/run_cross_switch_test.sh
├── 验收标准: 跨交换机 AllReduce ~317 GB/s (与同交换机持平即正常)
└── AllToAll 跨交换机较同交换机下降 ≤20% 属正常 (多一跳 IB spine)
``` ```
**关键环境变量说明 (nccl_env.sh):**
| 变量 | 值 | 原因 |
|---|---|---|
| `NCCL_IB_HCA` | `mlx5_0,1,2,3,4,5,8,9` | 仅 GPU 直连的 8 张 IB 卡;mlx5_6/7 为存储 NIC,不参与 NCCL |
| `NCCL_SOCKET_IFNAME` | `bond0` | 节点有 k8s 虚拟网卡,不指定会 bootstrap 选错导致 connection refused |
| `MPI_HOME` | `/usr/mpi/gcc/openmpi-4.1.7rc1` | 非系统默认路径,必须显式指定 |
**常见问题:**
| 现象 | 原因 | 解决 |
|---|---|---|
| busbw=0.00,每进程各打印一行结果 | nccl-tests 是 MPI=0 编译的 | `make clean && make MPI=1 ...` 重编 |
| `socketPollConnect: Connection refused` | NCCL 选了 k8s 虚拟网卡 | 确认 `NCCL_SOCKET_IFNAME=bond0` |
| AllToAll 在 8G 消息时 hang | 已知问题,16+ 节点大消息 hang | AllToAll 消息上限用 ≤2G |
| 新机 mpirun 连接失败 | SSH 互信未打通 | 互相追加 authorized_keys |
--- ---
### SOP-4: 故障诊断 ### SOP-4: 故障诊断
@ -683,27 +610,6 @@ python3 gpu_tester.py --test all --format html --output report.html
--- ---
## GPU 节点画像工具
`scripts/gpu_node_profile/` 提供独立、只读的 GPU 节点画像工具。采集器可在
一次执行中生成结构化 JSON,并由同一份 JSON 分别渲染运维画像和可逐层钻取
的 3D CT 页面。两个 HTML 源模板位于 `templates/`;`examples/` 包含脱敏
真机 JSON 和由其生成的两张自包含示例页面。支持 NUMA、PCIe、GPU、
NVLink/NVSwitch、RoCE Bond、无 Bond 的独立 RoCE/InfiniBand 端口、
物理网卡、CPU 亲和性和功耗信息。GPU 到 RDMA 端点的亲和关系来自
`nvidia-smi topo -m`,页面不会为矩阵未暴露的设备推测连线;详细命令与
采集边界见该目录的 `README.md`。
```bash
cd scripts/gpu_node_profile
python3 gpu_node_profile.py \
--snapshot-out gpu_node_snapshot.json \
--output gpu_node_profile.html \
--topology-output gpu_topology_3d.html
```
---
## 故障排除 ## 故障排除
| 问题 | 原因 | 解决方案 | | 问题 | 原因 | 解决方案 |

View File

@ -1,62 +0,0 @@
gpu_type: b300
tools:
install_dir: /root/test_gpu_scripts/.gpu-test-env/tools
benchmark:
memory:
size_mb: 4096
iterations: 10
nvbandwidth_buffer_mb: 512
nvbandwidth_samples: 3
compute:
dtypes: [fp32, tf32, fp16, bf16, fp8]
sweep_sizes:
- 3584
- 4608
- 5376
- 8192
- 11520
- [2048, 2048, 13312]
- [2048, 2048, 16384]
matrix_size: 8192
warmup: 20
iterations: 80
health:
temp_warning: 75
temp_critical: 85
power_limit: null
nccl:
min_bandwidth_gbps: null
test_allreduce: true
test_alltoall: true
test_broadcast: true
test_reduce_scatter: false
test_allgather: false
test_sendrecv: false
stress:
duration_sec: 600
use_doubles: false
use_tensor_cores: true
memory_pct: 90
gpus: all
rdma:
min_bandwidth_gbps: 50
max_latency_us: 10
cross_node:
enabled: false
training:
model: synthetic
batch_size: 8
seq_length: 2048
num_steps: 50
dtype: bf16
report:
output_dir: /root/test_gpu_scripts/reports/b300_single_node_20260827
format: json

View File

@ -1,86 +0,0 @@
gpu_type: bw1000
runtime:
name: DTK
version: 25.04.2
benchmark:
memory:
size_mb: 4096
iterations: 10
nvbandwidth_buffer_mb: 512
nvbandwidth_samples: 3
compute:
dtypes: [fp32, tf32, fp16, bf16, fp8]
sweep_sizes: [4096, 6144, 8192, [2048, 2048, 13312]]
matrix_size: 8192
warmup: 10
iterations: 30
health:
temp_warning: 75
temp_critical: 85
power_limit: 800
nccl:
min_bandwidth_gbps: 10
test_allreduce: true
test_alltoall: true
test_broadcast: true
test_reduce_scatter: true
test_allgather: true
test_sendrecv: false
torch_backend: nccl
torchrun_size_mb: 64
torchrun_warmup: 5
torchrun_iterations: 20
stress:
duration_sec: 600
use_doubles: false
use_tensor_cores: true
memory_pct: 80
gpus: all
rdma:
min_bandwidth_gbps: 200
max_latency_us: 6
ib_iterations: 5000
msg_size: 1048576
ib_device: null
ib_port: 1
cross_node:
enabled: false
allow_roce: true
server: null
server_addr: null
ssh_user: root
devices: [mlx5_bond_0, mlx5_bond_1, mlx5_bond_2, mlx5_bond_3]
ib_port: 1
gid_index: 3
msg_size: 1048576
iters: 5000
duration_sec: 10
queue_pairs: 4
base_oob_port: 18515
server_warmup_sec: 2.0
min_bandwidth_gbps: 200
max_latency_us: 6
training:
model: synthetic
batch_size: 2
seq_length: 1024
num_steps: 20
dtype: bf16
hidden_size: 2048
num_layers: 4
num_heads: 16
vocab_size: 8192
report:
output_dir: ./reports
format: json
tools:
install_dir: /opt/gpu-test-tools

View File

@ -1,15 +0,0 @@
gpu_type: bw1000
runtime:
name: DTK
version: 25.04.2
stress:
duration_sec: 60
use_doubles: false
use_tensor_cores: true
memory_pct: 80
gpus: all
tools:
install_dir: /opt/gpu-test-tools

View File

@ -1,4 +1,4 @@
# GPU type: auto-detect or override to a100/a800/h100/h800/h200/h20/b200/b300 # GPU type: auto-detect or override to a100/a800/h100/h200/b200/b300
gpu_type: auto gpu_type: auto
benchmark: benchmark:
@ -14,29 +14,13 @@ benchmark:
- fp16 - fp16
- bf16 - bf16
- fp8 - fp8
# MAMF-style shape sweep: measure each dtype at every shape below and keep the max matrix_size: 4096
# TFLOPS (the realistic achievable peak). A single fixed shape under-reports by warmup: 10
# ~7-12% and can't meet the MAMF-calibrated thresholds in gpu_specs.py. iterations: 100
# Each entry is either N (square N×N×N) or [M, N, K]. K-heavy non-square shapes
# (e.g. 2048×2048×13312) hit the true Hopper MAMF — bf16 ~790 vs ~755 square.
# Empty list => single matrix_size shape (legacy behaviour).
sweep_sizes:
- 3584
- 4608
- 5376
- 8192
- 11520
- [2048, 2048, 13312]
- [2048, 2048, 16384]
matrix_size: 8192 # fallback shape when sweep_sizes is empty
warmup: 20
iterations: 80
# NOTE: torch.compile was dropped — on H100 eager cuBLAS beats Triton for plain
# GEMM, and compiling would re-autotune per shape and make the sweep very slow.
health: health:
temp_warning: 75 temp_warning: 80
temp_critical: 85 temp_critical: 90
power_limit: null # null = auto-detect from GPU TDP per gpu_specs.py power_limit: null # null = auto-detect from GPU TDP per gpu_specs.py
nccl: nccl:
@ -49,7 +33,7 @@ nccl:
test_sendrecv: false test_sendrecv: false
stress: stress:
duration_sec: 600 # 10 min — reaches thermal steady state, validates throttle/jitter beyond warmup duration_sec: 60
use_doubles: false use_doubles: false
use_tensor_cores: true use_tensor_cores: true
memory_pct: 90 memory_pct: 90
@ -62,49 +46,6 @@ rdma:
msg_size: 65536 msg_size: 65536
ib_device: null ib_device: null
ib_port: 1 ib_port: 1
# Cross-node (two-host) RDMA via perftest, orchestrated over SSH from the CLIENT
# node. Replaces the old scripts/rdma_cross_node.sh. Run on the client; it starts
# ib_write_bw/ib_write_lat servers on `server` over SSH (passwordless required),
# then drives the local client per device.
cross_node:
enabled: false # set true on the client node to run cross-node RDMA
server: null # peer ssh address, e.g. 172.72.8.12 (server node)
server_addr: null # OOB addr client connects to (default: = server)
ssh_user: root
devices: [] # e.g. [mlx5_0, mlx5_1, mlx5_6, mlx5_7]; [] = auto-detect active IB
ib_port: 1
gid_index: null # -x <n> for RoCE; null for pure InfiniBand
msg_size: 1048576 # 1 MiB — large enough to reach NDR400 peak
iters: 5000
base_oob_port: 18515 # per-device OOB port = base + device index
server_warmup_sec: 2.0
min_bandwidth_gbps: 350 # per-port PASS floor (NDR400 ≈ 0.9 × 400)
max_latency_us: 5
multinode_nccl:
# Path to OpenMPI hostfile: one line per node, format "<ip> slots=8"
hosts_file: ~/nccl_kit/hosts_all
# MPI installation prefix (non-standard path, set empty to use system mpirun)
mpi_home: /usr/mpi/gcc/openmpi-4.1.7rc1
# IB HCAs for NCCL — GPU-direct (PIX) cards only; mlx5_6/7 are storage NICs, exclude them
# GPU↔NIC map: GPU0↔mlx5_3, GPU1↔mlx5_2, GPU2↔mlx5_1, GPU3↔mlx5_0
# GPU4↔mlx5_5, GPU5↔mlx5_4, GPU6↔mlx5_9, GPU7↔mlx5_8
nccl_ib_hca: "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_8,mlx5_9"
# NCCL bootstrap NIC — must be explicit (k8s virtual NICs cause connection refused)
nccl_socket_ifname: bond0
# Node counts to run scaling test at
scale_levels: [2, 4, 8]
# Operations to test
ops: [all_reduce, alltoall, all_gather, reduce_scatter]
# Message size for allreduce/allgather/reducescatter (bytes)
msg_size_bytes: 8589934592 # 8G
# Message size for alltoall — cap at 1G to avoid hang at 16+ nodes
msg_size_alltoall_bytes: 1073741824 # 1G
warmup: 5
iterations: 30
timeout_sec: 600 # per-op timeout (10 min)
# AllReduce busbw PASS floor (GB/s) — 4-node+ stable state on NDR400 cluster
allreduce_min_busbw_gbps: 310
training: training:
model: gpt2 model: gpt2

View File

@ -22,13 +22,11 @@ from modules.gpu_info import GPUInfo
from modules.health_check import HealthCheck from modules.health_check import HealthCheck
from modules.benchmark import Benchmark from modules.benchmark import Benchmark
from modules.nccl_test import NCCLTest from modules.nccl_test import NCCLTest
from modules.multinode_nccl_test import MultiNodeNCCLTest
from modules.training_sim import TrainingSim from modules.training_sim import TrainingSim
from modules.stress_test import StressTest from modules.stress_test import StressTest
from modules.rdma_test import RDMATest from modules.rdma_test import RDMATest
from modules.report import ReportGenerator from modules.report import ReportGenerator
from modules.gpu_specs import detect_gpu_type, get_gpu_specs, get_gpu_label, get_supported_gpus, validate_driver_compatibility from modules.gpu_specs import detect_gpu_type, get_gpu_specs, get_gpu_label, get_supported_gpus, validate_driver_compatibility
from modules.hygon_smi import find_hy_smi
DEFAULT_CONFIG = { DEFAULT_CONFIG = {
"benchmark": { "benchmark": {
@ -50,20 +48,6 @@ DEFAULT_CONFIG = {
"test_allgather": False, "test_allgather": False,
"test_sendrecv": False, "test_sendrecv": False,
}, },
"multinode_nccl": {
"hosts_file": "~/nccl_kit/hosts_all",
"mpi_home": "/usr/mpi/gcc/openmpi-4.1.7rc1",
"nccl_ib_hca": "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_8,mlx5_9",
"nccl_socket_ifname": "bond0",
"scale_levels": [2, 4, 8],
"ops": ["all_reduce", "alltoall", "all_gather", "reduce_scatter"],
"msg_size_bytes": 8589934592,
"msg_size_alltoall_bytes": 1073741824,
"warmup": 5,
"iterations": 30,
"timeout_sec": 600,
"allreduce_min_busbw_gbps": 310,
},
"stress": { "stress": {
"duration_sec": 60, "duration_sec": 60,
"use_doubles": False, "use_doubles": False,
@ -119,9 +103,9 @@ def check_prerequisites(console: Console) -> bool:
import shutil import shutil
ok = True ok = True
if not shutil.which("nvidia-smi") and not find_hy_smi(): if not shutil.which("nvidia-smi"):
console.print("[bold red]ERROR: neither nvidia-smi nor hy-smi was found![/bold red]") console.print("[bold red]ERROR: nvidia-smi not found![/bold red]")
console.print(" Please install the NVIDIA driver or Hygon DCU driver first.") console.print(" Please install NVIDIA drivers first.")
ok = False ok = False
return ok return ok
@ -158,7 +142,6 @@ def interactive_menu(config: dict):
("6", "GPU Stress Test (gpu-burn)", "stress"), ("6", "GPU Stress Test (gpu-burn)", "stress"),
("7", "RDMA/IB Test", "rdma"), ("7", "RDMA/IB Test", "rdma"),
("8", "Training Simulation", "training"), ("8", "Training Simulation", "training"),
("m", "Multi-node NCCL Scaling", "multinode_nccl"),
("9", "Full Test Suite (All Tests)", "all"), ("9", "Full Test Suite (All Tests)", "all"),
("0", "Generate Report", "report"), ("0", "Generate Report", "report"),
] ]
@ -181,7 +164,6 @@ def interactive_menu(config: dict):
"memory_bench": "HBM bandwidth via nvbandwidth", "memory_bench": "HBM bandwidth via nvbandwidth",
"compute_bench": "GEMM TFLOPS across FP32/TF32/FP16/BF16/FP8", "compute_bench": "GEMM TFLOPS across FP32/TF32/FP16/BF16/FP8",
"nccl": "AllReduce, AllToAll, Broadcast via nccl-tests", "nccl": "AllReduce, AllToAll, Broadcast via nccl-tests",
"multinode_nccl": "跨节点 NCCL Scaling via mpirun + hostfile",
"stress": "Long-running GPU stress via gpu-burn", "stress": "Long-running GPU stress via gpu-burn",
"rdma": "InfiniBand bandwidth & latency (ib_write_bw)", "rdma": "InfiniBand bandwidth & latency (ib_write_bw)",
"training": "Simulate LLM training with PyTorch", "training": "Simulate LLM training with PyTorch",
@ -263,12 +245,6 @@ def _run_test(test_name: str, config: dict, console: Console) -> dict:
m.print_results(result) m.print_results(result)
return result return result
elif test_name == "multinode_nccl":
m = MultiNodeNCCLTest(config)
result = m.run()
m.print_results(result)
return result
elif test_name == "stress": elif test_name == "stress":
m = StressTest(config) m = StressTest(config)
result = m.run() result = m.run()
@ -334,39 +310,14 @@ def _run_full_suite(config: dict, console: Console) -> dict:
console.print(f"[bold red]{name} FAILED: {e}[/bold red]") console.print(f"[bold red]{name} FAILED: {e}[/bold red]")
all_results[key] = {"error": str(e)} all_results[key] = {"error": str(e)}
# Summary: classify actual verdicts. The old implementation counted every # Summary
# result without an exception as PASS, even when passed=False or status=FAIL.
console.print("\n" + "=" * 60) console.print("\n" + "=" * 60)
# Only count test results, exclude metadata like timestamp
test_results = {k: v for k, v in all_results.items() if k != "timestamp"} test_results = {k: v for k, v in all_results.items() if k != "timestamp"}
statuses = {} passed = sum(1 for v in test_results.values() if not isinstance(v, dict) or "error" not in v)
for key, value in test_results.items(): total = len(test_results)
status = "PASS" color = "green" if passed == total else ("yellow" if passed > 0 else "red")
if isinstance(value, dict) and value.get("error"): console.print(f"[bold {color}]Suite complete: {passed}/{total} tests passed[/bold {color}]")
status = "FAIL"
elif key == "health":
status = "PASS" if value.get("passed") else "FAIL"
elif key == "memory_bench":
mem = value.get("memory", value)
if mem.get("error"):
status = "FAIL"
else:
eff = mem.get("d2d_efficiency_pct") or mem.get("efficiency_pct") or 0
status = "PASS" if eff >= 80 else ("WARN" if eff >= 60 else "FAIL")
elif key == "compute_bench":
status = Benchmark.judge_compute(value)["verdict"]
elif key in ("nccl", "rdma"):
status = value.get("status") or ("PASS" if value.get("passed") else "FAIL")
elif key == "stress":
status = "PASS" if value.get("passed") else "FAIL"
elif key == "training":
status = value.get("performance_verdict", "MEASURED")
statuses[key] = status
counts = {name: list(statuses.values()).count(name) for name in ("PASS", "WARN", "FAIL", "SKIP", "MEASURED")}
color = "red" if counts["FAIL"] else ("yellow" if counts["WARN"] else "green")
summary = ", ".join(f"{name} {count}" for name, count in counts.items() if count)
console.print(f"[bold {color}]Suite complete: {summary}[/bold {color}]")
all_results["suite_status"] = statuses
return all_results return all_results
@ -389,7 +340,7 @@ Examples:
python gpu_tester.py --report --format json --output report.json python gpu_tester.py --report --format json --output report.json
""", """,
) )
parser.add_argument("--test", choices=["gpu-info", "health", "benchmark", "nccl", "multinode-nccl", "stress", "rdma", "training", "all"], parser.add_argument("--test", choices=["gpu-info", "health", "benchmark", "nccl", "stress", "rdma", "training", "all"],
help="Run a specific test") help="Run a specific test")
parser.add_argument("--type", choices=["memory", "compute"], help="Benchmark type (with --test benchmark)") parser.add_argument("--type", choices=["memory", "compute"], help="Benchmark type (with --test benchmark)")
parser.add_argument("--dtype", choices=["fp32", "tf32", "fp16", "bf16", "fp8"], parser.add_argument("--dtype", choices=["fp32", "tf32", "fp16", "bf16", "fp8"],
@ -446,7 +397,6 @@ Examples:
"health": "health", "health": "health",
"benchmark": None, "benchmark": None,
"nccl": "nccl", "nccl": "nccl",
"multinode-nccl": "multinode_nccl",
"stress": "stress", "stress": "stress",
"rdma": "rdma", "rdma": "rdma",
"training": "training", "training": "training",
@ -472,14 +422,11 @@ Examples:
if args.report: if args.report:
ReportGenerator(config).generate(results, fmt=args.format, output=args.output) ReportGenerator(config).generate(results, fmt=args.format, output=args.output)
has_errors = any("error" in v for v in results.values() if isinstance(v, dict)) has_errors = any("error" in v for v in results.values() if isinstance(v, dict))
has_failures = any( sys.exit(1 if has_errors else 0)
status == "FAIL" for status in results.get("suite_status", {}).values()
)
sys.exit(1 if has_errors or has_failures else 0)
else: else:
result = _run_test(test_map[args.test], config, console) result = _run_test(test_map[args.test], config, console)
if args.report and result: if args.report and result:
ReportGenerator(config).generate({test_map[args.test]: result, "timestamp": datetime.now().isoformat()}, ReportGenerator(config).generate({args.test: result, "timestamp": datetime.now().isoformat()},
fmt=args.format, output=args.output) fmt=args.format, output=args.output)

View File

@ -17,10 +17,6 @@ from modules.gpu_specs import detect_gpu_type, get_gpu_specs, get_gpu_label, res
TORCH_AVAILABLE = False TORCH_AVAILABLE = False
try: try:
import torch import torch
try:
import torch_dcu # noqa: F401 - enables DTK's PyTorch device shim
except ImportError:
pass
if torch.cuda.is_available(): if torch.cuda.is_available():
TORCH_AVAILABLE = True TORCH_AVAILABLE = True
except ImportError: except ImportError:
@ -34,8 +30,7 @@ class Benchmark:
self.console = Console() self.console = Console()
self.bench_cfg = config.get("benchmark", {}) self.bench_cfg = config.get("benchmark", {})
self.tools_dir = resolve_tools_dir(config) self.tools_dir = resolve_tools_dir(config)
cfg_gpu_type = config.get("gpu_type", "auto") self.gpu_type = detect_gpu_type()
self.gpu_type = cfg_gpu_type if cfg_gpu_type != "auto" else detect_gpu_type()
self.specs = get_gpu_specs(self.gpu_type) self.specs = get_gpu_specs(self.gpu_type)
self.gpu_label = get_gpu_label(self.gpu_type) self.gpu_label = get_gpu_label(self.gpu_type)
@ -67,14 +62,12 @@ class Benchmark:
return None return None
def run_memory_benchmark(self) -> dict: def run_memory_benchmark(self) -> dict:
# nvbandwidth is NVIDIA-specific. DTK PyTorch provides the portable nvbw = self._find_nvbandwidth()
# transfer/copy path used below for Hygon DCUs.
nvbw = None if self.gpu_type == "bw1000" else self._find_nvbandwidth()
if nvbw: if nvbw:
return self._run_nvbandwidth(nvbw) return self._run_nvbandwidth(nvbw)
self.console.print("[yellow]Using PyTorch memory benchmark fallback[/yellow]") self.console.print("[yellow]nvbandwidth not found, falling back to PyTorch[/yellow]")
return self._run_memory_pytorch() return self._run_memory_pytorch()
def _run_nvbandwidth(self, nvbw_path: str) -> dict: def _run_nvbandwidth(self, nvbw_path: str) -> dict:
@ -132,12 +125,8 @@ class Benchmark:
continue continue
try: try:
# --disableAffinity skips nvbandwidth's CPU affinity setup, which cmd = [nvbw_path, "-t", tc, "-b", str(buffer_mb),
# calls nvmlDeviceGetHandleByUUID() — that lookup fails on hosts "-i", str(samples), "-j"]
# whose fabricmanager build doesn't expose the UUID format nvml
# expects (seen on H20-3e with custom 570.172.08-1 fabricmanager).
cmd = [nvbw_path, "--disableAffinity", "-t", tc,
"-b", str(buffer_mb), "-i", str(samples), "-j"]
r = subprocess.run(cmd, capture_output=True, text=True, timeout=120) r = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
if r.returncode == 0 and r.stdout.strip(): if r.returncode == 0 and r.stdout.strip():
@ -150,26 +139,14 @@ class Benchmark:
progress.advance(task) progress.advance(task)
# Compare like with like: the 900 GB/s reference is per direction, so do
# not select the bidirectional SM result as the acceptance value. Keep it
# as a separate informational metric.
d2d_bw = max( d2d_bw = max(
results_by_test.get("d2d_write", 0), results_by_test.get("d2d_write", 0),
results_by_test.get("d2d_read", 0), results_by_test.get("d2d_read", 0),
results_by_test.get("d2d_bidir", 0),
) )
d2d_bidir_bw = results_by_test.get("d2d_bidir", 0)
h2d_bw = results_by_test.get("h2d", 0) h2d_bw = results_by_test.get("h2d", 0)
d2h_bw = results_by_test.get("d2h", 0) d2h_bw = results_by_test.get("d2h", 0)
# If every subtest returned 0 the nvbandwidth binary is broken on this host
# (e.g. CUDA_ERROR_INVALID_CONTEXT, NVML mismatch). Fall back to PyTorch.
if all(v == 0 for v in results_by_test.values()):
self.console.print(
"[yellow]nvbandwidth returned no usable data — "
"falling back to PyTorch memory benchmark[/yellow]"
)
return self._run_memory_pytorch()
# D2D goes through NVLink — compare to NVLink per-direction bandwidth # D2D goes through NVLink — compare to NVLink per-direction bandwidth
# (nvlink_bandwidth_gbps is bidirectional, so per-direction = /2) # (nvlink_bandwidth_gbps is bidirectional, so per-direction = /2)
nvlink_bw = self.specs.get("nvlink_bandwidth_gbps", 0) nvlink_bw = self.specs.get("nvlink_bandwidth_gbps", 0)
@ -188,7 +165,6 @@ class Benchmark:
"h2d_bandwidth_gbps": round(h2d_bw, 1), "h2d_bandwidth_gbps": round(h2d_bw, 1),
"d2h_bandwidth_gbps": round(d2h_bw, 1), "d2h_bandwidth_gbps": round(d2h_bw, 1),
"d2d_bandwidth_gbps": round(d2d_bw, 1), "d2d_bandwidth_gbps": round(d2d_bw, 1),
"d2d_bidirectional_gbps": round(d2d_bidir_bw, 1),
"h2d_peak_gbps": pcie_peak if pcie_peak else None, "h2d_peak_gbps": pcie_peak if pcie_peak else None,
"d2h_peak_gbps": pcie_peak if pcie_peak else None, "d2h_peak_gbps": pcie_peak if pcie_peak else None,
"d2d_peak_gbps": round(d2d_peak, 1) if d2d_peak else None, "d2d_peak_gbps": round(d2d_peak, 1) if d2d_peak else None,
@ -199,8 +175,6 @@ class Benchmark:
"efficiency_pct": d2d_efficiency, "efficiency_pct": d2d_efficiency,
"results_by_test": results_by_test, "results_by_test": results_by_test,
"per_gpu": [], "per_gpu": [],
"scope_note": "GPU peer/PCIe transfer bandwidth; local HBM bandwidth not measured",
"accelerator_vendor": self.specs.get("vendor", "nvidia"),
} }
} }
@ -222,12 +196,9 @@ class Benchmark:
for cell in row: for cell in row:
try: try:
v = float(cell) v = float(cell)
values.append(v)
except (ValueError, TypeError): except (ValueError, TypeError):
continue continue
# Exclude diagonal entries (intra-device, reported as 0 or
# N/A) so they don't drag the off-diagonal average down.
if v > 0:
values.append(v)
if values: if values:
return sum(values) / len(values) return sum(values) / len(values)
return 0.0 return 0.0
@ -304,62 +275,29 @@ class Benchmark:
progress.advance(task) progress.advance(task)
best_d2d = max(v["d2d_gbps"] for v in bandwidth_by_size.values()) best_d2d = max(v["d2d_gbps"] for v in bandwidth_by_size.values())
best_h2d = max(v["h2d_gbps"] for v in bandwidth_by_size.values())
best_d2h = max(v["d2h_gbps"] for v in bandwidth_by_size.values())
peak_bw = self.specs["memory_bandwidth_gbps"] peak_bw = self.specs["memory_bandwidth_gbps"]
efficiency = round((best_d2d / peak_bw) * 100, 1) if peak_bw else None efficiency = round((best_d2d / peak_bw) * 100, 1) if peak_bw else None
pcie_gen = self.specs.get("pcie_gen", 0)
pcie_peak = {3: 16, 4: 32, 5: 64, 6: 128}.get(pcie_gen) if pcie_gen else None
return { return {
"memory": { "memory": {
"source": "pytorch", "source": "pytorch",
"h2d_bandwidth_gbps": round(best_h2d, 1), "h2d_bandwidth_gbps": round(max(v["h2d_gbps"] for v in bandwidth_by_size.values()), 1),
"d2h_bandwidth_gbps": round(best_d2h, 1), "d2h_bandwidth_gbps": round(max(v["d2h_gbps"] for v in bandwidth_by_size.values()), 1),
"d2d_bandwidth_gbps": round(best_d2d, 1), "d2d_bandwidth_gbps": round(best_d2d, 1),
"h2d_peak_gbps": pcie_peak,
"d2h_peak_gbps": pcie_peak,
"d2d_peak_gbps": None,
"h2d_efficiency_pct": round(best_h2d / pcie_peak * 100, 1) if pcie_peak else None,
"d2h_efficiency_pct": round(best_d2h / pcie_peak * 100, 1) if pcie_peak else None,
"d2d_efficiency_pct": efficiency,
"peak_bandwidth_gbps": self.specs["memory_bandwidth_gbps"], "peak_bandwidth_gbps": self.specs["memory_bandwidth_gbps"],
"efficiency_pct": efficiency, "efficiency_pct": efficiency,
"test_sizes_mb": test_sizes_mb, "test_sizes_mb": test_sizes_mb,
"bandwidth_by_size": bandwidth_by_size, "bandwidth_by_size": bandwidth_by_size,
"per_gpu": [], "per_gpu": [],
"accelerator_vendor": self.specs.get("vendor", "nvidia"),
} }
} }
def run_compute_benchmark(self, dtypes: Optional[List[str]] = None) -> dict: def run_compute_benchmark(self, dtypes: Optional[List[str]] = None) -> dict:
comp_cfg = self.bench_cfg.get("compute", {}) comp_cfg = self.bench_cfg.get("compute", {})
configured_dtypes = dtypes or comp_cfg.get("dtypes", ["fp32", "tf32", "fp16", "bf16", "fp8"]) configured_dtypes = dtypes or comp_cfg.get("dtypes", ["fp32", "tf32", "fp16", "bf16", "fp8"])
matrix_size = comp_cfg.get("matrix_size", 4096)
# MAMF-style shape sweep (à la stas00's mamf-finder): a single fixed matmul warmup = comp_cfg.get("warmup", 10)
# shape under-reports the achievable peak by ~7-12% and therefore can't meet iterations = comp_cfg.get("iterations", 100)
# the MAMF-calibrated PASS thresholds in gpu_specs.compute_pass_thresholds_tflops.
# So for each dtype we time several matmul shapes and keep the MAXIMUM TFLOPS
# (the realistic peak). matrix_size is the fallback when sweep_sizes is empty.
matrix_size = comp_cfg.get("matrix_size", 8192)
sweep_sizes = comp_cfg.get("sweep_sizes") or [matrix_size]
warmup = comp_cfg.get("warmup", 20)
iterations = comp_cfg.get("iterations", 80)
# Each sweep entry is either an int N (square N×N×N) or an [M, N, K] triple.
# Non-square / K-heavy shapes (e.g. 2048×2048×13312) reach the true MAMF peak
# on Hopper — square-only tops out ~5% lower — so the default set mixes both.
def _to_shape(entry):
if isinstance(entry, (list, tuple)):
if len(entry) == 3:
return tuple(int(x) for x in entry)
if len(entry) == 1:
n = int(entry[0])
return (n, n, n)
raise ValueError(f"sweep size {entry!r} must be an int or [M, N, K]")
n = int(entry)
return (n, n, n)
shapes = [_to_shape(e) for e in sweep_sizes]
if not TORCH_AVAILABLE: if not TORCH_AVAILABLE:
self.console.print("[yellow]PyTorch not available - skipping compute benchmark[/yellow]") self.console.print("[yellow]PyTorch not available - skipping compute benchmark[/yellow]")
@ -367,22 +305,16 @@ class Benchmark:
gpu_count = torch.cuda.device_count() gpu_count = torch.cuda.device_count()
self.console.print(f"[cyan]Compute Benchmark - {gpu_count} GPU(s)[/cyan]") self.console.print(f"[cyan]Compute Benchmark - {gpu_count} GPU(s)[/cyan]")
if len(sweep_sizes) > 1:
self.console.print(
f"[cyan] MAMF shape sweep over {len(sweep_sizes)} sizes: "
f"{', '.join(str(s) for s in sweep_sizes)}[/cyan]"
)
dtype_map = { dtype_map = {
"fp32": (torch.float32, self.specs["fp32_tflops"]), "fp32": (torch.float32, self.specs["fp32_tflops"]),
"tf32": ("tf32", self.specs["tf32_tflops"]), "tf32": ("tf32", self.specs["tf32_tflops"]),
"fp16": (torch.float16, self.specs["fp16_tflops"]), "fp16": (torch.float16, self.specs["fp16_tflops"]),
"bf16": (torch.bfloat16, self.specs["bf16_tflops"]), "bf16": (torch.bfloat16, self.specs["bf16_tflops"]),
"fp8": (getattr(torch, "float8_e4m3fn", None), self.specs["fp8_tflops"]), "fp8": (torch.float8_e4m3fn, self.specs["fp8_tflops"]),
} }
results_by_dtype = {} results_by_dtype = {}
best_shapes = {}
per_gpu_results = [{"index": i} for i in range(gpu_count)] per_gpu_results = [{"index": i} for i in range(gpu_count)]
with Progress( with Progress(
@ -390,78 +322,69 @@ class Benchmark:
BarColumn(), TextColumn("{task.completed}/{task.total}"), BarColumn(), TextColumn("{task.completed}/{task.total}"),
TimeElapsedColumn(), console=self.console, TimeElapsedColumn(), console=self.console,
) as progress: ) as progress:
task = progress.add_task( task = progress.add_task("Testing dtypes...", total=len(configured_dtypes))
"Testing dtypes on every GPU...", total=len(configured_dtypes) * gpu_count
)
for dtype_name in configured_dtypes: for dtype_name in configured_dtypes:
if dtype_name not in dtype_map: if dtype_name not in dtype_map:
progress.advance(task, advance=gpu_count) progress.advance(task)
continue continue
# Skip FP8 if GPU architecture doesn't support it # Skip FP8 if GPU architecture doesn't support it
if dtype_name == "fp8" and ( if dtype_name == "fp8" and self.specs.get("fp8_tflops", 0) == 0:
self.specs.get("fp8_tflops", 0) == 0 or dtype_map["fp8"][0] is None
):
arch = self.specs.get("architecture", "unknown") arch = self.specs.get("architecture", "unknown")
results_by_dtype["fp8"] = f"skipped ({arch} does not support FP8)" results_by_dtype["fp8"] = f"skipped ({arch} does not support FP8)"
self.console.print(f"[dim] fp8: skipped - {arch} architecture has no FP8 support[/dim]") self.console.print(f"[dim] fp8: skipped - {arch} architecture has no FP8 support[/dim]")
progress.advance(task, advance=gpu_count) progress.advance(task)
continue continue
dtype_val, peak_tflops = dtype_map[dtype_name] dtype_val, peak_tflops = dtype_map[dtype_name]
# allow_tf32 only affects float32 matmuls: ON for the TF32 run, OFF for try:
# the true-FP32 run so the two stay distinct.
old_tf32 = torch.backends.cuda.matmul.allow_tf32
if dtype_name == "tf32": if dtype_name == "tf32":
old_tf32 = torch.backends.cuda.matmul.allow_tf32
torch.backends.cuda.matmul.allow_tf32 = True torch.backends.cuda.matmul.allow_tf32 = True
dtype_val = torch.float32 dtype_val = torch.float32
elif dtype_name == "fp32":
torch.backends.cuda.matmul.allow_tf32 = False
gpu_values = [] M = N = K = matrix_size
gpu_shapes = {}
last_err = None if dtype_name == "fp8":
fastest = (0.0, None) a = torch.randn(M, K, device="cuda", dtype=torch.float32).to(torch.float8_e4m3fn)
for gpu_idx in range(gpu_count): b = torch.randn(K, N, device="cuda", dtype=torch.float32).to(torch.float8_e4m3fn)
best_tflops, best_shape = 0.0, None
for (M, N, K) in shapes:
try:
t = self._bench_matmul_once(
dtype_name, dtype_val, M, N, K,
warmup, iterations, gpu_idx,
)
if t > best_tflops:
best_tflops, best_shape = t, (M, N, K)
except Exception as e: # noqa: BLE001 - record and try next shape
last_err = e
if best_shape is not None:
value = round(best_tflops, 1)
per_gpu_results[gpu_idx][dtype_name] = value
gpu_values.append(value)
gpu_shapes[str(gpu_idx)] = "x".join(str(d) for d in best_shape)
if best_tflops > fastest[0]:
fastest = (best_tflops, best_shape)
else: else:
per_gpu_results[gpu_idx][dtype_name] = f"error: {last_err}" a = torch.randn(M, K, device="cuda", dtype=dtype_val)
progress.advance(task) b = torch.randn(K, N, device="cuda", dtype=dtype_val)
for _ in range(warmup):
torch.matmul(a, b)
torch.cuda.synchronize()
start_event = torch.cuda.Event(enable_timing=True)
end_event = torch.cuda.Event(enable_timing=True)
start_event.record()
for _ in range(iterations):
c = torch.matmul(a, b)
end_event.record()
torch.cuda.synchronize()
elapsed_ms = start_event.elapsed_time(end_event)
flops = 2 * M * N * K * iterations
tflops = flops / (elapsed_ms / 1000) / 1e12
results_by_dtype[dtype_name] = round(tflops, 1)
for pg in per_gpu_results:
pg[dtype_name] = round(tflops, 1)
if dtype_name == "tf32":
torch.backends.cuda.matmul.allow_tf32 = old_tf32 torch.backends.cuda.matmul.allow_tf32 = old_tf32
if not gpu_values: del a, b, c
results_by_dtype[dtype_name] = f"error: {last_err}" torch.cuda.empty_cache()
self.console.print(f"[yellow] {dtype_name}: {last_err}[/yellow]")
else: except Exception as e:
avg_value = round(sum(gpu_values) / len(gpu_values), 1) results_by_dtype[dtype_name] = f"error: {e}"
results_by_dtype[dtype_name] = avg_value self.console.print(f"[yellow] {dtype_name}: {e}[/yellow]")
best_shapes[dtype_name] = "x".join(str(d) for d in fastest[1])
if len(shapes) > 1: progress.advance(task)
self.console.print(
f"[dim] {dtype_name}: avg {avg_value:.1f}, "
f"min {min(gpu_values):.1f}, max {max(gpu_values):.1f} TFLOPS[/dim]"
)
best_shapes.setdefault("per_gpu", {})[dtype_name] = gpu_shapes
efficiency = {} efficiency = {}
for dt, achieved in results_by_dtype.items(): for dt, achieved in results_by_dtype.items():
@ -470,98 +393,18 @@ class Benchmark:
if peak_tp: if peak_tp:
efficiency[dt] = round((achieved / peak_tp) * 100, 1) efficiency[dt] = round((achieved / peak_tp) * 100, 1)
worst_by_dtype = {}
best_by_dtype = {}
spread_by_dtype = {}
for dt in results_by_dtype:
values = [pg.get(dt) for pg in per_gpu_results]
values = [v for v in values if isinstance(v, (int, float))]
if values:
worst_by_dtype[dt] = min(values)
best_by_dtype[dt] = max(values)
avg = sum(values) / len(values)
spread_by_dtype[dt] = round((max(values) - min(values)) / avg * 100, 2) if avg else 0
return { return {
"compute": { "compute": {
"per_dtype_tflops": results_by_dtype, "per_dtype_tflops": results_by_dtype,
"peak_tflops": {dt: dtype_map[dt][1] for dt in dtype_map}, "peak_tflops": {dt: dtype_map[dt][1] for dt in dtype_map},
"efficiency_pct": efficiency, "efficiency_pct": efficiency,
# Absolute TFLOPS PASS thresholds (decoupled from peak). When present,
# report.py judges PASS/WARN/FAIL against these directly instead of
# using % of peak. Empty dict => fall back to legacy 80% rule.
"pass_thresholds_tflops": dict(
self.specs.get("compute_pass_thresholds_tflops") or {}
),
"per_gpu": per_gpu_results, "per_gpu": per_gpu_results,
"worst_per_dtype_tflops": worst_by_dtype,
"best_per_dtype_tflops": best_by_dtype,
"spread_pct": spread_by_dtype,
"sweep_sizes": list(sweep_sizes),
"best_shapes": best_shapes,
"matrix_size": matrix_size, "matrix_size": matrix_size,
"warmup": warmup, "warmup": warmup,
"iterations": iterations, "iterations": iterations,
"accelerator_backend": "dtk" if self.gpu_type == "bw1000" else "cuda",
} }
} }
def _bench_matmul_once(self, dtype_name: str, dtype_val, M: int, N: int, K: int,
warmup: int, iterations: int, gpu_index: int = 0) -> float:
"""Time one (M×K)·(K×N) matmul for a dtype and return achieved TFLOPS.
Uses an L2-cache-busting pool of matrix pairs (total > 256 MB) so operands
can't be served from L2 across iterations, and CUDA events for timing. FP8
goes through torch._scaled_mm (e4m3); all others through torch.matmul — eager
cuBLAS, which on H100 beats torch.compile/Triton for plain GEMM and avoids the
per-shape recompile cost that would make a sweep pathologically slow.
"""
device = f"cuda:{gpu_index}"
with torch.cuda.device(gpu_index):
elem_bytes = 1 if dtype_name == "fp8" else torch.tensor([], dtype=dtype_val).element_size()
pair_bytes = (M * K + K * N) * elem_bytes
num_pools = max(4, -(-256 * 1024 * 1024 // pair_bytes)) # ceil(256MB / pair)
if dtype_name == "fp8":
if not hasattr(torch, "_scaled_mm"):
raise RuntimeError("torch._scaled_mm unavailable — upgrade to PyTorch >= 2.1")
pools_a = [torch.randn(M, K, device=device, dtype=torch.float32).to(torch.float8_e4m3fn) for _ in range(num_pools)]
pools_b = [torch.randn(N, K, device=device, dtype=torch.float32).to(torch.float8_e4m3fn) for _ in range(num_pools)]
scale_a = torch.tensor(1.0, device=device)
scale_b = torch.tensor(1.0, device=device)
def op(i):
return torch._scaled_mm(pools_a[i], pools_b[i].T, scale_a=scale_a, scale_b=scale_b, out_dtype=torch.bfloat16)
else:
pools_a = [torch.randn(M, K, device=device, dtype=dtype_val) for _ in range(num_pools)]
pools_b = [torch.randn(K, N, device=device, dtype=dtype_val) for _ in range(num_pools)]
def op(i):
return torch.matmul(pools_a[i], pools_b[i])
try:
torch.cuda.set_device(gpu_index)
# Probe once so a broken/unsupported kernel raises before the timed loop.
_probe = op(0)
torch.cuda.synchronize(gpu_index)
del _probe
for i in range(warmup):
op(i % num_pools)
torch.cuda.synchronize(gpu_index)
start_event = torch.cuda.Event(enable_timing=True)
end_event = torch.cuda.Event(enable_timing=True)
start_event.record()
for i in range(iterations):
op(i % num_pools)
end_event.record()
torch.cuda.synchronize(gpu_index)
elapsed_ms = start_event.elapsed_time(end_event)
finally:
del pools_a, pools_b
torch.cuda.empty_cache()
return (2 * M * N * K * iterations) / (elapsed_ms / 1000) / 1e12
@staticmethod @staticmethod
def print_results(results: dict, console: Console = None): def print_results(results: dict, console: Console = None):
c = console or Console() c = console or Console()
@ -625,124 +468,22 @@ class Benchmark:
table = Table(box=None, padding=(0, 1)) table = Table(box=None, padding=(0, 1))
table.add_column("DType", style="bold") table.add_column("DType", style="bold")
table.add_column("Avg", justify="right") table.add_column("Achieved (TFLOPS)", justify="right")
table.add_column("Min", justify="right")
table.add_column("Max", justify="right")
table.add_column("Spread", justify="right")
table.add_column("Peak", justify="right") table.add_column("Peak", justify="right")
table.add_column("Efficiency", justify="right") table.add_column("Efficiency", justify="right")
peak = comp.get("peak_tflops", {}) peak = comp.get("peak_tflops", {})
per_dtype = comp.get("per_dtype_tflops", {}) per_dtype = comp.get("per_dtype_tflops", {})
eff = comp.get("efficiency_pct", {}) eff = comp.get("efficiency_pct", {})
worst = comp.get("worst_per_dtype_tflops", {})
best = comp.get("best_per_dtype_tflops", {})
spread = comp.get("spread_pct", {})
for dt in per_dtype: for dt in per_dtype:
achieved = per_dtype[dt] achieved = per_dtype[dt]
if isinstance(achieved, str): if isinstance(achieved, str):
table.add_row(dt, f"[red]{achieved}[/red]", "-", "-", "-", str(peak.get(dt, "N/A")), "N/A") table.add_row(dt, f"[red]{achieved}[/red]", str(peak.get(dt, "N/A")), "N/A")
continue continue
pk = peak.get(dt, 0) pk = peak.get(dt, 0)
ef = eff.get(dt, 0) ef = eff.get(dt, 0)
if pk:
ec = "green" if ef >= 80 else ("yellow" if ef >= 50 else "red") ec = "green" if ef >= 80 else ("yellow" if ef >= 50 else "red")
peak_text = f"{pk:.0f}" table.add_row(dt.upper(), f"{achieved:.1f}", f"{pk:.0f}",
eff_text = f"[{ec}]{ef:.1f}%[/{ec}]" f"[{ec}]{ef:.1f}%[/{ec}]")
else:
peak_text = "N/A"
eff_text = "[cyan]MEASURED[/cyan]"
table.add_row(
dt.upper(), f"{achieved:.1f}", f"{worst.get(dt, achieved):.1f}",
f"{best.get(dt, achieved):.1f}", f"{spread.get(dt, 0):.2f}%",
peak_text, eff_text,
)
c.print(table) c.print(table)
@staticmethod
def judge_compute(results: dict) -> dict:
"""Judge compute results against pass_thresholds_tflops.
Single source of truth for the PASS/WARN/FAIL rule (same one report.py uses):
achieved >= thr -> PASS; >= 0.9*thr -> WARN; else FAIL. A string achieved value
(skipped/error) -> SKIP. A dtype without a threshold falls back to efficiency
(>=80 PASS / >=50 WARN / else FAIL). If neither threshold nor peak is
published, the numeric result is retained as MEASURED without a verdict.
Returns {"rows": [(dtype, achieved, threshold, status), ...], "verdict": str}.
"""
comp = results.get("compute", results)
per_dtype = comp.get("per_dtype_tflops", {})
worst_per_dtype = comp.get("worst_per_dtype_tflops", {}) or {}
thresholds = comp.get("pass_thresholds_tflops", {}) or {}
eff = comp.get("efficiency_pct", {})
rank = {"PASS": 0, "MEASURED": 0, "WARN": 1, "FAIL": 2, "SKIP": 0}
rows, verdict = [], "PASS"
evaluated = False
measured = False
for dt, val in per_dtype.items():
judged_val = worst_per_dtype.get(dt, val)
thr = thresholds.get(dt)
if isinstance(val, str):
status = "SKIP"
elif thr:
status = "PASS" if judged_val >= thr else ("WARN" if judged_val >= thr * 0.9 else "FAIL")
evaluated = True
elif dt not in eff:
status = "MEASURED"
measured = True
else:
e = eff.get(dt, 0)
status = "PASS" if e >= 80 else ("WARN" if e >= 50 else "FAIL")
evaluated = True
rows.append((dt, judged_val, thr, status))
if rank[status] > rank[verdict]:
verdict = status
if measured and not evaluated:
verdict = "MEASURED"
return {"rows": rows, "verdict": verdict}
@staticmethod
def print_compute_verdict(results: dict, console: Console = None) -> str:
"""Print the PASS/WARN/FAIL table for compute results; return the verdict."""
c = console or Console()
judged = Benchmark.judge_compute(results)
color = {"PASS": "green", "MEASURED": "cyan", "WARN": "yellow", "FAIL": "red", "SKIP": "dim"}
c.print("\n[bold cyan]Compute Verdict (vs thresholds)[/bold cyan]")
for dt, val, thr, status in judged["rows"]:
val_s = f"{val:.1f}" if isinstance(val, (int, float)) else str(val)
thr_s = f">= {thr}" if thr else "(efficiency)"
c.print(f" {dt.upper():>4}: {val_s:>8} {thr_s:<12} [{color[status]}]{status}[/{color[status]}]")
v = judged["verdict"]
c.print(f" [bold]VERDICT: [{color[v]}]{v}[/{color[v]}][/bold]")
return v
def _run_cli() -> None:
"""`python -m modules.benchmark` — run ONLY the compute-throughput benchmark."""
import argparse
from pathlib import Path
import yaml
repo_root = Path(__file__).resolve().parent.parent
parser = argparse.ArgumentParser(description="Run the compute-throughput benchmark only.")
parser.add_argument("--config", default=str(repo_root / "configs" / "default.yaml"),
help="path to config YAML (default: configs/default.yaml)")
parser.add_argument("--json", action="store_true", help="also print raw JSON of the compute results")
args = parser.parse_args()
with open(args.config) as f:
config = yaml.safe_load(f) or {}
results = Benchmark(config).run_compute_benchmark()
Benchmark.print_results(results)
Benchmark.print_compute_verdict(results)
if args.json:
print("JSON_RESULT:" + json.dumps(results["compute"]))
if __name__ == "__main__":
_run_cli()

View File

@ -1,4 +1,4 @@
"""Accelerator information detection for NVIDIA GPUs and Hygon DCUs.""" """GPU information detection module for NVIDIA datacenter GPUs (A100/A800/H100/H200/B200/B300)."""
import subprocess import subprocess
import shutil import shutil
@ -11,7 +11,6 @@ from rich.panel import Panel
from rich.text import Text from rich.text import Text
from modules.gpu_specs import detect_gpu_type, get_gpu_specs, get_gpu_label from modules.gpu_specs import detect_gpu_type, get_gpu_specs, get_gpu_label
from modules.hygon_smi import collect_hygon_info, find_hy_smi
class GPUInfo: class GPUInfo:
@ -19,8 +18,7 @@ class GPUInfo:
def __init__(self, config: dict): def __init__(self, config: dict):
self.config = config self.config = config
self.console = Console() self.console = Console()
configured = config.get("gpu_type", "auto") self.gpu_type = detect_gpu_type()
self.gpu_type = configured if configured != "auto" else detect_gpu_type()
self.specs = get_gpu_specs(self.gpu_type) self.specs = get_gpu_specs(self.gpu_type)
self.gpu_label = get_gpu_label(self.gpu_type) self.gpu_label = get_gpu_label(self.gpu_type)
@ -37,19 +35,6 @@ class GPUInfo:
return None return None
def run(self) -> dict: def run(self) -> dict:
if self.gpu_type == "bw1000" or (not shutil.which("nvidia-smi") and find_hy_smi()):
result = collect_hygon_info(include_topology=True)
runtime_cfg = self.config.get("runtime", {}) or {}
result.update({
"runtime_name": runtime_cfg.get("name", result.get("runtime_name", "DTK")),
"runtime_version": runtime_cfg.get("version", result.get("runtime_version", "N/A")),
"cuda_version": runtime_cfg.get("version", result.get("runtime_version", "N/A")),
"timestamp": datetime.now().isoformat(),
"detected_gpu_type": "bw1000",
"gpu_label": get_gpu_label("bw1000"),
})
return result
if not shutil.which("nvidia-smi"): if not shutil.which("nvidia-smi"):
self.console.print("[bold red]nvidia-smi not found![/bold red]") self.console.print("[bold red]nvidia-smi not found![/bold red]")
return {"error": "nvidia-smi not found", "gpu_count": 0} return {"error": "nvidia-smi not found", "gpu_count": 0}
@ -174,9 +159,7 @@ class GPUInfo:
c.print(f"\n[bold cyan]GPU Information[/bold cyan]") c.print(f"\n[bold cyan]GPU Information[/bold cyan]")
c.print(f" Driver Version : {results.get('driver_version', 'N/A')}") c.print(f" Driver Version : {results.get('driver_version', 'N/A')}")
runtime_name = results.get("runtime_name", "CUDA") c.print(f" CUDA Version : {results.get('cuda_version', 'N/A')}")
runtime_version = results.get("runtime_version", results.get("cuda_version", "N/A"))
c.print(f" {runtime_name} Version{' ' * max(0, 4-len(runtime_name))}: {runtime_version}")
c.print(f" GPU Count : {results.get('gpu_count', 0)}") c.print(f" GPU Count : {results.get('gpu_count', 0)}")
c.print(f" Detected GPU : {results.get('gpu_label', 'Unknown')} ({results.get('detected_gpu_type', 'unknown')})") c.print(f" Detected GPU : {results.get('gpu_label', 'Unknown')} ({results.get('detected_gpu_type', 'unknown')})")
c.print(f" Timestamp : {results.get('timestamp', 'N/A')}") c.print(f" Timestamp : {results.get('timestamp', 'N/A')}")
@ -198,7 +181,7 @@ class GPUInfo:
for g in gpus: for g in gpus:
name = g["name"] name = g["name"]
if any(k in name for k in ("A100", "A800", "H100", "H200", "B200", "B300", "BW1000")): if any(k in name for k in ("A100", "A800", "H100", "H200", "B200", "B300")):
name = f"[bold green]{name}[/bold green]" name = f"[bold green]{name}[/bold green]"
vram = f"{g['vram_used_mb']}/{g['vram_total_mb']} MB" vram = f"{g['vram_used_mb']}/{g['vram_total_mb']} MB"
temp = f"{g['temperature']}°C" temp = f"{g['temperature']}°C"
@ -221,8 +204,7 @@ class GPUInfo:
topology = results.get("topology", "") topology = results.get("topology", "")
if topology and topology != "Unavailable": if topology and topology != "Unavailable":
title = "HSW/XHCL Topology" if results.get("vendor") == "hygon" else "NVLink/NVSwitch Topology" c.print(Panel(topology, title="[bold]NVLink/NVSwitch Topology[/bold]", border_style="cyan"))
c.print(Panel(topology, title=f"[bold]{title}[/bold]", border_style="cyan"))
warnings = [] warnings = []
for g in gpus: for g in gpus:

View File

@ -1,4 +1,4 @@
"""Accelerator specifications for NVIDIA GPUs and Hygon DCUs.""" """GPU specifications database for NVIDIA datacenter GPUs."""
import os import os
import shutil import shutil
@ -6,52 +6,18 @@ import subprocess
from typing import List, Optional from typing import List, Optional
# GPU name patterns -> internal key mapping # GPU name patterns -> internal key mapping
# Order matters: longer/more-specific patterns must come before shorter ones.
GPU_NAME_PATTERNS = { GPU_NAME_PATTERNS = {
"BW1000": "bw1000",
"A100": "a100", "A100": "a100",
"A800": "a800", "A800": "a800",
"H100": "h100", "H100": "h100",
"H800": "h800", # H800 = H100 SXM with NVLink halved (400 GB/s) and FP64 restricted
"H200": "h200", "H200": "h200",
"H20": "h20", # H20 / H20-3e is the China-compliance export variant, REDUCED peaks
"B200": "b200", "B200": "b200",
"B30-E": "b300", # nvidia-smi product name used by the B300 systems under test
"B300": "b300", "B300": "b300",
} }
# Specs database — ALL values are DENSE (non-sparse) TFLOPS # Specs database — ALL values are DENSE (non-sparse) TFLOPS
GPU_SPECS = { GPU_SPECS = {
"bw1000": {
# Public Hygon material identifies the supported precisions but does not
# publish stable dense peak figures for BW1000_H. Keep unknown peaks at
# zero so reports show measured throughput without inventing efficiency.
"full_name": "Hygon BW1000_H DCU",
"vendor": "hygon",
"architecture": "C-3000 GPGPU",
"compute_capability": 0.0,
"hbm_capacity_gb": 64,
"hbm_type": "HBM",
"memory_bandwidth_gbps": 0,
"fp32_tflops": 0,
"tf32_tflops": 0,
"fp16_tflops": 0,
"bf16_tflops": 0,
"fp8_tflops": 0,
"compute_pass_thresholds_tflops": {},
"tdp_watts": 800,
"nvlink_gen": 0,
"nvlink_bandwidth_gbps": 0,
"interconnect": "HSW/XHCL",
"pcie_gen": 5,
"min_driver_version": "",
"min_cuda_version": "",
},
"h100": { "h100": {
# Peaks below are NVIDIA marketing dense peaks (theoretical Tensor Core max).
# `compute_pass_thresholds_tflops` carries the absolute PASS thresholds used
# by report.py — decoupled from peaks so marketing-spec changes (dense vs
# sparse vs FP8-sparsity) don't shift the validation bar.
"full_name": "NVIDIA H100 SXM5", "full_name": "NVIDIA H100 SXM5",
"architecture": "Hopper", "architecture": "Hopper",
"compute_capability": 9.0, "compute_capability": 9.0,
@ -63,18 +29,6 @@ GPU_SPECS = {
"fp16_tflops": 990, # dense (1979 sparse w/ 2:4) "fp16_tflops": 990, # dense (1979 sparse w/ 2:4)
"bf16_tflops": 990, # dense "bf16_tflops": 990, # dense
"fp8_tflops": 1979, # dense "fp8_tflops": 1979, # dense
"compute_pass_thresholds_tflops": {
# Recalibrated 2026-05-25 to the H100 eager-cuBLAS achievable floor (each
# threshold ~2-4% below the sustained value measured across 16 GPUs via the
# MAMF shape sweep: fp32 ~52 / tf32 ~405 / fp16 ~732-748 / bf16 ~747-758 /
# fp8 ~1248-1271). The old marketing/MAMF-derived values (fp32 54, tf32 444,
# fp16 734, bf16 745, fp8 1400) sat ON or ABOVE what PyTorch cuBLAS reaches
# on H100, so healthy cards flaked to WARN/FAIL. fp8 1400 in particular was
# an H200/rowwise-scaling figure; H100 tensorwise _scaled_mm tops out ~1310.
"fp32": 50, "tf32": 385, "fp16": 720, "bf16": 730, "fp8": 1200,
# FP64 63 / INT8 1536 — listed for documentation; benchmark module
# doesn't currently exercise these dtypes.
},
"tdp_watts": 700, "tdp_watts": 700,
"nvlink_gen": 4, "nvlink_gen": 4,
"nvlink_bandwidth_gbps": 900, # bidirectional "nvlink_bandwidth_gbps": 900, # bidirectional
@ -94,70 +48,6 @@ GPU_SPECS = {
"fp16_tflops": 990, # dense "fp16_tflops": 990, # dense
"bf16_tflops": 990, # dense "bf16_tflops": 990, # dense
"fp8_tflops": 1979, # dense "fp8_tflops": 1979, # dense
# PASS thresholds aligned with H200_production_acceptance.md v2 (2026-05-21):
# calibrated against Semianalysis & stas00 MAMF — H200 shares H100 SMs so
# achievable TFLOPS in PyTorch is in the same band.
"compute_pass_thresholds_tflops": {
"fp32": 50, "tf32": 400, "fp16": 720, "bf16": 720, "fp8": 1400,
},
"tdp_watts": 700,
"nvlink_gen": 4,
"nvlink_bandwidth_gbps": 900,
"pcie_gen": 5,
"min_driver_version": "545",
"min_cuda_version": "12.4",
},
"h800": {
# H800 = China-compliance export variant of H100 SXM5. SAME chip / SMs /
# clocks / HBM as H100 SXM5 — Tensor Core peaks (FP16 / BF16 / FP8 / TF32 /
# FP32) are identical to H100. Two restrictions vs H100:
# 1. NVLink bandwidth halved: 400 GB/s bidirectional (vs H100 900 GB/s)
# 2. FP64 throughput severely cut to ~1 TFLOPS (vs H100 34/67 TFLOPS)
# All other interfaces (PCIe Gen5, NVSwitch, HBM3 80GB @ 3.35 TB/s) match H100.
# NCCL multi-GPU thresholds MUST be downscaled because NVLink BW is halved.
"full_name": "NVIDIA H800 SXM5",
"architecture": "Hopper",
"compute_capability": 9.0,
"hbm_capacity_gb": 80,
"hbm_type": "HBM3",
"memory_bandwidth_gbps": 3350, # GB/s (3.35 TB/s) — same as H100 SXM
"fp32_tflops": 67,
"tf32_tflops": 495, # dense (same as H100)
"fp16_tflops": 990, # dense (same as H100)
"bf16_tflops": 990, # dense (same as H100)
"fp8_tflops": 1979, # dense (same as H100)
# Tensor Core peaks identical to H100, so PASS thresholds reuse the H100
# eager-cuBLAS calibration (2026-05-25). Measured on 8×H800: fp32 ~52 /
# tf32 ~420 / fp16 ~741 / bf16 ~745 / fp8 ~1249 — all clear these. fp8 was
# 1400 (an H200/rowwise-scaling figure) which PyTorch tensorwise _scaled_mm
# can't reach on H100-class silicon (~1310 ceiling); lowered to 1200 to match
# h100. FP64 deliberately NOT listed — H800 is restricted to ~1 TFLOPS FP64.
"compute_pass_thresholds_tflops": {
"fp32": 50, "tf32": 385, "fp16": 720, "bf16": 730, "fp8": 1200,
},
"tdp_watts": 700,
"nvlink_gen": 4,
"nvlink_bandwidth_gbps": 400, # bidirectional — HALF of H100 (export restriction)
"pcie_gen": 5,
"min_driver_version": "535",
"min_cuda_version": "12.1",
},
"h20": {
# China-compliance export variant of H200 (reported as "H20" / "H20-3e" by nvidia-smi).
# Same silicon family / HBM as H200, but Tensor Core peaks are throttled.
# Peaks below are sourced from supplier / NVIDIA China and confirmed against
# measured throughput on 8x H20-3e (FP16 ~741, BF16 ~770, FP8 ~1328 TFLOPS).
"full_name": "NVIDIA H20 / H20-3e",
"architecture": "Hopper",
"compute_capability": 9.0,
"hbm_capacity_gb": 141,
"hbm_type": "HBM3e",
"memory_bandwidth_gbps": 4800,
"fp32_tflops": 54, # China spec (matches measured ~51-52)
"tf32_tflops": 372, # ~75% of H200 (matches measured ~362)
"fp16_tflops": 744, # dense, China spec
"bf16_tflops": 739, # dense, China spec
"fp8_tflops": 1420, # dense, China spec
"tdp_watts": 700, "tdp_watts": 700,
"nvlink_gen": 4, "nvlink_gen": 4,
"nvlink_bandwidth_gbps": 900, "nvlink_bandwidth_gbps": 900,
@ -229,33 +119,15 @@ GPU_SPECS = {
"hbm_capacity_gb": 288, "hbm_capacity_gb": 288,
"hbm_type": "HBM3e", "hbm_type": "HBM3e",
"memory_bandwidth_gbps": 8000, # GB/s (8 TB/s) "memory_bandwidth_gbps": 8000, # GB/s (8 TB/s)
# Dense, per-GPU reference peaks. NVIDIA publishes DGX B300 FP8 as "fp32_tflops": 125,
# 72 PFLOPS sparse for eight GPUs and explicitly states dense is half, "tf32_tflops": 1750, # dense (estimated)
# i.e. 4.5 PFLOPS/GPU. The previous 7 PFLOPS dense value mixed product "fp16_tflops": 3500, # dense
# variants/modes and made healthy B300 cards fail by construction. "bf16_tflops": 3500, # dense
"fp32_tflops": 75, "fp8_tflops": 7000, # dense
"tf32_tflops": 1100,
"fp16_tflops": 2250,
"bf16_tflops": 2250,
"fp8_tflops": 4500,
# Acceptance floors are achievable eager-PyTorch GEMM values rather than
# a blanket percentage of marketing peak. BF16/FP8 are calibrated against
# NVIDIA Transformer Engine's public B300 GEMM examples; the remaining
# modes use conservative floors below sustained measurements on this path.
"compute_pass_thresholds_tflops": {
"fp32": 60, "tf32": 750, "fp16": 1450, "bf16": 1450, "fp8": 2800,
},
# Single-node 8-GPU floors derived from public B300 nccl-tests results.
# These are operation-specific because collectives do not reach the same
# fraction of raw NVLink bandwidth.
"nccl_pass_thresholds_gbps": {
"allreduce": 750, "alltoall": 620, "broadcast": 600,
"allgather": 700, "reducescatter": 700, "sendrecv": 600,
},
"tdp_watts": 1200, "tdp_watts": 1200,
"nvlink_gen": 5, "nvlink_gen": 5,
"nvlink_bandwidth_gbps": 1800, "nvlink_bandwidth_gbps": 1800,
"pcie_gen": 6, "pcie_gen": 5,
"min_driver_version": "550", "min_driver_version": "550",
"min_cuda_version": "12.4", "min_cuda_version": "12.4",
}, },
@ -274,7 +146,6 @@ _UNKNOWN_SPECS = {
"fp16_tflops": 0, "fp16_tflops": 0,
"bf16_tflops": 0, "bf16_tflops": 0,
"fp8_tflops": 0, "fp8_tflops": 0,
"compute_pass_thresholds_tflops": {}, # empty => report.py falls back to 80% of peak
"tdp_watts": 700, "tdp_watts": 700,
"nvlink_gen": 0, "nvlink_gen": 0,
"nvlink_bandwidth_gbps": 0, "nvlink_bandwidth_gbps": 0,
@ -285,40 +156,28 @@ _UNKNOWN_SPECS = {
def detect_gpu_type() -> str: def detect_gpu_type() -> str:
"""Detect accelerator type via nvidia-smi or hy-smi. """Detect GPU type via nvidia-smi and return the internal key (e.g. 'h200').
Returns 'unknown' if nvidia-smi is unavailable or the GPU is not recognized. Returns 'unknown' if nvidia-smi is unavailable or the GPU is not recognized.
""" """
nvidia_smi = shutil.which("nvidia-smi") nvidia_smi = shutil.which("nvidia-smi")
if nvidia_smi: if not nvidia_smi:
return "unknown"
try: try:
r = subprocess.run( r = subprocess.run(
[nvidia_smi, "--query-gpu=name", "--format=csv,noheader"], [nvidia_smi, "--query-gpu=name", "--format=csv,noheader"],
capture_output=True, text=True, timeout=10, capture_output=True, text=True, timeout=10,
) )
if r.returncode == 0 and r.stdout.strip(): if r.returncode != 0:
first_line = r.stdout.strip().splitlines()[0].strip().upper() return "unknown"
# Iterate longest-pattern-first so "H200" doesn't match "H20".
for pattern, key in sorted(GPU_NAME_PATTERNS.items(), key=lambda kv: -len(kv[0])):
if pattern in first_line:
return key
except (subprocess.TimeoutExpired, FileNotFoundError, OSError):
pass
# Hygon DTK hosts expose hy-smi on PATH or under /opt/hyhal. first_line = r.stdout.strip().splitlines()[0].strip()
try: for pattern, key in GPU_NAME_PATTERNS.items():
from modules.hygon_smi import find_hy_smi if pattern in first_line.upper():
hy_smi = find_hy_smi() return key
if hy_smi: return "unknown"
r = subprocess.run( except (subprocess.TimeoutExpired, FileNotFoundError, OSError):
[hy_smi, "--showproductname"], capture_output=True,
text=True, timeout=10,
)
product = (r.stdout or "").upper()
if r.returncode == 0 and "BW1000" in product:
return "bw1000"
except (ImportError, subprocess.TimeoutExpired, FileNotFoundError, OSError):
pass
return "unknown" return "unknown"
@ -342,7 +201,8 @@ def get_gpu_label(gpu_type: str) -> str:
specs = GPU_SPECS.get(gpu_type) specs = GPU_SPECS.get(gpu_type)
if specs: if specs:
full = specs["full_name"] full = specs["full_name"]
return full.replace("NVIDIA ", "").replace("Hygon ", "") # Strip the "NVIDIA " prefix for display
return full.replace("NVIDIA ", "")
return "Unknown GPU" return "Unknown GPU"

View File

@ -1,4 +1,4 @@
"""Hardware health monitoring for NVIDIA GPUs and Hygon DCUs.""" """Hardware health monitoring module for NVIDIA datacenter GPUs."""
import subprocess import subprocess
import shutil import shutil
@ -12,7 +12,6 @@ from rich.panel import Panel
from rich.text import Text from rich.text import Text
from modules.gpu_specs import detect_gpu_type, get_gpu_specs from modules.gpu_specs import detect_gpu_type, get_gpu_specs
from modules.hygon_smi import collect_hygon_info, find_hy_smi
class HealthCheck: class HealthCheck:
@ -21,8 +20,7 @@ class HealthCheck:
self.config = config self.config = config
self.console = Console() self.console = Console()
self.health_cfg = config.get("health", {}) self.health_cfg = config.get("health", {})
configured = config.get("gpu_type", "auto") self.gpu_type = detect_gpu_type()
self.gpu_type = configured if configured != "auto" else detect_gpu_type()
self.specs = get_gpu_specs(self.gpu_type) self.specs = get_gpu_specs(self.gpu_type)
def _run_smi(self, query: str) -> Optional[str]: def _run_smi(self, query: str) -> Optional[str]:
@ -57,9 +55,6 @@ class HealthCheck:
return default return default
def run(self) -> dict: def run(self) -> dict:
if self.gpu_type == "bw1000" or (not shutil.which("nvidia-smi") and find_hy_smi()):
return self._run_hygon()
if not shutil.which("nvidia-smi"): if not shutil.which("nvidia-smi"):
return {"error": "nvidia-smi not found", "passed": False} return {"error": "nvidia-smi not found", "passed": False}
@ -185,102 +180,6 @@ class HealthCheck:
"detected_gpu_type": self.gpu_type, "detected_gpu_type": self.gpu_type,
} }
def _run_hygon(self) -> dict:
info = collect_hygon_info(include_topology=False)
if info.get("error"):
return {"error": info["error"], "passed": False}
temp_warn = self.health_cfg.get("temp_warning", 80)
temp_crit = self.health_cfg.get("temp_critical", 90)
configured_power = self.health_cfg.get("power_limit")
gpu_health = []
overall_pass = True
for gpu in info.get("gpus", []):
checks = {}
temp = gpu.get("temperature", 0)
if temp >= temp_crit:
temp_status = "FAIL"
elif temp >= temp_warn:
temp_status = "WARN"
else:
temp_status = "PASS"
checks["temperature"] = {
"value": temp, "status": temp_status,
"threshold": temp_crit if temp_status == "FAIL" else temp_warn,
}
draw = gpu.get("power_draw", 0.0)
limit = configured_power or gpu.get("power_limit") or self.specs.get("tdp_watts", 800)
checks["power"] = {
"value": draw, "limit": limit,
"status": "PASS" if draw <= limit * 1.05 else "WARN",
}
ecc_enabled = bool(gpu.get("ecc_enabled"))
checks["ecc_errors"] = {
"single": gpu.get("ecc_errors_single", 0),
"double": gpu.get("ecc_errors_double", 0),
"enabled": ecc_enabled,
"status": "PASS" if ecc_enabled else "FAIL",
}
xgmi = int(gpu.get("xgmi_errors", 0))
replay = int(gpu.get("pcie_replay_count", 0))
healthy = bool(gpu.get("healthy"))
checks["memory_errors"] = {
"xgmi_errors": xgmi,
"pcie_replay_count": replay,
"device_healthy": healthy,
"status": "PASS" if healthy and xgmi == 0 and replay == 0 else "FAIL",
}
generation = int(gpu.get("pcie_link_gen", 0))
width = int(gpu.get("pcie_link_width", 0))
expected = self.specs.get("pcie_gen", 0)
pcie_ok = generation >= expected and width >= 16
checks["pcie_link"] = {
"gen": generation, "width": width,
"status": "PASS" if pcie_ok else "WARN",
}
sm = int(gpu.get("clock_sm", 0))
mem = int(gpu.get("clock_mem", 0))
checks["clock_speed"] = {
"sm": sm, "mem": mem,
"status": "PASS" if sm > 0 and mem > 0 else "WARN",
}
throttling = bool(gpu.get("throttling_active"))
checks["throttling"] = {
"status": "FAIL" if throttling else "PASS",
"reasons": [gpu.get("throttling_text")] if throttling else [],
}
# Hygon has no NVIDIA-style persistence mode; a loaded /dev/kfd
# driver and successful hy-smi health query are the equivalent check.
checks["persistence_mode"] = {
"enabled": healthy, "status": "PASS" if healthy else "FAIL"
}
worst = "PASS"
for check in checks.values():
if check["status"] == "FAIL":
worst = "FAIL"
break
if check["status"] == "WARN":
worst = "WARN"
if worst == "FAIL":
overall_pass = False
gpu_health.append({"index": gpu["index"], "status": worst, "checks": checks})
return {
"passed": overall_pass,
"gpu_health": gpu_health,
"system_health": self._check_system(),
"timestamp": datetime.now().isoformat(),
"detected_gpu_type": "bw1000",
"driver_version": info.get("driver_version", "N/A"),
"vendor": "hygon",
}
def _check_system(self) -> dict: def _check_system(self) -> dict:
persistd = shutil.which("nvidia-persistenced") is not None persistd = shutil.which("nvidia-persistenced") is not None
persistd_running = False persistd_running = False
@ -328,10 +227,7 @@ class HealthCheck:
if os.path.isdir("/sys/class/infiniband_verbs"): if os.path.isdir("/sys/class/infiniband_verbs"):
rdma_devs = os.listdir("/sys/class/infiniband_verbs") rdma_devs = os.listdir("/sys/class/infiniband_verbs")
nccl_env = { nccl_env = {k: v for k, v in os.environ.items() if k.startswith("NCCL_")}
k: v for k, v in os.environ.items()
if k.startswith(("NCCL_", "RCCL_", "HCCL_"))
}
return { return {
"nvidia_persistenced": {"installed": persistd, "running": persistd_running}, "nvidia_persistenced": {"installed": persistd, "running": persistd_running},

View File

@ -1,190 +0,0 @@
"""Helpers for querying Hygon DCU devices through ``hy-smi``."""
import os
import re
import shutil
import subprocess
from typing import Dict, List, Optional
def find_hy_smi() -> Optional[str]:
"""Return the hy-smi executable path when available."""
path = shutil.which("hy-smi")
if path:
return path
path = "/opt/hyhal/bin/hy-smi"
return path if os.path.isfile(path) and os.access(path, os.X_OK) else None
def run_hy_smi(*args: str, timeout: int = 30) -> Optional[str]:
"""Run hy-smi and return stdout, or ``None`` on failure."""
binary = find_hy_smi()
if not binary:
return None
try:
result = subprocess.run(
[binary, *args], capture_output=True, text=True, timeout=timeout
)
except (FileNotFoundError, OSError, subprocess.TimeoutExpired):
return None
return result.stdout.strip() if result.returncode == 0 else None
def _indexed_values(text: Optional[str], pattern: str) -> Dict[int, str]:
if not text:
return {}
values: Dict[int, str] = {}
for match in re.finditer(pattern, text, re.MULTILINE | re.IGNORECASE):
values[int(match.group(1))] = match.group(2).strip()
return values
def _number(value, default=0.0):
try:
return float(value)
except (TypeError, ValueError):
return default
def _pcie_gen(gt_per_second: float) -> int:
if gt_per_second >= 32:
return 5
if gt_per_second >= 16:
return 4
if gt_per_second >= 8:
return 3
if gt_per_second >= 5:
return 2
return 1 if gt_per_second > 0 else 0
def collect_hygon_info(include_topology: bool = True) -> dict:
"""Collect normalized per-device information for GPUInfo and HealthCheck."""
summary = run_hy_smi()
if not summary:
return {"error": "hy-smi query failed", "gpu_count": 0, "gpus": []}
rows: Dict[int, dict] = {}
row_re = re.compile(
r"^\s*(\d+)\s+([\d.]+)C\s+([\d.]+)W\s+(\S+)\s+"
r"([\d.]+)W\s+(\d+)%\s+([\d.]+)%",
re.MULTILINE,
)
for match in row_re.finditer(summary):
index = int(match.group(1))
rows[index] = {
"index": index,
"temperature": int(round(float(match.group(2)))),
"power_draw": float(match.group(3)),
"performance_level": match.group(4),
"power_limit": float(match.group(5)),
"vram_used_pct": int(match.group(6)),
"utilization_pct": float(match.group(7)),
}
product = run_hy_smi("--showproductname")
driver = run_hy_smi("--showdriverversion")
memory = run_hy_smi("--showmemavailable")
bus = run_hy_smi("--showbus")
unique = run_hy_smi("--showuniqueid")
serial = run_hy_smi("--showserial")
clocks = run_hy_smi("--showclocks")
ecc = run_hy_smi("--showmemeccinfo")
xgmi = run_hy_smi("--showxgmierr")
replay = run_hy_smi("--showreplaycount")
health = run_hy_smi("--healthcheck")
throttle = run_hy_smi("--showfdr")
names = _indexed_values(product, r"HCU\[(\d+)\].*Card Series:\s*([^\s]+)")
memories = _indexed_values(memory, r"HCU\[(\d+)\].*Available memory size \(MiB\):\s*(\d+)")
buses = _indexed_values(bus, r"HCU\[(\d+)\].*PCI Bus:\s*([0-9a-fA-F:.]+)")
unique_ids = _indexed_values(unique, r"HCU\[(\d+)\].*Unique ID:\s*(\S+)")
serials = _indexed_values(serial, r"HCU\[(\d+)\].*Serial Number:\s*(\S+)")
sclk = _indexed_values(clocks, r"HCU\[(\d+)\].*sclk clock level:.*\((\d+)Mhz\)")
mclk = _indexed_values(clocks, r"HCU\[(\d+)\].*mclk clock level:.*\((\d+)Mhz\)")
pcie = {}
if clocks:
for match in re.finditer(
r"HCU\[(\d+)\].*pcie clock level:.*\(([\d.]+)GT/s,\s*x(\d+)", clocks
):
pcie[int(match.group(1))] = (float(match.group(2)), int(match.group(3)))
ecc_enabled = {
idx: value.upper() == "ON"
for idx, value in _indexed_values(
ecc, r"HCU\[(\d+)\].*Vram memory ECC status:\s*(\S+)"
).items()
}
xgmi_errors = {
idx: int(value)
for idx, value in _indexed_values(
xgmi, r"HCU\[(\d+)\].*XGMI Error count\s+(\d+)"
).items()
}
replay_counts = {
idx: int(value)
for idx, value in _indexed_values(
replay, r"HCU\[(\d+)\].*PCIe Replay Count:\s*(\d+)"
).items()
}
healthy = {
idx: value.lower() == "healthy"
for idx, value in _indexed_values(
health, r"HCU\[(\d+)\].*:\s*(Healthy|Unhealthy)\s*$"
).items()
}
throttle_status = _indexed_values(
throttle, r"HCU\[(\d+)\].*Throttler Status:\s*(.+?)\s*$"
)
driver_match = re.search(r"Driver Version:\s*(\S+)", driver or "")
driver_version = driver_match.group(1) if driver_match else "N/A"
gpu_count = max(rows.keys(), default=-1) + 1
gpus: List[dict] = []
for index in range(gpu_count):
row = rows.get(index, {"index": index})
total_mb = int(memories.get(index, 0))
used_mb = int(total_mb * int(row.get("vram_used_pct", 0)) / 100)
gt_s, width = pcie.get(index, (0.0, 0))
throttle_text = throttle_status.get(index, "")
real_throttle = bool(
throttle_text
and any(token not in {"IDLE", "ACTIVE", "NONE"}
for token in re.findall(r"[A-Z_]+", throttle_text.upper()))
)
gpus.append({
**row,
"name": names.get(index, "Hygon DCU"),
"uuid": unique_ids.get(index, "N/A"),
"pci_bus_id": buses.get(index, "N/A"),
"pcie_link_gen": _pcie_gen(gt_s),
"pcie_link_width": width,
"pcie_gt_s": gt_s,
"vram_total_mb": total_mb,
"vram_used_mb": used_mb,
"vram_free_mb": max(total_mb - used_mb, 0),
"clock_sm": int(_number(sclk.get(index))),
"clock_mem": int(_number(mclk.get(index))),
"fan_speed": 0,
"persistence_mode": True,
"compute_mode": row.get("performance_level", "auto"),
"serial_number": serials.get(index, "N/A"),
"ecc_enabled": ecc_enabled.get(index, False),
"ecc_errors_single": 0,
"ecc_errors_double": 0,
"xgmi_errors": xgmi_errors.get(index, 0),
"pcie_replay_count": replay_counts.get(index, 0),
"healthy": healthy.get(index, False),
"throttling_active": real_throttle,
"throttling_text": throttle_text,
})
return {
"driver_version": driver_version,
"runtime_name": "DTK",
"runtime_version": os.environ.get("DTK_VERSION", "N/A"),
"gpu_count": gpu_count,
"gpus": gpus,
"topology": run_hy_smi("--showtopo") if include_topology else "",
"vendor": "hygon",
}

View File

@ -1,553 +0,0 @@
"""Multi-node NCCL benchmark wrapper for nccl-tests via mpirun."""
import json
import os
import re
import shlex
import shutil
import subprocess
from datetime import datetime
from typing import Optional
from rich.console import Console
from rich.table import Table
from modules.gpu_specs import resolve_tools_dir
_TEST_ALIASES = {
"allreduce": "all_reduce_perf",
"all_reduce": "all_reduce_perf",
"all_reduce_perf": "all_reduce_perf",
"allgather": "all_gather_perf",
"all_gather": "all_gather_perf",
"all_gather_perf": "all_gather_perf",
"alltoall": "alltoall_perf",
"all_to_all": "alltoall_perf",
"alltoall_perf": "alltoall_perf",
"broadcast": "broadcast_perf",
"broadcast_perf": "broadcast_perf",
"reducescatter": "reduce_scatter_perf",
"reduce_scatter": "reduce_scatter_perf",
"reduce_scatter_perf": "reduce_scatter_perf",
"sendrecv": "sendrecv_perf",
"send_recv": "sendrecv_perf",
"sendrecv_perf": "sendrecv_perf",
}
_OP_LABELS = {
"all_reduce_perf": "allreduce",
"all_gather_perf": "allgather",
"alltoall_perf": "alltoall",
"broadcast_perf": "broadcast",
"reduce_scatter_perf": "reducescatter",
"sendrecv_perf": "sendrecv",
}
class MultiNodeNCCLTest:
"""Run cross-node NCCL tests with a PDF-style message-size sweep."""
def __init__(self, config: dict):
self.config = config
self.cfg = config.get("multinode_nccl", {}) or {}
self.tools_dir = resolve_tools_dir(config)
self.console = Console()
self.artifact_dir = os.environ.get("MULTINODE_NCCL_ARTIFACT_DIR") or self.cfg.get("artifact_dir")
def _find_mpirun(self) -> Optional[str]:
configured = self.cfg.get("mpirun_path")
if configured and os.path.isfile(str(configured)) and os.access(str(configured), os.X_OK):
return str(configured)
for cmd in ["mpirun", "mpiexec", os.path.join(self.tools_dir, "mpi", "bin", "mpirun")]:
found = shutil.which(cmd)
if found:
return found
return None
def _find_nccl_test(self, binary_name: str) -> Optional[str]:
configured = self.cfg.get("nccl_tests_dir")
candidates = []
if configured:
candidates.append(os.path.join(configured, binary_name))
candidates.append(os.path.join(self.tools_dir, "nccl-tests", "build", binary_name))
found = shutil.which(binary_name)
if found:
candidates.insert(0, found)
for path in candidates:
if path and os.path.isfile(path) and os.access(path, os.X_OK):
return path
return None
def _tests(self) -> list[str]:
configured = self.cfg.get("tests") or ["all_reduce_perf", "alltoall_perf"]
tests = []
for name in configured:
binary = _TEST_ALIASES.get(str(name).lower())
if binary and binary not in tests:
tests.append(binary)
return tests
def _hosts(self) -> list[dict]:
hosts = self.cfg.get("hosts") or []
normalized = []
for host in hosts:
if isinstance(host, str):
normalized.append({"addr": host, "slots": 8})
elif isinstance(host, dict):
normalized.append({
"name": host.get("name") or host.get("addr"),
"addr": host.get("addr") or host.get("host") or host.get("ip"),
"slots": int(host.get("slots", 8)),
})
return [h for h in normalized if h.get("addr")]
def _topologies(self) -> list[dict]:
topologies = self.cfg.get("topologies") or [{"nodes": 2, "gpus_per_node": 8}]
normalized = []
for topo in topologies:
nodes = int(topo.get("nodes", 2))
gpus_per_node = int(topo.get("gpus_per_node", topo.get("gpn", 8)))
normalized.append({
"nodes": nodes,
"gpus_per_node": gpus_per_node,
"label": topo.get("label") or f"{nodes} nodes x {gpus_per_node} GPUs",
"cuda_visible_devices": topo.get("cuda_visible_devices"),
"env": topo.get("env") or {},
"op_env": topo.get("op_env") or topo.get("test_env") or {},
"min_peak_busbw_gbps": topo.get("min_peak_busbw_gbps"),
})
return normalized
def _env_exports(self, topo: dict = None, label: str = None, binary: str = None) -> list[tuple[str, str]]:
env_cfg = {
"NCCL_DEBUG": self.cfg.get("debug", "WARN"),
"NCCL_SOCKET_IFNAME": self.cfg.get("socket_ifname"),
"NCCL_IB_GID_INDEX": self.cfg.get("ib_gid_index"),
"NCCL_IB_SL": self.cfg.get("ib_sl"),
"NCCL_IB_TC": self.cfg.get("ib_tc"),
"NCCL_IB_HCA": self.cfg.get("ib_hca"),
"NCCL_IB_TIMEOUT": self.cfg.get("ib_timeout"),
"NCCL_IB_QPS_PER_CONNECTION": self.cfg.get("qps_per_connection"),
"NCCL_MIN_NCHANNELS": self.cfg.get("min_nchannels"),
"NCCL_NET_PLUGIN": self.cfg.get("net_plugin"),
"NCCL_NVLS_ENABLE": self.cfg.get("nvls_enable"),
"NCCL_IB_SPLIT_DATA_ON_QPS": self.cfg.get("split_data_on_qps"),
}
mpi_ld_preload = self._mpi_ld_preload()
if mpi_ld_preload:
env_cfg["LD_PRELOAD"] = mpi_ld_preload
extra_ld_library_path = self._extra_ld_library_path()
if extra_ld_library_path:
existing = os.environ.get("LD_LIBRARY_PATH", "")
env_cfg["LD_LIBRARY_PATH"] = ":".join(
[extra_ld_library_path] + ([existing] if existing else [])
)
extra_env = self.cfg.get("extra_env") or {}
if isinstance(extra_env, dict):
self._merge_env(env_cfg, extra_env)
if topo:
if topo.get("cuda_visible_devices"):
env_cfg["CUDA_VISIBLE_DEVICES"] = str(topo["cuda_visible_devices"])
if isinstance(topo.get("env"), dict):
self._merge_env(env_cfg, topo["env"])
op_env = topo.get("op_env")
if isinstance(op_env, dict):
for key in (label, binary):
overrides = op_env.get(key)
if isinstance(overrides, dict):
self._merge_env(env_cfg, overrides)
return [(k, str(v)) for k, v in env_cfg.items() if v is not None]
@staticmethod
def _merge_env(env_cfg: dict, overrides: dict):
for key, value in overrides.items():
key = str(key)
if value is None:
env_cfg.pop(key, None)
else:
env_cfg[key] = str(value)
def _mpi_ld_preload(self) -> str:
preload = self.cfg.get("mpi_ld_preload")
if isinstance(preload, list):
return " ".join(str(p) for p in preload if p)
return str(preload) if preload else ""
def _runtime_env(self) -> dict:
env = os.environ.copy()
mpi_ld_preload = self._mpi_ld_preload()
if mpi_ld_preload:
env["LD_PRELOAD"] = mpi_ld_preload
extra_ld_library_path = self._extra_ld_library_path()
if extra_ld_library_path:
existing = env.get("LD_LIBRARY_PATH", "")
env["LD_LIBRARY_PATH"] = ":".join(
[extra_ld_library_path] + ([existing] if existing else [])
)
return env
def _extra_ld_library_path(self) -> str:
paths = self.cfg.get("extra_ld_library_path")
if isinstance(paths, list):
return ":".join(str(p) for p in paths if p)
return str(paths) if paths else ""
def _preflight(self, mpirun: Optional[str], tests: list[str], hosts: list[dict]) -> dict:
checks = []
checks.append({"name": "mpirun", "status": "PASS" if mpirun else "FAIL", "detail": mpirun or "not found"})
checks.append({"name": "hosts", "status": "PASS" if len(hosts) >= 2 else "FAIL", "detail": f"{len(hosts)} configured"})
for binary in tests:
path = self._find_nccl_test(binary)
checks.append({"name": binary, "status": "PASS" if path else "FAIL", "detail": path or "not found"})
if self.cfg.get("ssh_preflight", True):
user = self.cfg.get("ssh_user", "root")
for host in hosts:
target = f"{user}@{host['addr']}"
cmd = [
"ssh",
"-o", "BatchMode=yes",
"-o", "ConnectTimeout=5",
"-o", "StrictHostKeyChecking=accept-new",
]
plm_rsh_args = self.cfg.get("plm_rsh_args")
if plm_rsh_args:
cmd.extend(shlex.split(str(plm_rsh_args)))
cmd.extend([target, "hostname"])
try:
r = subprocess.run(cmd, capture_output=True, text=True, timeout=8, env=self._runtime_env())
detail = r.stdout.strip() or r.stderr.strip()[:120]
checks.append({
"name": f"ssh {host['addr']}",
"status": "PASS" if r.returncode == 0 else "WARN",
"detail": detail,
})
except Exception as e:
checks.append({"name": f"ssh {host['addr']}", "status": "WARN", "detail": str(e)})
return {
"checks": checks,
"passed": all(c["status"] == "PASS" for c in checks if not c["name"].startswith("ssh ")),
}
def run(self) -> dict:
mpirun = self._find_mpirun()
tests = self._tests()
hosts = self._hosts()
topologies = self._topologies()
preflight = self._preflight(mpirun, tests, hosts)
if not preflight["passed"]:
return {
"passed": False,
"source": "nccl-tests-mpirun",
"mode": self.cfg.get("mode", "sweep"),
"hosts": hosts,
"preflight": preflight,
"tests": {},
"error": "multinode NCCL preflight failed",
"timestamp": datetime.now().isoformat(),
}
results = {}
for binary in tests:
label = _OP_LABELS[binary]
binary_path = self._find_nccl_test(binary)
op_results = []
for topo in topologies:
op_results.append(self._run_topology(mpirun, binary_path, label, hosts, topo))
results[label] = {"binary": binary_path, "topologies": op_results}
passed = all(
topo.get("status") == "PASS"
for op in results.values()
for topo in op.get("topologies", [])
)
return {
"passed": passed,
"source": "nccl-tests-mpirun",
"mode": self.cfg.get("mode", "sweep"),
"hosts": hosts,
"preflight": preflight,
"tests": results,
"artifact_dir": self.artifact_dir,
"timestamp": datetime.now().isoformat(),
}
def _run_topology(self, mpirun: str, binary: str, label: str, hosts: list[dict], topo: dict) -> dict:
nodes = topo["nodes"]
gpus_per_node = topo["gpus_per_node"]
selected_hosts = hosts[:nodes]
host_arg = ",".join(f"{h['addr']}:{gpus_per_node}" for h in selected_hosts)
ranks = nodes * gpus_per_node
cmd = [
mpirun,
"--allow-run-as-root",
"--mca", "btl_openib_warn_no_device_params_found", "0",
"--mca", "btl_tcp_if_include", str(self.cfg.get("socket_ifname", "bond0")),
"--mca", "oob_tcp_if_include", str(self.cfg.get("oob_tcp_ifname", self.cfg.get("socket_ifname", "bond0"))),
]
# Keep the MPI launcher/control plane separately configurable from the
# NCCL data plane. For example, PML/UCX can be restricted to TCP/eth0
# while NCCL uses the configured RoCE HCAs for benchmark traffic.
mpi_pml = self.cfg.get("mpi_pml", "ob1")
if mpi_pml:
cmd.extend(["--mca", "pml", str(mpi_pml)])
mpi_btl = self.cfg.get("mpi_btl", "self,tcp,vader")
if mpi_btl:
cmd.extend(["--mca", "btl", str(mpi_btl)])
# HCOLL does an eager UCX IB wireup during MPI_Init that deadlocks on nodes
# where UCX only sees mlx5_2/mlx5_3 instead of mlx5_0/mlx5_1. HCOLL
# collective optimisations are irrelevant for NCCL tests (NCCL handles its
# own collectives), so disable it unconditionally unless the caller opts in.
if self.cfg.get("disable_hcoll", True):
cmd.extend(["--mca", "coll", "^hcoll"])
cmd.extend([
"-H", host_arg,
"--map-by", f"ppr:{gpus_per_node}:node",
"-np", str(ranks),
])
plm_rsh_args = self.cfg.get("plm_rsh_args")
if plm_rsh_args:
cmd.extend(["--mca", "plm_rsh_args", str(plm_rsh_args)])
for key, value in self._env_exports(topo=topo, label=label, binary=os.path.basename(binary)):
cmd.extend(["-x", f"{key}={value}"])
cmd.extend([
binary,
"-b", str(self.cfg.get("begin_size", "1k")),
"-e", str(self.cfg.get("end_size", "16g")),
"-g", str(self.cfg.get("gpus_per_rank", 1)),
"-f", str(self.cfg.get("step_factor", 2)),
"-w", str(self.cfg.get("warmup_iters", 10)),
])
if self.cfg.get("iters") is not None:
cmd.extend(["-n", str(self.cfg["iters"])])
timeout = int(self.cfg.get("timeout_sec", 1800))
started = datetime.now().isoformat()
try:
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, env=self._runtime_env())
except subprocess.TimeoutExpired:
result = {
"label": topo["label"],
"nodes": nodes,
"gpus_per_node": gpus_per_node,
"ranks": ranks,
"hosts": selected_hosts,
"command": " ".join(cmd),
"status": "FAIL",
"error": f"timeout after {timeout}s",
"started_at": started,
}
self._write_artifacts(label, topo, result, "", "")
return result
parsed = self._parse_nccl_output(r.stdout)
net_diag = self._parse_network_diagnostics(r.stdout + "\n" + r.stderr)
threshold = self._threshold_for(label, topo)
wrong = sum(row.get("wrong", 0) for row in parsed["by_size"])
has_bw = parsed["peak_busbw_gbps"] > 0
status = "PASS" if r.returncode == 0 and has_bw and wrong == 0 and parsed["peak_busbw_gbps"] >= threshold else "FAIL"
result = {
"label": topo["label"],
"nodes": nodes,
"gpus_per_node": gpus_per_node,
"ranks": ranks,
"hosts": selected_hosts,
"cuda_visible_devices": topo.get("cuda_visible_devices"),
"command": " ".join(cmd),
"returncode": r.returncode,
"status": status,
"peak_busbw_gbps": parsed["peak_busbw_gbps"],
"peak_algbw_gbps": parsed["peak_algbw_gbps"],
"peak_size": parsed["peak_size"],
"avg_busbw_gbps": parsed["avg_busbw_gbps"],
"min_required_gbps": threshold,
"wrong_count": wrong,
"network": net_diag,
"by_size": parsed["by_size"],
"stderr_tail": r.stderr[-1200:],
"stdout_tail": r.stdout[-1200:],
"started_at": started,
"finished_at": datetime.now().isoformat(),
}
self._write_artifacts(label, topo, result, r.stdout, r.stderr)
return result
def _write_artifacts(self, label: str, topo: dict, result: dict, stdout: str, stderr: str):
if not self.artifact_dir:
return
os.makedirs(self.artifact_dir, exist_ok=True)
prefix = _safe_name(f"{label}_{topo.get('nodes')}x{topo.get('gpus_per_node')}_{topo.get('label')}")
base = os.path.join(self.artifact_dir, prefix)
with open(base + ".cmd.txt", "w") as f:
f.write(result.get("command", ""))
f.write("\n")
with open(base + ".stdout.txt", "w") as f:
f.write(stdout)
with open(base + ".stderr.txt", "w") as f:
f.write(stderr)
artifact_result = {k: v for k, v in result.items() if k not in ("stdout_tail", "stderr_tail")}
with open(base + ".json", "w") as f:
json.dump(artifact_result, f, indent=2, default=str)
result["artifact_prefix"] = base
def _threshold_for(self, label: str, topo: dict = None) -> float:
if topo and topo.get("min_peak_busbw_gbps") is not None:
topo_thresholds = topo.get("min_peak_busbw_gbps")
if isinstance(topo_thresholds, dict):
return float(topo_thresholds.get(label, 0) or 0)
return float(topo_thresholds or 0)
thresholds = self.cfg.get("min_peak_busbw_gbps") or {}
if isinstance(thresholds, dict):
op_threshold = thresholds.get(label, 0)
if isinstance(op_threshold, dict):
keys = []
if topo:
keys.extend([
topo.get("label"),
f"{topo.get('nodes')}x{topo.get('gpus_per_node')}",
f"{topo.get('nodes')} nodes x {topo.get('gpus_per_node')} GPUs",
str(topo.get("gpus_per_node")),
])
keys.append("default")
for key in keys:
if key in op_threshold:
return float(op_threshold.get(key) or 0)
return 0.0
return float(op_threshold or 0)
return float(thresholds or 0)
@staticmethod
def _parse_nccl_output(stdout: str) -> dict:
rows = []
avg_bus = 0.0
for line in stdout.splitlines():
stripped = line.strip()
if not stripped:
continue
avg_match = re.search(r"Avg bus bandwidth\s*:\s*([0-9.]+)", stripped)
if avg_match:
avg_bus = float(avg_match.group(1))
continue
if stripped.startswith("#"):
continue
parts = stripped.split()
if len(parts) < 9:
continue
try:
size_bytes = int(parts[0])
time_us = float(parts[5])
algbw = float(parts[6])
busbw = float(parts[7])
wrong = int(parts[8])
except (ValueError, IndexError):
continue
rows.append({
"size_bytes": size_bytes,
"size": _format_size(size_bytes),
"time_us": time_us,
"algbw_gbps": algbw,
"busbw_gbps": busbw,
"wrong": wrong,
})
peak_row = max(rows, key=lambda r: r["busbw_gbps"], default={})
return {
"peak_busbw_gbps": round(float(peak_row.get("busbw_gbps", 0)), 2),
"peak_algbw_gbps": round(float(peak_row.get("algbw_gbps", 0)), 2),
"peak_size": peak_row.get("size", ""),
"avg_busbw_gbps": round(avg_bus, 2),
"by_size": rows,
}
@staticmethod
def _parse_network_diagnostics(output: str) -> dict:
networks = sorted(set(re.findall(r"Using network (\S+)", output)))
gdr_enabled = sorted(set(re.findall(r"GPU Direct RDMA Enabled for HCA \d+ '([^']+)'", output)))
gdr_disabled = sorted(set(re.findall(r"GPU Direct RDMA Disabled for HCA \d+ '([^']+)'", output)))
ib_using = []
for line in output.splitlines():
if "NET/IB : Using" in line:
text = line.split("NET/IB : ", 1)[-1].strip()
if text not in ib_using:
ib_using.append(text)
if gdr_disabled:
gdr_state = "DISABLED"
elif gdr_enabled or "/GDRDMA" in output:
gdr_state = "ENABLED"
elif networks:
gdr_state = "NOT_DISABLED_IN_LOG"
else:
gdr_state = "UNKNOWN"
return {
"networks": networks,
"ib_using": ib_using[:8],
"gdr_enabled_hcas": gdr_enabled,
"gdr_disabled_hcas": gdr_disabled,
"gpu_direct_rdma": gdr_state,
}
@staticmethod
def print_results(results: dict, console: Console = None):
c = console or Console()
if results.get("error"):
c.print(f"[bold red]Multi-node NCCL failed: {results['error']}[/bold red]")
else:
c.print("[bold green]Multi-node NCCL complete[/bold green]" if results.get("passed") else "[bold red]Multi-node NCCL failed[/bold red]")
preflight = results.get("preflight", {})
if preflight.get("checks"):
table = Table(title="Preflight")
table.add_column("Check")
table.add_column("Status")
table.add_column("Detail")
for check in preflight["checks"]:
table.add_row(check["name"], check["status"], str(check.get("detail", "")))
c.print(table)
for op, data in (results.get("tests") or {}).items():
table = Table(title=f"Multi-node NCCL {op}")
table.add_column("Topology")
table.add_column("Peak Bus BW")
table.add_column("Peak Size")
table.add_column("Threshold")
table.add_column("Status")
for topo in data.get("topologies", []):
table.add_row(
topo.get("label", ""),
f"{topo.get('peak_busbw_gbps', 0):.2f} GB/s",
str(topo.get("peak_size", "")),
f">= {_format_gbps(topo.get('min_required_gbps', 0))} GB/s" if topo.get("min_required_gbps") else "-",
topo.get("status", "?"),
)
c.print(table)
def _format_size(size_bytes: int) -> str:
units = [("G", 1024 ** 3), ("M", 1024 ** 2), ("K", 1024)]
for suffix, factor in units:
if size_bytes >= factor and size_bytes % factor == 0:
return f"{size_bytes // factor}{suffix}"
return str(size_bytes)
def _format_gbps(value) -> str:
try:
numeric = float(value)
except (TypeError, ValueError):
return str(value)
if numeric.is_integer():
return f"{numeric:.0f}"
return f"{numeric:.2f}"
def _safe_name(value: str) -> str:
text = re.sub(r"[^A-Za-z0-9_.-]+", "_", value.strip())
text = re.sub(r"_+", "_", text).strip("_")
return text[:160] or "case"

View File

@ -5,12 +5,9 @@ import os
import re import re
import shutil import shutil
import subprocess import subprocess
import sysconfig
import tempfile
from datetime import datetime from datetime import datetime
from typing import Optional from typing import Optional
from rich import box
from rich.console import Console from rich.console import Console
from rich.table import Table from rich.table import Table
from rich.progress import Progress, SpinnerColumn, TextColumn, TimeElapsedColumn from rich.progress import Progress, SpinnerColumn, TextColumn, TimeElapsedColumn
@ -20,30 +17,12 @@ from modules.gpu_specs import detect_gpu_type, get_gpu_specs, resolve_tools_dir
TORCH_AVAILABLE = False TORCH_AVAILABLE = False
try: try:
import torch import torch
try:
import torch_dcu # noqa: F401 - enables DTK's PyTorch device shim
except ImportError:
pass
if torch.cuda.is_available(): if torch.cuda.is_available():
TORCH_AVAILABLE = True TORCH_AVAILABLE = True
except ImportError: except ImportError:
pass pass
# Per-operation bandwidth thresholds, as a fraction of NVLink bidirectional BW.
# Values aligned with the H100 production acceptance criteria (acceptance doc §5).
# AllToAll runs ~10-20% lower than AllReduce on 8-GPU NVSwitch, so its fraction is
# set lower; broadcast/sendrecv sit between.
_OP_BW_FRACTIONS = {
"allreduce": 0.45,
"allgather": 0.45,
"reducescatter": 0.45,
"broadcast": 0.40,
"sendrecv": 0.40,
"alltoall": 0.35,
}
class NCCLTest: class NCCLTest:
def __init__(self, config: dict): def __init__(self, config: dict):
@ -51,8 +30,7 @@ class NCCLTest:
self.console = Console() self.console = Console()
self.nccl_cfg = config.get("nccl", {}) self.nccl_cfg = config.get("nccl", {})
self.tools_dir = resolve_tools_dir(config) self.tools_dir = resolve_tools_dir(config)
configured = config.get("gpu_type", "auto") self.gpu_type = detect_gpu_type()
self.gpu_type = configured if configured != "auto" else detect_gpu_type()
self.specs = get_gpu_specs(self.gpu_type) self.specs = get_gpu_specs(self.gpu_type)
def _find_nccl_test(self, name: str) -> Optional[str]: def _find_nccl_test(self, name: str) -> Optional[str]:
@ -78,30 +56,6 @@ class NCCLTest:
return p return p
return None return None
@staticmethod
def _runtime_env() -> dict:
"""Return an environment that can load NCCL bundled in the active venv.
CUDA PyTorch wheels install libnccl under site-packages/nvidia/nccl/lib,
which is intentionally isolated from the system linker path. nccl-tests
is an external executable, so it needs that directory explicitly even
though importing torch works in the same virtual environment.
"""
env = os.environ.copy()
purelib = sysconfig.get_paths().get("purelib", "")
candidates = [
os.path.join(purelib, "nvidia", "nccl", "lib"),
os.path.join(purelib, "torch", "lib"),
]
library_dirs = [path for path in candidates if os.path.isdir(path)]
existing = env.get("LD_LIBRARY_PATH", "")
if existing:
library_dirs.append(existing)
if library_dirs:
env["LD_LIBRARY_PATH"] = os.pathsep.join(library_dirs)
env["NCCL_DEBUG"] = "WARN"
return env
def run(self) -> dict: def run(self) -> dict:
gpu_count = 0 gpu_count = 0
if TORCH_AVAILABLE: if TORCH_AVAILABLE:
@ -121,26 +75,17 @@ class NCCLTest:
if self.nccl_cfg.get("test_reduce_scatter", False): if self.nccl_cfg.get("test_reduce_scatter", False):
tests.append(("reduce_scatter_perf", "ReduceScatter")) tests.append(("reduce_scatter_perf", "ReduceScatter"))
if self.nccl_cfg.get("test_allgather", False): if self.nccl_cfg.get("test_allgather", False):
tests.append(("all_gather_perf", "AllGather")) tests.append(("allgather_perf", "AllGather"))
if self.nccl_cfg.get("test_sendrecv", False): if self.nccl_cfg.get("test_sendrecv", False):
tests.append(("sendrecv_perf", "SendRecv")) tests.append(("sendrecv_perf", "SendRecv"))
nvlink_bw = self.specs.get("nvlink_bandwidth_gbps", 0) nvlink_bw = self.specs.get("nvlink_bandwidth_gbps", 0)
calibrated_thresholds = self.specs.get("nccl_pass_thresholds_gbps", {}) or {} if nvlink_bw > 0:
# User-provided override applies uniformly across all ops; otherwise default_min_bw = nvlink_bw * 0.4
# each op gets its own threshold from _OP_BW_FRACTIONS. else:
user_override = self.nccl_cfg.get("min_bandwidth_gbps") # Conservative floor: any working NVLink should exceed 10 GB/s
default_min_bw = 10
def threshold_for(label: str) -> float: min_bw = self.nccl_cfg.get("min_bandwidth_gbps") or round(default_min_bw)
if user_override:
return float(user_override)
calibrated = calibrated_thresholds.get(label.lower())
if calibrated:
return float(calibrated)
if nvlink_bw <= 0:
return 10.0 # conservative floor
frac = _OP_BW_FRACTIONS.get(label.lower(), 0.45)
return round(nvlink_bw * frac)
if self.gpu_type == "unknown": if self.gpu_type == "unknown":
self.console.print("[yellow]Unknown GPU — using conservative bandwidth thresholds[/yellow]") self.console.print("[yellow]Unknown GPU — using conservative bandwidth thresholds[/yellow]")
@ -158,9 +103,8 @@ class NCCLTest:
for binary, label in tests: for binary, label in tests:
progress.update(task, description=f"NCCL {label}...") progress.update(task, description=f"NCCL {label}...")
op_min_bw = threshold_for(label)
result = self._run_one_nccl_test_direct( result = self._run_one_nccl_test_direct(
binary, label, gpu_count, op_min_bw binary, label, gpu_count, min_bw
) )
if result.get("status") not in ("SKIP", None) and "error" not in result: if result.get("status") not in ("SKIP", None) and "error" not in result:
any_binary_worked = True any_binary_worked = True
@ -170,7 +114,7 @@ class NCCLTest:
mpirun = self._find_mpirun() mpirun = self._find_mpirun()
if mpirun: if mpirun:
result = self._run_one_nccl_test_mpirun( result = self._run_one_nccl_test_mpirun(
binary, label, gpu_count, mpirun, op_min_bw binary, label, gpu_count, mpirun, min_bw
) )
if result.get("status") not in ("SKIP", None) and "error" not in result: if result.get("status") not in ("SKIP", None) and "error" not in result:
any_binary_worked = True any_binary_worked = True
@ -181,23 +125,16 @@ class NCCLTest:
self.console.print("[yellow]nccl-tests binaries failed, falling back to torchrun[/yellow]") self.console.print("[yellow]nccl-tests binaries failed, falling back to torchrun[/yellow]")
return self._run_torchrun_fallback(gpu_count) return self._run_torchrun_fallback(gpu_count)
statuses = [ all_passed = all(
r.get("status") for r in results.values() r.get("status") == "PASS"
if isinstance(r, dict) and r.get("status") for r in results.values()
] if isinstance(r, dict) and "status" in r
overall_status = (
"FAIL" if "FAIL" in statuses else
"WARN" if "WARN" in statuses else
"PASS" if statuses else "SKIP"
) )
return { return {
"passed": overall_status in ("PASS", "WARN"), "passed": all_passed,
"status": overall_status,
"source": "nccl-tests", "source": "nccl-tests",
"min_bandwidth_gbps": { "min_bandwidth_gbps": min_bw,
lbl.lower(): threshold_for(lbl) for _, lbl in tests
},
"tests": results, "tests": results,
"gpu_count": gpu_count, "gpu_count": gpu_count,
"timestamp": datetime.now().isoformat(), "timestamp": datetime.now().isoformat(),
@ -213,8 +150,8 @@ class NCCLTest:
cmd = [ cmd = [
binary, binary,
"-b", "8M", "-b", "8",
"-e", "8G", "-e", "256M",
"-f", "2", "-f", "2",
"-g", str(gpu_count), "-g", str(gpu_count),
"-w", "5", "-w", "5",
@ -222,7 +159,8 @@ class NCCLTest:
] ]
try: try:
env = self._runtime_env() env = os.environ.copy()
env["NCCL_DEBUG"] = "WARN"
r = subprocess.run(cmd, capture_output=True, text=True, timeout=180, env=env) r = subprocess.run(cmd, capture_output=True, text=True, timeout=180, env=env)
combined = r.stdout + r.stderr combined = r.stdout + r.stderr
@ -267,7 +205,8 @@ class NCCLTest:
] ]
try: try:
env = self._runtime_env() env = os.environ.copy()
env["NCCL_DEBUG"] = "WARN"
r = subprocess.run(cmd, capture_output=True, text=True, timeout=180, env=env) r = subprocess.run(cmd, capture_output=True, text=True, timeout=180, env=env)
combined = r.stdout + r.stderr combined = r.stdout + r.stderr
@ -300,15 +239,12 @@ class NCCLTest:
if not line or line.startswith("#"): if not line or line.startswith("#"):
continue continue
parts = line.split() parts = line.split()
# nccl-tests data lines: size count type redop root time algbw busbw #wrong [time algbw busbw #wrong] if len(parts) >= 7:
if len(parts) >= 9:
try: try:
size = int(parts[0]) size = int(parts[0])
# parts[2] is dtype string ('float'/'int32'/etc.), not a number algbw = float(parts[-3]) if len(parts) >= 3 else 0
# out-of-place columns: time=parts[5], algbw=parts[6], busbw=parts[7] busbw = float(parts[-2]) if len(parts) >= 2 else 0
time_us = float(parts[5]) time_us = float(parts[2]) if len(parts) >= 3 else 0
algbw = float(parts[6])
busbw = float(parts[7])
size_results.append({ size_results.append({
"size": size, "size": size,
"time_us": time_us, "time_us": time_us,
@ -332,78 +268,60 @@ class NCCLTest:
} }
def _run_torchrun_fallback(self, gpu_count: int) -> dict: def _run_torchrun_fallback(self, gpu_count: int) -> dict:
"""Benchmark collectives through PyTorch when nccl-tests is unavailable.""" """Basic NCCL connectivity test via torchrun — verifies NCCL works but does not benchmark performance."""
self.console.print("[yellow]nccl-tests not available, benchmarking collectives via torchrun[/yellow]") self.console.print("[yellow]nccl-tests not available, running basic NCCL connectivity check[/yellow]")
size_mb = int(self.nccl_cfg.get("torchrun_size_mb", 64))
warmup = int(self.nccl_cfg.get("torchrun_warmup", 5))
iterations = int(self.nccl_cfg.get("torchrun_iterations", 20))
backend = self.nccl_cfg.get("torch_backend", "nccl")
enabled_ops = []
for key, enabled in [
("allreduce", self.nccl_cfg.get("test_allreduce", True)),
("alltoall", self.nccl_cfg.get("test_alltoall", True)),
("broadcast", self.nccl_cfg.get("test_broadcast", True)),
("reducescatter", self.nccl_cfg.get("test_reduce_scatter", False)),
("allgather", self.nccl_cfg.get("test_allgather", False)),
]:
if enabled:
enabled_ops.append(key)
code = f""" code = f"""
import os, time import torch, torch.distributed as dist, os
import torch
try:
import torch_dcu
except ImportError:
pass
import torch.distributed as dist
os.environ.setdefault("MASTER_ADDR","127.0.0.1") os.environ.setdefault("MASTER_ADDR","127.0.0.1")
os.environ.setdefault("MASTER_PORT","29500") os.environ.setdefault("MASTER_PORT","29500")
rank=int(os.environ.get("LOCAL_RANK",0)) rank=int(os.environ.get("LOCAL_RANK",0))
ws={gpu_count} ws={gpu_count}
dist.init_process_group({backend!r},rank=rank,world_size=ws) dist.init_process_group("nccl",rank=rank,world_size=ws)
torch.cuda.set_device(rank) torch.cuda.set_device(rank)
size_bytes={size_mb}*1024*1024
numel=size_bytes//4
x=torch.ones(numel,device=f"cuda:{{rank}}",dtype=torch.float32)
allgather_out=torch.empty(numel*ws,device=f"cuda:{{rank}}",dtype=torch.float32)
rs_out=torch.empty(numel//ws,device=f"cuda:{{rank}}",dtype=torch.float32)
a2a_out=torch.empty_like(x)
def run_op(name): x=torch.randn(1024*1024,device=f"cuda:{{rank}}",dtype=torch.float32)
if name == "allreduce":
return dist.all_reduce(x) # Test AllReduce
if name == "broadcast": try:
return dist.broadcast(x, src=0) dist.all_reduce(x.clone())
if name == "allgather": if rank==0: print("allreduce:ok")
return dist.all_gather_into_tensor(allgather_out, x) except Exception as e:
if name == "reducescatter": if rank==0: print(f"allreduce:fail:{{e}}")
return dist.reduce_scatter_tensor(rs_out, x)
if name == "alltoall": # Test Broadcast
return dist.all_to_all_single(a2a_out, x) try:
raise ValueError(name) dist.broadcast(x.clone(),src=0)
if rank==0: print("broadcast:ok")
except Exception as e:
if rank==0: print(f"broadcast:fail:{{e}}")
# Test AllGather
try:
tensor_list=[torch.empty_like(x) for _ in range(ws)]
dist.all_gather(tensor_list,x.clone())
if rank==0: print("allgather:ok")
except Exception as e:
if rank==0: print(f"allgather:fail:{{e}}")
# Test ReduceScatter
try:
chunks=list(x.chunk(ws))
output=torch.empty_like(chunks[0])
dist.reduce_scatter(output,chunks)
if rank==0: print("reducescatter:ok")
except Exception as e:
if rank==0: print(f"reducescatter:fail:{{e}}")
# Test AllToAll
try:
chunks=list(x.chunk(ws))
output_list=[torch.empty_like(c) for c in chunks]
dist.all_to_all(output_list,chunks)
if rank==0: print("alltoall:ok")
except Exception as e:
if rank==0: print(f"alltoall:fail:{{e}}")
for name in {enabled_ops!r}:
try:
for _ in range({warmup}):
run_op(name)
torch.cuda.synchronize()
dist.barrier()
start=time.perf_counter()
for _ in range({iterations}):
run_op(name)
torch.cuda.synchronize()
elapsed=(time.perf_counter()-start)/{iterations}
algbw=size_bytes/elapsed/1e9
factor={{"allreduce":2*(ws-1)/ws,"allgather":(ws-1)/ws,
"reducescatter":(ws-1)/ws,"alltoall":(ws-1)/ws,
"broadcast":1.0}}[name]
busbw=algbw*factor
if rank==0:
print(f"METRIC|{{name}}|{{elapsed*1e6:.3f}}|{{algbw:.3f}}|{{busbw:.3f}}", flush=True)
except Exception as exc:
if rank==0:
print(f"ERROR|{{name}}|{{exc}}", flush=True)
dist.destroy_process_group() dist.destroy_process_group()
""" """
import tempfile import tempfile
@ -419,50 +337,41 @@ dist.destroy_process_group()
r = subprocess.run( r = subprocess.run(
[torchrun_cmd, f"--nproc_per_node={gpu_count}", tmp.name], [torchrun_cmd, f"--nproc_per_node={gpu_count}", tmp.name],
capture_output=True, text=True, timeout=300, capture_output=True, text=True, timeout=120,
env={**os.environ, "NCCL_DEBUG": "WARN", "RCCL_DEBUG": "WARN"}, env={**os.environ, "NCCL_DEBUG": "WARN"},
) )
os.unlink(tmp.name) os.unlink(tmp.name)
# Parse connectivity results — format: op_name:ok or op_name:fail:error
tests = {} tests = {}
all_passed = True
for line in r.stdout.split("\n"): for line in r.stdout.split("\n"):
line = line.strip() line = line.strip()
if line.startswith("METRIC|"): if not line:
_, name, time_us, algbw, busbw = line.split("|", 4) continue
threshold = self.nccl_cfg.get("min_bandwidth_gbps") or 10.0 parts = line.split(":")
busbw_f = float(busbw) op_name = parts[0]
tests[name] = { result = parts[1] if len(parts) > 1 else "unknown"
"status": "PASS" if busbw_f >= float(threshold) else "WARN",
"best_algbw_gbps": round(float(algbw), 2), if result == "ok":
"best_busbw_gbps": round(busbw_f, 2), status = "PASS"
"min_required_gbps": float(threshold), else:
"by_size": [{ status = "FAIL"
"size": size_mb * 1024 * 1024, all_passed = False
"time_us": round(float(time_us), 2),
"algbw_gbps": round(float(algbw), 2), tests[op_name] = {
"busbw_gbps": round(busbw_f, 2), "status": status,
}], "error": ":".join(parts[2:]) if len(parts) > 2 and result == "fail" else None,
} }
elif line.startswith("ERROR|"):
_, name, error = line.split("|", 2)
tests[name] = {"status": "FAIL", "error": error}
if not tests:
error = (r.stderr or r.stdout or "torchrun produced no metrics")[-1000:]
return {"passed": False, "source": "torchrun_collective", "error": error}
all_passed = all(value.get("status") == "PASS" for value in tests.values())
return { return {
"passed": all_passed, "passed": all_passed,
"source": "torchrun_collective", "source": "torchrun_fallback",
"tests": tests, "tests": tests,
"gpu_count": gpu_count, "gpu_count": gpu_count,
"message_size_mb": size_mb,
"warmup": warmup,
"iterations": iterations,
"backend": backend,
"timestamp": datetime.now().isoformat(),
} }
except Exception as e: except Exception as e:
return {"passed": False, "source": "torchrun_collective", "error": str(e)} return {"passed": False, "source": "torchrun_fallback", "error": str(e)}
@staticmethod @staticmethod
def print_results(results: dict, console: Console = None): def print_results(results: dict, console: Console = None):
@ -472,7 +381,6 @@ dist.destroy_process_group()
return return
passed = results.get("passed", False) passed = results.get("passed", False)
overall_status = results.get("status") or ("PASS" if passed else "FAIL")
source = results.get("source", "unknown") source = results.get("source", "unknown")
if source == "torchrun_fallback": if source == "torchrun_fallback":
@ -497,12 +405,7 @@ dist.destroy_process_group()
c.print("\n[yellow]Note: functional connectivity test only (no performance data)[/yellow]") c.print("\n[yellow]Note: functional connectivity test only (no performance data)[/yellow]")
else: else:
# nccl-tests mode # nccl-tests mode
if overall_status == "PASS": verdict = "[bold green]✓ NCCL tests PASSED[/bold green]" if passed else "[bold yellow]⚠ NCCL tests WARNING[/bold yellow]"
verdict = "[bold green]✓ NCCL tests PASSED[/bold green]"
elif overall_status == "WARN":
verdict = "[bold yellow]⚠ NCCL tests WARNING[/bold yellow]"
else:
verdict = "[bold red]✗ NCCL tests FAILED[/bold red]"
c.print(f"{verdict} [dim](via {source})[/dim]") c.print(f"{verdict} [dim](via {source})[/dim]")
tests = results.get("tests", {}) tests = results.get("tests", {})
@ -529,251 +432,3 @@ dist.destroy_process_group()
t.add_row(sz_str, f"{r.get('time_us',0):.1f}", t.add_row(sz_str, f"{r.get('time_us',0):.1f}",
f"{r.get('algbw_gbps',0):.1f}", f"{r.get('busbw_gbps',0):.1f}") f"{r.get('algbw_gbps',0):.1f}", f"{r.get('busbw_gbps',0):.1f}")
c.print(t) c.print(t)
class MultiNodeNCCLTest:
"""Multi-node NCCL scaling test — runs nccl-tests via mpirun across a cluster hostfile.
Config section: multinode_nccl
hosts_file path to OpenMPI hostfile (one line: "<ip> slots=8")
mpi_home MPI installation prefix (non-default location)
nccl_ib_hca comma-separated IB HCAs (GPU-direct PIX cards only)
nccl_socket_ifname bootstrap NIC name (e.g. bond0)
scale_levels list of node counts to test
ops list of operations (all_reduce, alltoall, all_gather, reduce_scatter)
msg_size_bytes message size for most ops (default 8G)
msg_size_alltoall_bytes message size for alltoall (default 1G, avoids hang at 16+ nodes)
warmup / iterations nccl-tests -w / -n
timeout_sec per-op subprocess timeout
allreduce_min_busbw_gbps PASS threshold for AllReduce
"""
_OP_BINARY = {
"all_reduce": "all_reduce_perf",
"alltoall": "alltoall_perf",
"all_gather": "all_gather_perf",
"reduce_scatter": "reduce_scatter_perf",
}
_OP_LABEL = {
"all_reduce": "AllReduce",
"alltoall": "AllToAll",
"all_gather": "AllGather",
"reduce_scatter": "ReduceScatter",
}
def __init__(self, config: dict):
self.config = config
self.console = Console()
self.mn_cfg = config.get("multinode_nccl", {})
self.tools_dir = resolve_tools_dir(config)
def _find_mpirun(self) -> Optional[str]:
mpi_home = os.path.expanduser(self.mn_cfg.get("mpi_home", ""))
if mpi_home:
candidate = os.path.join(mpi_home, "bin", "mpirun")
if os.path.isfile(candidate):
return candidate
return shutil.which("mpirun") or shutil.which("mpiexec")
def _find_nccl_build_dir(self) -> Optional[str]:
candidates = [
"/opt/nccl-tests/build",
os.path.join(self.tools_dir, "nccl-tests", "build"),
]
for d in candidates:
probe = os.path.join(d, "all_reduce_perf")
if os.path.isfile(probe) and os.access(probe, os.X_OK):
return d
return None
def _build_env(self) -> dict:
env = os.environ.copy()
mpi_home = os.path.expanduser(self.mn_cfg.get("mpi_home", ""))
if mpi_home:
env["PATH"] = f"{mpi_home}/bin:{env.get('PATH', '')}"
env["LD_LIBRARY_PATH"] = f"{mpi_home}/lib:{env.get('LD_LIBRARY_PATH', '')}"
env["NCCL_IB_HCA"] = self.mn_cfg.get(
"nccl_ib_hca", "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_8,mlx5_9"
)
env["NCCL_IB_DISABLE"] = "0"
env["NCCL_SOCKET_IFNAME"] = self.mn_cfg.get("nccl_socket_ifname", "bond0")
env["NCCL_DEBUG"] = "WARN"
return env
def _load_hosts(self, hosts_file: str) -> list:
path = os.path.expanduser(hosts_file)
if not os.path.isfile(path):
return []
lines = []
with open(path) as f:
for line in f:
line = line.strip()
if line and not line.startswith("#"):
lines.append(line)
return lines
def _run_op(self, mpirun: str, nt_dir: str, op: str, n_nodes: int,
host_lines: list, env: dict) -> dict:
binary = os.path.join(nt_dir, self._OP_BINARY[op])
if not os.path.isfile(binary):
return {"busbw_gbps": None, "status": "SKIP", "error": f"{op} binary not found"}
if op == "alltoall":
msg_bytes = self.mn_cfg.get("msg_size_alltoall_bytes", 1073741824)
else:
msg_bytes = self.mn_cfg.get("msg_size_bytes", 8589934592)
with tempfile.NamedTemporaryFile(mode="w", suffix=".hosts", delete=False) as tf:
tf.write("\n".join(host_lines[:n_nodes]) + "\n")
hf_path = tf.name
cmd = [
mpirun,
"--allow-run-as-root",
"--mca", "plm_rsh_args", "-oStrictHostKeyChecking=no",
"-np", str(n_nodes * 8),
"-N", "8",
"--hostfile", hf_path,
"-x", "PATH",
"-x", "LD_LIBRARY_PATH",
"-x", "NCCL_IB_HCA",
"-x", "NCCL_IB_DISABLE",
"-x", "NCCL_SOCKET_IFNAME",
"-x", "NCCL_DEBUG",
binary,
"-b", "8",
"-e", str(msg_bytes),
"-f", "2",
"-g", "1",
"-w", str(self.mn_cfg.get("warmup", 5)),
"-n", str(self.mn_cfg.get("iterations", 30)),
]
timeout = self.mn_cfg.get("timeout_sec", 600)
try:
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, env=env)
os.unlink(hf_path)
if r.returncode != 0:
return {"busbw_gbps": None, "status": "ERR", "error": r.stderr[:200]}
return self._parse_op_output(r.stdout, msg_bytes)
except subprocess.TimeoutExpired:
os.unlink(hf_path)
return {"busbw_gbps": None, "status": "ERR", "error": "timeout"}
except Exception as e:
os.unlink(hf_path)
return {"busbw_gbps": None, "status": "ERR", "error": str(e)}
@staticmethod
def _parse_op_output(stdout: str, target_bytes: int) -> dict:
for line in stdout.split("\n"):
line = line.strip()
if not line or line.startswith("#"):
continue
parts = line.split()
if len(parts) < 9:
continue
try:
if int(parts[0]) == target_bytes:
return {"busbw_gbps": round(float(parts[7]), 2), "status": "OK"}
except (ValueError, IndexError):
continue
return {"busbw_gbps": None, "status": "ERR", "error": "size not found in output"}
def run(self) -> dict:
hosts_file = self.mn_cfg.get("hosts_file", "~/nccl_kit/hosts_all")
scale_levels = self.mn_cfg.get("scale_levels", [2, 4, 8])
ops = self.mn_cfg.get("ops", ["all_reduce", "alltoall", "all_gather", "reduce_scatter"])
min_busbw = self.mn_cfg.get("allreduce_min_busbw_gbps", 310)
mpirun = self._find_mpirun()
if not mpirun:
return {"error": "mpirun not found — set multinode_nccl.mpi_home in config"}
nt_dir = self._find_nccl_build_dir()
if not nt_dir:
return {"error": "nccl-tests not found (need MPI=1 build at /opt/nccl-tests/build)"}
host_lines = self._load_hosts(hosts_file)
if not host_lines:
return {"error": f"hostfile not found or empty: {hosts_file}"}
env = self._build_env()
results = {}
runnable = [n for n in scale_levels if n <= len(host_lines)]
skipped = [n for n in scale_levels if n > len(host_lines)]
if skipped:
self.console.print(f"[yellow]⚠ 跳过节点数 {skipped}:hostfile 只有 {len(host_lines)} 台[/yellow]")
with Progress(
SpinnerColumn(), TextColumn("[progress.description]{task.description}"),
TimeElapsedColumn(), console=self.console,
) as progress:
task = progress.add_task("Multi-node NCCL...", total=len(runnable) * len(ops))
for n in runnable:
results[n] = {}
for op in ops:
progress.update(task, description=f"{n} nodes / {self._OP_LABEL.get(op, op)}...")
results[n][op] = self._run_op(mpirun, nt_dir, op, n, host_lines, env)
progress.advance(task)
passed = all(
(results.get(n, {}).get("all_reduce", {}).get("busbw_gbps") or 0) >= min_busbw
for n in runnable
)
return {
"passed": passed,
"hosts_file": hosts_file,
"available_nodes": len(host_lines),
"scale_levels": scale_levels,
"ops": ops,
"allreduce_min_busbw_gbps": min_busbw,
"results": results,
"timestamp": datetime.now().isoformat(),
}
def print_results(self, results: dict):
c = self.console
if "error" in results:
c.print(f"[bold red]Multi-node NCCL Error: {results['error']}[/bold red]")
return
passed = results.get("passed", False)
verdict = "[bold green]✓ PASS[/bold green]" if passed else "[bold red]✗ FAIL[/bold red]"
c.print(f"\n[bold cyan]Multi-node NCCL Scaling[/bold cyan] — {verdict}")
c.print(f" Hostfile: {results['hosts_file']} ({results['available_nodes']} nodes available)")
c.print(f" AllReduce PASS threshold: ≥ {results['allreduce_min_busbw_gbps']} GB/s")
data = results.get("results", {})
ops = results.get("ops", [])
scale_levels = [n for n in results.get("scale_levels", []) if n in data]
min_busbw = results.get("allreduce_min_busbw_gbps", 310)
t = Table(box=box.SIMPLE_HEAD, border_style="cyan")
t.add_column("Nodes / GPUs", style="bold", justify="center")
for op in ops:
t.add_column(self._OP_LABEL.get(op, op) + " (GB/s)", justify="right")
t.add_column("", justify="center")
for n in scale_levels:
row_data = data.get(n, {})
cells = []
ok = True
for op in ops:
r = row_data.get(op, {})
bw = r.get("busbw_gbps")
if bw is None:
err = r.get("error", "ERR")[:8]
cells.append(f"[red]{err}[/red]")
ok = False
else:
is_alltoall = op == "alltoall"
fail = (op == "all_reduce" and bw < min_busbw)
color = "red" if fail else "green"
suffix = "¹" if is_alltoall else ""
cells.append(f"[{color}]{bw:.1f}[/{color}]{suffix}")
status_str = "[green]PASS[/green]" if ok else "[red]FAIL[/red]"
t.add_row(f"{n} / {n * 8}", *cells, status_str)
c.print(t)
c.print("[dim]¹ AllToAll 使用 1G 消息(避免大消息 hang) | 其余操作使用 8G 消息[/dim]")

View File

@ -3,7 +3,6 @@
import os import os
import shutil import shutil
import subprocess import subprocess
import time
from datetime import datetime from datetime import datetime
from typing import Optional, List from typing import Optional, List
@ -38,111 +37,29 @@ class RDMATest:
ports = sorted(os.listdir(ports_dir)) ports = sorted(os.listdir(ports_dir))
return ports return ports
@staticmethod
def _read_sys(path: str) -> str:
try:
with open(path) as f:
return f.read().strip()
except (FileNotFoundError, PermissionError, OSError):
return ""
def run(self) -> dict: def run(self) -> dict:
devices = self._get_ib_devices() devices = self._get_ib_devices()
if not devices: if not devices:
self.console.print( self.console.print("[yellow]No InfiniBand devices found[/yellow]")
"[yellow]No InfiniBand devices found — skipping RDMA test[/yellow]" return {"error": "no_ib_devices", "passed": False}
)
return {
"status": "SKIP", "skipped": True,
"reason": "no IB hardware detected",
"timestamp": datetime.now().isoformat(),
}
# Only consider ports whose link_layer is InfiniBand — Ethernet self.console.print(f"[cyan]RDMA Test - Devices: {', '.join(devices)}[/cyan]")
# bond/management interfaces (e.g. mlx5_bond_0) can show ACTIVE state
# without actually providing IB fabric connectivity.
ib_devices = []
active_ib_devices = []
active_ib_port = False
for dev in devices:
for port in self._get_ib_ports(dev):
link_layer = self._read_sys(
f"/sys/class/infiniband/{dev}/ports/{port}/link_layer")
if link_layer != "InfiniBand":
continue
ib_devices.append((dev, port))
state = self._read_sys(
f"/sys/class/infiniband/{dev}/ports/{port}/state")
if "ACTIVE" in state.upper():
active_ib_port = True
active_ib_devices.append(dev)
device_info = self._collect_device_info(devices) device_info = self._collect_device_info(devices)
cross_enabled = (self.rdma_cfg.get("cross_node", {}) or {}).get("enabled") bw_results = self._run_bandwidth_tests(devices)
if not ib_devices and cross_enabled: latency_results = self._run_latency_tests(devices)
# RoCE ports report link_layer=Ethernet. They are valid for configured
# cross-node perftest even though local InfiniBand loopback is skipped.
cross = self.run_cross_node()
return {
"passed": cross.get("status") == "PASS",
"status": cross.get("status", "FAIL"),
"reason": "RoCE cross-node mode",
"devices": device_info,
"bandwidth_tests": [],
"latency_tests": [],
"cross_node": cross,
"timestamp": datetime.now().isoformat(),
}
if not ib_devices:
self.console.print(
"[yellow]No InfiniBand-link_layer ports present — "
"skipping RDMA benchmarks[/yellow]"
)
return {
"status": "SKIP", "skipped": True,
"reason": "no InfiniBand link_layer ports (only Ethernet/RoCE)",
"devices": device_info,
"timestamp": datetime.now().isoformat(),
}
if not active_ib_port:
self.console.print(
f"[yellow]{len(ib_devices)} IB port(s) detected but all DOWN — "
f"fabric not wired, skipping RDMA benchmarks[/yellow]"
)
return {
"status": "SKIP", "skipped": True,
"reason": f"{len(ib_devices)} IB port(s) found but all DOWN (fabric not wired)",
"devices": device_info,
"timestamp": datetime.now().isoformat(),
}
# perftest is a two-endpoint benchmark. Running both endpoints on all_passed = all(
# localhost can select a DOWN HCA, bypass the fabric, or return a zero r.get("status") == "PASS"
# result that looks like hardware failure. Inventory-only is objective for r in bw_results + latency_results
# until a peer is explicitly configured. if isinstance(r, dict)
if not cross_enabled:
self.console.print(
"[yellow]Active IB ports detected, but no cross-node peer is configured — "
"skipping bandwidth/latency benchmarks[/yellow]"
) )
return {
"status": "SKIP", "skipped": True,
"reason": "cross-node RDMA peer not configured; local loopback is not an acceptance test",
"devices": device_info,
"active_ib_devices": sorted(set(active_ib_devices)),
"timestamp": datetime.now().isoformat(),
}
cross = self.run_cross_node()
return { return {
"passed": cross.get("status") == "PASS", "passed": all_passed,
"status": cross.get("status", "FAIL"),
"reason": "cross-node RDMA benchmark",
"devices": device_info, "devices": device_info,
"active_ib_devices": sorted(set(active_ib_devices)), "bandwidth_tests": bw_results,
"bandwidth_tests": [], "latency_tests": latency_results,
"latency_tests": [],
"cross_node": cross,
"timestamp": datetime.now().isoformat(), "timestamp": datetime.now().isoformat(),
} }
@ -281,216 +198,9 @@ class RDMATest:
return results return results
# ------------------------------------------------------------------
# Cross-node (two-host) RDMA over perftest, orchestrated via SSH.
# Runs FROM the client host: for each IB device it launches the matching
# perftest server on the peer over SSH (held open in a live ssh channel),
# then runs the local client against the peer's OOB address and parses the
# result. Replaces the old standalone scripts/rdma_cross_node.sh.
# ------------------------------------------------------------------
def _active_ib_devices(self) -> List[str]:
"""Active IB devices, optionally including Ethernet/RoCE devices."""
out = []
allow_roce = bool((self.rdma_cfg.get("cross_node", {}) or {}).get("allow_roce"))
for dev in self._get_ib_devices():
for port in self._get_ib_ports(dev):
ll = self._read_sys(f"/sys/class/infiniband/{dev}/ports/{port}/link_layer")
st = self._read_sys(f"/sys/class/infiniband/{dev}/ports/{port}/state")
if (ll == "InfiniBand" or allow_roce) and "ACTIVE" in st.upper():
out.append(dev)
break
return out
def run_cross_node(self) -> dict:
cn = self.rdma_cfg.get("cross_node", {}) or {}
if not cn.get("enabled"):
return {"status": "SKIP", "skipped": True,
"reason": "rdma.cross_node.enabled is false"}
server = cn.get("server")
if not server:
return {"status": "SKIP", "skipped": True,
"reason": "rdma.cross_node.server (peer ssh address) not set"}
ssh_user = cn.get("ssh_user", "root")
server_target = server if "@" in server else f"{ssh_user}@{server}"
# OOB address the client's perftest connects to (defaults to the ssh host).
server_addr = cn.get("server_addr") or server.split("@")[-1]
ib_port = cn.get("ib_port", 1)
gid_index = cn.get("gid_index")
msg_size = cn.get("msg_size", 1048576)
iters = cn.get("iters", 5000)
duration_sec = cn.get("duration_sec")
queue_pairs = int(cn.get("queue_pairs", 1))
base_port = cn.get("base_oob_port", 18515)
warmup = cn.get("server_warmup_sec", 2.0)
min_bw = cn.get("min_bandwidth_gbps", 350)
max_lat = cn.get("max_latency_us", 5)
devices = cn.get("devices") or self._active_ib_devices()
if not devices:
return {"status": "SKIP", "skipped": True,
"reason": "no active InfiniBand devices to test"}
has_bw = self._find_tool("ib_write_bw") is not None
has_lat = self._find_tool("ib_write_lat") is not None
if not has_bw and not has_lat:
return {"status": "SKIP", "skipped": True,
"reason": "perftest (ib_write_bw / ib_write_lat) not installed"}
self.console.print(
f"[cyan]Cross-node RDMA — client → {server_addr}, "
f"devices: {', '.join(devices)}[/cyan]")
per_device = []
for idx, dev in enumerate(devices):
oob = base_port + idx
entry = {"device": dev}
if has_bw:
bw_extra = ["--report_gbits", "-s", str(msg_size),
"-q", str(queue_pairs)]
if duration_sec:
bw_extra += ["-D", str(duration_sec)]
else:
bw_extra += ["-n", str(iters)]
bw = self._cross_node_perftest(
"ib_write_bw", dev, server_target, server_addr, ib_port,
oob, gid_index, warmup,
extra=bw_extra,
parse="bw")
entry["bandwidth_gbps"] = bw
if isinstance(bw, (int, float)):
entry["bw_status"] = "PASS" if bw >= min_bw else "WARN"
else:
entry["bw_status"] = "FAIL"
if has_lat:
lat = self._cross_node_perftest(
"ib_write_lat", dev, server_target, server_addr, ib_port,
oob, gid_index, warmup, extra=[], parse="lat")
if isinstance(lat, dict):
entry["latency_us"] = lat.get("typical")
entry["latency_p99_us"] = lat.get("p99")
t = lat.get("typical")
entry["lat_status"] = ("PASS" if isinstance(t, (int, float)) and 0 < t <= max_lat
else ("WARN" if isinstance(t, (int, float)) else "FAIL"))
else:
entry["latency_us"] = lat
entry["lat_status"] = "FAIL"
per_device.append(entry)
statuses = [e.get(k) for e in per_device for k in ("bw_status", "lat_status") if e.get(k)]
verdict = "PASS"
for s in statuses:
if s == "FAIL":
verdict = "FAIL"
break
if s == "WARN" and verdict == "PASS":
verdict = "WARN"
return {
"status": verdict,
"server": server_addr,
"min_bandwidth_gbps": min_bw,
"max_latency_us": max_lat,
"per_device": per_device,
"timestamp": datetime.now().isoformat(),
}
def _cross_node_perftest(self, tool: str, dev: str, server_target: str,
server_addr: str, ib_port: int, oob_port: int,
gid_index, warmup: float, extra: List[str], parse: str):
"""Start `tool` server on the peer via SSH, run the local client, parse output.
Returns a float (bw, Gb/s), a dict {typical, p99} (lat, µs), or an error string.
"""
tool_path = self._find_tool(tool)
if not tool_path:
return f"{tool} not installed"
flags = ["-d", dev, "-i", str(ib_port), "-p", str(oob_port), "-F"]
if gid_index is not None:
flags += ["-x", str(gid_index)]
flags += extra
server_cmd = " ".join([tool] + flags) # server: no host argument
server_proc = None
try:
server_proc = subprocess.Popen(
["ssh", "-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=no",
server_target, server_cmd],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
time.sleep(warmup) # let the remote server bind before the client connects
client = subprocess.run([tool_path] + flags + [server_addr],
capture_output=True, text=True, timeout=120)
out = client.stdout + "\n" + (client.stderr or "")
return self._parse_perftest_lat(out) if parse == "lat" else self._parse_perftest_bw(out)
except subprocess.TimeoutExpired:
return "timeout"
except Exception as e: # noqa: BLE001
return f"error: {e}"
finally:
if server_proc and server_proc.poll() is None:
server_proc.terminate()
try:
server_proc.wait(timeout=5)
except Exception:
server_proc.kill()
# ib_write_* server normally exits after one run; pkill cleans up a
# leftover one if the client failed mid-handshake. -x matches the exact
# process name so it never kills this ssh command itself.
try:
subprocess.run(
["ssh", "-o", "BatchMode=yes", server_target, f"pkill -x {tool}"],
capture_output=True, timeout=10)
except Exception:
pass
@staticmethod
def _parse_perftest_bw(output: str) -> float:
"""Parse ib_write_bw rows (#bytes #iter BW_peak BW_avg ...); return max BW avg."""
best = 0.0
for line in output.splitlines():
parts = line.split()
if len(parts) >= 4:
try:
int(parts[0]) # #bytes column
best = max(best, float(parts[3])) # BW average[Gb/sec]
except ValueError:
continue
return round(best, 2) if best else 0.0
@staticmethod
def _parse_perftest_lat(output: str) -> dict:
"""Parse ib_write_lat row (#bytes #iter t_min t_max t_typical t_avg ... 99%)."""
for line in output.splitlines():
parts = line.split()
if len(parts) >= 6:
try:
int(parts[0]); int(parts[1])
typical = float(parts[4]) # t_typical[usec]
except ValueError:
continue
p99 = None
if len(parts) >= 8:
try:
p99 = float(parts[7]) # 99% percentile[usec]
except ValueError:
p99 = None
return {"typical": round(typical, 2), "p99": round(p99, 2) if p99 else None}
return {"typical": None, "p99": None}
@staticmethod @staticmethod
def print_results(results: dict, console: Console = None): def print_results(results: dict, console: Console = None):
c = console or Console() c = console or Console()
if results.get("skipped") or results.get("status") == "SKIP":
c.print(f"\n[bold yellow]RDMA/InfiniBand: SKIPPED[/bold yellow] "
f"[dim]({results.get('reason', 'no IB hardware')})[/dim]")
return
if "error" in results: if "error" in results:
c.print(f"[bold red]Error: {results['error']}[/bold red]") c.print(f"[bold red]Error: {results['error']}[/bold red]")
return return
@ -528,29 +238,3 @@ class RDMATest:
c.print(f" {t['test']}: [{sc}]{status}[/{sc}] " c.print(f" {t['test']}: [{sc}]{status}[/{sc}] "
f"({lat:.2f} us, max: {t.get('max_allowed_us', 'N/A')} us)" if status != "SKIP" f"({lat:.2f} us, max: {t.get('max_allowed_us', 'N/A')} us)" if status != "SKIP"
else f" {t['test']}: [dim]SKIPPED[/dim]") else f" {t['test']}: [dim]SKIPPED[/dim]")
cn = results.get("cross_node")
if cn:
if cn.get("skipped"):
c.print(f"\n [bold]Cross-node RDMA[/bold]: [dim]SKIPPED "
f"({cn.get('reason', '')})[/dim]")
else:
v = cn.get("status", "?")
vc = "green" if v == "PASS" else ("yellow" if v == "WARN" else "red")
c.print(f"\n [bold]Cross-node RDMA[/bold] (server {cn.get('server')}) "
f"[{vc}]{v}[/{vc}] "
f"[dim]min {cn.get('min_bandwidth_gbps')} Gb/s, "
f"max {cn.get('max_latency_us')} µs[/dim]")
for e in cn.get("per_device", []):
bw = e.get("bandwidth_gbps")
lat = e.get("latency_us")
bws = e.get("bw_status", "")
lts = e.get("lat_status", "")
bc = "green" if bws == "PASS" else ("yellow" if bws == "WARN" else "red")
lc = "green" if lts == "PASS" else ("yellow" if lts == "WARN" else "red")
bw_s = f"{bw:.1f} Gb/s" if isinstance(bw, (int, float)) else str(bw)
lat_s = f"{lat:.2f} µs" if isinstance(lat, (int, float)) else str(lat)
p99 = e.get("latency_p99_us")
p99_s = f", p99 {p99:.2f}" if isinstance(p99, (int, float)) else ""
c.print(f" {e['device']}: BW [{bc}]{bw_s}[/{bc}] | "
f"lat [{lc}]{lat_s}[/{lc}]{p99_s}")

View File

@ -2,7 +2,6 @@
import json import json
import os import os
import re
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
from typing import Optional from typing import Optional
@ -101,10 +100,6 @@ class ReportGenerator:
if "gpu_info" in results: if "gpu_info" in results:
gpus = results["gpu_info"].get("gpus", []) gpus = results["gpu_info"].get("gpus", [])
runtime_name = results["gpu_info"].get("runtime_name", "CUDA")
runtime_version = results["gpu_info"].get(
"runtime_version", results["gpu_info"].get("cuda_version", "N/A")
)
rows = "" rows = ""
for g in gpus: for g in gpus:
rows += f"<tr><td>GPU {g['index']}</td><td>{g['name']}</td>" rows += f"<tr><td>GPU {g['index']}</td><td>{g['name']}</td>"
@ -114,7 +109,7 @@ class ReportGenerator:
sections.append( sections.append(
f'<div class="section"><h2>GPU Information</h2>' f'<div class="section"><h2>GPU Information</h2>'
f'<p>Driver: {results["gpu_info"].get("driver_version", "N/A")} | ' f'<p>Driver: {results["gpu_info"].get("driver_version", "N/A")} | '
f'{runtime_name}: {runtime_version} | ' f'CUDA: {results["gpu_info"].get("cuda_version", "N/A")} | '
f'Count: {len(gpus)}</p>' f'Count: {len(gpus)}</p>'
f'<table><tr><th>GPU</th><th>Model</th><th>VRAM</th><th>Temp</th><th>SM Clock</th></tr>' f'<table><tr><th>GPU</th><th>Model</th><th>VRAM</th><th>Temp</th><th>SM Clock</th></tr>'
f'{rows}</table></div>' f'{rows}</table></div>'
@ -199,10 +194,8 @@ class ReportGenerator:
gpus = gpu_info.get("gpus", []) gpus = gpu_info.get("gpus", [])
gpu_name = gpus[0]["name"] if gpus else "Unknown" gpu_name = gpus[0]["name"] if gpus else "Unknown"
lines.append(f"- **GPU:** {gpu_name} x{gpu_info.get('gpu_count', len(gpus))}") lines.append(f"- **GPU:** {gpu_name} x{gpu_info.get('gpu_count', len(gpus))}")
runtime_name = gpu_info.get("runtime_name", "CUDA")
runtime_version = gpu_info.get("runtime_version", gpu_info.get("cuda_version", "N/A"))
lines.append(f"- **Driver:** {gpu_info.get('driver_version', 'N/A')} | " lines.append(f"- **Driver:** {gpu_info.get('driver_version', 'N/A')} | "
f"**{runtime_name}:** {runtime_version}") f"**CUDA:** {gpu_info.get('cuda_version', 'N/A')}")
lines.append("") lines.append("")
# --- Summary table --- # --- Summary table ---
@ -259,53 +252,29 @@ class ReportGenerator:
# --- Memory Bandwidth --- # --- Memory Bandwidth ---
mem_data = self._extract_memory_results(results) mem_data = self._extract_memory_results(results)
if mem_data and not mem_data.get("error"): if mem_data and not mem_data.get("error"):
lines.append("## GPU Interconnect and PCIe Bandwidth\n") lines.append("## Memory Bandwidth\n")
lines.append(f"Source: {mem_data.get('source', 'unknown')}\n") lines.append(f"Source: {mem_data.get('source', 'unknown')}\n")
if mem_data.get("scope_note"):
lines.append(f"> Scope: {mem_data['scope_note']}.\n")
lines.append("| Metric | Value | Peak | Efficiency |") lines.append("| Metric | Value | Peak | Efficiency |")
lines.append("|--------|-------|------|------------|") lines.append("|--------|-------|------|------------|")
d2d = mem_data.get("d2d_bandwidth_gbps") or 0 d2d = mem_data.get("d2d_bandwidth_gbps", 0)
h2d = mem_data.get("h2d_bandwidth_gbps") or 0 h2d = mem_data.get("h2d_bandwidth_gbps", 0)
d2h = mem_data.get("d2h_bandwidth_gbps") or 0 d2h = mem_data.get("d2h_bandwidth_gbps", 0)
# New format with per-metric peaks # New format with per-metric peaks
h2d_peak = mem_data.get("h2d_peak_gbps") or 0 h2d_peak = mem_data.get("h2d_peak_gbps", 0)
d2h_peak = mem_data.get("d2h_peak_gbps") or 0 d2h_peak = mem_data.get("d2h_peak_gbps", 0)
d2d_peak = mem_data.get("d2d_peak_gbps") or 0 d2d_peak = mem_data.get("d2d_peak_gbps", 0)
h2d_eff = mem_data.get("h2d_efficiency_pct") or 0 h2d_eff = mem_data.get("h2d_efficiency_pct", 0)
d2h_eff = mem_data.get("d2h_efficiency_pct") or 0 d2h_eff = mem_data.get("d2h_efficiency_pct", 0)
d2d_eff = mem_data.get("d2d_efficiency_pct") or 0 d2d_eff = mem_data.get("d2d_efficiency_pct", 0)
# Fallback for old format # Fallback for old format
if not d2d_peak: if not d2d_peak:
d2d_peak = mem_data.get("peak_bandwidth_gbps") or 0 d2d_peak = mem_data.get("peak_bandwidth_gbps", 0)
d2d_eff = mem_data.get("efficiency_pct") or 0 d2d_eff = mem_data.get("efficiency_pct", 0)
def peak_text(value): lines.append(f"| H2D (PCIe) | {h2d:.1f} GB/s | {h2d_peak:.0f} GB/s | {h2d_eff:.1f}% |")
return f"{value:.0f} GB/s" if value else "N/A" lines.append(f"| D2H (PCIe) | {d2h:.1f} GB/s | {d2h_peak:.0f} GB/s | {d2h_eff:.1f}% |")
lines.append(f"| D2D (NVLink) | {d2d:.1f} GB/s | {d2d_peak:.0f} GB/s | {d2d_eff:.1f}% |")
def efficiency_text(value):
return f"{value:.1f}%" if value else "N/A"
lines.append(f"| H2D (PCIe) | {h2d:.1f} GB/s | {peak_text(h2d_peak)} | {efficiency_text(h2d_eff)} |")
lines.append(f"| D2H (PCIe) | {d2h:.1f} GB/s | {peak_text(d2h_peak)} | {efficiency_text(d2h_eff)} |")
d2d_label = "D2D (HBM copy)" if mem_data.get("accelerator_vendor") == "hygon" else "D2D (NVLink, unidirectional)"
lines.append(f"| {d2d_label} | {d2d:.1f} GB/s | {peak_text(d2d_peak)} | {efficiency_text(d2d_eff)} |")
if mem_data.get("d2d_bidirectional_gbps"):
lines.append(
f"| D2D (NVLink, bidirectional informational) | "
f"{mem_data['d2d_bidirectional_gbps']:.1f} GB/s | N/A | N/A |"
)
lines.append("") lines.append("")
# PyTorch fallback can't accurately measure HBM peak (intra-GPU copy_() verdict = "PASS" if d2d_eff >= 50 else ("WARN" if d2d_eff >= 30 else "FAIL")
# only reaches ~20% of HBM bandwidth). When fallback is used, report
# the number but mark as WARN with a note instead of evaluating as FAIL.
if mem_data.get("source") == "pytorch":
lines.append(
f"**Verdict: WARN** (D2D {d2d:.1f} GB/s via PyTorch fallback; "
"nvbandwidth unavailable — figure is indicative only, not a true HBM peak)\n"
)
else:
# Tightened to match production acceptance: PASS >= 80%, WARN 60–80%, FAIL < 60%.
verdict = "PASS" if d2d_eff >= 80 else ("WARN" if d2d_eff >= 60 else "FAIL")
lines.append(f"**Verdict: {verdict}** (D2D efficiency {d2d_eff:.1f}%)\n") lines.append(f"**Verdict: {verdict}** (D2D efficiency {d2d_eff:.1f}%)\n")
# --- Compute Throughput --- # --- Compute Throughput ---
@ -315,71 +284,23 @@ class ReportGenerator:
per_dtype = comp_data.get("per_dtype_tflops", {}) per_dtype = comp_data.get("per_dtype_tflops", {})
peak_tflops = comp_data.get("peak_tflops", {}) peak_tflops = comp_data.get("peak_tflops", {})
eff_pct = comp_data.get("efficiency_pct", {}) eff_pct = comp_data.get("efficiency_pct", {})
# Absolute PASS thresholds (TFLOPS) from gpu_specs.compute_pass_thresholds_tflops. lines.append("| DType | Achieved (TFLOPS) | Peak | Efficiency | Status |")
# When present, override the legacy 80%-of-peak rule on a per-dtype basis. lines.append("|-------|-------------------|------|------------|--------|")
pass_thresholds = comp_data.get("pass_thresholds_tflops", {}) or {}
use_abs = bool(pass_thresholds)
has_peak_reference = any(
isinstance(value, (int, float)) and value > 0
for value in peak_tflops.values()
)
worst_values = comp_data.get("worst_per_dtype_tflops", {}) or {}
best_values = comp_data.get("best_per_dtype_tflops", {}) or {}
spreads = comp_data.get("spread_pct", {}) or {}
if use_abs:
lines.append("| DType | Avg | Min | Max | Spread | Dense Peak | Pass Floor | Status |")
else:
lines.append("| DType | Avg | Min | Max | Spread | Dense Peak | Efficiency | Status |")
lines.append("|-------|-----|-----|-----|--------|------------|------------|--------|")
worst_eff = 100.0 worst_eff = 100.0
overall_status = "PASS"
rank = {"PASS": 0, "MEASURED": 0, "WARN": 1, "FAIL": 2, "SKIP": 0}
for dt, val in per_dtype.items(): for dt, val in per_dtype.items():
if isinstance(val, str): if isinstance(val, str):
# skipped or error # skipped or error
lines.append(f"| {dt.upper()} | {val} | - | - | - | - | N/A | SKIP |") lines.append(f"| {dt.upper()} | {val} | - | N/A | SKIP |")
else: else:
pk = peak_tflops.get(dt, 0) pk = peak_tflops.get(dt, 0)
ef = eff_pct.get(dt, 0) ef = eff_pct.get(dt, 0)
worst = worst_values.get(dt, val)
best = best_values.get(dt, val)
spread = spreads.get(dt, 0)
if isinstance(ef, (int, float)) and ef > 0: if isinstance(ef, (int, float)) and ef > 0:
worst_eff = min(worst_eff, ef) worst_eff = min(worst_eff, ef)
thr = pass_thresholds.get(dt)
if use_abs and thr:
if worst >= thr:
status = "PASS"
elif worst >= thr * 0.9:
status = "WARN"
else:
status = "FAIL"
lines.append(
f"| {dt.upper()} | {val:.1f} | {worst:.1f} | {best:.1f} | "
f"{spread:.2f}% | {pk:.0f} | >= {thr} | {status} |"
)
elif has_peak_reference:
status = "PASS" if ef >= 80 else ("WARN" if ef >= 50 else "FAIL") status = "PASS" if ef >= 80 else ("WARN" if ef >= 50 else "FAIL")
lines.append( lines.append(f"| {dt.upper()} | {val:.1f} | {pk:.0f} | {ef:.1f}% | {status} |")
f"| {dt.upper()} | {val:.1f} | {worst:.1f} | {best:.1f} | "
f"{spread:.2f}% | {pk:.0f} | {ef:.1f}% | {status} |"
)
else:
status = "MEASURED"
lines.append(
f"| {dt.upper()} | {val:.1f} | {worst:.1f} | {best:.1f} | "
f"{spread:.2f}% | N/A | N/A | {status} |"
)
if rank.get(status, 0) > rank.get(overall_status, 0):
overall_status = status
lines.append("") lines.append("")
if use_abs: overall = "PASS" if worst_eff >= 80 else ("WARN" if worst_eff >= 50 else "FAIL")
lines.append(f"**Verdict: {overall_status}** (each dtype is judged by the slowest GPU against an achievable B300 floor)\n") lines.append(f"**Verdict: {overall}** (worst efficiency {worst_eff:.1f}%)\n")
elif has_peak_reference:
overall_status = "PASS" if worst_eff >= 80 else ("WARN" if worst_eff >= 50 else "FAIL")
lines.append(f"**Verdict: {overall_status}** (worst efficiency {worst_eff:.1f}%)\n")
else:
lines.append("**Verdict: MEASURED** (no published peak/acceptance threshold configured)\n")
# --- NCCL --- # --- NCCL ---
nccl = results.get("nccl") nccl = results.get("nccl")
@ -400,99 +321,25 @@ class ReportGenerator:
elif isinstance(data, dict) and data.get("error"): elif isinstance(data, dict) and data.get("error"):
lines.append(f"| {op} | - | - | ERROR: {data['error']} |") lines.append(f"| {op} | - | - | ERROR: {data['error']} |")
lines.append("") lines.append("")
nccl_status = nccl.get("status") or ("PASS" if nccl.get("passed") else "FAIL") passed = nccl.get("passed", False)
lines.append(f"**Overall: {nccl_status}**\n") lines.append(f"**Overall: {'PASS' if passed else 'FAIL'}**\n")
# --- Multi-node NCCL ---
multinode = results.get("multinode_nccl")
if multinode:
lines.append("## Multi-node NCCL\n")
lines.append(f"Source: {multinode.get('source', 'unknown')} | "
f"Mode: {multinode.get('mode', 'unknown')}\n")
hosts = multinode.get("hosts", []) or []
if hosts:
host_text = ", ".join(
f"{host.get('name', host.get('addr', '?'))} "
f"({host.get('addr', '?')}, {host.get('slots', '?')} GPUs)"
for host in hosts
)
lines.append(f"- **Hosts:** {host_text}")
lines.append(f"- **Overall: {'PASS' if multinode.get('passed') else 'FAIL'}**")
if multinode.get("artifact_dir"):
lines.append(f"- **Artifacts:** `{multinode['artifact_dir']}`")
lines.append("")
tests = multinode.get("tests", {}) or {}
if tests:
lines.append("| Operation | Topology | Peak Bus BW | Avg Bus BW | Peak Size | Threshold | Wrong | Network | GDR | Status |")
lines.append("|-----------|----------|-------------|------------|-----------|-----------|-------|---------|-----|--------|")
for op, test_data in tests.items():
for topo in test_data.get("topologies", []):
network = topo.get("network", {}) or {}
networks = ", ".join(network.get("networks", [])) or "N/A"
ib_using = " ".join(network.get("ib_using", []))
roce_hcas = sorted(set(re.findall(r"mlx5_bond_\d+", ib_using)))
if roce_hcas:
networks = f"{networks} ({len(roce_hcas)}x RoCE)"
threshold = topo.get("min_required_gbps", 0)
lines.append(
f"| {op} | {topo.get('label', '')} | "
f"{topo.get('peak_busbw_gbps', 0):.2f} GB/s | "
f"{topo.get('avg_busbw_gbps', 0):.2f} GB/s | "
f"{topo.get('peak_size', '')} | >= {threshold:g} GB/s | "
f"{topo.get('wrong_count', 0)} | {networks} | "
f"{network.get('gpu_direct_rdma', 'UNKNOWN')} | {topo.get('status', '?')} |"
)
lines.append("")
preflight = multinode.get("preflight", {}) or {}
checks = preflight.get("checks", []) or []
if checks:
lines.append("### Preflight\n")
lines.append("| Check | Status | Detail |")
lines.append("|-------|--------|--------|")
for check in checks:
detail = str(check.get("detail", "")).replace("|", "\\|").replace("\n", " ")
lines.append(f"| {check.get('name', '')} | {check.get('status', '?')} | {detail} |")
lines.append("")
# --- Stress Test --- # --- Stress Test ---
stress = results.get("stress") stress = results.get("stress")
if stress and not stress.get("error"): if stress and not stress.get("error"):
lines.append("## Stress Test\n") lines.append("## Stress Test\n")
passed = stress.get("passed", False) passed = stress.get("passed", False)
duration = stress.get("duration_sec") or 0 duration = stress.get("duration_sec", 0)
elapsed = stress.get("elapsed_sec") or 0 elapsed = stress.get("elapsed_sec", 0)
source = stress.get("source", "unknown") source = stress.get("source", "unknown")
lines.append(f"- **Source:** {source}") lines.append(f"- **Source:** {source}")
lines.append(f"- **Duration:** {elapsed:.0f}s (requested {duration}s)") lines.append(f"- **Duration:** {elapsed:.0f}s (requested {duration}s)")
lines.append(f"- **Result: {'PASS' if passed else 'FAIL'}**") lines.append(f"- **Result: {'PASS' if passed else 'FAIL'}**")
lines.append(f"- **Validation scope:** {stress.get('validation_scope', 'runtime completion')}")
if stress.get("numerical_validation") is False:
lines.append("- **Numerical result validation:** Not performed by PyTorch fallback")
ecc_delta = stress.get("uncorrected_ecc_delta", {}) or {}
if ecc_delta:
lines.append(f"- **Uncorrected ECC delta:** {sum(ecc_delta.values())}")
max_temp = stress.get("max_temperature_c", {}) or {}
max_power = stress.get("max_power_w", {}) or {}
if max_temp or max_power:
lines.append("")
lines.append("| GPU | Peak Temp | Peak Power |")
lines.append("|-----|-----------|------------|")
for gpu in sorted(set(max_temp) | set(max_power), key=lambda value: int(value)):
lines.append(
f"| {gpu} | {max_temp.get(gpu, 'N/A')} C | "
f"{max_power.get(gpu, 'N/A')} W |"
)
lines.append("") lines.append("")
# --- RDMA --- # --- RDMA ---
rdma = results.get("rdma") rdma = results.get("rdma")
if rdma and (rdma.get("skipped") or rdma.get("status") == "SKIP"): if rdma and not rdma.get("error"):
lines.append("## RDMA/InfiniBand\n")
lines.append(f"**Overall: SKIP** [{rdma.get('reason', 'no IB hardware detected')}]\n")
elif rdma and not rdma.get("error"):
lines.append("## RDMA/InfiniBand\n") lines.append("## RDMA/InfiniBand\n")
bw_tests = rdma.get("bandwidth_tests", []) bw_tests = rdma.get("bandwidth_tests", [])
lat_tests = rdma.get("latency_tests", []) lat_tests = rdma.get("latency_tests", [])
@ -523,10 +370,7 @@ class ReportGenerator:
lines.append(f"| Avg Step Time | {training.get('avg_step_time_ms', 0):.1f} ms |") lines.append(f"| Avg Step Time | {training.get('avg_step_time_ms', 0):.1f} ms |")
lines.append(f"| Peak Memory | {training.get('peak_memory_gb', 0):.1f} GB |") lines.append(f"| Peak Memory | {training.get('peak_memory_gb', 0):.1f} GB |")
lines.append(f"| Final Loss | {training.get('final_loss', 'N/A')} |") lines.append(f"| Final Loss | {training.get('final_loss', 'N/A')} |")
lines.append(f"| GPU Scope | {training.get('gpu_scope', 'framework-defined')} |")
lines.append(f"| SDPA Policy | {training.get('sdpa_backend_policy', 'framework_default')} |")
lines.append("") lines.append("")
lines.append("**Verdict: MEASURED** (synthetic workload; not an 8-GPU scaling acceptance test)\n")
# --- Footer --- # --- Footer ---
lines.append("---") lines.append("---")
@ -586,15 +430,11 @@ class ReportGenerator:
mem = self._extract_memory_results(results) mem = self._extract_memory_results(results)
if mem: if mem:
if mem.get("error"): if mem.get("error"):
items.append(("GPU Interconnect / PCIe", f"ERROR: {mem['error']}")) items.append(("Memory Bandwidth", f"ERROR: {mem['error']}"))
elif mem.get("source") == "pytorch":
# PyTorch fallback can't reach HBM peak — report as WARN, not FAIL.
d2d = mem.get("d2d_bandwidth_gbps") or 0
items.append(("GPU Interconnect / PCIe", f"WARN ({d2d:.0f} GB/s via PyTorch fallback)"))
else: else:
eff = mem.get("efficiency_pct") or 0 eff = mem.get("efficiency_pct", 0)
verdict = "PASS" if eff >= 80 else ("WARN" if eff >= 60 else "FAIL") verdict = "PASS" if eff >= 80 else ("WARN" if eff >= 50 else "FAIL")
items.append(("GPU Interconnect / PCIe", f"{verdict} ({eff:.1f}% unidirectional D2D)")) items.append(("Memory Bandwidth", f"{verdict} ({eff:.1f}%)"))
# Compute # Compute
comp = self._extract_compute_results(results) comp = self._extract_compute_results(results)
@ -602,66 +442,24 @@ class ReportGenerator:
if comp.get("error"): if comp.get("error"):
items.append(("Compute Throughput", f"ERROR: {comp['error']}")) items.append(("Compute Throughput", f"ERROR: {comp['error']}"))
else: else:
per_dtype = comp.get("per_dtype_tflops", {})
worst_per_dtype = comp.get("worst_per_dtype_tflops", {}) or {}
eff_pct = comp.get("efficiency_pct", {}) eff_pct = comp.get("efficiency_pct", {})
pass_thresholds = comp.get("pass_thresholds_tflops", {}) or {}
if pass_thresholds:
# Absolute TFLOPS judgment, mirroring the per-dtype table above.
rank = {"PASS": 0, "WARN": 1, "FAIL": 2}
worst_status = "PASS"
worst_dt = None
for dt, thr in pass_thresholds.items():
avg_val = per_dtype.get(dt)
val = worst_per_dtype.get(dt, avg_val)
if not isinstance(val, (int, float)):
continue
if val >= thr:
st = "PASS"
elif val >= thr * 0.9:
st = "WARN"
else:
st = "FAIL"
if rank[st] > rank[worst_status]:
worst_status = st
worst_dt = dt
if worst_dt:
items.append((
"Compute Throughput",
f"{worst_status} (worst {worst_dt.upper()} "
f"min {worst_per_dtype.get(worst_dt, per_dtype[worst_dt]):.0f} "
f"vs >= {pass_thresholds[worst_dt]})"
))
else:
items.append(("Compute Throughput", f"{worst_status}"))
else:
valid_effs = [v for v in eff_pct.values() if isinstance(v, (int, float)) and v > 0] valid_effs = [v for v in eff_pct.values() if isinstance(v, (int, float)) and v > 0]
if valid_effs: if valid_effs:
worst = min(valid_effs) worst = min(valid_effs)
verdict = "PASS" if worst >= 80 else ("WARN" if worst >= 50 else "FAIL") verdict = "PASS" if worst >= 80 else ("WARN" if worst >= 50 else "FAIL")
items.append(("Compute Throughput", f"{verdict} (worst {worst:.1f}%)")) items.append(("Compute Throughput", f"{verdict} (worst {worst:.1f}%)"))
else: else:
measured = [v for v in per_dtype.values() if isinstance(v, (int, float))] items.append(("Compute Throughput", "N/A"))
items.append(("Compute Throughput", "MEASURED (no published threshold)" if measured else "N/A"))
# NCCL # NCCL
if "nccl" in results: if "nccl" in results:
n = results["nccl"] n = results["nccl"]
if n.get("error"): if n.get("error"):
items.append(("NCCL", f"ERROR: {n['error']}")) items.append(("NCCL", f"ERROR: {n['error']}"))
else:
status = n.get("status") or ("PASS" if n.get("passed") else "FAIL")
items.append(("NCCL", status))
# Multi-node NCCL
if "multinode_nccl" in results:
n = results["multinode_nccl"]
if n.get("error"):
items.append(("Multi-node NCCL", f"ERROR: {n['error']}"))
elif n.get("passed"): elif n.get("passed"):
items.append(("Multi-node NCCL", "PASS")) items.append(("NCCL", "PASS"))
else: else:
items.append(("Multi-node NCCL", "FAIL")) items.append(("NCCL", "FAIL"))
# Stress # Stress
if "stress" in results: if "stress" in results:
@ -676,9 +474,7 @@ class ReportGenerator:
# RDMA # RDMA
if "rdma" in results: if "rdma" in results:
r = results["rdma"] r = results["rdma"]
if r.get("skipped") or r.get("status") == "SKIP": if r.get("error"):
items.append(("RDMA", f"SKIP ({r.get('reason', 'no IB hardware')})"))
elif r.get("error"):
items.append(("RDMA", f"ERROR: {r['error']}")) items.append(("RDMA", f"ERROR: {r['error']}"))
elif r.get("passed"): elif r.get("passed"):
items.append(("RDMA", "PASS")) items.append(("RDMA", "PASS"))
@ -692,7 +488,6 @@ class ReportGenerator:
items.append(("Training", f"ERROR: {t['error']}")) items.append(("Training", f"ERROR: {t['error']}"))
else: else:
tps = t.get("throughput_tokens_per_sec", 0) tps = t.get("throughput_tokens_per_sec", 0)
scope = t.get("gpu_scope", "framework-defined") items.append(("Training", f"PASS ({tps:.0f} tokens/sec)"))
items.append(("Training", f"MEASURED ({tps:.0f} tokens/sec; {scope})"))
return items return items

View File

@ -2,7 +2,6 @@
import glob import glob
import os import os
import re
import shutil import shutil
import subprocess import subprocess
import time import time
@ -14,7 +13,6 @@ from rich.live import Live
from rich.text import Text from rich.text import Text
from modules.gpu_specs import resolve_tools_dir from modules.gpu_specs import resolve_tools_dir
from modules.hygon_smi import run_hy_smi
class StressTest: class StressTest:
@ -58,12 +56,12 @@ class StressTest:
if not result.get("passed") and result.get("elapsed_sec", 0) < duration_sec * 0.5: if not result.get("passed") and result.get("elapsed_sec", 0) < duration_sec * 0.5:
self.console.print("\n[yellow]gpu-burn exited early (possible OOM), switching to PyTorch stress test[/yellow]") self.console.print("\n[yellow]gpu-burn exited early (possible OOM), switching to PyTorch stress test[/yellow]")
self.console.print("[dim]PyTorch mode dynamically adapts to available memory[/dim]\n") self.console.print("[dim]PyTorch mode dynamically adapts to available memory[/dim]\n")
return self._run_pytorch_stress(duration_sec, memory_pct, use_tensor_cores) return self._run_pytorch_stress(duration_sec, memory_pct)
return result return result
self.console.print("[yellow]gpu_burn not found, using PyTorch stress test[/yellow]") self.console.print("[yellow]gpu_burn not found, using PyTorch stress test[/yellow]")
return self._run_pytorch_stress(duration_sec, memory_pct, use_tensor_cores) return self._run_pytorch_stress(duration_sec, memory_pct)
def _run_gpu_burn(self, gpu_burn: str, duration: int, def _run_gpu_burn(self, gpu_burn: str, duration: int,
doubles: bool, tensor_cores: bool, target_gpus: str) -> dict: doubles: bool, tensor_cores: bool, target_gpus: str) -> dict:
@ -118,50 +116,9 @@ class StressTest:
"timestamp": datetime.now().isoformat(), "timestamp": datetime.now().isoformat(),
} }
@staticmethod def _run_pytorch_stress(self, duration: int, memory_pct: int = 90) -> dict:
def _accelerator_snapshot() -> dict:
if shutil.which("nvidia-smi"):
try:
r = subprocess.run(
["nvidia-smi", "--query-gpu=index,temperature.gpu,power.draw,"
"ecc.errors.uncorrected.volatile.total", "--format=csv,noheader,nounits"],
capture_output=True, text=True, timeout=15,
)
temps, powers, ecc = {}, {}, {}
if r.returncode == 0:
for line in r.stdout.splitlines():
parts = [part.strip() for part in line.split(",")]
if len(parts) < 4:
continue
idx = int(parts[0])
temps[idx] = float(parts[1])
powers[idx] = float(parts[2])
ecc[idx] = int(parts[3]) if parts[3].isdigit() else 0
return {
"temperatures_c": temps, "power_w": powers,
"uncorrected_ecc": ecc,
}
except (OSError, ValueError, subprocess.SubprocessError):
pass
output = run_hy_smi()
temps, powers = {}, {}
if output:
pattern = re.compile(r"^\s*(\d+)\s+([\d.]+)C\s+([\d.]+)W", re.MULTILINE)
for match in pattern.finditer(output):
idx = int(match.group(1))
temps[idx] = float(match.group(2))
powers[idx] = float(match.group(3))
return {"temperatures_c": temps, "power_w": powers, "uncorrected_ecc": {}}
def _run_pytorch_stress(self, duration: int, memory_pct: int = 90,
tensor_cores: bool = True) -> dict:
try: try:
import torch import torch
try:
import torch_dcu # noqa: F401
except ImportError:
pass
if not torch.cuda.is_available(): if not torch.cuda.is_available():
return {"error": "pytorch_not_available"} return {"error": "pytorch_not_available"}
except ImportError: except ImportError:
@ -172,10 +129,9 @@ class StressTest:
gpu_status = {} gpu_status = {}
t0 = time.time() t0 = time.time()
snapshots = [self._accelerator_snapshot()]
try: try:
work_a, work_b, outputs, ballast = {}, {}, {}, {} tensors = {}
for i in range(gpu_count): for i in range(gpu_count):
with torch.cuda.device(i): with torch.cuda.device(i):
# Get actual free memory (accounting for other processes) # Get actual free memory (accounting for other processes)
@ -187,23 +143,10 @@ class StressTest:
# Cap at actual free memory with 5% safety margin # Cap at actual free memory with 5% safety margin
alloc_bytes = min(target_mem, int(free_mem * 0.95)) alloc_bytes = min(target_mem, int(free_mem * 0.95))
dtype = torch.bfloat16 if tensor_cores else torch.float32 # matmul(A, A.T) needs 2x input memory (input + output)
elem_size = torch.tensor([], dtype=dtype).element_size() side = int((alloc_bytes / 4 / 2) ** 0.5) # float32 = 4 bytes
mem_side = int((alloc_bytes / elem_size / 3) ** 0.5)
max_compute_side = 8192 if tensor_cores else 4096
side = min(mem_side, max_compute_side)
compute_bytes = side * side * elem_size * 3
ballast_target = max(0, alloc_bytes - compute_bytes - 1024**3)
ballast[i] = []
chunk_size = 1024**3
while ballast_target > 0:
this_chunk = min(chunk_size, ballast_target)
tensor = torch.empty(this_chunk, device=f"cuda:{i}", dtype=torch.uint8)
tensor.fill_(0x5A)
ballast[i].append(tensor)
ballast_target -= this_chunk
actual_mem_mb = (alloc_bytes - max(0, ballast_target)) / 1024 / 1024 actual_mem_mb = side * side * 4 / 1024 / 1024
total_mem_mb = total_mem / 1024 / 1024 total_mem_mb = total_mem / 1024 / 1024
free_mem_mb = free_mem / 1024 / 1024 free_mem_mb = free_mem / 1024 / 1024
@ -212,29 +155,22 @@ class StressTest:
f"alloc {actual_mem_mb:.0f}MB ({actual_mem_mb/total_mem_mb*100:.0f}%) - " f"alloc {actual_mem_mb:.0f}MB ({actual_mem_mb/total_mem_mb*100:.0f}%) - "
f"matrix {side}x{side}[/dim]" f"matrix {side}x{side}[/dim]"
) )
work_a[i] = torch.randn(side, side, device=f"cuda:{i}", dtype=dtype) tensors[i] = torch.randn(side, side, device=f"cuda:{i}", dtype=torch.float32)
work_b[i] = torch.randn(side, side, device=f"cuda:{i}", dtype=dtype)
outputs[i] = torch.empty(side, side, device=f"cuda:{i}", dtype=dtype)
self.console.print(f"\n[cyan]Starting stress test for {duration} seconds...[/cyan]") self.console.print(f"\n[cyan]Starting stress test for {duration} seconds...[/cyan]")
elapsed_check = 0 elapsed_check = 0
while time.time() - t0 < duration: while time.time() - t0 < duration:
# Dispatch matmul on all GPUs in parallel — do NOT synchronize between
# GPUs, otherwise the 8 GPUs run serially and overshoot the duration.
for i in range(gpu_count):
with torch.cuda.device(i):
outputs[i] = torch.matmul(work_a[i], work_b[i])
# Single sync per pass — waits for all 8 streams concurrently
for i in range(gpu_count): for i in range(gpu_count):
with torch.cuda.device(i): with torch.cuda.device(i):
tensors[i] = torch.matmul(tensors[i], tensors[i].T)
torch.cuda.synchronize() torch.cuda.synchronize()
time.sleep(0.1)
# Show progress every 10 seconds # Show progress every 10 seconds
current_elapsed = time.time() - t0 current_elapsed = time.time() - t0
if int(current_elapsed) != int(elapsed_check) and int(current_elapsed) % 10 == 0: if int(current_elapsed) != int(elapsed_check) and int(current_elapsed) % 10 == 0:
self.console.print(f" [dim]Running {int(current_elapsed)}s / {duration}s[/dim]") self.console.print(f" [dim]Running {int(current_elapsed)}s / {duration}s[/dim]")
snapshots.append(self._accelerator_snapshot())
elapsed_check = current_elapsed elapsed_check = current_elapsed
for i in range(gpu_count): for i in range(gpu_count):
@ -254,42 +190,16 @@ class StressTest:
"gpu_status": gpu_status, "gpu_status": gpu_status,
} }
finally: finally:
for mapping in (work_a, work_b, outputs, ballast): tensors.clear()
mapping.clear()
torch.cuda.empty_cache() torch.cuda.empty_cache()
elapsed = round(time.time() - t0, 1) elapsed = round(time.time() - t0, 1)
snapshots.append(self._accelerator_snapshot())
max_temp = {}
max_power = {}
for snapshot in snapshots:
for idx, value in snapshot.get("temperatures_c", {}).items():
max_temp[idx] = max(max_temp.get(idx, value), value)
for idx, value in snapshot.get("power_w", {}).items():
max_power[idx] = max(max_power.get(idx, value), value)
first_ecc = snapshots[0].get("uncorrected_ecc", {}) if snapshots else {}
last_ecc = snapshots[-1].get("uncorrected_ecc", {}) if snapshots else {}
ecc_delta = {
idx: max(0, last_ecc.get(idx, 0) - first_ecc.get(idx, 0))
for idx in set(first_ecc) | set(last_ecc)
}
passed = not any(ecc_delta.values())
if not passed:
for idx, delta in ecc_delta.items():
if delta:
gpu_status[idx] = "FAIL"
return { return {
"source": "pytorch", "source": "pytorch",
"passed": passed, "passed": True,
"duration_sec": duration, "duration_sec": duration,
"elapsed_sec": elapsed, "elapsed_sec": elapsed,
"gpu_status": gpu_status, "gpu_status": gpu_status,
"memory_pct": memory_pct,
"max_temperature_c": max_temp,
"max_power_w": max_power,
"uncorrected_ecc_delta": ecc_delta,
"numerical_validation": False,
"validation_scope": "runtime stability, allocation, temperature, power, and ECC delta",
"timestamp": datetime.now().isoformat(), "timestamp": datetime.now().isoformat(),
} }
@ -325,15 +235,5 @@ class StressTest:
color = "green" if status == "PASS" else "red" color = "green" if status == "PASS" else "red"
c.print(f" GPU {gid}: [{color}]{status}[/{color}]") c.print(f" GPU {gid}: [{color}]{status}[/{color}]")
max_temp = results.get("max_temperature_c", {})
max_power = results.get("max_power_w", {})
if max_temp or max_power:
c.print("\n Peak telemetry:")
for gid in sorted(set(max_temp) | set(max_power)):
c.print(
f" GPU {gid}: {max_temp.get(gid, 'N/A')} C, "
f"{max_power.get(gid, 'N/A')} W"
)
if results.get("error"): if results.get("error"):
c.print(f" [red]Error: {results['error']}[/red]") c.print(f" [red]Error: {results['error']}[/red]")

View File

@ -3,7 +3,6 @@
import time import time
import subprocess import subprocess
import shutil import shutil
from contextlib import nullcontext
from datetime import datetime from datetime import datetime
from typing import Optional from typing import Optional
@ -14,10 +13,6 @@ from rich.progress import Progress, SpinnerColumn, BarColumn, TextColumn, TimeEl
TORCH_AVAILABLE = False TORCH_AVAILABLE = False
try: try:
import torch import torch
try:
import torch_dcu # noqa: F401 - enables DTK's PyTorch device shim
except ImportError:
pass
if torch.cuda.is_available(): if torch.cuda.is_available():
TORCH_AVAILABLE = True TORCH_AVAILABLE = True
except ImportError: except ImportError:
@ -31,73 +26,6 @@ class TrainingSim:
self.console = Console() self.console = Console()
self.train_cfg = config.get("training", {}) self.train_cfg = config.get("training", {})
@staticmethod
def _requires_b300_sdpa_workaround(
configured_type: str, capability, cudnn_version: int | None,
) -> bool:
is_b300 = str(configured_type).lower() == "b300" or capability == (10, 3)
# cuDNN 9.20 restores SM103 plan generation and PyTorch 2.13 correctly
# rejects unsupported cuDNN shapes (for example dropout=0.1) before
# dispatch, allowing another SDPA backend to run. Keep the workaround
# only for the affected older stack.
return is_b300 and (not cudnn_version or cudnn_version < 92000)
def _sdpa_context(self):
"""Select a reliable SDPA backend policy for B300/SM103.
PyTorch 2.11 + cuDNN 9.19 can dispatch BF16 attention on B300 to the
cuDNN frontend even though that backend cannot build an execution plan.
Keep the workaround scoped to the training operation and retain ordered
Flash/Efficient/Math fallbacks instead of changing global process state.
"""
configured_type = str(self.config.get("gpu_type", "")).lower()
capability = None
cudnn_version = None
if torch.cuda.is_available():
try:
capability = tuple(torch.cuda.get_device_capability(0))
except Exception: # noqa: BLE001 - backend discovery must not abort training
capability = None
try:
cudnn_version = torch.backends.cudnn.version()
except Exception: # noqa: BLE001 - backend discovery must not abort training
cudnn_version = None
requires_workaround = self._requires_b300_sdpa_workaround(
configured_type, capability, cudnn_version,
)
if not requires_workaround:
if (configured_type == "b300" or capability == (10, 3)) and cudnn_version:
return nullcontext(), (
f"framework_default (cuDNN {cudnn_version}; "
"unsupported SDPA shapes may fall back automatically)"
)
return nullcontext(), "framework_default"
try:
from torch.nn.attention import SDPBackend, sdpa_kernel
context = sdpa_kernel(
[
SDPBackend.FLASH_ATTENTION,
SDPBackend.EFFICIENT_ATTENTION,
SDPBackend.MATH,
],
set_priority=True,
)
return context, "flash>efficient>math (cuDNN SDPA excluded on B300)"
except (ImportError, AttributeError, TypeError):
# Compatibility path for older PyTorch versions that do not expose
# the context-manager API. Restore the global flag when leaving.
class _DisableCudnnSDPA:
def __enter__(self_inner):
self_inner.old = torch.backends.cuda.cudnn_sdp_enabled()
torch.backends.cuda.enable_cudnn_sdp(False)
def __exit__(self_inner, exc_type, exc, traceback):
torch.backends.cuda.enable_cudnn_sdp(self_inner.old)
return _DisableCudnnSDPA(), "framework fallback (cuDNN SDPA disabled on B300)"
def run(self) -> dict: def run(self) -> dict:
if not TORCH_AVAILABLE: if not TORCH_AVAILABLE:
self.console.print("[yellow]PyTorch not available - skipping training simulation[/yellow]") self.console.print("[yellow]PyTorch not available - skipping training simulation[/yellow]")
@ -121,9 +49,6 @@ class TrainingSim:
self.console.print(f" Model: {model_name} | Batch: {batch_size} | Seq: {seq_length} | " self.console.print(f" Model: {model_name} | Batch: {batch_size} | Seq: {seq_length} | "
f"DType: {dtype_str} | Steps: {num_steps} | GPUs: {gpu_count}") f"DType: {dtype_str} | Steps: {num_steps} | GPUs: {gpu_count}")
if model_name == "synthetic":
return self._run_synthetic(gpu_count, batch_size, seq_length, num_steps, dtype)
try: try:
from transformers import AutoModelForCausalLM, AutoTokenizer from transformers import AutoModelForCausalLM, AutoTokenizer
except ImportError: except ImportError:
@ -157,8 +82,6 @@ class TrainingSim:
step_times = [] step_times = []
mem_usage = [] mem_usage = []
sdpa_context, sdpa_policy = self._sdpa_context()
self.console.print(f" SDPA policy: {sdpa_policy}")
with Progress( with Progress(
SpinnerColumn(), TextColumn("[progress.description]{task.description}"), SpinnerColumn(), TextColumn("[progress.description]{task.description}"),
BarColumn(), TextColumn("{task.completed}/{task.total}"), BarColumn(), TextColumn("{task.completed}/{task.total}"),
@ -166,7 +89,6 @@ class TrainingSim:
) as progress: ) as progress:
task = progress.add_task("Training steps...", total=num_steps) task = progress.add_task("Training steps...", total=num_steps)
with sdpa_context:
for step in range(num_steps): for step in range(num_steps):
torch.cuda.synchronize() torch.cuda.synchronize()
t0 = time.perf_counter() t0 = time.perf_counter()
@ -213,7 +135,6 @@ class TrainingSim:
"throughput_samples_per_sec": round(batch_size / avg_step_time, 2), "throughput_samples_per_sec": round(batch_size / avg_step_time, 2),
"peak_memory_gb": round(max(mem_usage) if mem_usage else 0, 2), "peak_memory_gb": round(max(mem_usage) if mem_usage else 0, 2),
"final_loss": round(loss.item(), 4) if hasattr(loss, 'item') else None, "final_loss": round(loss.item(), 4) if hasattr(loss, 'item') else None,
"sdpa_backend_policy": sdpa_policy,
"timestamp": datetime.now().isoformat(), "timestamp": datetime.now().isoformat(),
} }
@ -224,10 +145,10 @@ class TrainingSim:
def _run_synthetic(self, gpu_count, batch_size, seq_length, num_steps, dtype) -> dict: def _run_synthetic(self, gpu_count, batch_size, seq_length, num_steps, dtype) -> dict:
self.console.print(" Running synthetic training benchmark...") self.console.print(" Running synthetic training benchmark...")
hidden_size = int(self.train_cfg.get("hidden_size", 4096)) hidden_size = 4096
num_layers = int(self.train_cfg.get("num_layers", 6)) num_layers = 6
num_heads = int(self.train_cfg.get("num_heads", 32)) num_heads = 32
vocab_size = int(self.train_cfg.get("vocab_size", 32000)) vocab_size = 32000
class SyntheticTransformer(torch.nn.Module): class SyntheticTransformer(torch.nn.Module):
def __init__(self): def __init__(self):
@ -262,8 +183,6 @@ class TrainingSim:
step_times = [] step_times = []
mem_usage = [] mem_usage = []
sdpa_context, sdpa_policy = self._sdpa_context()
self.console.print(f" SDPA policy: {sdpa_policy}")
with Progress( with Progress(
SpinnerColumn(), TextColumn("[progress.description]{task.description}"), SpinnerColumn(), TextColumn("[progress.description]{task.description}"),
BarColumn(), TextColumn("{task.completed}/{task.total}"), BarColumn(), TextColumn("{task.completed}/{task.total}"),
@ -271,7 +190,6 @@ class TrainingSim:
) as progress: ) as progress:
task = progress.add_task("Synthetic training...", total=num_steps) task = progress.add_task("Synthetic training...", total=num_steps)
with sdpa_context:
for step in range(num_steps): for step in range(num_steps):
torch.cuda.synchronize() torch.cuda.synchronize()
t0 = time.perf_counter() t0 = time.perf_counter()
@ -303,9 +221,6 @@ class TrainingSim:
"num_layers": num_layers, "num_layers": num_layers,
"hidden_size": hidden_size, "hidden_size": hidden_size,
"gpu_count": gpu_count, "gpu_count": gpu_count,
"gpu_scope": "single_gpu:0",
"performance_verdict": "MEASURED",
"sdpa_backend_policy": sdpa_policy,
"dtype": str(dtype).replace("torch.", ""), "dtype": str(dtype).replace("torch.", ""),
"batch_size": batch_size, "batch_size": batch_size,
"seq_length": seq_length, "seq_length": seq_length,

View File

@ -17,12 +17,10 @@ dependencies = [
# uv sync --extra torch-cu121 # uv sync --extra torch-cu121
# uv sync --extra torch-cu124 (recommended for CUDA 12.x + Driver 535+) # uv sync --extra torch-cu124 (recommended for CUDA 12.x + Driver 535+)
# uv sync --extra torch-cu128 (for CUDA 12.8+ / Driver 570+) # uv sync --extra torch-cu128 (for CUDA 12.8+ / Driver 570+)
# uv sync --extra torch-cu130 (recommended for B300 / Driver 580+)
torch-cu118 = ["torch>=2.1.0"] torch-cu118 = ["torch>=2.1.0"]
torch-cu121 = ["torch>=2.1.0"] torch-cu121 = ["torch>=2.1.0"]
torch-cu124 = ["torch>=2.1.0"] torch-cu124 = ["torch>=2.1.0"]
torch-cu128 = ["torch>=2.1.0"] torch-cu128 = ["torch>=2.1.0"]
torch-cu130 = ["torch>=2.13.0"]
[build-system] [build-system]
requires = ["hatchling"] requires = ["hatchling"]
@ -49,11 +47,6 @@ name = "pytorch-cu128"
url = "https://download.pytorch.org/whl/cu128" url = "https://download.pytorch.org/whl/cu128"
explicit = true explicit = true
[[tool.uv.index]]
name = "pytorch-cu130"
url = "https://download.pytorch.org/whl/cu130"
explicit = true
# Default: resolve torch from cu124 index (most broadly compatible with 535+ drivers) # Default: resolve torch from cu124 index (most broadly compatible with 535+ drivers)
# To use a different CUDA version, override: # To use a different CUDA version, override:
# uv pip install torch --index-url https://download.pytorch.org/whl/cu128 # uv pip install torch --index-url https://download.pytorch.org/whl/cu128

View File

@ -1,46 +0,0 @@
#!/usr/bin/env python3
"""Minimal DTK/PyTorch smoke test used before the full BW1000 suite."""
import json
import torch
try:
import torch_dcu # noqa: F401
except ImportError:
pass
def main() -> None:
count = torch.cuda.device_count()
if not torch.cuda.is_available() or count < 1:
raise SystemExit("DTK PyTorch reports no available accelerator")
devices = []
for index in range(count):
props = torch.cuda.get_device_properties(index)
devices.append({
"index": index,
"name": torch.cuda.get_device_name(index),
"memory_gib": round(props.total_memory / 1024**3, 2),
})
a = torch.randn(2048, 2048, device="cuda:0", dtype=torch.bfloat16)
b = torch.randn(2048, 2048, device="cuda:0", dtype=torch.bfloat16)
c = a @ b
torch.cuda.synchronize()
if not torch.isfinite(c).all().item():
raise SystemExit("BF16 matmul produced non-finite output")
print(json.dumps({
"torch_version": torch.__version__,
"cuda_api_available": torch.cuda.is_available(),
"device_count": count,
"devices": devices,
"bf16_matmul": "PASS",
"checksum": round(float(c.float().mean().item()), 6),
}, indent=2))
if __name__ == "__main__":
main()

View File

@ -1,5 +0,0 @@
/gpu_node_snapshot*.json
/gpu_node_profile*.html
/gpu_topology_3d*.html
/b300_*_live.json
/b300_*_live.html

View File

@ -1,155 +0,0 @@
# GPU Node Profile
`gpu_node_profile.py` 通过只读命令采集 GPU、NVLink、PCIe/NUMA、RoCE
Bond、RDMA 和 InfiniBand 信息,生成通用结构化 JSON,并使用 `templates/`
中的两个独立 HTML 源模板,从同一份 JSON 渲染完全离线、无 Mermaid、
无 CDN 依赖的运维画像与 3D 拓扑 CT。
页面顶部按“系统与软件 / GPU / CPU与内存”展示整机概览;中部的
GPU-RDMA 拓扑支持“总览 / NUMA 分览”即时切换,并继续展示 InfiniBand、
OS 可见的带内管理网卡和可读取的 BMC 带外信息;最底部横向展示健康检查
与验证结论。
3D CT 页面使用原生 Canvas 对结构化 JSON 动态布局,包含 NUMA 半透明地域、
CPU Socket、PCIe Root Complex、PCIe Switch 分组、GPU、NVSwitch、RoCE、
InfiniBand 和管理网卡。NUMA 与 PCIe 均来自数据,不包含 B300 专用坐标,
因此同一渲染器也可用于 A800 等其他 GPU 服务器。
页面提供递归内部镜头,可按“整机 → NUMA → PCIe 分组 → Bond → 物理网卡”
逐层钻取。进入任一层后会隐藏外层与无关的同级设备,只保留当前壳体和后代
节点;顶部面包屑说明当前位置,“上一层”按原路径返回。双击壳体继续深入,
双击普通设备时也会沿该设备的父级路径每次深入一层,例如连续双击 GPU 会
依次进入整机、NUMA 和对应 PCIe 分组;到达最深层后才执行设备聚焦。
`Shift + 拖动`可平移,`Esc`返回整机外观。
Canvas 标签使用高对比浅色底与黑色粗体。每个 RoCE Bond 会作为透明聚合
容器展示,并在内部继续拆出物理成员网卡、速率和 BDF。NVLink/NVSwitch
连线在设备盒子上层高亮绘制,GPU 端点带编号;该 Fabric 只连接 GPU,
不会连接网卡。
网络模型同时支持两种形态:存在 Bond 时渲染
`GPU → RoCE Bond → 物理成员端口`;不存在 Bond 时根据
`nvidia-smi topo -m`、RDMA 名称和 PCIe 路径渲染
`GPU → 独立 RoCE/InfiniBand 端点`。多张 GPU 与多个等距端点会进行
全局最优、确定性的唯一分配,避免顺序贪心导致所有 GPU 错误指向同一张
网卡;端点少于 GPU 时才保留可共享的最近路径语义。亲和关系只接受
`nvidia-smi topo -m` 暴露的 `PIX/PXB/PHB/NODE/SYS` 证据;若独立 IB
未出现在矩阵中,会保留设备和 PCIe/NUMA 信息并标记 `not-exposed`,但不会
推测一条 GPU 连线。
JSON 中的 `network_endpoints` 是两张页面共用的标准网络模型:
```text
network_endpoints[]
id / type(bond|standalone) / protocol(RoCE|InfiniBand)
rdma / numa / bdf / state / topology_visibility
members[] # Bond 为多个物理口,直连端点为一个物理口
gpu_links[] # GPU、PIX/PXB/...、用途及证据来源
gpus[].primary_network_id # 指向主要计算 RDMA 端点
gpus[].network_links[] # 保留主要端点和额外最近端点
```
CPU 与内存概览、NUMA 标尺和每张 GPU 卡片会同时展示 `nvidia-smi topo -m`
解析出的推荐 CPU 范围及内存 NUMA。该信息描述硬件拓扑的近端路径,不表示
训练或推理进程已经通过 `taskset`、cgroup、Slurm 或 Kubernetes 完成实际绑核。
GPU 概览和单卡画像会展示采集时刻的 `power.draw`、当前 `power.limit` 与
P-State。Power Limit 表示当前生效的 GPU 功耗上限,不等于整机额定功耗;
P0 表示最高性能状态,也不单独等价于 GPU 持续满载。
## 在 GPU 节点执行
```bash
python3 gpu_node_profile.py \
--output gpu_node_profile.html \
--topology-output gpu_topology_3d.html \
--snapshot-out gpu_node_snapshot.json \
--expected-gpus 8 \
--expected-bonds 8 \
--expected-ib 4
```
脚本不会修改网卡、驱动、Bond 或系统配置。唯一写操作是生成用户指定的
HTML/JSON 文件。`--expected-bonds` 校验的是 RDMA/RoCE Bond;普通管理
Bond 会保留在原始网卡信息中,但不会计入加速网络端点数量。
三个产物的关系:
```text
只读采集 → gpu_node_snapshot.json
├─→ gpu_node_profile.html
└─→ gpu_topology_3d.html
```
源码与演示数据的关系:
```text
gpu_node_profile.py # 只读采集、校验和渲染入口
templates/gpu_node_profile.html # 运维画像源模板
templates/gpu_topology_3d.html # 3D CT 源模板
examples/b300_demo.json # 由真机快照脱敏得到的演示数据
examples/b300_profile_demo.html # demo.json 生成的自包含示例
examples/b300_topology_3d_demo.html
```
最终生成的两张 HTML 会把 JSON 嵌入页面,因此可以直接双击、复制到其他机器
或离线归档。它们不会在打开时读取旁边的 `demo.json`;修改或替换 JSON 后,
需要重新运行 Python 渲染:
```bash
python3 gpu_node_profile.py \
--snapshot-in examples/b300_demo.json \
--output examples/b300_profile_demo.html \
--topology-output examples/b300_topology_3d_demo.html \
--expected-gpus 8 \
--expected-bonds 8 \
--expected-ib 4
```
## 使用已有快照离线渲染
```bash
python3 gpu_node_profile.py \
--snapshot-in examples/b300_demo.json \
--output b300_profile_demo.html \
--topology-output b300_topology_3d_demo.html \
--expected-gpus 8 \
--expected-bonds 8 \
--expected-ib 4
```
只采集 JSON 并写到标准输出,适合通过 SSH 在远端只读采集、在本地渲染:
```bash
python3 gpu_node_profile.py --snapshot-stdout
```
## 测试
```bash
python3 -m unittest discover -s tests -v
```
## 采集边界
- `nvcc` 表示当前 PATH 指向的 CUDA Toolkit,不代表容器或应用实际加载的
CUDA Runtime。
- LACP Bond 展示的是成员链路聚合容量,不承诺单连接可达到聚合带宽。
- 虚拟机中的 PCIe/NUMA 数据表示 Guest 可见拓扑,不能代替宿主机物理布线
和交换机侧验证。
- `pci_fabric` 保存 Guest 可见的 PCIe 祖先路径、桥和端点。3D 页面中的空间
坐标由这些逻辑关系动态生成,不代表设备在机箱中的厘米级真实位置。
- `PIX/PXB/PHB/NODE/SYS` 表示拓扑距离。`PIX` 可作为 GPUDirect RDMA 的
良好候选路径,但只有完成驱动能力和实际 RDMA 传输验证后才能标记为
`GDR Verified`。
- CPU 亲和表示 NVIDIA 拓扑给出的推荐 CPU/内存范围,不是运行中进程的
`Cpus_allowed_list`。评估实际 H2D 性能还需结合进程绑核、内存落点、
pinned memory 和带宽实测。
- GPU 功耗为报告生成时的一次瞬时采样;周期性训练负载的平均功耗和峰值
需要连续采样。整机功耗还需结合 BMC/DCMI 或机柜 PDU,不能简单等同于
所有 GPU Power Limit 之和。
- 独立BMC口通常不进入宿主OS网络命名空间。脚本会尝试从常见LAN通道只读
执行 `ipmitool lan print`;读取不到时会在画像中明确标记“OS不可见”。
- 管理网络区域只渲染物理 PCI 网卡、Bond 或默认路由接口;CNI、veth 等
容器虚拟接口仍可保留在原始 JSON 的 `nics` 中,但不会污染整机画像。

File diff suppressed because it is too large Load Diff

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because it is too large Load Diff

View File

@ -1,246 +0,0 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="color-scheme" content="dark">
<title>@@TITLE@@</title>
<style>
:root {
--bg:#07100e; --surface:#0d1916; --surface-2:#12231e; --line:#284238;
--text:#edf8f3; --muted:#8da79b; --green:#76b900; --green-2:#a5e34b;
--cyan:#39d5d0; --amber:#ffbd4a; --red:#ff6b6b; --shadow:0 24px 60px rgba(0,0,0,.28);
}
* { box-sizing:border-box; }
html { background:var(--bg); }
body {
margin:0; color:var(--text); background:
radial-gradient(circle at 18% -5%,rgba(118,185,0,.14),transparent 29rem),
radial-gradient(circle at 92% 8%,rgba(57,213,208,.08),transparent 24rem), var(--bg);
font:14px/1.5 Inter,ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;
}
body::before {
content:""; position:fixed; inset:0; pointer-events:none; opacity:.26;
background-image:linear-gradient(rgba(255,255,255,.025) 1px,transparent 1px),
linear-gradient(90deg,rgba(255,255,255,.025) 1px,transparent 1px);
background-size:32px 32px;
}
.hero,main,footer { width:min(1680px,calc(100% - 48px)); margin-inline:auto; position:relative; }
.hero { padding:44px 0 28px; display:flex; align-items:flex-end; justify-content:space-between; gap:24px; }
.kicker,.eyebrow { color:var(--green-2); font-size:11px; font-weight:800; letter-spacing:.18em; }
h1 { margin:6px 0 2px; font-size:clamp(34px,4vw,58px); letter-spacing:-.045em; line-height:1; }
.hero p,.gpu-card p,.nic-card p { color:var(--muted); margin:8px 0 0; }
.hero-actions { display:flex; align-items:center; gap:12px; }
button {
border:1px solid var(--line); background:var(--surface-2); color:var(--text);
border-radius:12px; padding:10px 14px; font-weight:750; cursor:pointer;
}
button:hover { border-color:var(--green); }
.overall { display:flex; gap:9px; align-items:center; font-weight:800; padding:10px 14px; border-radius:999px; border:1px solid var(--line); }
.overall i { width:9px; height:9px; border-radius:50%; box-shadow:0 0 18px currentColor; }
.overall.healthy { color:var(--green-2); }.overall.healthy i{background:var(--green-2)}
.overall.attention { color:var(--amber); }.overall.attention i{background:var(--amber)}
.overall.critical { color:var(--red); }.overall.critical i{background:var(--red)}
.overview-grid { display:grid; grid-template-columns:1.15fr .9fr 1fr; gap:12px; margin-bottom:14px; }
.overview-card,.panel {
background:linear-gradient(145deg,rgba(18,35,30,.94),rgba(10,21,18,.96));
border:1px solid var(--line); box-shadow:var(--shadow);
}
.overview-card { min-height:290px; border-radius:18px; padding:20px; overflow:hidden; }
.overview-head { display:flex; align-items:flex-start; justify-content:space-between; gap:14px; }
.overview-lead { color:var(--muted); margin:12px 0 0; min-height:42px; }
.overview-primary { display:flex; align-items:baseline; gap:9px; margin:8px 0 17px; }
.overview-primary b { font-size:52px; line-height:1; letter-spacing:-.055em; }
.overview-primary span { color:var(--muted); font-weight:750; }
.gpu-power-summary { margin:-5px 0 12px; }
.gpu-power-summary>div:first-child { display:flex; justify-content:space-between; align-items:baseline; gap:10px; font-size:10px; color:var(--muted); }
.gpu-power-summary>div:first-child b { color:var(--text); font-size:12px; }
.gpu-power-summary>small { display:block; margin-top:4px; color:var(--muted); font-size:8px; }
.overview-metrics { display:grid; grid-template-columns:repeat(2,1fr); gap:1px; background:var(--line); border:1px solid var(--line); border-radius:10px; overflow:hidden; }
.overview-metrics span { background:rgba(7,16,14,.72); padding:10px; min-width:0; }
.overview-metrics b,.overview-metrics small { display:block; }
.overview-metrics b { font-size:17px; }.overview-metrics small { color:var(--muted); margin-top:2px; font-size:9px; }
.overview-facts { display:grid; grid-template-columns:repeat(2,1fr); gap:1px; background:var(--line); border:1px solid var(--line); border-radius:10px; overflow:hidden; margin-top:12px; }
.overview-facts div { background:rgba(7,16,14,.72); padding:9px 10px; min-width:0; }
.overview-facts dt { font-size:9px; letter-spacing:.08em; text-transform:uppercase; }
.overview-facts dd { text-align:left; margin-top:3px; font-size:11px; font-weight:700; overflow-wrap:anywhere; }
.overview-facts small { color:var(--muted); }
.compact { padding:7px 9px; font-size:11px; }
.memory-row { display:flex; justify-content:space-between; gap:10px; color:var(--muted); font-size:10px; }
.memory-row b { color:var(--text); }
.memory-meter { height:6px; background:#172821; border-radius:10px; margin:7px 0 14px; overflow:hidden; }
.memory-meter i { display:block; height:100%; background:linear-gradient(90deg,var(--cyan),var(--green-2)); }
.cpu-metrics { grid-template-columns:repeat(3,1fr); }
.cpu-affinity-summary { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:6px; margin-top:9px; }
.cpu-affinity-summary span { border:1px solid var(--line); background:rgba(7,16,14,.55); border-radius:8px; padding:7px 8px; min-width:0; }
.cpu-affinity-summary b,.cpu-affinity-summary small { display:block; }
.cpu-affinity-summary b { color:var(--green-2); font-size:10px; }.cpu-affinity-summary small { color:var(--muted); font-size:8px; overflow-wrap:anywhere; }
.panel { border-radius:20px; padding:22px; margin-bottom:14px; overflow:hidden; }
.section-head { display:flex; justify-content:space-between; align-items:flex-start; gap:20px; margin-bottom:20px; }
h2 { margin:3px 0 0; font-size:24px; letter-spacing:-.025em; }
.legend { display:flex; flex-wrap:wrap; gap:7px; justify-content:flex-end; }
.badge { display:inline-flex; align-items:center; width:max-content; border:1px solid var(--line); color:var(--muted); border-radius:999px; padding:3px 8px; font-size:10px; font-weight:850; letter-spacing:.05em; white-space:nowrap; }
.badge.ok { color:var(--green-2); border-color:rgba(165,227,75,.34); background:rgba(118,185,0,.09); }
.badge.warn { color:var(--amber); border-color:rgba(255,189,74,.35); background:rgba(255,189,74,.08); }
.badge.bad { color:var(--red); border-color:rgba(255,107,107,.35); background:rgba(255,107,107,.08); }
.topology-viewbar { display:flex; align-items:center; justify-content:space-between; gap:14px; margin:-5px 0 14px; padding:10px 12px; border:1px solid var(--line); border-radius:12px; background:rgba(7,16,14,.48); }
.topology-viewbar b,.topology-viewbar small { display:block; }
.topology-viewbar b { font-size:11px; }.topology-viewbar small { color:var(--muted); font-size:9px; margin-top:2px; }
.numa-switcher { display:flex; align-items:center; gap:4px; padding:3px; border:1px solid var(--line); border-radius:10px; background:#07100e; }
.numa-switch { min-width:72px; padding:6px 10px; border:0; border-radius:7px; background:transparent; color:var(--muted); font-size:10px; }
.numa-switch:hover { border-color:transparent; color:var(--text); }
.numa-switch.active { color:#07100e; background:var(--green-2); box-shadow:0 0 18px rgba(165,227,75,.12); }
.numa-ruler { display:grid; grid-template-columns:1fr 1fr; margin:0 0 8px; gap:2px; }
.numa-ruler span { text-align:center; padding:5px; color:var(--muted); font-size:11px; font-weight:800; letter-spacing:.14em; border-bottom:1px solid var(--line); }
.numa-ruler small { display:block; margin-top:2px; color:var(--green-2); font-size:9px; letter-spacing:.04em; }
.affinity-note { color:var(--muted); text-align:center; font-size:9px; margin:3px 0 10px; }
.nvlink-fabric {
border:1px solid rgba(118,185,0,.5); background:linear-gradient(90deg,rgba(118,185,0,.18),rgba(57,213,208,.08),rgba(118,185,0,.18));
border-radius:13px; padding:11px 16px; display:flex; align-items:center; justify-content:center; gap:18px; color:var(--green-2); margin-bottom:16px;
}
.nvlink-fabric span { font-size:11px; letter-spacing:.12em; font-weight:900; }
.nvlink-fabric small { color:var(--muted); }
.rails { display:grid; gap:10px; }
.topology-panel { overflow-x:auto; }
.rail { position:relative; display:grid; grid-template-rows:auto 54px 1fr; }
.rail[hidden],.numa-ruler [hidden] { display:none!important; }
.rail::before { content:""; position:absolute; z-index:0; left:50%; top:150px; bottom:150px; width:2px; background:linear-gradient(var(--green),var(--cyan)); opacity:.6; }
.gpu-card,.nic-card,.ib-card { position:relative; z-index:1; border:1px solid var(--line); border-radius:14px; background:rgba(7,16,14,.92); }
.gpu-card { padding:13px; border-top:3px solid var(--green); min-height:278px; }
.card-head { display:flex; justify-content:space-between; align-items:center; gap:8px; }
.gpu-card h3,.nic-card h4,.ib-card h4 { margin:10px 0 0; line-height:1.14; }
.gpu-card h3 { min-height:33px; font-size:15px; }
.mono { font:10px/1.4 ui-monospace,SFMono-Regular,Menlo,monospace; }
.gpu-affinity { display:grid; grid-template-columns:1.4fr .6fr; gap:5px; margin-top:8px; }
.gpu-affinity span { min-width:0; background:var(--surface); border:1px solid var(--line); border-radius:7px; padding:5px 6px; }
.gpu-affinity small,.gpu-affinity b { display:block; }
.gpu-affinity small { color:var(--muted); font-size:7px; letter-spacing:.08em; }
.gpu-affinity b { color:var(--green-2); margin-top:1px; font:9px/1.25 ui-monospace,monospace; overflow-wrap:anywhere; }
.gpu-stats { display:grid; grid-template-columns:1.05fr .75fr .9fr; gap:3px; margin-top:15px; }
.gpu-stats span { min-width:0; }.gpu-stats b,.gpu-stats small { display:block; }
.gpu-stats b { font-size:12px; }.gpu-stats small { color:var(--muted); font-size:9px; }
.meter { height:4px; background:#172821; border-radius:10px; margin-top:11px; overflow:hidden; }
.meter i { display:block; height:100%; background:linear-gradient(90deg,var(--green),var(--green-2)); }
.gpu-power { margin-top:10px; padding-top:8px; border-top:1px solid var(--line); }
.gpu-power>div:first-child { display:flex; align-items:flex-start; justify-content:space-between; gap:5px; }
.gpu-power span,.gpu-power b,.gpu-power small { display:block; min-width:0; }
.gpu-power b { font-size:10px; }.gpu-power small { color:var(--muted); font-size:7px; margin-top:1px; }
.gpu-power em { flex:0 0 auto; color:var(--cyan); border:1px solid rgba(57,213,208,.3); border-radius:999px; padding:2px 5px; font:normal 800 8px/1.2 ui-monospace,monospace; }
.power-meter { height:5px; margin-top:6px; overflow:hidden; border-radius:999px; background:#172821; }
.power-meter i { display:block; height:100%; background:linear-gradient(90deg,var(--cyan),var(--green-2),var(--amber)); }
.relation { position:relative; z-index:2; align-self:center; justify-self:center; display:flex; align-items:center; flex-direction:column; gap:3px; color:var(--muted); font-size:9px; }
.nic-card { padding:13px; border-top:3px solid var(--cyan); min-height:234px; }
.nic-card h4,.ib-card h4 { font-size:18px; }.nic-card p { font:10px ui-monospace,monospace; }
.bond-capacity { display:flex; align-items:baseline; gap:6px; margin:14px 0 10px; }
.bond-capacity b { font-size:24px; }.bond-capacity span { color:var(--muted); font-size:9px; }
.ports { display:grid; gap:6px; }
.port { display:flex; align-items:center; gap:8px; background:var(--surface); border:1px solid var(--line); border-radius:8px; padding:7px; }
.port-dot { width:7px; height:7px; flex:0 0 auto; border-radius:50%; background:var(--muted); }
.port.up .port-dot { background:var(--green-2); box-shadow:0 0 9px rgba(165,227,75,.6); }
.port.down .port-dot { background:var(--red); }
.port b,.port small { display:block; }.port b { font-size:11px; }.port small { color:var(--muted); font:8px ui-monospace,monospace; margin-top:2px; }
.capacity-note { color:var(--muted); margin:16px 0 0; font-size:11px; }
.network-grid { display:grid; grid-template-columns:1fr 1fr; gap:14px; }
.ib-grid { display:grid; grid-template-columns:repeat(4,minmax(0,1fr)); gap:10px; }
.ib-card { min-width:0; padding:14px; border-top:3px solid var(--amber); overflow:hidden; }
.ib-card .card-head { align-items:flex-start; flex-wrap:wrap; row-gap:7px; }
.ib-card .card-head .eyebrow { min-width:0; overflow-wrap:anywhere; font-size:9px; letter-spacing:.13em; }
.ib-card .card-head .badge { flex:0 0 auto; margin-left:auto; max-width:100%; padding:3px 6px; font-size:8px; letter-spacing:.03em; }
.ib-rate { color:var(--amber); font-size:30px; font-weight:850; margin:13px 0; }
.ib-rate small { font-size:11px; margin-left:3px; color:var(--muted); }
dl { margin:0; }.ib-card dl { display:grid; gap:5px; }
.ib-card dl div { display:flex; justify-content:space-between; gap:8px; font-size:10px; }
.ib-card dd { min-width:0; overflow-wrap:anywhere; }
dt { color:var(--muted); } dd { margin:0; text-align:right; }
.management-grid { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:10px; }
.management-card { border:1px solid var(--line); border-top:3px solid var(--cyan); border-radius:14px; background:rgba(7,16,14,.92); padding:14px; min-height:205px; }
.management-card.bmc-card { border-top-color:#8da79b; }
.management-card.unavailable { border-top-color:var(--line); }
.management-title { display:flex; justify-content:space-between; align-items:baseline; gap:10px; margin-top:12px; }
.management-title h4 { margin:0; font-size:19px; }.management-title b { color:var(--cyan); font-size:18px; }
.management-card .address { color:var(--text); background:var(--surface); border:1px solid var(--line); border-radius:8px; padding:8px; min-height:38px; font:10px/1.5 ui-monospace,monospace; }
.management-card dl { display:grid; gap:4px; }
.management-card dl div { display:flex; justify-content:space-between; gap:8px; font-size:9px; }
.management-card .card-note { color:var(--muted); font-size:10px; margin-top:12px; }
.health-panel { margin-bottom:14px; }
.checks { list-style:none; padding:0; margin:0; display:grid; grid-template-columns:repeat(auto-fit,minmax(175px,1fr)); gap:8px; }
.check { display:flex; align-items:center; gap:11px; padding:12px; min-height:68px; border:1px solid var(--line); border-radius:10px; background:rgba(7,16,14,.55); }
.check-icon { width:25px; height:25px; display:grid; place-items:center; border-radius:50%; font-weight:900; flex:0 0 auto; }
.check.ok .check-icon { color:var(--green-2); background:rgba(118,185,0,.13); }
.check.warning .check-icon { color:var(--amber); background:rgba(255,189,74,.12); }
.check.critical .check-icon { color:var(--red); background:rgba(255,107,107,.12); }
.check b,.check small { display:block; }.check small { color:var(--muted); }
.ghost { background:transparent; }
pre { white-space:pre-wrap; word-break:break-word; background:#050b09; color:#b8cdc3; border:1px solid var(--line); border-radius:12px; padding:16px; max-height:480px; overflow:auto; font:11px/1.6 ui-monospace,monospace; }
dialog { width:min(1100px,calc(100% - 32px)); max-height:86vh; color:var(--text); background:var(--surface); border:1px solid var(--line); border-radius:18px; padding:20px; box-shadow:0 35px 100px rgba(0,0,0,.65); }
dialog::backdrop { background:rgba(0,0,0,.72); backdrop-filter:blur(5px); }
.dialog-head { display:flex; align-items:flex-start; justify-content:space-between; gap:20px; }
dialog pre { max-height:68vh; }
.empty-state { color:var(--muted); border:1px dashed var(--line); border-radius:12px; padding:30px; text-align:center; grid-column:1/-1; }
footer { padding:8px 0 30px; color:var(--muted); display:flex; justify-content:space-between; font-size:11px; }
@media (max-width:1100px) {
.overview-grid { grid-template-columns:1fr 1fr; }
.system-overview { grid-column:1/-1; }
.network-grid { grid-template-columns:1fr; }
}
@media (max-width:640px) {
.hero,main,footer { width:min(100% - 24px,1680px); }
.hero { align-items:flex-start; flex-direction:column; padding-top:28px; }
.overview-grid { grid-template-columns:1fr; }.system-overview { grid-column:auto; }
.panel { padding:15px; }.section-head { flex-direction:column; }
.ib-grid,.management-grid { grid-template-columns:1fr; }
.topology-viewbar { align-items:flex-start; flex-direction:column; }
.numa-switcher { width:100%; }.numa-switch { flex:1; min-width:0; }
.nvlink-fabric { align-items:flex-start; flex-direction:column; gap:2px; }
}
@media print {
:root { --bg:#fff; --surface:#fff; --surface-2:#f3f6f4; --line:#cbd6d0; --text:#102018; --muted:#52675d; --shadow:none; }
body { background:#fff; }.hero,main,footer { width:100%; }.hero { padding-top:10px; }
button,dialog { display:none!important; }.panel,.overview-card { break-inside:avoid; }
.topology-panel { overflow:visible; zoom:.72; }.rails { min-width:0; }
}
</style>
</head>
<body>
@@BODY@@
<script>
const rawDialog=document.getElementById('raw-dialog');
document.querySelector('[data-open-raw]')?.addEventListener('click',()=>rawDialog?.showModal());
document.querySelector('[data-close-raw]')?.addEventListener('click',()=>rawDialog?.close());
rawDialog?.addEventListener('click',event=>{if(event.target===rawDialog)rawDialog.close();});
const topologyRails=document.querySelector('.rails');
const numaRuler=document.querySelector('.numa-ruler');
const numaSwitches=[...document.querySelectorAll('[data-numa-filter]')];
function setNumaView(value){
const rails=[...document.querySelectorAll('.rail[data-numa]')];
const rulers=[...document.querySelectorAll('.numa-ruler [data-numa]')];
let visibleRails=0;
rails.forEach(rail=>{
const visible=value==='all'||rail.dataset.numa===value;
rail.hidden=!visible;
if(visible)visibleRails+=1;
});
const visibleRulers=[];
rulers.forEach(ruler=>{
const visible=value==='all'||ruler.dataset.numa===value;
ruler.hidden=!visible;
if(visible)visibleRulers.push(ruler);
});
if(topologyRails){
const count=Math.max(1,visibleRails);
topologyRails.style.gridTemplateColumns=`repeat(${count},minmax(132px,1fr))`;
topologyRails.style.minWidth=`${count*132+Math.max(0,count-1)*10}px`;
}
if(numaRuler){
numaRuler.style.gridTemplateColumns=visibleRulers.map(item=>`${item.dataset.count||1}fr`).join(' ')||'1fr';
}
numaSwitches.forEach(button=>{
const active=button.dataset.numaFilter===value;
button.classList.toggle('active',active);
button.setAttribute('aria-pressed',String(active));
});
}
numaSwitches.forEach(button=>button.addEventListener('click',()=>setNumaView(button.dataset.numaFilter)));
</script>
</body>
</html>

View File

@ -1,980 +0,0 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="color-scheme" content="dark">
<title>@@TITLE@@</title>
<style>
:root {
--bg:#050c0a; --surface:#0b1713; --surface-2:#10241d; --line:#28493d;
--text:#eef9f4; --muted:#8fa99e; --green:#a5e34b; --cyan:#51d9d2;
--amber:#ffbd4a; --purple:#b592ff; --red:#ff6b6b;
}
* { box-sizing:border-box; }
html,body { margin:0; min-height:100%; background:var(--bg); color:var(--text); }
body {
font:14px/1.45 Inter,ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;
background:
radial-gradient(circle at 16% 0,rgba(118,185,0,.15),transparent 30rem),
radial-gradient(circle at 90% 8%,rgba(81,217,210,.08),transparent 28rem),
var(--bg);
}
button { font:inherit; }
.shell { min-height:100vh; display:grid; grid-template-rows:auto 1fr auto; }
.topbar {
min-width:0; display:flex; align-items:flex-end; justify-content:space-between;
gap:24px; padding:22px 28px 16px; border-bottom:1px solid var(--line);
}
.kicker { color:var(--green); font-size:10px; font-weight:850; letter-spacing:.19em; }
h1 { margin:4px 0 0; font-size:clamp(25px,3vw,42px); letter-spacing:-.045em; line-height:1; }
.subtitle { margin:7px 0 0; color:var(--muted); }
.top-stats { display:flex; flex-wrap:wrap; justify-content:flex-end; gap:8px; }
.stat {
min-width:96px; padding:8px 11px; border:1px solid var(--line); border-radius:10px;
background:rgba(11,23,19,.76);
}
.stat b,.stat small { display:block; }.stat b { font-size:15px; }.stat small { color:var(--muted); font-size:9px; }
.workspace { min-height:0; display:grid; grid-template-columns:minmax(0,1fr) 330px; gap:0; }
.stage { min-width:0; min-height:720px; position:relative; overflow:hidden; }
#topology-canvas { display:block; width:100%; height:100%; min-height:720px; outline:none; cursor:grab; }
#topology-canvas:active { cursor:grabbing; }
#topology-canvas:focus-visible { box-shadow:inset 0 0 0 2px var(--green); }
.toolbar {
position:absolute; z-index:4; left:18px; right:18px; top:16px;
display:flex; align-items:flex-start; justify-content:space-between; gap:12px;
pointer-events:none;
}
.toolbar-group {
pointer-events:auto; display:flex; flex-wrap:wrap; align-items:center; gap:5px;
padding:5px; border:1px solid var(--line); border-radius:11px;
background:rgba(5,12,10,.86); backdrop-filter:blur(10px);
}
.toolbar-label { padding:0 7px; color:var(--muted); font-size:9px; font-weight:800; letter-spacing:.12em; }
.tool-button {
border:0; border-radius:8px; padding:7px 10px; color:var(--muted);
background:transparent; cursor:pointer; font-size:10px; font-weight:800;
}
.tool-button:hover { color:var(--text); background:rgba(255,255,255,.05); }
.tool-button[aria-pressed="true"] { color:#07100e; background:var(--green); }
.tool-button:disabled { opacity:.35; cursor:not-allowed; }
.tool-button.reset { color:var(--text); border:1px solid var(--line); }
.camera-status {
position:absolute; z-index:3; left:50%; top:72px; transform:translateX(-50%);
padding:6px 10px; border:1px solid rgba(165,227,75,.32); border-radius:999px;
color:var(--green); background:rgba(5,12,10,.76); backdrop-filter:blur(8px);
font:800 9px ui-monospace,monospace; letter-spacing:.1em; pointer-events:none;
}
.canvas-note {
position:absolute; z-index:3; left:22px; bottom:18px; max-width:min(630px,calc(100% - 44px));
padding:8px 11px; color:var(--muted); font-size:10px; border:1px solid var(--line);
border-radius:9px; background:rgba(5,12,10,.8); backdrop-filter:blur(8px);
}
.legend {
position:absolute; z-index:3; right:20px; bottom:18px; display:flex; flex-wrap:wrap;
justify-content:flex-end; gap:9px 12px; max-width:52%; padding:8px 10px;
border:1px solid var(--line); border-radius:9px; background:rgba(5,12,10,.8);
}
.legend span { display:flex; align-items:center; gap:5px; color:var(--muted); font-size:9px; }
.legend i { width:18px; height:3px; border-radius:9px; background:var(--line-color); }
.inspector {
min-width:0; padding:20px; border-left:1px solid var(--line);
background:linear-gradient(180deg,rgba(13,28,23,.97),rgba(7,16,13,.97));
overflow:auto;
}
.inspector h2 { margin:4px 0 5px; font-size:21px; }.inspector>p { color:var(--muted); margin:0 0 18px; }
.scope {
margin-bottom:16px; padding:10px; color:var(--muted); border:1px solid var(--line);
border-left:3px solid var(--amber); border-radius:9px; background:rgba(255,189,74,.04); font-size:10px;
}
.mini-grid { display:grid; grid-template-columns:1fr 1fr; gap:7px; margin-bottom:18px; }
.mini-grid div { min-width:0; padding:9px; border:1px solid var(--line); border-radius:8px; background:rgba(5,12,10,.5); }
.mini-grid b,.mini-grid small { display:block; }.mini-grid b { font-size:16px; }.mini-grid small { color:var(--muted); font-size:9px; }
.detail {
padding-top:15px; border-top:1px solid var(--line);
}
.detail-head { display:flex; justify-content:space-between; gap:12px; align-items:flex-start; }
.detail h3 { margin:3px 0 0; font-size:19px; }.detail-kind { color:var(--green); font-size:9px; font-weight:850; letter-spacing:.15em; }
.detail-status { color:var(--cyan); font:800 10px ui-monospace,monospace; }
.detail dl { display:grid; gap:1px; margin:14px 0 0; border:1px solid var(--line); border-radius:9px; overflow:hidden; }
.detail dl div { display:grid; grid-template-columns:92px minmax(0,1fr); gap:8px; padding:8px 9px; background:rgba(5,12,10,.52); border-bottom:1px solid var(--line); }
.detail dl div:last-child { border-bottom:0; }
.detail dt { color:var(--muted); font-size:9px; }.detail dd { margin:0; text-align:right; overflow-wrap:anywhere; font:10px/1.4 ui-monospace,monospace; }
.path-guide { margin-top:18px; }.path-guide h3 { margin:0 0 9px; font-size:13px; }
.path-guide ul { margin:0; padding-left:17px; color:var(--muted); font-size:10px; }
.path-guide li+li { margin-top:5px; }
footer { display:flex; justify-content:space-between; gap:16px; padding:10px 28px 16px; color:var(--muted); font-size:9px; border-top:1px solid var(--line); }
@media (max-width:980px) {
.workspace { grid-template-columns:1fr; }
.inspector { border-left:0; border-top:1px solid var(--line); }
.stage,#topology-canvas { min-height:620px; }
.mini-grid { grid-template-columns:repeat(4,minmax(0,1fr)); }
}
@media (max-width:640px) {
.topbar { align-items:flex-start; flex-direction:column; padding-inline:16px; }
.top-stats { justify-content:flex-start; }.workspace { display:block; }
.stage,#topology-canvas { min-height:560px; }
.toolbar { left:10px; right:10px; flex-direction:column; }
.toolbar-group { width:100%; }.tool-button { flex:1; }
.legend { display:none; }.canvas-note { left:10px; bottom:10px; max-width:calc(100% - 20px); }
.mini-grid { grid-template-columns:1fr 1fr; }
footer { padding-inline:16px; flex-direction:column; }
}
</style>
</head>
<body>
<div class="shell">
<header class="topbar">
<div>
<span class="kicker">MACHINE CT · GUEST-VISIBLE TOPOLOGY</span>
<h1 id="machine-title">GPU Machine 3D CT</h1>
<p class="subtitle" id="machine-subtitle"></p>
</div>
<div class="top-stats" id="top-stats"></div>
</header>
<main class="workspace">
<section class="stage" aria-label="交互式 3D 硬件拓扑">
<div class="toolbar">
<div class="toolbar-group" id="view-controls"><span class="toolbar-label">视图</span></div>
<div class="toolbar-group" id="camera-controls">
<span class="toolbar-label">镜头</span>
<button class="tool-button" type="button" data-camera="outside" aria-pressed="true">外观</button>
<button class="tool-button" type="button" data-camera="inside" aria-pressed="false">进入内部</button>
<button class="tool-button" type="button" id="camera-back" disabled>上一层</button>
<button class="tool-button" type="button" id="focus-selected" disabled>聚焦所选</button>
</div>
<div class="toolbar-group" id="layer-controls">
<span class="toolbar-label">图层</span>
<button class="tool-button" type="button" data-layer="cpu" aria-pressed="true">CPU</button>
<button class="tool-button" type="button" data-layer="pcie" aria-pressed="true">PCIe</button>
<button class="tool-button" type="button" data-layer="gpu" aria-pressed="true">GPU</button>
<button class="tool-button" type="button" data-layer="fabric" aria-pressed="true">NVSwitch</button>
<button class="tool-button" type="button" data-layer="network" aria-pressed="true">网络</button>
<button class="tool-button reset" type="button" id="reset-view">复位</button>
</div>
</div>
<canvas id="topology-canvas" tabindex="0" aria-label="可旋转的 GPU、CPU、PCIe、NVSwitch 和网卡三维拓扑"></canvas>
<div class="camera-status" id="camera-status">整机外观 · 双击壳体逐层进入</div>
<div class="canvas-note">双击壳体逐层进入 · 上一层返回 · 拖动旋转 · Shift+拖动平移 · Esc 返回整机</div>
<div class="legend" aria-label="连线图例">
<span><i style="--line-color:#a5e34b"></i>NVLink(仅连 GPU)</span>
<span><i style="--line-color:#51d9d2"></i>GPUDirect 候选</span>
<span><i style="--line-color:#ffbd4a"></i>PCIe</span>
<span><i style="--line-color:#b592ff"></i>CPU 亲和</span>
<span><i style="--line-color:#ff6b6b"></i>跨 NUMA / SYS</span>
</div>
</section>
<aside class="inspector">
<span class="kicker">TOPOLOGY INSPECTOR</span>
<h2>逻辑拓扑切片</h2>
<p>NUMA 是地域,PCIe 是道路,GPUDirect 是设备间数据路径。</p>
<div class="scope">当前视图来自 Guest 可见的 PCIe/NUMA 拓扑。直通或虚拟 PCIe 可能隐藏宿主机真实插槽几何位置。</div>
<div class="mini-grid" id="machine-metrics"></div>
<section class="detail" id="selection-detail" aria-live="polite"></section>
<section class="path-guide">
<h3>距离语义</h3>
<ul>
<li>PIX:最多经过一个 PCIe Switch,通常是最优 GPU–NIC 路径。</li>
<li>PXB/PHB:经过更多 Switch 或 Host Bridge。</li>
<li>NODE:同 NUMA 内跨 Host Bridge。</li>
<li>SYS:跨 NUMA/CPU 互联,路径最长。</li>
</ul>
</section>
</aside>
</main>
<footer><span>GPU Machine CT v@@VERSION@@ · 原生 Canvas · 无外部依赖</span><span id="generated-at"></span></footer>
</div>
<script type="application/json" id="machine-data">@@MACHINE_DATA@@</script>
<script>
(() => {
'use strict';
const data = JSON.parse(document.getElementById('machine-data').textContent);
const canvas = document.getElementById('topology-canvas');
const ctx = canvas.getContext('2d');
const system = data.system || {};
const gpus = data.gpus || [];
const networkEndpoints = data.network_endpoints || [];
const allNics = data.nics || [];
const endpointById = new Map(networkEndpoints.map(endpoint => [endpoint.id,endpoint]));
const networkByGpu = new Map(gpus.map(gpu => [Number(gpu.index),endpointById.get(gpu.primary_network_id)]).filter(([,endpoint]) => endpoint));
const numaIds = [...new Set(gpus.map(gpu => gpu.numa).filter(value => value !== null && value !== undefined))];
if (!numaIds.length) numaIds.push(0);
document.getElementById('machine-title').textContent = `${system.hostname || 'GPU Node'} · 3D Hardware CT`;
document.getElementById('machine-subtitle').textContent = `${system.manufacturer || ''} ${system.product || ''} · ${system.os || ''}`.trim();
document.getElementById('generated-at').textContent = `采集时间 ${data.generated_at || '—'} · ${data.pci_fabric?.scope || 'guest-visible'}`;
const totalPower = gpus.reduce((sum,gpu) => sum + (Number(gpu.power_w) || 0),0);
const totalLimit = gpus.reduce((sum,gpu) => sum + (Number(gpu.power_limit_w) || 0),0);
const metrics = [
['GPU', gpus.length],
['CPU', system.cpu_physical_cores || system.cpu_logical || '—'],
['NUMA', numaIds.length],
['GPU Power', totalPower ? `${(totalPower/1000).toFixed(2)} kW` : '—']
];
document.getElementById('top-stats').innerHTML = metrics.map(([label,value]) => `<span class="stat"><b>${escapeHtml(value)}</b><small>${escapeHtml(label)}</small></span>`).join('');
document.getElementById('machine-metrics').innerHTML = [
['CPU Sockets', system.cpu_sockets || '—'],
['PCI Bridges', data.pci_fabric?.bridges?.length || '—'],
['NVLinks', data.nvlink?.active_links || '—'],
['Power Limit', totalLimit ? `${(totalLimit/1000).toFixed(2)} kW` : '—']
].map(([label,value]) => `<div><b>${escapeHtml(value)}</b><small>${escapeHtml(label)}</small></div>`).join('');
const state = {
yaw:-0.64, pitch:0.52, zoom:1, focus:'all', selected:null,
cameraMode:'outside', target:{x:0,y:3.4,z:0}, scopeStack:[],
layers:new Set(['cpu','pcie','gpu','fabric','network'])
};
let scene = {boxes:[],edges:[]};
let hitTargets = [];
let pointer = null;
let cameraAnimation = 0;
function escapeHtml(value) {
return String(value ?? '').replace(/[&<>"']/g, char => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[char]));
}
function deepestCommonPath(paths) {
const valid = paths.filter(path => Array.isArray(path) && path.length);
if (valid.length < 2) return '';
const shortest = Math.min(...valid.map(path => path.length));
let common = '';
for (let index=0; index<shortest; index+=1) {
const value = valid[0][index];
if (valid.every(path => path[index] === value)) common = value;
else break;
}
return common;
}
function addBox(box) {
scene.boxes.push({
selectable:true, opacity:1, ...box,
size:box.size || {x:2,y:1,z:1}
});
return box.id;
}
function addEdge(from,to,type,label,layer,meta={}) {
scene.edges.push({from,to,type,label,layer,meta});
}
function buildScene() {
scene = {boxes:[],edges:[]};
const visibleNumas = state.focus === 'all' ? numaIds : numaIds.filter(id => String(id) === String(state.focus));
const domainGap = 18;
const totalWidth = Math.max(18,visibleNumas.length * domainGap);
addBox({
id:'chassis', kind:'chassis', layer:'pcie', selectable:true,
label:`${system.virtualization || 'HOST'} · GUEST VISIBLE`,
position:{x:0,y:3.4,z:0}, size:{x:totalWidth,y:10,z:17},
color:'#56756a', opacity:.08, wire:true
});
const gpuNodeIds = [];
visibleNumas.forEach((numa,domainIndex) => {
const centerX = (domainIndex - (visibleNumas.length-1)/2) * domainGap;
const domainGpus = gpus.filter(gpu => String(gpu.numa) === String(numa));
const cpuAffinity = domainGpus.find(gpu => gpu.cpu_affinity)?.cpu_affinity || '未解析';
const domainColor = numaIds.indexOf(numa) % 2 ? '#4fc6e8' : '#76b900';
addBox({
id:`numa-${numa}`, kind:'numa', layer:'cpu', selectable:true,
parentId:'chassis',
label:`NUMA ${numa}`, sublabel:`CPU ${cpuAffinity}`,
position:{x:centerX,y:3.2,z:0}, size:{x:16,y:8,z:15},
color:domainColor, opacity:.11, wire:true,
meta:{'CPU Affinity':cpuAffinity,'Memory NUMA':numa,'GPU Count':domainGpus.length}
});
addBox({
id:`cpu-${numa}`, kind:'cpu', layer:'cpu', label:`CPU / NUMA ${numa}`,
parentId:`numa-${numa}`,
sublabel:`Cores ${cpuAffinity}`, position:{x:centerX,y:1.4,z:4.7},
size:{x:5.8,y:1.3,z:2.3}, color:'#b592ff',
meta:{NUMA:numa,'CPU Affinity':cpuAffinity,Model:system.cpu_model || '—'}
});
addBox({
id:`root-${numa}`, kind:'pcie-root', layer:'pcie', label:'PCIe Root Complex',
parentId:`numa-${numa}`,
sublabel:`NUMA ${numa}`, position:{x:centerX,y:2.6,z:2.2},
size:{x:6.8,y:.8,z:1.6}, color:'#ffbd4a',
meta:{NUMA:numa,Scope:'Guest-visible host bridge'}
});
addEdge(`cpu-${numa}`,`root-${numa}`,'control','CPU / Host Bridge','pcie');
const columns = Math.max(1,Math.min(4,domainGpus.length));
domainGpus.forEach((gpu,index) => {
const row = Math.floor(index/columns);
const column = index%columns;
const x = centerX + (column-(columns-1)/2)*3.45;
const z = -.7-row*4.4;
const gpuId = `gpu-${gpu.index}`;
const endpoint = networkByGpu.get(Number(gpu.index));
const nicId = endpoint ? `network-${endpoint.id}` : '';
const primaryOwners = (endpoint?.gpu_links || [])
.filter(link => link.role === 'primary')
.map(link => Number(link.gpu_index))
.sort((left,right) => left-right);
const endpointOwner = primaryOwners.length > 0
&& primaryOwners[0] === Number(gpu.index)
&& (endpoint?.type === 'bond' || primaryOwners.length === 1);
const memberPaths = (endpoint?.members || []).map(member => member.pci_path);
const sharedBridge = deepestCommonPath([gpu.pci_path,...memberPaths]);
const relation = gpu.network_relation || '未解析';
addBox({
id:`pair-${gpu.index}`, kind:'pcie-group', layer:'pcie', selectable:true,
parentId:`numa-${numa}`,
label:sharedBridge ? `PCIe Switch ${sharedBridge}` : `PCIe Group · ${relation}`,
sublabel:`GPU${gpu.index} ↔ ${endpoint?.name || 'RDMA ?'}`,
position:{x,y:1.55,z:z-1.05}, size:{x:3.15,y:3.4,z:4.35},
color:relation === 'PIX' ? '#51d9d2' : '#ffbd4a', opacity:.08, wire:true,
meta:{Relation:relation,'Shared Bridge':sharedBridge || '未暴露','GPU BDF':gpu.bdf || '—'}
});
addBox({
id:gpuId, kind:'gpu', layer:'gpu', label:`GPU ${gpu.index}`,
parentId:`pair-${gpu.index}`,
sublabel:gpu.name || 'NVIDIA GPU', position:{x,y:2.35,z},
size:{x:2.55,y:1.45,z:1.7}, color:'#76b900',
meta:{
NUMA:gpu.numa,'CPU Affinity':gpu.cpu_affinity || '—',BDF:gpu.bdf || '—',
'PCI Path':(gpu.pci_path || []).join(' → ') || '未采集',
Power:gpu.power_w ? `${Number(gpu.power_w).toFixed(0)} W / ${Number(gpu.power_limit_w || 0).toFixed(0)} W` : '—',
PState:gpu.pstate || '—',Utilization:gpu.utilization_pct !== null ? `${gpu.utilization_pct}%` : '—'
}
});
gpuNodeIds.push(gpuId);
addEdge(`root-${numa}`,gpuId,'pcie',relation,'pcie',{relation});
addEdge(`cpu-${numa}`,gpuId,'affinity',`CPU ${cpuAffinity}`,'cpu');
if (endpoint?.type === 'bond' && endpointOwner) {
const slaves=endpoint.members || [];
const capacity=Number(endpoint.capacity_mbps)||0;
addBox({
id:nicId, kind:'bond', layer:'network', label:endpoint.name,
parentId:`pair-${gpu.index}`,
sublabel:`${slaves.length} ports · ${endpoint.rdma || 'RoCE / RDMA'}`,
position:{x,y:.62,z:z-2.15}, size:{x:2.85,y:1.45,z:1.65},
color:'#51d9d2',opacity:.1,wire:true,labelOffset:.85,
meta:{
NUMA:slaves[0]?.numa ?? numa,Relation:relation,
RDMA:endpoint.rdma || '—',Capacity:capacity ? `${capacity/1000} Gb/s` : '—',
Members:slaves.map(slave => `${slave.name} ${slave.bdf || ''}`).join(', ') || '—',
'Shared Bridge':sharedBridge || '未暴露'
}
});
slaves.forEach((slave,slaveIndex) => {
const portId=`nic-${endpoint.name}-${slave.name}`;
const offset=(slaveIndex-(slaves.length-1)/2)*1.18;
const speed=Number(slave.speed_mbps) ? `${Number(slave.speed_mbps)/1000}G` : '—';
addBox({
id:portId,kind:'roce-port',layer:'network',label:slave.name,
parentId:nicId,
sublabel:speed,position:{x:x+offset,y:.68,z:z-2.15},
size:{x:1.02,y:.78,z:1.08},color:'#79f4ee',
meta:{
Bond:endpoint.name,RDMA:endpoint.rdma || '—',NUMA:slave.numa ?? numa,
BDF:slave.bdf || '—',Speed:speed,State:slave.status || '—',
Driver:slave.driver || '—','PCI Path':(slave.pci_path || []).join(' → ') || '未采集'
}
});
addEdge(portId,nicId,'bond-member','Bond member','network');
addEdge(`root-${numa}`,portId,'pcie',relation,'pcie',{relation});
});
}
if (endpoint?.type === 'bond') {
(endpoint.members || []).forEach(slave => {
addEdge(
gpuId,
`nic-${endpoint.name}-${slave.name}`,
'gdr',
`${relation} · GDR candidate`,
'network',
{relation,port:slave.name}
);
});
} else if (endpoint && endpointOwner) {
const member=(endpoint.members || [])[0] || endpoint;
const speed=Number(endpoint.capacity_mbps) ? `${Number(endpoint.capacity_mbps)/1000}G` : '—';
const kind=endpoint.protocol === 'InfiniBand' ? 'infiniband' : 'roce-port';
addBox({
id:nicId,kind,layer:'network',label:endpoint.name,
parentId:`pair-${gpu.index}`,
sublabel:`${endpoint.protocol || 'RDMA'} · ${speed}`,
position:{x,y:.68,z:z-2.15},size:{x:2.35,y:1.05,z:1.35},
color:endpoint.protocol === 'InfiniBand' ? '#ff9f43' : '#79f4ee',
meta:{
Mode:'Standalone RDMA',RDMA:endpoint.rdma || '—',
NUMA:endpoint.numa ?? member.numa ?? numa,Relation:relation,
Evidence:endpoint.topology_visibility === 'nvidia-smi-topo'
? 'nvidia-smi topo -m' : 'Topology relation not exposed',
BDF:endpoint.bdf || member.bdf || '—',Speed:speed,
State:endpoint.state || member.status || '—',
'PCI Path':(endpoint.pci_path || member.pci_path || []).join(' → ') || '未采集'
}
});
addEdge(`root-${numa}`,nicId,'pcie',relation,'pcie',{relation});
}
if (endpoint?.type !== 'bond' && endpoint) {
addEdge(gpuId,nicId,'gdr',`${relation} · GDR candidate`,'network',{relation,port:endpoint.name});
}
});
const unpairedEndpoints = networkEndpoints.filter(endpoint =>
endpoint.type === 'standalone'
&& String(endpoint.numa) === String(numa)
&& (endpoint.gpu_links || []).filter(link => link.role === 'primary').length !== 1
);
unpairedEndpoints.forEach((endpoint,index) => {
const x = centerX - 5.4 + index*2.8;
const id = `network-${endpoint.id}`;
const isIb = endpoint.protocol === 'InfiniBand';
const speed = Number(endpoint.capacity_mbps)
? `${Number(endpoint.capacity_mbps)/1000} Gb/s` : '—';
addBox({
id,kind:isIb ? 'infiniband' : 'roce-port',
layer:'network',label:endpoint.name,
parentId:`numa-${numa}`,
sublabel:`${endpoint.protocol || 'RDMA'} ${speed} · GPU relation ${endpoint.topology_visibility === 'nvidia-smi-topo' ? 'exposed' : 'not exposed'}`,
position:{x,y:.75,z:6.35},
size:{x:2.35,y:1.05,z:1.4},color:isIb ? '#ff9f43' : '#79f4ee',
meta:{
NUMA:endpoint.numa,BDF:endpoint.bdf || '—',
RDMA:endpoint.rdma || '—',State:endpoint.state || '—',
'GPU Topology':endpoint.topology_visibility === 'nvidia-smi-topo'
? 'nvidia-smi topo -m' : 'Not exposed; no affinity inferred',
'PCI Path':(endpoint.pci_path || []).join(' → ') || '未采集'
}
});
addEdge(`root-${numa}`,id,'pcie',isIb ? 'Storage / IB' : 'Standalone RoCE','pcie');
(endpoint?.gpu_links || []).filter(link => link.role==='nearest').forEach(link => {
const gpuId=`gpu-${link.gpu_index}`;
if (scene.boxes.some(box => box.id===gpuId)) {
addEdge(gpuId,id,'gdr',`${link.relation} · nearest ${endpoint.protocol}`,'network',{relation:link.relation,port:endpoint.name});
}
});
});
const acceleratorMembers = new Set(networkEndpoints.flatMap(endpoint => (endpoint.members || []).map(member => member.name)));
const acceleratorGroups = new Set(networkEndpoints.filter(endpoint => endpoint.type==='bond').map(endpoint => endpoint.name));
const management = allNics.filter(nic =>
String(nic.numa) === String(numa)
&& !acceleratorMembers.has(nic.name)
&& !acceleratorGroups.has(nic.name)
&& (nic.bdf || nic.default_route || nic.is_bond)
);
management.slice(0,2).forEach((nic,index) => {
const id = `mgmt-${nic.name}`;
addBox({
id,kind:'management',layer:'network',label:nic.name,
parentId:`numa-${numa}`,
sublabel:nic.default_route ? 'In-band management' : 'OS network',
position:{x:centerX+5.4-index*2.7,y:.65,z:6.35},size:{x:2.25,y:1,z:1.35},color:'#7d9b90',
meta:{NUMA:nic.numa,BDF:nic.bdf || '—',Driver:nic.driver || '—',Address:(nic.addresses || []).join(', ') || '—'}
});
addEdge(`root-${numa}`,id,'pcie','Management','pcie');
});
});
if (gpuNodeIds.length > 1 && data.nvlink?.active_links) {
addBox({
id:'nvswitch',kind:'nvswitch',layer:'fabric',label:data.nvlink.fabric_type || 'NVLink Fabric',
parentId:'chassis',
sublabel:`${gpuNodeIds.length}/${gpus.length} GPU · GPU endpoints only`,
position:{x:0,y:8.15,z:-1.2},size:{x:Math.max(9,totalWidth-3),y:.8,z:2.1},color:'#a5e34b',
meta:{
Connection:'GPU endpoints only · not NIC',
'Active Links':data.nvlink.active_links,
'Links / GPU':data.nvlink.links_per_gpu || '—',
'Speed / Link':`${data.nvlink.speed_gbs || '—'} GB/s`
}
});
gpuNodeIds.forEach(id => {
const gpuIndex=Number(id.replace('gpu-',''));
addEdge(id,'nvswitch','nvlink',`GPU ${gpuIndex} ↔ NVSwitch`,'fabric',{gpuIndex});
});
}
renderDetail();
}
const viewControls = document.getElementById('view-controls');
function buildViewControls() {
const views = [['all','总览'],...numaIds.map(id => [String(id),`NUMA ${id}`])];
viewControls.querySelectorAll('button').forEach(button => button.remove());
views.forEach(([value,label]) => {
const button = document.createElement('button');
button.type='button'; button.className='tool-button'; button.textContent=label;
button.dataset.view=value; button.setAttribute('aria-pressed',String(state.focus === value));
button.addEventListener('click',() => {
state.focus=value; state.selected=null;
state.scopeStack=state.cameraMode==='inside' ? ['chassis'] : [];
buildViewControls(); buildScene();
setCameraMode(state.cameraMode,null,false);
});
viewControls.appendChild(button);
});
}
function cameraPreset(mode,box=null) {
if (mode === 'inside') {
const zoomByKind={chassis:1.48,numa:1.72,'pcie-group':2.5,bond:3.25};
return {
yaw:-.2,pitch:.27,zoom:box ? (zoomByKind[box.kind] || 2.15) : 1.48,
target:box ? {...box.position} : {x:0,y:2.25,z:-1.2}
};
}
return {yaw:-.64,pitch:.52,zoom:1,target:{x:0,y:3.4,z:0}};
}
function animateCamera(next,duration=380) {
const animation=++cameraAnimation;
const start={
yaw:state.yaw,pitch:state.pitch,zoom:state.zoom,target:{...state.target}
};
const started=performance.now();
function frame(now) {
if (animation!==cameraAnimation) return;
const raw=Math.min(1,(now-started)/duration);
const progress=1-Math.pow(1-raw,3);
state.yaw=start.yaw+(next.yaw-start.yaw)*progress;
state.pitch=start.pitch+(next.pitch-start.pitch)*progress;
state.zoom=start.zoom+(next.zoom-start.zoom)*progress;
state.target={
x:start.target.x+(next.target.x-start.target.x)*progress,
y:start.target.y+(next.target.y-start.target.y)*progress,
z:start.target.z+(next.target.z-start.target.z)*progress
};
draw();
if (raw<1) requestAnimationFrame(frame);
}
requestAnimationFrame(frame);
}
function updateCameraControls() {
document.querySelectorAll('[data-camera]').forEach(button => {
button.setAttribute('aria-pressed',String(button.dataset.camera===state.cameraMode));
});
const selectedBox=scene.boxes.find(box => box.id===state.selected && box.selectable);
const scopeBoxes=state.scopeStack.map(id => scene.boxes.find(box => box.id===id)).filter(Boolean);
const breadcrumb=['整机',...scopeBoxes.filter(box => box.kind!=='chassis').map(box => box.label)].join(' › ');
document.getElementById('focus-selected').disabled=!selectedBox;
document.getElementById('camera-back').disabled=state.cameraMode!=='inside';
document.getElementById('camera-status').textContent=state.cameraMode==='inside'
? `${breadcrumb} · 当前层之外已隐藏`
: '整机外观 · 双击壳体逐层进入';
}
function setCameraMode(mode,box=null,animate=true) {
state.cameraMode=mode;
if (mode==='outside') state.scopeStack=[];
const next=cameraPreset(mode,box);
updateCameraControls();
if (animate) animateCamera(next);
else {
cameraAnimation+=1;
state.yaw=next.yaw;state.pitch=next.pitch;state.zoom=next.zoom;
state.target={...next.target};draw();
}
}
const shellKinds=new Set(['chassis','numa','pcie-group','bond']);
function isShell(box) {
return Boolean(box && shellKinds.has(box.kind));
}
function activeScopeId() {
return state.scopeStack[state.scopeStack.length-1] || null;
}
function isDescendant(boxId,scopeId) {
let current=scene.boxes.find(box => box.id===boxId);
const visited=new Set();
while (current?.parentId && !visited.has(current.id)) {
if (current.parentId===scopeId) return true;
visited.add(current.id);
current=scene.boxes.find(box => box.id===current.parentId);
}
return false;
}
function scopePath(box) {
const path=[];
let current=box;
const visited=new Set();
while (current && !visited.has(current.id)) {
visited.add(current.id);
if (isShell(current)) path.unshift(current.id);
current=current.parentId ? scene.boxes.find(item => item.id===current.parentId) : null;
}
return path;
}
function nextScopeToward(box) {
const path=scopePath(box);
if (!path.length) return null;
const current=activeScopeId();
if (!current) return scene.boxes.find(item => item.id===path[0]) || null;
const currentIndex=path.indexOf(current);
const nextId=currentIndex>=0 ? path[currentIndex+1] : null;
return nextId ? scene.boxes.find(item => item.id===nextId) || null : null;
}
function enterScope(box,animate=true) {
const scope=isShell(box) ? box : scene.boxes.find(item => item.id==='chassis');
if (!scope) return;
const current=activeScopeId();
const targetPath=scopePath(scope);
if (state.cameraMode!=='inside' || !current) {
state.scopeStack=targetPath;
} else if (scope.id!==current && isDescendant(scope.id,current)) {
const currentIndex=targetPath.indexOf(current);
state.scopeStack=[
...state.scopeStack,
...targetPath.slice(Math.max(0,currentIndex+1)).filter(id => !state.scopeStack.includes(id))
];
}
state.cameraMode='inside';
state.selected=scope.kind==='chassis' ? null : scope.id;
renderDetail();setCameraMode('inside',scope,animate);
}
function leaveScope() {
if (state.cameraMode!=='inside') return;
state.scopeStack.pop();
const parentId=activeScopeId();
if (!parentId) {
state.selected=null;renderDetail();setCameraMode('outside');
return;
}
const parent=scene.boxes.find(box => box.id===parentId);
state.selected=parent?.kind==='chassis' ? null : parent?.id || null;
renderDetail();setCameraMode('inside',parent);
}
document.querySelectorAll('[data-camera]').forEach(button => button.addEventListener('click',() => {
const selectedBox=scene.boxes.find(box => box.id===state.selected && box.selectable);
const nextScope=selectedBox ? nextScopeToward(selectedBox) : null;
if (button.dataset.camera==='outside') {
state.selected=null;renderDetail();setCameraMode('outside');
} else if (nextScope) {
enterScope(nextScope);
} else if (isShell(selectedBox)) {
enterScope(selectedBox);
} else if (state.cameraMode!=='inside') {
enterScope(scene.boxes.find(box => box.id==='chassis'));
} else if (selectedBox) {
setCameraMode('inside',selectedBox);
}
}));
document.getElementById('camera-back').addEventListener('click',leaveScope);
document.getElementById('focus-selected').addEventListener('click',() => {
const selectedBox=scene.boxes.find(box => box.id===state.selected && box.selectable);
if (selectedBox) setCameraMode('inside',selectedBox);
});
document.querySelectorAll('[data-layer]').forEach(button => button.addEventListener('click',() => {
const layer = button.dataset.layer;
if (state.layers.has(layer)) state.layers.delete(layer); else state.layers.add(layer);
button.setAttribute('aria-pressed',String(state.layers.has(layer)));
if (state.selected && !visibleBox(scene.boxes.find(box => box.id === state.selected))) state.selected=null;
renderDetail(); draw();
}));
document.getElementById('reset-view').addEventListener('click',() => {
state.selected=null;setCameraMode('outside');renderDetail();
});
function visibleBox(box) {
if (!box) return false;
if (state.cameraMode==='outside') {
return box.kind==='chassis' || box.kind==='numa' || state.layers.has(box.layer);
}
const scopeId=activeScopeId() || 'chassis';
if (box.id===scopeId) return box.kind!=='chassis';
if (!isDescendant(box.id,scopeId)) return false;
return box.kind==='numa' || state.layers.has(box.layer);
}
function visibleEdge(edge) {
const from=scene.boxes.find(box => box.id===edge.from);
const to=scene.boxes.find(box => box.id===edge.to);
return state.layers.has(edge.layer) && visibleBox(from) && visibleBox(to);
}
function cameraPoint(point) {
const cy=Math.cos(state.yaw), sy=Math.sin(state.yaw);
const cp=Math.cos(state.pitch), sp=Math.sin(state.pitch);
const translated={
x:point.x-state.target.x,y:point.y-state.target.y,z:point.z-state.target.z
};
const x=translated.x*cy-translated.z*sy;
const z=translated.x*sy+translated.z*cy;
return {x,y:translated.y*cp-z*sp,z:translated.y*sp+z*cp};
}
function project(point,scale) {
const p=cameraPoint(point);
return {x:canvas.clientWidth/2+p.x*scale,y:canvas.clientHeight*.58-p.y*scale,z:p.z};
}
function corners(box) {
const {position:p,size:s}=box;
const points=[];
[-1,1].forEach(dx => [-1,1].forEach(dy => [-1,1].forEach(dz => points.push({x:p.x+dx*s.x/2,y:p.y+dy*s.y/2,z:p.z+dz*s.z/2}))));
return points;
}
const faceIndexes=[[0,1,3,2],[4,6,7,5],[0,4,5,1],[2,3,7,6],[0,2,6,4],[1,5,7,3]];
function hexToRgba(hex,alpha) {
const value=hex.replace('#','');
const number=parseInt(value.length===3?value.split('').map(x=>x+x).join(''):value,16);
return `rgba(${(number>>16)&255},${(number>>8)&255},${number&255},${alpha})`;
}
function boxCenter(id) {
const box=scene.boxes.find(item => item.id===id);
return box?.position || null;
}
function relationColor(edge) {
if (edge.meta?.relation === 'SYS') return '#ff6b6b';
return {
nvlink:'#b8ff56',gdr:'#51d9d2',pcie:'#ffbd4a',affinity:'#b592ff',
control:'#8fa99e','bond-member':'#79f4ee'
}[edge.type] || '#8fa99e';
}
function roundedRectPath(x,y,width,height,radius) {
const r=Math.min(radius,width/2,height/2);
ctx.beginPath();
ctx.moveTo(x+r,y);ctx.lineTo(x+width-r,y);ctx.quadraticCurveTo(x+width,y,x+width,y+r);
ctx.lineTo(x+width,y+height-r);ctx.quadraticCurveTo(x+width,y+height,x+width-r,y+height);
ctx.lineTo(x+r,y+height);ctx.quadraticCurveTo(x,y+height,x,y+height-r);
ctx.lineTo(x,y+r);ctx.quadraticCurveTo(x,y,x+r,y);ctx.closePath();
}
function drawLabelChip(box,point,dimmed) {
const isDomain=box.kind==='numa';
const mainFont=isDomain?'900 12px ui-sans-serif,sans-serif':'900 10px ui-sans-serif,sans-serif';
const subFont='800 8px ui-monospace,monospace';
ctx.save();ctx.globalAlpha=dimmed ? .34 : 1;ctx.textAlign='center';
ctx.font=mainFont;
const mainWidth=ctx.measureText(box.label).width;
ctx.font=subFont;
const subWidth=box.sublabel ? ctx.measureText(box.sublabel).width : 0;
const width=Math.max(mainWidth,subWidth)+14;
const height=box.sublabel ? 29 : 18;
const left=point.x-width/2,top=point.y-13;
roundedRectPath(left,top,width,height,5);
ctx.fillStyle=isDomain ? hexToRgba(box.color,.96) : 'rgba(238,249,244,.96)';
ctx.fill();
ctx.strokeStyle=hexToRgba(box.color,.9);ctx.lineWidth=1.2;ctx.stroke();
ctx.fillStyle='#06100c';ctx.font=mainFont;
ctx.fillText(box.label,point.x,top+12);
if (box.sublabel) {
ctx.fillStyle='#17251f';ctx.font=subFont;
ctx.fillText(box.sublabel,point.x,top+23);
}
ctx.restore();
}
function drawEdgeTag(text,x,y,color) {
ctx.save();ctx.font='900 8px ui-monospace,monospace';ctx.textAlign='center';
const width=ctx.measureText(text).width+9;
roundedRectPath(x-width/2,y-8,width,14,4);
ctx.fillStyle=color;ctx.fill();
ctx.fillStyle='#06100c';ctx.fillText(text,x,y+2);
ctx.restore();
}
function drawEdge(edge,scale,selected,overlay=false) {
const from=boxCenter(edge.from),to=boxCenter(edge.to);
if (!from || !to) return;
const a=project(from,scale),b=project(to,scale);
const related=!selected || edge.from===selected || edge.to===selected;
const color=relationColor(edge);
ctx.save();
ctx.strokeStyle=hexToRgba(color,related ? (overlay ? 1 : .85) : (overlay ? .24 : .13));
ctx.lineWidth=related
? (overlay ? 4.8 : edge.type==='gdr' ? 3 : edge.type==='bond-member' ? 2 : 1.3)
: (overlay ? 2 : 1);
if (overlay) {
ctx.shadowColor=color;ctx.shadowBlur=related ? 12 : 3;
ctx.lineCap='round';
}
if (edge.type==='affinity') ctx.setLineDash([5,5]);
if (edge.meta?.relation==='SYS') ctx.setLineDash([8,4]);
ctx.beginPath();ctx.moveTo(a.x,a.y);ctx.lineTo(b.x,b.y);ctx.stroke();
ctx.shadowBlur=0;
if (overlay) {
[a,b].forEach(point => {
ctx.beginPath();ctx.arc(point.x,point.y,related ? 4.5 : 3,0,Math.PI*2);
ctx.fillStyle=hexToRgba(color,related ? 1 : .3);ctx.fill();
ctx.strokeStyle='#06100c';ctx.lineWidth=1.5;ctx.stroke();
});
if (related) {
const markerX=a.x+(b.x-a.x)*.18,markerY=a.y+(b.y-a.y)*.18;
drawEdgeTag(`GPU ${edge.meta?.gpuIndex ?? '?'}`,markerX,markerY,color);
}
} else if (related && selected && Math.hypot(a.x-b.x,a.y-b.y)>70) {
drawEdgeTag(edge.label,(a.x+b.x)/2,(a.y+b.y)/2-4,color);
}
ctx.restore();
}
function drawGrid(scale) {
ctx.save(); ctx.strokeStyle='rgba(122,164,147,.08)'; ctx.lineWidth=1;
for (let value=-24; value<=24; value+=2) {
const a=project({x:value,y:0,z:-12},scale),b=project({x:value,y:0,z:12},scale);
const c=project({x:-24,y:0,z:value},scale),d=project({x:24,y:0,z:value},scale);
ctx.beginPath();ctx.moveTo(a.x,a.y);ctx.lineTo(b.x,b.y);ctx.stroke();
ctx.beginPath();ctx.moveTo(c.x,c.y);ctx.lineTo(d.x,d.y);ctx.stroke();
}
ctx.restore();
}
function draw() {
const width=canvas.clientWidth,height=canvas.clientHeight;
const dpr=Math.min(2,window.devicePixelRatio || 1);
if (canvas.width!==Math.round(width*dpr) || canvas.height!==Math.round(height*dpr)) {
canvas.width=Math.round(width*dpr);canvas.height=Math.round(height*dpr);
}
ctx.setTransform(dpr,0,0,dpr,0,0);
ctx.clearRect(0,0,width,height);
const domainCount=state.focus==='all'?Math.max(1,numaIds.length):1;
const scale=Math.max(18,Math.min(width/(domainCount*22),height/23))*state.zoom;
drawGrid(scale);
const selected=state.selected;
scene.edges.filter(edge => visibleEdge(edge) && edge.layer!=='fabric').forEach(edge => {
drawEdge(edge,scale,selected);
});
const faces=[];
const labels=[];
hitTargets=[];
scene.boxes.filter(visibleBox).forEach(box => {
const projected=corners(box).map(point => project(point,scale));
const alpha = box.opacity ?? 1;
faceIndexes.forEach((indexes,faceIndex) => {
const points=indexes.map(index => projected[index]);
faces.push({box,points,depth:points.reduce((sum,p)=>sum+p.z,0)/points.length,faceIndex,alpha});
});
const xs=projected.map(p=>p.x),ys=projected.map(p=>p.y);
hitTargets.push({box,rect:{left:Math.min(...xs),right:Math.max(...xs),top:Math.min(...ys),bottom:Math.max(...ys)},depth:cameraPoint(box.position).z});
labels.push({
box,
point:project({
x:box.position.x,
y:box.position.y+box.size.y/2+(box.labelOffset ?? .18),
z:box.position.z
},scale)
});
});
faces.sort((a,b) => a.depth-b.depth).forEach(face => {
const isSelected=face.box.id===selected;
const dimmed=selected && !isSelected && face.box.kind!=='numa' && face.box.kind!=='chassis';
const internalBoundary=state.cameraMode==='inside' && (
face.box.kind==='numa' || face.box.kind==='pcie-group' || face.box.kind==='bond'
);
const baseAlpha=internalBoundary
? face.alpha*.22
: face.box.wire ? face.alpha : Math.min(.86,face.alpha*(face.faceIndex===4?1:.72));
ctx.save();
ctx.fillStyle=hexToRgba(face.box.color,dimmed?baseAlpha*.16:baseAlpha);
ctx.strokeStyle=hexToRgba(face.box.color,isSelected ? 1 : internalBoundary ? .26 : face.box.wire ? .55 : .75);
ctx.lineWidth=isSelected ? 2.8 : face.box.wire ? 1.1 : 1;
if (face.box.wire) ctx.setLineDash([7,5]);
ctx.beginPath();face.points.forEach((point,index) => index?ctx.lineTo(point.x,point.y):ctx.moveTo(point.x,point.y));ctx.closePath();
ctx.fill();ctx.stroke();ctx.restore();
});
scene.edges.filter(edge => visibleEdge(edge) && edge.layer==='fabric').forEach(edge => {
drawEdge(edge,scale,selected,true);
});
labels.sort((a,b)=>a.point.z-b.point.z).forEach(({box,point}) => {
if (box.kind==='chassis') return;
if (state.cameraMode==='inside' && box.kind==='numa') return;
const dimmed=selected && box.id!==selected && box.kind!=='numa';
drawLabelChip(box,point,dimmed);
});
}
function renderDetail() {
const root=document.getElementById('selection-detail');
const box=scene.boxes.find(item => item.id===state.selected);
if (!box) {
root.innerHTML=`<span class="detail-kind">MACHINE OVERVIEW</span><h3>${escapeHtml(system.hostname || 'GPU Node')}</h3><dl>
<div><dt>CPU Model</dt><dd>${escapeHtml(system.cpu_model || '—')}</dd></div>
<div><dt>CPU Topology</dt><dd>${escapeHtml(`${system.cpu_sockets || '—'} sockets · ${system.cpu_logical || '—'} logical`)}</dd></div>
<div><dt>GPU</dt><dd>${escapeHtml(`${gpus.length} × ${gpus[0]?.name || '—'}`)}</dd></div>
<div><dt>Virtualization</dt><dd>${escapeHtml(system.virtualization || 'bare metal / unknown')}</dd></div>
<div><dt>PCI Scope</dt><dd>${escapeHtml(data.pci_fabric?.scope || 'guest-visible')}</dd></div>
</dl>`;
return;
}
const meta=box.meta || {};
const fields=Object.entries(meta).map(([key,value]) => `<div><dt>${escapeHtml(key)}</dt><dd>${escapeHtml(value)}</dd></div>`).join('');
root.innerHTML=`<div class="detail-head"><div><span class="detail-kind">${escapeHtml(box.kind.toUpperCase())}</span><h3>${escapeHtml(box.label)}</h3></div><span class="detail-status">${escapeHtml(box.sublabel || '')}</span></div><dl>${fields || '<div><dt>Info</dt><dd>无附加字段</dd></div>'}</dl>`;
}
function pickTarget(x,y) {
return hitTargets
.filter(hit => hit.box.selectable && x>=hit.rect.left && x<=hit.rect.right && y>=hit.rect.top && y<=hit.rect.bottom)
.sort((a,b) => {
const areaA=(a.rect.right-a.rect.left)*(a.rect.bottom-a.rect.top);
const areaB=(b.rect.right-b.rect.left)*(b.rect.bottom-b.rect.top);
return areaA-areaB || b.depth-a.depth;
})[0];
}
canvas.addEventListener('pointerdown',event => {
cameraAnimation+=1;
canvas.setPointerCapture(event.pointerId);
pointer={
id:event.pointerId,x:event.clientX,y:event.clientY,startX:event.clientX,startY:event.clientY,
yaw:state.yaw,pitch:state.pitch,target:{...state.target},pan:event.shiftKey || event.button===1
};
});
canvas.addEventListener('pointermove',event => {
if (!pointer || pointer.id!==event.pointerId) return;
if (pointer.pan) {
const factor=.022/state.zoom;
state.target.x=pointer.target.x-(event.clientX-pointer.x)*factor;
state.target.y=pointer.target.y+(event.clientY-pointer.y)*factor;
} else {
state.yaw=pointer.yaw+(event.clientX-pointer.x)*.008;
state.pitch=Math.max(-.15,Math.min(1.1,pointer.pitch+(event.clientY-pointer.y)*.006));
}
draw();
});
canvas.addEventListener('pointerup',event => {
if (!pointer || pointer.id!==event.pointerId) return;
const moved=Math.hypot(event.clientX-pointer.startX,event.clientY-pointer.startY);
if (moved<6) {
const rect=canvas.getBoundingClientRect(),x=event.clientX-rect.left,y=event.clientY-rect.top;
const target=pickTarget(x,y);
state.selected=target?.box.id || null;renderDetail();updateCameraControls();draw();
}
pointer=null;
});
canvas.addEventListener('dblclick',event => {
const rect=canvas.getBoundingClientRect(),x=event.clientX-rect.left,y=event.clientY-rect.top;
const target=pickTarget(x,y);
if (!target) return;
state.selected=target.box.id;
const nextScope=nextScopeToward(target.box);
if (nextScope) enterScope(nextScope);
else if (isShell(target.box)) enterScope(target.box);
else { renderDetail();setCameraMode('inside',target.box); }
});
canvas.addEventListener('wheel',event => {
event.preventDefault();
cameraAnimation+=1;
const limits=state.cameraMode==='inside' ? [.72,3.8] : [.55,2.3];
state.zoom=Math.max(limits[0],Math.min(limits[1],state.zoom*(event.deltaY>0 ? .92 : 1.08)));draw();
},{passive:false});
canvas.addEventListener('keydown',event => {
const key=event.key;
if (['ArrowLeft','ArrowRight','ArrowUp','ArrowDown'].includes(key)) {
event.preventDefault();
cameraAnimation+=1;
if (key==='ArrowLeft') state.yaw-=.08;if (key==='ArrowRight') state.yaw+=.08;
if (key==='ArrowUp') state.pitch=Math.min(1.1,state.pitch+.06);if (key==='ArrowDown') state.pitch=Math.max(-.15,state.pitch-.06);
draw();
}
if (key==='Escape') {
event.preventDefault();state.selected=null;renderDetail();setCameraMode('outside');
}
});
new ResizeObserver(draw).observe(canvas);
buildViewControls();buildScene();updateCameraControls();draw();
})();
</script>
</body>
</html>

View File

@ -1,133 +0,0 @@
{
"schema_version": 1,
"generated_at": "2026-07-25T09:45:54+08:00",
"collector_version": "1.0.0",
"system": {
"hostname": "root-2",
"os": "Ubuntu 24.04.1 LTS",
"kernel": "6.8.0-71-generic",
"architecture": "x86_64",
"manufacturer": "Baidu Cloud",
"product": "Baidu Cloud BCC",
"product_version": "6.10.0",
"virtualization": "kvm",
"cpu_model": "Intel(R) Xeon(R) 6787P",
"numa_nodes": 2,
"cpu_logical": 332,
"cpu_sockets": 2,
"cpu_cores_per_socket": 83,
"cpu_physical_cores": 166,
"cpu_threads_per_core": 2,
"memory_total_kib": 4227858432,
"memory_available_kib": 3549581076,
"memory_used_kib": 678277356
},
"software": {
"nvidia_driver": "595.71.05",
"cuda_toolkit": "12.8",
"cuda_note": "Toolkit selected by nvcc; application/container runtime may differ."
},
"gpus": [
{"index": 0, "name": "NVIDIA B30Z", "uuid": "GPU-0", "bdf": "63:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 34720, "temperature_c": 48, "utilization_pct": 12, "pstate": "P0", "power_w": 312, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 0},
{"index": 1, "name": "NVIDIA B30Z", "uuid": "GPU-1", "bdf": "67:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 50812, "temperature_c": 51, "utilization_pct": 18, "pstate": "P0", "power_w": 326, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 0},
{"index": 2, "name": "NVIDIA B30Z", "uuid": "GPU-2", "bdf": "6b:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 19236, "temperature_c": 47, "utilization_pct": 8, "pstate": "P0", "power_w": 296, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 0},
{"index": 3, "name": "NVIDIA B30Z", "uuid": "GPU-3", "bdf": "6f:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 62840, "temperature_c": 54, "utilization_pct": 23, "pstate": "P0", "power_w": 341, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 0},
{"index": 4, "name": "NVIDIA B30Z", "uuid": "GPU-4", "bdf": "a3:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 41560, "temperature_c": 50, "utilization_pct": 15, "pstate": "P0", "power_w": 318, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 1},
{"index": 5, "name": "NVIDIA B30Z", "uuid": "GPU-5", "bdf": "a7:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 28410, "temperature_c": 49, "utilization_pct": 10, "pstate": "P0", "power_w": 307, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 1},
{"index": 6, "name": "NVIDIA B30Z", "uuid": "GPU-6", "bdf": "ab:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 73320, "temperature_c": 56, "utilization_pct": 29, "pstate": "P0", "power_w": 359, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 1},
{"index": 7, "name": "NVIDIA B30Z", "uuid": "GPU-7", "bdf": "af:00.0", "driver": "595.71.05", "memory_total_mib": 285184, "memory_used_mib": 22350, "temperature_c": 48, "utilization_pct": 9, "pstate": "P0", "power_w": 301, "power_limit_w": 1100, "power_default_limit_w": 1100, "power_min_limit_w": 200, "power_max_limit_w": 1100, "numa": 1}
],
"topology": {
"rows": {},
"nic_legend": {
"NIC1": "mlx5_bond_0", "NIC2": "mlx5_bond_1", "NIC3": "mlx5_bond_2", "NIC4": "mlx5_bond_3",
"NIC5": "mlx5_bond_4", "NIC6": "mlx5_bond_5", "NIC7": "mlx5_bond_6", "NIC8": "mlx5_bond_7"
},
"gpu_nic": {
"GPU0": [{"slot": "NIC1", "rdma": "mlx5_bond_0", "relation": "PIX"}],
"GPU1": [{"slot": "NIC2", "rdma": "mlx5_bond_1", "relation": "PIX"}],
"GPU2": [{"slot": "NIC3", "rdma": "mlx5_bond_2", "relation": "PIX"}],
"GPU3": [{"slot": "NIC4", "rdma": "mlx5_bond_3", "relation": "PIX"}],
"GPU4": [{"slot": "NIC5", "rdma": "mlx5_bond_4", "relation": "PIX"}],
"GPU5": [{"slot": "NIC6", "rdma": "mlx5_bond_5", "relation": "PIX"}],
"GPU6": [{"slot": "NIC7", "rdma": "mlx5_bond_6", "relation": "PIX"}],
"GPU7": [{"slot": "NIC8", "rdma": "mlx5_bond_7", "relation": "PIX"}]
},
"gpu_affinity": {
"GPU0": {"cpu_affinity": "0-165", "numa_affinity": "0", "gpu_numa_id": "N/A"},
"GPU1": {"cpu_affinity": "0-165", "numa_affinity": "0", "gpu_numa_id": "N/A"},
"GPU2": {"cpu_affinity": "0-165", "numa_affinity": "0", "gpu_numa_id": "N/A"},
"GPU3": {"cpu_affinity": "0-165", "numa_affinity": "0", "gpu_numa_id": "N/A"},
"GPU4": {"cpu_affinity": "166-331", "numa_affinity": "1", "gpu_numa_id": "N/A"},
"GPU5": {"cpu_affinity": "166-331", "numa_affinity": "1", "gpu_numa_id": "N/A"},
"GPU6": {"cpu_affinity": "166-331", "numa_affinity": "1", "gpu_numa_id": "N/A"},
"GPU7": {"cpu_affinity": "166-331", "numa_affinity": "1", "gpu_numa_id": "N/A"}
}
},
"pci_fabric": {
"scope": "guest-visible",
"virtualization": "passthrough-or-virtual-pci",
"bridges": [
{"bdf": "61:00.0", "parent_bdf": "60:00.0", "numa": 0, "description": "PCI bridge", "children": ["63:00.0", "64:00.0"]},
{"bdf": "a1:00.0", "parent_bdf": "a0:00.0", "numa": 1, "description": "PCI bridge", "children": ["a3:00.0", "a4:00.0"]}
],
"endpoints": [
{"kind": "gpu", "label": "GPU0", "bdf": "63:00.0", "numa": 0, "path": ["60:00.0", "61:00.0", "63:00.0"], "description": "NVIDIA GPU"},
{"kind": "nic", "label": "eth1", "bdf": "64:00.0", "numa": 0, "path": ["60:00.0", "61:00.0", "64:00.0"], "description": "Ethernet controller"},
{"kind": "gpu", "label": "GPU4", "bdf": "a3:00.0", "numa": 1, "path": ["a0:00.0", "a1:00.0", "a3:00.0"], "description": "NVIDIA GPU"},
{"kind": "nic", "label": "eth9", "bdf": "a4:00.0", "numa": 1, "path": ["a0:00.0", "a1:00.0", "a4:00.0"], "description": "Ethernet controller"}
]
},
"nvlink": {
"fabric_type": "NVSwitch Fabric",
"per_gpu": {
"GPU0": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
"GPU1": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
"GPU2": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
"GPU3": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
"GPU4": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
"GPU5": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
"GPU6": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125],
"GPU7": [53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125, 53.125]
},
"active_links": 144,
"links_per_gpu": 18,
"speed_gbs": 53.125
},
"nics": [
{
"name": "eth0",
"driver": "mlx5_core",
"bdf": "02:00.0",
"numa": 0,
"speed_mbps": 200000,
"state": "up",
"carrier": 1,
"mac": "fa:20:20:2e:4a:3f",
"addresses": ["192.0.2.10/24"],
"default_route": true
}
],
"bonds": [
{"name": "bond0", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_0", "slaves": [{"name": "eth1", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "64:00.0", "numa": 0}, {"name": "eth2", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "64:00.1", "numa": 0}]},
{"name": "bond1", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_1", "slaves": [{"name": "eth3", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "68:00.0", "numa": 0}, {"name": "eth4", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "68:00.1", "numa": 0}]},
{"name": "bond2", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_2", "slaves": [{"name": "eth5", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "6c:00.0", "numa": 0}, {"name": "eth6", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "6c:00.1", "numa": 0}]},
{"name": "bond3", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_3", "slaves": [{"name": "eth7", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "70:00.0", "numa": 0}, {"name": "eth8", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "70:00.1", "numa": 0}]},
{"name": "bond4", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_4", "slaves": [{"name": "eth9", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "a4:00.0", "numa": 1}, {"name": "eth10", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "a4:00.1", "numa": 1}]},
{"name": "bond5", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_5", "slaves": [{"name": "eth11", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "a8:00.0", "numa": 1}, {"name": "eth12", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "a8:00.1", "numa": 1}]},
{"name": "bond6", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_6", "slaves": [{"name": "eth13", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "ac:00.0", "numa": 1}, {"name": "eth14", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "ac:00.1", "numa": 1}]},
{"name": "bond7", "mode": "IEEE 802.3ad Dynamic link aggregation", "hash_policy": "layer3+4 (1)", "status": "up", "rdma": "mlx5_bond_7", "slaves": [{"name": "eth15", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "b0:00.0", "numa": 1}, {"name": "eth16", "status": "up", "speed_mbps": 400000, "failures": 0, "bdf": "b0:00.1", "numa": 1}]}
],
"ib_devices": [
{"name": "mlx5_17", "ca_type": "MT4129", "firmware": "28.46.5500", "state": "Active", "physical_state": "LinkUp", "rate_gbps": 100, "lid": "1", "link_layer": "InfiniBand", "bdf": "b3:00.0", "numa": 1},
{"name": "mlx5_18", "ca_type": "MT4129", "firmware": "28.46.5500", "state": "Active", "physical_state": "LinkUp", "rate_gbps": 100, "lid": "4", "link_layer": "InfiniBand", "bdf": "b4:00.0", "numa": 1},
{"name": "mlx5_19", "ca_type": "MT4129", "firmware": "28.46.5500", "state": "Active", "physical_state": "LinkUp", "rate_gbps": 100, "lid": "5", "link_layer": "InfiniBand", "bdf": "b5:00.0", "numa": 1},
{"name": "mlx5_20", "ca_type": "MT4129", "firmware": "28.46.5500", "state": "Active", "physical_state": "LinkUp", "rate_gbps": 100, "lid": "6", "link_layer": "InfiniBand", "bdf": "b6:00.0", "numa": 1}
],
"bmc": {
"detected": false,
"note": "独立 BMC 带外口未向当前 OS/IPMI 通道暴露"
},
"rdma_map": {},
"notes": []
}

View File

@ -1,430 +0,0 @@
import io
import json
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
import gpu_node_profile as profile
ROOT = Path(__file__).resolve().parents[1]
FIXTURE = ROOT / "tests" / "fixtures" / "b300_snapshot.json"
class ParserTests(unittest.TestCase):
def test_normalize_bdf_uses_endpoint_from_sysfs_path(self):
path = (
"/sys/devices/pci0000:a0/0000:a0:00.4/0000:b1:00.0/"
"0000:b2:00.0/0000:b3:00.0"
)
self.assertEqual(profile.normalize_bdf(path), "b3:00.0")
def test_cuda_build_version_is_concise(self):
self.assertEqual(profile.concise_version("12.8.20250303"), "12.8")
def test_parse_topology_and_mapping(self):
text = """
GPU0 GPU1 NIC0 NIC1 CPU Affinity NUMA Affinity
GPU0 X NV18 PIX SYS 0-165 0
GPU1 NV18 X SYS PIX 166-331 1
NIC Legend:
NIC0: mlx5_bond_0
NIC1: mlx5_bond_1
"""
parsed = profile.parse_topology(text)
self.assertEqual(parsed["nic_legend"]["NIC0"], "mlx5_bond_0")
self.assertEqual(parsed["gpu_nic"]["GPU0"][0]["relation"], "PIX")
self.assertEqual(
parsed["gpu_affinity"]["GPU0"]["cpu_affinity"], "0-165"
)
self.assertEqual(
parsed["gpu_affinity"]["GPU1"]["numa_affinity"], "1"
)
self.assertNotIn("CPU", parsed["rows"]["GPU0"])
def test_parse_bond(self):
text = """
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
Slave Interface: eth1
MII Status: up
Speed: 400000 Mbps
Link Failure Count: 0
"""
bond = profile.parse_bond_text("bond0", text)
self.assertEqual(bond["status"], "up")
self.assertEqual(bond["slaves"][0]["speed_mbps"], 400000)
def test_nvlink_fabric_type_does_not_assume_nvswitch(self):
direct = {"rows": {"GPU0": {"GPU1": "NV4"}, "GPU1": {"GPU0": "NV4"}}}
self.assertEqual(
profile.infer_nvlink_fabric_type(direct, 2), "NVLink Fabric"
)
switched = {
"rows": {
f"GPU{source}": {
f"GPU{target}": "NV18"
for target in range(4)
if target != source
}
for source in range(4)
}
}
self.assertEqual(
profile.infer_nvlink_fabric_type(switched, 4), "NVSwitch Fabric"
)
def test_render_is_offline_native_html(self):
snapshot = json.loads(FIXTURE.read_text(encoding="utf-8"))
profile.attach_gpu_bond_mapping(snapshot)
report = profile.render_html(
snapshot, {"gpus": 8, "bonds": 8, "ib": 4}
)
self.assertIn("GPU ↔ RDMA 亲和画像", report)
self.assertIn("系统与软件", report)
self.assertIn("CPU 与内存", report)
self.assertIn("带内与带外管理网络", report)
self.assertIn("eth0", report)
self.assertIn("带外 · BMC", report)
self.assertIn("HEALTH 与验证结果", report)
self.assertIn("CPU / NUMA 亲和", report)
self.assertIn("GPU 功耗遥测", report)
self.assertIn("GPU 瞬时功耗", report)
self.assertIn("2.56 kW / 8.80 kW", report)
self.assertIn("瞬时 / 1.10 kW 上限", report)
self.assertIn(">P0<", report)
self.assertIn("GPU0–3", report)
self.assertIn("推荐 CPU 0-165", report)
self.assertIn("不代表当前训练进程已经绑核", report)
self.assertIn('data-numa-filter="all"', report)
self.assertIn('data-numa-filter="0"', report)
self.assertIn('data-numa-filter="1"', report)
self.assertIn("function setNumaView(value)", report)
self.assertIn(".rail[hidden],.numa-ruler [hidden]", report)
self.assertIn(".ib-card .card-head { align-items:flex-start; flex-wrap:wrap;", report)
self.assertIn("eth0 承载默认路由且链路 Up", report)
self.assertIn("mlx5_bond_7", report)
self.assertIn("NVLINK / NVSWITCH FABRIC", report)
self.assertLess(report.index("系统与软件"), report.index("GPU ↔ RDMA 亲和画像"))
self.assertLess(report.index("带内与带外管理网络"), report.index("HEALTH 与验证结果"))
self.assertNotIn("mermaid", report.lower())
self.assertNotIn("https://", report.lower())
self.assertNotIn("http://", report.lower())
def test_render_3d_topology_is_offline_and_json_driven(self):
snapshot = json.loads(FIXTURE.read_text(encoding="utf-8"))
profile.attach_gpu_bond_mapping(snapshot)
report = profile.render_topology_3d_html(snapshot)
self.assertIn("GPU Machine 3D CT", report)
self.assertIn('id="machine-data"', report)
self.assertIn('id="topology-canvas"', report)
self.assertIn("function buildScene()", report)
self.assertIn("function deepestCommonPath(paths)", report)
self.assertIn('data-camera="inside"', report)
self.assertIn("function setCameraMode(mode,box=null,animate=true)", report)
self.assertIn("双击壳体逐层进入", report)
self.assertIn("state.cameraMode==='inside'", report)
self.assertIn("kind:'bond'", report)
self.assertIn("kind:'roce-port'", report)
self.assertIn("GPU endpoints only · not NIC", report)
self.assertIn("function drawEdge(edge,scale,selected,overlay=false)", report)
self.assertIn("function scopePath(box)", report)
self.assertIn("function nextScopeToward(box)", report)
self.assertIn("function enterScope(box,animate=true)", report)
self.assertIn('id="camera-back"', report)
self.assertIn("parentId:`pair-${gpu.index}`", report)
self.assertIn("PCIe Root Complex", report)
self.assertIn("GDR candidate", report)
self.assertIn('"scope": "guest-visible"', report)
self.assertNotIn("https://", report.lower())
self.assertNotIn("http://", report.lower())
def test_html_templates_are_separate_source_files(self):
operations = profile.load_html_template("gpu_node_profile.html")
topology = profile.load_html_template("gpu_topology_3d.html")
self.assertIn("@@BODY@@", operations)
self.assertIn("@@MACHINE_DATA@@", topology)
self.assertNotIn("https://", operations.lower())
self.assertNotIn("https://", topology.lower())
def test_build_pci_fabric_preserves_bridge_hierarchy(self):
gpus = [
{
"index": 0,
"bdf": "63:00.0",
"numa": 0,
"pci_path": ["60:00.0", "61:00.0", "63:00.0"],
}
]
nics = [
{
"name": "eth1",
"bdf": "64:00.0",
"numa": 0,
"pci_path": ["60:00.0", "61:00.0", "64:00.0"],
}
]
with patch.object(profile, "pci_numa", return_value=0):
fabric = profile.build_pci_fabric(gpus, nics, [], {})
self.assertEqual(fabric["scope"], "guest-visible")
bridge = next(item for item in fabric["bridges"] if item["bdf"] == "61:00.0")
self.assertEqual(bridge["children"], ["63:00.0", "64:00.0"])
def test_standalone_roce_endpoints_map_uniquely_to_gpus(self):
nics = [
{
"name": "eth0",
"bdf": "0b:00.0",
"numa": 0,
"pci_path": ["08:00.0", "0b:00.0"],
"speed_mbps": 100000,
"state": "up",
"carrier": 1,
"default_route": True,
"is_bond": False,
},
{
"name": "eth1",
"bdf": "5a:00.0",
"numa": 0,
"pci_path": ["4c:00.0", "54:00.0", "5a:00.0"],
"speed_mbps": 100000,
"state": "up",
"carrier": 1,
"default_route": False,
"is_bond": False,
},
{
"name": "eth2",
"bdf": "5a:00.1",
"numa": 0,
"pci_path": ["4c:00.0", "54:00.0", "5a:00.1"],
"speed_mbps": 100000,
"state": "up",
"carrier": 1,
"default_route": False,
"is_bond": False,
},
]
rdma_map = {"mlx5_0": "eth0", "mlx5_1": "eth1", "mlx5_2": "eth2"}
endpoints = profile.build_network_endpoints(
nics,
[],
[],
rdma_map,
{
"mlx5_1": {"state": "ACTIVE"},
"mlx5_2": {"state": "ACTIVE"},
},
)
self.assertEqual([item["name"] for item in endpoints], ["eth1", "eth2"])
self.assertTrue(all(item["type"] == "standalone" for item in endpoints))
snapshot = {
"gpus": [
{"index": 0, "pci_path": ["4c:00.0", "53:00.0"], "numa": 0},
{"index": 1, "pci_path": ["4c:00.0", "58:00.0"], "numa": 0},
],
"nics": nics,
"bonds": [],
"ib_devices": [],
"rdma_map": rdma_map,
"network_endpoints": endpoints,
"topology": {
"gpu_affinity": {},
"gpu_nic": {
"GPU0": [
{"rdma": "mlx5_1", "relation": "PXB"},
{"rdma": "mlx5_2", "relation": "PXB"},
],
"GPU1": [
{"rdma": "mlx5_1", "relation": "PXB"},
{"rdma": "mlx5_2", "relation": "PXB"},
],
},
},
}
profile.attach_gpu_network_mapping(snapshot)
primary_ids = {
gpu.get("primary_network_id") for gpu in snapshot["gpus"]
}
self.assertEqual(primary_ids, {"rdma:mlx5_1", "rdma:mlx5_2"})
self.assertTrue(
all(gpu.get("network_relation") == "PXB" for gpu in snapshot["gpus"])
)
health = profile.evaluate(
snapshot, {"gpus": 2, "bonds": None, "ib": None}
)
affinity = next(
item for item in health["checks"] if item["title"] == "GPU ↔ RDMA 亲和"
)
self.assertEqual(affinity["level"], "ok")
self.assertIn("PXB 2", affinity["detail"])
operations = profile.render_html(
snapshot, {"gpus": 2, "bonds": None, "ib": None}
)
topology = profile.render_topology_3d_html(snapshot)
self.assertEqual(operations.count("ROCE · DIRECT"), 2)
self.assertNotIn("未发现对应网络路径", operations)
self.assertIn("Mode:'Standalone RDMA'", topology)
self.assertIn("endpoint.topology_visibility", topology)
def test_unique_mapping_uses_global_minimum_not_greedy_first_match(self):
endpoints = [
{
"id": "rdma:mlx5_1",
"type": "standalone",
"protocol": "RoCE",
"name": "eth1",
"rdma": "mlx5_1",
"role": "compute",
"members": [{"name": "eth1", "pci_path": []}],
},
{
"id": "rdma:mlx5_2",
"type": "standalone",
"protocol": "RoCE",
"name": "eth2",
"rdma": "mlx5_2",
"role": "compute",
"members": [{"name": "eth2", "pci_path": []}],
},
]
snapshot = {
"gpus": [
{"index": 0, "pci_path": [], "numa": 0},
{"index": 1, "pci_path": [], "numa": 0},
],
"bonds": [],
"network_endpoints": endpoints,
"topology": {
"gpu_affinity": {},
"gpu_nic": {
"GPU0": [
{"rdma": "mlx5_1", "relation": "PIX"},
{"rdma": "mlx5_2", "relation": "PIX"},
],
"GPU1": [
{"rdma": "mlx5_1", "relation": "PIX"},
{"rdma": "mlx5_2", "relation": "PHB"},
],
},
},
}
profile.attach_gpu_network_mapping(snapshot)
self.assertEqual(snapshot["gpus"][0]["network"], "eth2")
self.assertEqual(snapshot["gpus"][1]["network"], "eth1")
self.assertTrue(
all(gpu["network_relation"] == "PIX" for gpu in snapshot["gpus"])
)
def test_non_rdma_bond_is_not_an_accelerator_endpoint(self):
management_bond = {
"name": "bond-mgmt",
"rdma": "",
"status": "up",
"slaves": [{"name": "eno1", "status": "up"}],
}
endpoints = profile.build_network_endpoints(
[],
[management_bond],
[],
{},
)
self.assertEqual(endpoints, [])
health = profile.evaluate(
{
"gpus": [],
"bonds": [management_bond],
"ib_devices": [],
"network_endpoints": [],
"topology": {},
},
{"gpus": None, "bonds": 0, "ib": None},
)
self.assertFalse(
any(item["title"] == "Bond 数量不符合预期" for item in health["checks"])
)
def test_power_telemetry_health_does_not_flag_p0_as_an_error(self):
snapshot = json.loads(FIXTURE.read_text(encoding="utf-8"))
profile.attach_gpu_bond_mapping(snapshot)
health = profile.evaluate(snapshot, {"gpus": 8, "bonds": 8, "ib": 4})
power_check = next(
item for item in health["checks"] if item["title"] == "GPU 功耗遥测"
)
self.assertEqual(power_check["level"], "ok")
self.assertIn("P0 8/8", power_check["detail"])
def test_collect_gpus_keeps_power_when_optional_range_is_unsupported(self):
core = (
"0, NVIDIA Test, GPU-0, 00000000:63:00.0, 595.71.05, "
"275040, 1024, 42, 75"
)
power = "0, P0, 875.5, 1100"
with (
patch.object(profile, "run", side_effect=[core, power, ""]),
patch.object(profile, "pci_numa", return_value=0),
):
gpus = profile.collect_gpus()
self.assertEqual(len(gpus), 1)
self.assertEqual(gpus[0]["pstate"], "P0")
self.assertEqual(gpus[0]["power_w"], 875.5)
self.assertEqual(gpus[0]["power_limit_w"], 1100)
self.assertNotIn("power_max_limit_w", gpus[0])
def test_incomplete_power_telemetry_is_a_warning(self):
snapshot = json.loads(FIXTURE.read_text(encoding="utf-8"))
snapshot["gpus"][0].pop("power_limit_w")
profile.attach_gpu_bond_mapping(snapshot)
health = profile.evaluate(snapshot, {"gpus": 8, "bonds": 8, "ib": 4})
power_check = next(
item for item in health["checks"] if item["title"] == "GPU功耗信息不完整"
)
self.assertEqual(power_check["level"], "warning")
report = profile.render_html(snapshot, {"gpus": 8, "bonds": 8, "ib": 4})
self.assertIn("遥测不完整", report)
def test_cli_renders_snapshot(self):
with tempfile.TemporaryDirectory() as directory:
output = Path(directory) / "profile.html"
topology_output = Path(directory) / "topology-3d.html"
result = profile.main(
[
"--snapshot-in",
str(FIXTURE),
"--output",
str(output),
"--topology-output",
str(topology_output),
"--expected-gpus",
"8",
"--expected-bonds",
"8",
"--expected-ib",
"4",
]
)
self.assertEqual(result, 0)
self.assertTrue(output.exists())
self.assertGreater(output.stat().st_size, 30_000)
self.assertTrue(topology_output.exists())
self.assertIn(
'id="topology-canvas"',
topology_output.read_text(encoding="utf-8"),
)
def test_cli_can_emit_structured_snapshot_only(self):
buffer = io.StringIO()
with patch.object(profile.sys, "stdout", buffer):
result = profile.main(["--snapshot-in", str(FIXTURE), "--snapshot-stdout"])
self.assertEqual(result, 0)
snapshot = json.loads(buffer.getvalue())
self.assertEqual(snapshot["schema_version"], 1)
self.assertEqual(snapshot["pci_fabric"]["scope"], "guest-visible")
if __name__ == "__main__":
unittest.main()

View File

@ -1,34 +0,0 @@
10.184.0.20 slots=8
10.184.0.21 slots=8
10.184.0.22 slots=8
10.184.0.23 slots=8
10.184.0.24 slots=8
10.184.0.25 slots=8
10.184.0.26 slots=8
10.184.0.27 slots=8
10.184.0.28 slots=8
10.184.0.29 slots=8
10.184.0.30 slots=8
10.184.0.31 slots=8
10.184.0.32 slots=8
10.184.0.33 slots=8
10.184.0.34 slots=8
10.184.0.35 slots=8
10.184.0.36 slots=8
10.184.0.37 slots=8
10.184.0.38 slots=8
10.184.0.39 slots=8
10.184.0.40 slots=8
10.184.0.41 slots=8
10.184.0.42 slots=8
10.184.0.43 slots=8
10.184.0.44 slots=8
10.184.0.45 slots=8
10.184.0.46 slots=8
10.184.0.47 slots=8
10.184.0.48 slots=8
10.184.0.49 slots=8
10.184.0.50 slots=8
10.184.0.51 slots=8
10.184.0.52 slots=8
10.184.0.55 slots=8

View File

@ -1,28 +0,0 @@
#!/bin/bash
# ===== 多机 NCCL 测试环境 — H20-3e 集群 (34台, 8xNDR400 per node) =====
# 主节点: gpu-20 (10.184.0.20), 在此节点 source 后执行 mpirun
# 验证日期: 2026-06-06
# OpenMPI 不在默认 loader 路径,必须显式加
export PATH=/usr/mpi/gcc/openmpi-4.1.7rc1/bin:$PATH
export LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi-4.1.7rc1/lib:$LD_LIBRARY_PATH
# NCCL: 使用 8 张 GPU 直连 IB 网卡 (PIX 级别直连)
# 排除 mlx5_6/7: 虽为 IB Active, 但无 GPU PIX 直连, 为存储 NIC
# 排除 mlx5_bond_0: 25G 以太管理口
# GPU↔NIC 对应: GPU0↔mlx5_3, GPU1↔mlx5_2, GPU2↔mlx5_1, GPU3↔mlx5_0
# GPU4↔mlx5_5, GPU5↔mlx5_4, GPU6↔mlx5_9, GPU7↔mlx5_8
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_8,mlx5_9
export NCCL_IB_DISABLE=0
# NCCL bootstrap 网卡: 节点上有 k8s 虚拟网卡 (lvscare/kube-ipvs0),
# 不指定会选错导致 connection refused, 内网 IP 在 bond0 上
# 注意: 不能用 ^bond (会把 bond0 也排掉), 必须明确写 bond0
export NCCL_SOCKET_IFNAME=bond0
export NCCL_IB_DISABLE=0
export NCCL_DEBUG=${NCCL_DEBUG:-WARN}
# IB 为 InfiniBand link_layer (非 RoCE), GID 一般不用设
# 如报 GID 问题再放开:
# export NCCL_IB_GID_INDEX=3

View File

@ -1,76 +0,0 @@
#!/bin/bash
# ===== 跨交换机多机 NCCL 通信测试 =====
# 适用场景: 验证不同交换机的节点加入集群后跨交换机通信是否正常
# 用法: bash run_cross_switch_test.sh
# 主节点运行: gpu-20 (需与目标节点已打通免密 SSH)
#
# 默认分三个阶段:
# old2node — 旧集群内 2 台基准 (gpu-20/21)
# new2node — 新机 2 台同交换机 (gpu-51/55)
# cross4node — 跨交换机 4 台联合
#
# 新机接入前置步骤:
# 1. 四台节点互相打通 authorized_keys
# 2. 新机上重编 nccl-tests: cd /opt/nccl-tests && make clean &&
# make MPI=1 MPI_HOME=/usr/mpi/gcc/openmpi-4.1.7rc1 CUDA_HOME=/usr/local/cuda -j
set -euo pipefail
source "$(dirname "$0")/nccl_env.sh"
NT=/opt/nccl-tests/build
OUT=~/nccl_kit/results_cross_$(date +%Y%m%d_%H%M%S)
mkdir -p "$OUT"
SUMMARY="$OUT/summary.csv"
echo "test,nodes,op,size,busbw_GBps" > "$SUMMARY"
MPI=(mpirun --allow-run-as-root
--mca plm_rsh_args "-oStrictHostKeyChecking=no"
-x PATH -x LD_LIBRARY_PATH
-x NCCL_IB_HCA -x NCCL_IB_DISABLE
-x NCCL_SOCKET_IFNAME -x NCCL_DEBUG)
run_test() {
local tag="$1" hostfile="$2"
local np=$(( $(wc -l < "$hostfile") * 8 ))
echo ""
echo "======== $tag ($np GPUs) ========"
for op in all_reduce alltoall all_gather reduce_scatter; do
log="$OUT/${tag}_${op}.log"
"${MPI[@]}" -np "$np" -N 8 --hostfile "$hostfile" \
"$NT/${op}_perf" -b 8 -e 4G -f 2 -g 1 -w 5 -n 30 > "$log" 2>&1
row=$(awk '$1==4294967296{print}' "$log" | tail -1)
busbw=$(echo "$row" | awk '{print $(NF-1)}')
[ -z "$busbw" ] && busbw="ERR(见 $log)"
printf " %-14s 4G busbw=%s GB/s\n" "$op" "$busbw"
echo "$tag,$(wc -l < "$hostfile"),$op,4G,$busbw" >> "$SUMMARY"
done
}
# === 阶段1: 旧集群内 2 台 (基准) ===
cat > "$OUT/hosts_old2" << HEOF
10.184.0.20 slots=8
10.184.0.21 slots=8
HEOF
run_test "old2node" "$OUT/hosts_old2"
# === 阶段2: 新机 2 台同交换机 ===
cat > "$OUT/hosts_new2" << HEOF
10.184.0.51 slots=8
10.184.0.55 slots=8
HEOF
run_test "new2node" "$OUT/hosts_new2"
# === 阶段3: 跨交换机 4 台 ===
cat > "$OUT/hosts_cross4" << HEOF
10.184.0.20 slots=8
10.184.0.21 slots=8
10.184.0.51 slots=8
10.184.0.55 slots=8
HEOF
run_test "cross4node" "$OUT/hosts_cross4"
echo ""
echo "==== 汇总 ===="
column -t -s, "$SUMMARY"
echo "结果目录: $OUT"

View File

@ -1,63 +0,0 @@
#!/bin/bash
# ===== 多机 NCCL Scaling 测试 =====
# 用法: bash run_scaling.sh [节点级别...]
# 默认: "2 4 8 16 32"
# 示例: bash run_scaling.sh 2 4 8
# 前置: source nccl_env.sh, hosts_all 文件与本脚本同目录或指定 HOSTS_ALL
# 主节点运行: gpu-20
set -euo pipefail
source "$(dirname "$0")/nccl_env.sh"
NT=/opt/nccl-tests/build
HOSTS_ALL="${HOSTS_ALL:-$(dirname "$0")/hosts_all}"
LEVELS="${*:-2 4 8 16 32}"
OUT=~/nccl_kit/results_$(date +%Y%m%d_%H%M%S)
mkdir -p "$OUT"
SUMMARY="$OUT/summary.csv"
echo "nodes,gpus,op,size,busbw_GBps" > "$SUMMARY"
MPI=(mpirun --allow-run-as-root
--mca plm_rsh_args "-oStrictHostKeyChecking=no"
-x PATH -x LD_LIBRARY_PATH
-x NCCL_IB_HCA -x NCCL_IB_DISABLE
-x NCCL_SOCKET_IFNAME -x NCCL_DEBUG)
echo "Scaling levels: $LEVELS"
echo "结果目录: $OUT"
for N in $LEVELS; do
hf="$OUT/hosts_$N"
head -"$N" "$HOSTS_ALL" > "$hf"
avail=$(wc -l < "$hf")
if [ "$avail" -lt "$N" ]; then
echo "!! hosts_all 只有 $avail 台, 跳过 $N 节点"
continue
fi
np=$((N * 8))
echo ""
echo "================ $N nodes / $np GPUs ================"
for op in all_reduce alltoall all_gather reduce_scatter; do
log="$OUT/${op}_${N}n.log"
"${MPI[@]}" -np "$np" -N 8 --hostfile "$hf" \
"$NT/${op}_perf" -b 8 -e 8G -f 2 -g 1 -w 5 -n 50 > "$log" 2>&1
# AllToAll 在大消息 (>2G) + 多节点会 hang, 优先取 1G 行
if [[ "$op" == "alltoall" ]]; then
row=$(awk '$1==1073741824{print}' "$log" | tail -1)
sz="1G"
else
row=$(awk '$1==8589934592{print}' "$log" | tail -1)
sz="8G"
fi
busbw=$(echo "$row" | awk '{print $(NF-1)}')
[ -z "$busbw" ] && busbw="ERR(见 $log)"
printf " %-14s %s busbw=%s GB/s\n" "$op" "$sz" "$busbw"
echo "$N,$np,$op,$sz,$busbw" >> "$SUMMARY"
done
done
echo ""
echo "==== 汇总 (AllReduce 应随节点数基本持平 ~317 GB/s) ===="
column -t -s, "$SUMMARY"
echo "结果目录: $OUT"

View File

@ -1,90 +0,0 @@
import os
import tempfile
import unittest
from unittest.mock import patch
from modules.benchmark import Benchmark
from modules.gpu_specs import get_gpu_specs
from modules.nccl_test import NCCLTest
from modules.rdma_test import RDMATest
from modules.report import ReportGenerator
from modules.training_sim import TrainingSim
class B300JudgementTests(unittest.TestCase):
def test_b300_training_excludes_cudnn_sdpa(self):
self.assertTrue(TrainingSim._requires_b300_sdpa_workaround("b300", None, 91900))
self.assertTrue(TrainingSim._requires_b300_sdpa_workaround("auto", (10, 3), 91900))
self.assertFalse(TrainingSim._requires_b300_sdpa_workaround("b300", (10, 3), 92000))
self.assertFalse(TrainingSim._requires_b300_sdpa_workaround("h200", (9, 0), 91900))
def test_b300_uses_dense_peaks_and_calibrated_floors(self):
specs = get_gpu_specs("b300")
self.assertEqual(specs["fp8_tflops"], 4500)
self.assertEqual(specs["bf16_tflops"], 2250)
self.assertEqual(specs["compute_pass_thresholds_tflops"]["fp8"], 2800)
self.assertEqual(specs["nccl_pass_thresholds_gbps"]["allreduce"], 750)
def test_compute_verdict_uses_slowest_gpu_not_average(self):
result = {
"compute": {
"per_dtype_tflops": {"fp8": 3000.0},
"worst_per_dtype_tflops": {"fp8": 2500.0},
"pass_thresholds_tflops": {"fp8": 2800.0},
"efficiency_pct": {"fp8": 66.7},
}
}
judged = Benchmark.judge_compute(result)
self.assertEqual(judged["verdict"], "FAIL")
self.assertEqual(judged["rows"][0][1], 2500.0)
def test_rdma_without_peer_is_skip_not_false_failure(self):
rdma = RDMATest({"rdma": {}})
with (
patch.object(rdma, "_get_ib_devices", return_value=["mlx5_0"]),
patch.object(rdma, "_get_ib_ports", return_value=["1"]),
patch.object(rdma, "_read_sys", side_effect=["InfiniBand", "4: ACTIVE"]),
patch.object(rdma, "_collect_device_info", return_value=[]),
):
result = rdma.run()
self.assertEqual(result["status"], "SKIP")
self.assertTrue(result["skipped"])
self.assertIn("peer not configured", result["reason"])
def test_nccl_runtime_env_preserves_and_prepends_venv_libraries(self):
with (
patch("modules.nccl_test.sysconfig.get_paths", return_value={"purelib": "/venv/site"}),
patch("modules.nccl_test.os.path.isdir", return_value=True),
patch.dict(os.environ, {"LD_LIBRARY_PATH": "/system/lib"}),
):
env = NCCLTest._runtime_env()
self.assertEqual(
env["LD_LIBRARY_PATH"],
"/venv/site/nvidia/nccl/lib:/venv/site/torch/lib:/system/lib",
)
self.assertEqual(env["NCCL_DEBUG"], "WARN")
def test_markdown_summary_keeps_warn_and_measured_semantics(self):
results = {
"nccl": {"status": "WARN", "passed": True, "tests": {}},
"training": {
"throughput_tokens_per_sec": 1234,
"gpu_scope": "single_gpu:0",
},
}
generator = ReportGenerator({"report": {"output_dir": "."}})
summary = dict(generator._build_summary(results))
self.assertEqual(summary["NCCL"], "WARN")
self.assertTrue(summary["Training"].startswith("MEASURED"))
with tempfile.TemporaryDirectory() as tmpdir:
path = os.path.join(tmpdir, "report.md")
generator.generate(results, fmt="md", output=path)
with open(path, encoding="utf-8") as report_file:
report = report_file.read()
self.assertIn("| NCCL | WARN |", report)
self.assertIn("synthetic workload; not an 8-GPU scaling acceptance test", report)
if __name__ == "__main__":
unittest.main()