diff --git a/.gitignore b/.gitignore index 99f18a6..2347ffb 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,12 @@ __pycache__/ dist/ build/ reports/ +reports_* +H100*.md +test_all*.md +docs/h100_test_all_metrics_guide_cn.md +docs/multinode_nccl_concepts.md +docs/multinode_nccl_deep_diagnose_runbook.md *.egg .eggs/ *.log @@ -14,5 +20,6 @@ reports/ .venv/ venv/ .qoder/* +.playwright-mcp/ .claude/settings.local.json .omx/ diff --git a/README.md b/README.md index 21aad0d..ebe1ae6 100644 --- a/README.md +++ b/README.md @@ -6,53 +6,10 @@ > **支持 GPU 架构：** Ampere (A100/A800) · Hopper (H100/H200) · Blackwell (B200/B300) > 系统自动检测 GPU 型号并使用对应的规格参数进行基准对比。 -## H100 当前验收入口 - -当前分支 `h100-acceptance-current` 已补齐 H100 单节点、多节点 NCCL、跨节点 RDMA 的主要证据链。按现有 PDF/配置口径，当前结论仍是 **FAIL**：脚本和证据基本可交付，但机器尚未达到生产验收阈值。 - -| 优先级 | 文件 | 用途 | -|---|---|---| -| 1 | [reports_h100_acceptance_current_status_20260523.md](reports_h100_acceptance_current_status_20260523.md) | 当前总状态：已测项、失败项、阻塞项、下一步 | -| 2 | [reports_h100_acceptance_closure_checklist_20260523.md](reports_h100_acceptance_closure_checklist_20260523.md) | 收尾检查清单：可交付项、未关闭门禁、最短收尾路径 | -| 3 | [reports_h100_acceptance_delivery_manifest_20260523.md](reports_h100_acceptance_delivery_manifest_20260523.md) | 交付包 manifest：入口、脚本、远端 artifacts、checksum | -| 4 | [reports_h100_acceptance_pr_summary_20260523.md](reports_h100_acceptance_pr_summary_20260523.md) | PR/审阅摘要：变更范围、验证、风险、合并说明 | -| 5 | [reports_h100_network_hardware_escalation_request_20260523.md](reports_h100_network_hardware_escalation_request_20260523.md) | 给网络/硬件/环境侧的闭环请求和回填表 | -| 6 | [reports_multinode_nccl_latest_index_20260523.md](reports_multinode_nccl_latest_index_20260523.md) | 多节点 NCCL 相关报告索引 | -| 7 | [reports_multinode_nccl_handoff_plan_20260523.md](reports_multinode_nccl_handoff_plan_20260523.md) | 接手人复跑和继续定位计划 | -| 8 | [reports_test_all_latest_summary_cn_20260523.md](reports_test_all_latest_summary_cn_20260523.md) | 单节点 `test all` 中文原始汇总 | -| 9 | [reports_rdma_cross_node_mlx5_0_20260523.md](reports_rdma_cross_node_mlx5_0_20260523.md) | 跨节点 RDMA `mlx5_0` 双向结果 | - -当前主要阻塞： - -- 单节点 `test all`：两台节点均为 `6/10 PASS`，Compute、NCCL、Stress、RDMA 未过。 -- 跨节点 RDMA：`mlx5_0` 写带宽接近/达到阈值，但读带宽和读写延迟未过。 -- 多节点 NCCL：`2x8 allreduce`、`2x8 alltoall` 按 PDF 阈值未过；NCCL `wrong_count=0`，主要是性能不达标。 -- 环境差异：当前可用 400G IB rail 主要是 `mlx5_0,mlx5_1,mlx5_6,mlx5_7`，未发现外部 NCCL net plugin / SHARP / HCOLL。 - -### H100 复跑入口 - -远端默认路径为 `/root/test_gpu_scripts`，建议在 `nccl-gpu-1` 作为发起节点执行多节点测试。 - -```bash -# 单节点全量验收，分别在每台机器执行 -bash scripts/run_h100_single_node_all.sh - -# 多节点 NCCL PDF 矩阵：allreduce/alltoall x 2x1/2x2/2x4/2x8 -bash scripts/run_multinode_nccl_pdf_matrix.sh - -# 多节点 NCCL 六类 collective：2 节点 x 8 GPU -bash scripts/run_multinode_nccl_all_collectives.sh - -# 多节点 NCCL 深度诊断和环境证据抓取 -bash scripts/multinode_nccl_deep_diagnose.sh preflight -bash scripts/multinode_nccl_deep_diagnose.sh all -``` - --- ## 目录 -- [H100 当前验收入口](#h100-当前验收入口) - [项目结构](#项目结构) - [环境要求](#环境要求) - [快速开始](#快速开始) @@ -69,31 +26,23 @@ bash scripts/multinode_nccl_deep_diagnose.sh all ## 项目结构 ``` -test_gpu_scripts/ -├── gpu_tester.py # 主入口：CLI + 交互式菜单 -├── install_deps.sh # 一键安装三方工具 +servertest/ +├── gpu_tester.py # 主入口：CLI + 交互式菜单 +├── install_deps.sh # 一键安装三方工具 ├── configs/ -│ ├── default.yaml # 默认配置 -│ ├── multinode_nccl_nccl227_pdf_matrix.yaml # H100 多节点 PDF 矩阵配置 -│ └── multinode_nccl_nccl227_all_collectives_2x8.yaml +│ └── default.yaml # 默认配置 ├── modules/ -│ ├── gpu_specs.py # GPU 规格数据库 -│ ├── gpu_info.py # GPU 检测 & 信息 -│ ├── health_check.py # 健康诊断 -│ ├── benchmark.py # 内存带宽 + 计算吞吐 -│ ├── nccl_test.py # NCCL 多卡/多节点通信 -│ ├── stress_test.py # GPU 压力/稳定性 -│ ├── rdma_test.py # RDMA/InfiniBand -│ ├── training_sim.py # 训练模拟 -│ └── report.py # 报告生成 -├── scripts/ -│ ├── run_h100_single_node_all.sh # H100 单节点全量复跑 -│ ├── run_multinode_nccl_pdf_matrix.sh # 多节点 NCCL PDF 矩阵复跑 -│ ├── run_multinode_nccl_all_collectives.sh # 多节点 NCCL 六类 collective 复跑 -│ └── multinode_nccl_deep_diagnose.sh # 多节点 NCCL 深度诊断 -├── docs/ # 指标说明和 runbook -├── reports_*20260523*.md # 当前 H100 验收证据和汇总报告 -└── requirements.txt +│ ├── gpu_specs.py # GPU 规格数据库 (A100/A800/H100/H200/B200/B300) +│ ├── gpu_info.py # GPU 检测 & 信息 +│ ├── health_check.py # 健康诊断 +│ ├── benchmark.py # 内存带宽 + 计算吞吐 +│ ├── nccl_test.py # NCCL 多卡通信 +│ ├── stress_test.py # GPU 压力/稳定性 +│ ├── rdma_test.py # RDMA/InfiniBand +│ ├── training_sim.py # 训练模拟 +│ └── report.py # 报告生成 +├── requirements.txt +└── 调研.md # 行业框架调研 ``` --- @@ -210,7 +159,7 @@ python3 gpu_tester.py [3] Memory Benchmark (nvbandwidth) [4] Compute Benchmark [5] NCCL Multi-GPU Test - [6] GPU Stress Test (PyTorch/gpu-burn) + [6] GPU Stress Test (gpu-burn) [7] RDMA/IB Test [8] Training Simulation [9] Full Test Suite (All Tests) @@ -330,35 +279,33 @@ python3 gpu_tester.py --config /path/to/config.yaml --test all | FP16 | 312 TFLOPS | 990 TFLOPS | 2,250 TFLOPS | 3,500 TFLOPS | | BF16 | 312 TFLOPS | 990 TFLOPS | 2,250 TFLOPS | 3,500 TFLOPS | | FP8 | N/A | 1,979 TFLOPS | 4,500 TFLOPS | 7,000 TFLOPS | -| FP64 | 9.7 TFLOPS | 67 TFLOPS | TBD | TBD | -| INT8 | 624 TOPS | 1,979 TOPS | TBD | TBD | -默认配置：8192×8192 矩阵，50 次 warmup，500 次迭代；逐 GPU 跑 FP32/TF32/FP16/BF16/FP8/FP64/INT8，并按同 dtype 的极差/均值判断一致性。 +默认配置：4096×4096 矩阵，10 次 warmup，100 次迭代。 ### 5. NCCL Multi-GPU Test（多卡通信） -优先使用官方 nccl-tests（通过 mpirun 调用）并解析真实 bus BW；如果只能走 torchrun fallback，验收结果会标记 FAIL。 +优先使用官方 nccl-tests（通过 mpirun 调用），不可用时 torchrun fallback。 | 操作 | 说明 | |---|---| | AllReduce | 最常用的集合通信 | | AllToAll | 模型并行关键操作 | | Broadcast | 参数同步 | -| ReduceScatter | 必测 | -| AllGather | 必测 | -| SendRecv | 必测 | +| ReduceScatter | 可选 | +| AllGather | 可选 | +| SendRecv | 可选 | -默认按 PDF 口径测试 1MB、256MB、2GB 三个 size，每个 op 重复 3 次，取 worst bus BW 和标准差；标准差超过 3% 判 FAIL。 +默认测试数据量范围 8B ~ 256MB，5 次 warmup，20 次迭代。 **NVLink 参考带宽：** A100/A800 ≥ 240 GB/s | H100/H200 ≥ 360 GB/s | B200/B300 ≥ 720 GB/s（40% NVLink 峰值） ### 6. GPU Stress Test（压力测试） -默认使用 PyTorch BF16/FP16 GEMM 进行长时高功耗满载测试；也可在配置中启用 gpu-burn。测试期间采集温度、功耗、throttle、XID，并计算稳态功耗、温差和 TFLOPS 抖动。 +使用 gpu-burn 进行长时满载测试，验证热稳定性和内存正确性。 | 参数 | 默认值 | 说明 | |---|---|---| -| duration_sec | 1800 | 测试时长（秒） | +| duration_sec | 60 | 测试时长（秒） | | use_tensor_cores | true | 使用 Tensor Core | | memory_pct | 90 | 内存占用比例 | @@ -373,18 +320,18 @@ python3 gpu_tester.py --config /path/to/config.yaml --test all | 写延迟 | ib_write_lat | | 读延迟 | ib_read_lat | -**参考阈值：** 端口 ACTIVE 且 ≥400Gbps；4MB 写/读带宽 ≥47GB/s；8B 写延迟 ≤2μs、读延迟 ≤3.5μs；PFC/ECN/CNP/congestion 计数为 0。 +**参考阈值：** 带宽 ≥ 50 GB/s, 延迟 ≤ 10 μs ### 8. Training Simulation（训练模拟） -默认跑 8 卡 DDP synthetic 1.5B Transformer 训练模拟。 +使用真实或合成模型模拟训练负载。 | 模式 | 说明 | |---|---| -| DDP 合成模型 | 约 1.5B 参数，8 卡 torchrun | -| 单进程 fallback | 仅用于调试；生产验收按 FAIL | +| 真实模型 | 加载 HuggingFace GPT-2（需安装 transformers） | +| 合成模型 | 6 层 Transformer（无需额外依赖） | -输出：tokens/sec、步时、warmup 后 step 抖动、峰值显存、最终 loss，并检查 loss 是否 NaN/Inf。 +输出：tokens/sec、步时、峰值显存、最终 loss。 --- @@ -404,14 +351,14 @@ benchmark: nvbandwidth_buffer_mb: 512 # nvbandwidth 缓冲区大小 nvbandwidth_samples: 3 # nvbandwidth 采样次数 compute: - dtypes: [fp32, tf32, fp16, bf16, fp8, fp64, int8] - matrix_size: 8192 # GEMM 矩阵维度 - warmup: 50 - iterations: 500 + dtypes: [fp32, tf32, fp16, bf16, fp8] + matrix_size: 4096 # GEMM 矩阵维度 + warmup: 10 + iterations: 100 health: - temp_warning: 75 # 温度警告阈值 °C - temp_critical: 85 # 温度严重阈值 °C + temp_warning: 80 # 温度警告阈值 °C + temp_critical: 90 # 温度严重阈值 °C power_limit: null # null = 自动匹配 GPU TDP nccl: @@ -419,83 +366,26 @@ nccl: test_allreduce: true test_alltoall: true test_broadcast: true - test_reduce_scatter: true - test_allgather: true - test_sendrecv: true - message_sizes: [1M, 256M, 2G] - repeats: 3 - max_stddev_pct: 3 - -multinode_nccl: - enabled: false # true 时纳入 --test all - hosts: - - {name: nccl-gpu-1, addr: 172.72.8.12, slots: 8} - - {name: nccl-gpu-2, addr: 172.72.8.16, slots: 8} - tests: [all_reduce_perf, alltoall_perf] - topologies: - - {nodes: 2, gpus_per_node: 8} - mpirun_path: /usr/mpi/gcc/openmpi-4.1.9a1/bin/mpirun - extra_ld_library_path: # 传给远端 rank 的 MPI/NCCL/CUDA 库路径 - - /usr/mpi/gcc/openmpi-4.1.9a1/lib - - /root/gpu-test-venv/lib/python3.10/site-packages/nvidia/nccl/lib - - /usr/local/cuda-12.4/targets/x86_64-linux/lib - begin_size: 1k - end_size: 16g - step_factor: 2 - warmup_iters: 10 - socket_ifname: bond0 - ib_gid_index: 3 - ib_hca: mlx5_0,mlx5_1,mlx5_6,mlx5_7 stress: - duration_sec: 1800 # 压力测试时长 - use_gpu_burn: false # 默认走 PyTorch GEMM stress - dtype: bf16 - matrix_size: 24576 - telemetry_interval_sec: 1 - min_power_watts: 630 - max_tflops_jitter_pct: 5 - require_tflops_jitter: true + duration_sec: 60 # 压力测试时长 use_tensor_cores: true rdma: - min_bandwidth_gbps: 47 # RDMA 最低可接受带宽 - min_port_rate_gbps: 400 # IB 端口最低速率 - max_write_latency_us: 2.0 - max_read_latency_us: 3.5 - msg_size: 4194304 # 4MB 带宽测试消息 - latency_msg_size: 8 # 8B 延迟测试消息 - server_addr: null # client 模式 perftest 对端 IP - ibping_target: null # ibping 对端 LID/GID，不是 IP - role: auto # auto / server / client - pfc_ecn_counters: true - -nvlink: - expected_links_per_gpu: 18 - expected_link_speed_gbps: 25 - require_zero_errors: true - -dcgm: - diag_level: 3 - timeout_sec: 3600 - expected_num_gpus: 8 - json_output: true - require_subtests: true + min_bandwidth_gbps: 50 # RDMA 最低可接受带宽 + max_latency_us: 10 # RDMA 最大可接受延迟 + msg_size: 65536 # 测试消息大小 training: - model: synthetic_1.5b # 8 卡 synthetic Transformer + model: gpt2 # HuggingFace 模型名 batch_size: 8 seq_length: 2048 num_steps: 50 - warmup_steps: 5 dtype: bf16 - mode: ddp - min_tokens_per_sec: 45000 - max_step_jitter_pct: 3 report: output_dir: ./reports - format: json # json / html / md + format: json # json 或 html ``` --- @@ -603,22 +493,22 @@ report: 步骤 2: RDMA 网络测试 ├── python3 gpu_tester.py --test rdma ├── 确认: IB 设备被识别 -├── 确认: 端口状态 ACTIVE 且 ≥400Gbps -├── 确认: 4MB 写/读带宽 ≥47 GB/s -├── 确认: 8B 写延迟 ≤2 μs、读延迟 ≤3.5 μs -├── 确认: ibping 双向连通 -├── 确认: PFC/ECN/CNP/congestion 计数为 0 +├── 确认: 端口状态 Active +├── 确认: 写带宽 ≥ 50 GB/s +├── 确认: 延迟 ≤ 10 μs └── 异常: 检查 IB 线缆、交换机配置、子网管理器 步骤 3: 多节点 NCCL 测试 -├── 在发起节点确认 mpirun、nccl-tests、跨节点 root SSH 可用 -├── 配置 configs/default.yaml 的 multinode_nccl.hosts / IB 参数 -├── 执行 PDF 风格 sweep: -│ python3 gpu_tester.py --test multinode-nccl --report --format md -├── 默认命令口径: -│ mpirun -H :8,:8 --map-by ppr:8:node -np 16 \ -│ all_reduce_perf/alltoall_perf -b 1k -e 16g -f 2 -g 1 -w 10 -└── 确认: Peak Bus BW、Peak Size、wrong_count 正常 +├── 在每个节点上配置: +│ export MASTER_ADDR=<主节点IP> +│ export MASTER_PORT=29500 +│ export NCCL_SOCKET_IFNAME=ib0 # IB 网卡名 +│ export NCCL_DEBUG=INFO +├── 运行 nccl-tests 手动测试: +│ mpirun -np <总GPU数> -hostfile hosts \ +│ /opt/gpu-test-tools/nccl-tests/build/all_reduce_perf \ +│ -b 8 -e 256M -f 2 -g 1 -w 5 -n 20 +└── 确认: 多节点 AllReduce 带宽正常 步骤 4: 训练验证 ├── python3 gpu_tester.py --test training @@ -626,17 +516,6 @@ report: └── 确认: 训练 loss 正常下降 ``` -#### 多节点 NCCL 深度诊断 - -当 SOP-3 的多节点 NCCL 结果与验收 PDF 不一致时，可以在发起节点运行深度诊断脚本，复现 counter 抓取、GRAPH/TUNING 日志和 PXN disabled sweep： - -```bash -bash scripts/multinode_nccl_deep_diagnose.sh preflight -bash scripts/multinode_nccl_deep_diagnose.sh all -``` - -详细参数、输出目录和解读方法见 [docs/multinode_nccl_deep_diagnose_runbook.md](/Users/d-robotics/lab/test_gpu_scripts/docs/multinode_nccl_deep_diagnose_runbook.md)。 - --- ### SOP-4: 故障诊断 diff --git a/docs/h100_test_all_metrics_guide_cn.md b/docs/h100_test_all_metrics_guide_cn.md deleted file mode 100644 index 37abd28..0000000 --- a/docs/h100_test_all_metrics_guide_cn.md +++ /dev/null @@ -1,255 +0,0 @@ -# H100 `test all` 指标说明 - -本文解释 `gpu_tester.py --test all` 报告里每一项指标的意义、它在验收中代表什么，以及异常时通常应该优先排查什么。 - -适用报告： - -- `reports_test_all_latest_aikubeworker0012_20260522_203246.md` -- `reports_test_all_latest_aikubeworker0016_20260522_203447.md` -- `reports_test_all_latest_summary_cn_20260523.md` - -## 总体判定 - -| 指标 | 意义 | 怎么看 | -|---|---|---| -| `Overall Acceptance Verdict` | 整机验收结论 | 按 PDF 生产验收规则，任一必测子项 FAIL，则整机 FAIL | -| `Suite complete: x/10 tests passed` | 10 个测试模块里通过了几个 | 用来快速看整体健康度，但最终以 `Overall Acceptance Verdict` 为准 | -| `PASS` | 达到当前配置阈值 | 表示该指标在当前测试口径下通过 | -| `FAIL` | 未达到当前配置阈值，或证据不足 | 表示该项不能作为生产验收通过证据 | -| `WARN` | 旧报告或非强制警告口径 | 当前 PDF 生产验收里，关键性能未达标应按 FAIL 处理 | - -## GPU Info - -GPU Info 是基础盘点项，用来确认机器硬件、驱动和 CUDA 环境是否符合预期。 - -| 指标 | 意义 | 异常影响 | -|---|---|---| -| GPU count | 当前系统识别到的 GPU 数量 | H100 8 卡机器如果不是 8 张，后续所有多卡测试都不可信 | -| GPU model | GPU 型号，例如 H100 | 型号不对会导致阈值、峰值、验收口径都不对 | -| Driver version | NVIDIA 驱动版本 | 版本过旧可能影响 CUDA、NCCL、DCGM、NVLink 工具 | -| CUDA version | CUDA 运行时或驱动支持版本 | CUDA 不匹配会导致 PyTorch、nccl-tests 或编译工具异常 | -| GPU UUID / PCI bus id | GPU 唯一标识和 PCIe 拓扑位置 | 用于定位具体故障卡、对应槽位和链路 | - -这项通常不直接代表性能好坏，它是确认“测的是不是目标机器、目标 GPU、目标软件栈”。 - -## Health Check - -Health Check 是空闲或轻负载状态下的基础健康检查。 - -| 指标 | 意义 | 怎么看 | -|---|---|---| -| Temperature | 当前 GPU 温度 | 空闲温度过高可能说明散热、风道、环境温度异常 | -| Power | 当前功耗 | 空闲功耗异常高可能说明有残留进程或功耗状态异常 | -| ECC errors | 显存纠错错误 | 单比特错误过多或双比特错误通常需要重点关注硬件稳定性 | -| PCIe | PCIe 代际和宽度，例如 Gen5 x16 | 降速或降宽会影响 CPU-GPU、RDMA、部分数据搬运性能 | -| Throttle | 当前是否触发限速 | 空闲状态下非 idle throttle 不正常，可能影响后续性能 | -| XID / NVRM events | 驱动或 GPU 错误事件 | 出现新 XID 通常说明硬件、驱动、供电或内核态异常 | - -Health PASS 只能说明基础状态正常，不代表满载性能一定达标。 - -## Memory Bandwidth - -Memory Bandwidth 衡量数据搬运能力，包括 CPU 到 GPU、GPU 到 CPU、GPU 到 GPU。 - -| 指标 | 意义 | 代表什么 | -|---|---|---| -| H2D | Host to Device，CPU 内存到 GPU 显存带宽 | 受 PCIe、NUMA、CPU 内存、驱动影响 | -| D2H | Device to Host，GPU 显存到 CPU 内存带宽 | 受 PCIe、NUMA、CPU 内存、驱动影响 | -| D2D | Device to Device，GPU 到 GPU 带宽 | 单节点多卡通常主要受 NVLink/NVSwitch 影响 | -| Efficiency | 实测值相对理论或配置阈值的比例 | 用于快速判断是否达到预期带宽 | - -H2D/D2H 主要看 PCIe 和 CPU 侧链路是否正常。D2D 更接近多卡训练、NCCL 和 P2P 通信的基础能力。 - -## Compute Throughput - -Compute Throughput 衡量 GPU 在不同数值格式下的矩阵计算吞吐，单位通常是 TFLOPS。 - -| 指标 | 意义 | 常见用途 | -|---|---|---| -| FP32 | 32 位浮点性能 | 传统科学计算、部分模型训练和验证 | -| TF32 | TensorFloat-32 Tensor Core 性能 | NVIDIA Ampere/Hopper 上常见的 FP32 加速路径 | -| FP16 | 16 位浮点 Tensor Core 性能 | 深度学习训练和推理常用 | -| BF16 | bfloat16 Tensor Core 性能 | 大模型训练常用，数值范围比 FP16 更稳 | -| FP8 | 8 位浮点 Tensor Core 性能 | 新一代低精度训练/推理加速 | -| FP64 | 64 位双精度性能 | HPC、科学计算、仿真 | -| INT8 | 8 位整数性能 | 推理、量化模型 | -| Achieved | 实测吞吐 | 越接近峰值越好 | -| Peak | 理论峰值或规格峰值 | 用来计算效率 | -| Threshold | 当前验收阈值 | 低于阈值则 FAIL | -| Efficiency | `Achieved / Peak` | 衡量实测利用率 | - -### Compute Consistency - -Consistency 是看同一种 dtype 下，不同 GPU 之间性能是否均衡。 - -| 指标 | 意义 | 异常含义 | -|---|---|---| -| Min | 8 张 GPU 里最慢卡的实测值 | 用于发现拖后腿的卡 | -| Mean | 8 张 GPU 平均值 | 用于看整体水平 | -| Max | 8 张 GPU 里最快卡的实测值 | 和 Min 一起计算离散度 | -| Spread | `(Max - Min) / Mean` | 反映卡间性能差异 | - -Spread 超过阈值通常说明某些卡受温度、功耗、PCIe、后台负载、时钟策略或硬件状态影响。即使平均性能还可以，卡间差异过大也会拖慢分布式训练。 - -## NVLink / NVSwitch - -NVLink/NVSwitch 测试确认 GPU 间高速互联是否完整、速率是否正确、错误计数是否干净。 - -| 指标 | 意义 | 怎么看 | -|---|---|---| -| Active Links | 每张 GPU 当前活跃 NVLink 数 | H100 8 卡 SXM 常见期望是每卡 18 条 | -| Expected Links | 配置期望链路数 | 少一条都可能影响拓扑和 NCCL 性能 | -| Link speed | 单条链路速率 | 速率不对说明链路降级或识别异常 | -| Error counters | NVLink 错误计数，例如 CRC/replay/recovery | 非零可能说明链路质量或硬件问题 | - -NVLink PASS 表示链路状态看起来正常，但 NCCL 仍可能因算法、拓扑、消息大小、NCCL 参数或系统噪声而不达标。 - -## DCGM Diagnostic - -DCGM 是 NVIDIA 官方诊断工具。`dcgmi diag -r 3` 是比较完整的生产诊断级别。 - -| 子项 | 意义 | -|---|---| -| Deployment/software | 驱动、库、系统软件依赖检查 | -| Hardware/memory | GPU 显存健康检查 | -| Hardware/diagnostic | GPU 硬件基础诊断 | -| Hardware/nvbandwidth | GPU/NVLink/NVSwitch 带宽诊断 | -| Integration/pcie | PCIe 集成和链路相关检查 | -| Stress/targeted_stress | DCGM 自带目标压力测试 | -| Stress/targeted_power | DCGM 自带目标功耗压力测试 | -| summary | 该分类汇总结果 | - -DCGM PASS 是强证据，说明官方诊断没有发现明显硬件故障。但它不替代项目里的 NCCL、RDMA、长时间 telemetry 和训练模拟验收。 - -## NCCL Multi-GPU - -NCCL 测试衡量单节点多 GPU 集合通信能力。它直接关系到多卡训练效率。 - -| 指标 | 意义 | 为什么重要 | -|---|---|---| -| source | 测试来源 | 必须是 `nccl-tests` 才有真实 bus BW；`torchrun_fallback` 只能说明功能连通，不是性能验收 | -| bus BW | NCCL 报告的总线等效带宽 | 用来衡量通信是否吃满 NVLink/NVSwitch | -| message size | 消息大小，例如 1M、256M、2G | 小消息看延迟和调度，中大消息看带宽 | -| repeats | 重复次数 | 减少偶然波动，当前按 3 次取样 | -| worst bus BW | 多次结果里的最差值 | 生产验收更关注最差情况 | -| mean bus BW | 多次平均值 | 反映稳定水平 | -| stddev | 标准差或波动 | 波动大说明通信稳定性不足 | - -### NCCL op 含义 - -| Op | 意义 | 常见场景 | -|---|---|---| -| allreduce | 每张卡都有一份数据，做规约后每张卡都拿到结果 | 数据并行梯度同步最常见 | -| allgather | 每张卡收集所有卡的数据分片 | 模型并行、张量并行、参数/激活收集 | -| reducescatter | 先规约再把结果切分给各卡 | ZeRO、优化器状态切分、分布式训练常用 | -| broadcast | 一张卡把数据广播给其他卡 | 参数同步、初始化权重分发 | -| sendrecv | 点对点发送和接收 | pipeline、定制通信、拓扑验证 | -| alltoall | 每张卡向每张卡交换不同数据 | MoE、专家并行、shuffle 类通信 | - -NCCL 小消息失败常见于延迟、调度或阈值口径较严；大消息失败更偏向链路带宽、拓扑、NCCL 参数或 NVSwitch/PCIe/NUMA 配置问题。 - -## Stress Test - -Stress Test 是长时间高负载稳定性测试。它不是只看“能不能跑完”，还要看满载期间的温度、功耗、限速和错误事件。 - -| 指标 | 意义 | 怎么看 | -|---|---|---| -| duration | 实际压力测试时长 | 生产验收通常需要 30/60 分钟 | -| source | 压力来源，例如 `pytorch` 或 `gpu-burn` | 说明用什么负载压 GPU | -| dtype | 压力计算的数据类型，例如 BF16 | 影响 Tensor Core、功耗和温度 | -| matrix_size | GEMM 矩阵边长 | 越大越容易形成持续高占用 | -| memory_pct | 目标显存占用比例 | 避免只测很小负载 | -| Avg steady power | 稳态平均功耗 | 判断是否真的把卡压起来 | -| Max steady temp | 稳态最高温度 | 判断散热上限 | -| Temp delta | 8 卡之间最高温和最低温的差 | 差异过大说明风道、散热或卡位不均衡 | -| TFLOPS jitter | 稳态吞吐波动 | 波动大说明性能不稳定 | -| Throttle events | 限速事件数量 | 非 idle throttle 会影响性能稳定性 | -| XID events | 压测期间新增 XID 错误 | 出现 XID 通常是严重风险 | - -### Throttle 常见含义 - -| 代码 | 常见含义 | 解释 | -|---|---|---| -| `0x1` | idle throttle | 空闲状态限速，通常不算真实问题 | -| `0x4` | `sw_power_cap` | 达到软件功耗上限，性能可能被功耗墙限制 | -| `0x8` | hardware slowdown | 硬件触发降速 | -| `0x10` | thermal slowdown | 温度触发降速 | -| `0x20` | power brake | 外部供电或硬件功率保护 | -| `0x40` | software thermal slowdown | 软件温度策略触发降速 | - -当前报告里的 `sw_power_cap` 表示负载确实压到了功耗墙附近，但验收口径把非 idle throttle 作为失败原因之一，因为它会影响长时间稳定输出。 - -## RDMA / InfiniBand - -RDMA 测试衡量 IB 网卡和网络链路性能。单节点 loopback 和跨节点 server/client 是两种不同证据，不能混用。 - -| 指标 | 意义 | 怎么看 | -|---|---|---| -| Device | IB 设备名，例如 `mlx5_0` | 对应具体 HCA/端口 | -| Port | 端口号 | 通常是 port 1 | -| State | 端口状态，例如 ACTIVE/DOWN | ACTIVE 才能作为可用链路 | -| Rate | 端口速率，例如 400 Gb/sec | 低于期望说明链路降级或接错网络 | -| GID/LID | IB 寻址信息 | `ibping` 和跨节点定位会用到 | -| ib_write_bw | RDMA write 带宽 | 客户端向远端写数据的吞吐 | -| ib_read_bw | RDMA read 带宽 | 客户端从远端读数据的吞吐 | -| ib_write_lat | RDMA write 延迟 | 小消息写延迟 | -| ib_read_lat | RDMA read 延迟 | 小消息读延迟 | -| ibping | IB 层连通性测试 | 看 LID/GID 层是否可达 | -| PFC/ECN/CNP counters | 拥塞和流控相关计数 | 非零或增长可能说明网络拥塞/丢包/流控问题 | - -### 单节点与跨节点的区别 - -| 口径 | 意义 | 能证明什么 | 不能证明什么 | -|---|---|---|---| -| `local_loopback` | 在同一台机器本地启动 perftest server/client | 工具、设备、单机端口基本可用 | 不能证明两台机器之间 RDMA 网络达标 | -| server/client 跨节点 | 一台做 server，另一台做 client | 能证明实际跨节点 RDMA 带宽/延迟 | 需要明确 server_addr、ib_device、ib_port、ibping_target | - -RDMA read 带宽低于 write 带宽很常见，但生产验收会给 read/write 各自设置阈值。read 不过线时，需要排查 HCA 固件、BIOS、PCIe、NUMA、RoCE/IB 配置、交换机、PFC/ECN、线缆和端口速率。 - -## Training Simulation - -Training Simulation 用一个合成 1.5B Transformer 训练负载验证 8 卡分布式训练是否能稳定运行。 - -| 指标 | 意义 | 怎么看 | -|---|---|---| -| Model | 模型类型 | 当前是 synthetic 1.5B，不依赖真实数据集 | -| Parameters | 参数量 | 用来确认负载规模是否达到预期 | -| GPU Count | 参与训练的 GPU 数 | 生产口径要求 8 卡 DDP | -| DType | 训练数值格式，例如 BF16 | 大模型训练常用 BF16 | -| Batch Size | 每步 batch 大小 | 影响吞吐和显存 | -| Seq Length | 序列长度 | 影响计算量和显存 | -| Steps | 计入统计的训练步数 | 步数太少会导致统计不稳 | -| Warmup Steps | 预热步数 | 避免把 CUDA 初始化、编译、缓存冷启动计入性能 | -| Avg Step Time | 平均每步耗时 | 越低越好 | -| Throughput | tokens/sec | 训练吞吐核心指标 | -| Samples/sec | 每秒样本数 | 辅助衡量数据处理速度 | -| Peak Memory | 峰值显存 | 看是否接近 OOM 或显存利用不足 | -| Final Loss | 最后 loss | 用于确认数值是有限值，没有 NaN/Inf | -| Step Jitter | step 时间抖动 | 抖动大说明训练不稳定 | -| Distributed Mode | 分布式模式 | 必须是 `ddp` 才满足 8 卡分布式口径 | - -Training PASS 说明 8 卡 DDP 训练路径、NCCL 功能连通、PyTorch CUDA 和基本数值稳定性都没问题。但它不能替代 NCCL 性能测试，因为训练负载可能没有覆盖所有通信模式和消息大小。 - -## 常见误读 - -1. `DCGM PASS` 不等于整机验收 PASS。DCGM 是官方诊断的一部分，不覆盖全部业务性能门槛。 -2. `Training PASS` 不等于 NCCL 性能 PASS。训练能跑，只说明功能链路通；NCCL bus BW 仍可能不达标。 -3. `NVLink PASS` 不等于 NCCL PASS。链路数量和错误计数正常，不代表所有 NCCL op/size 都达到阈值。 -4. `ibping PASS` 不等于 RDMA 带宽 PASS。`ibping` 只证明连通性，不证明吞吐和延迟达标。 -5. `local_loopback` 不能当作跨节点 RDMA 证据。跨节点验收必须有 server/client 两端证据。 -6. Stress 跑满 30 分钟不等于 PASS。温差、功耗、throttle、XID、jitter 都要一起看。 -7. 小消息 NCCL 低不一定是链路断了，可能是延迟、算法、启动开销或阈值口径导致；但生产验收仍按阈值判定。 - -## 排查优先级建议 - -| 失败项 | 优先看什么 | -|---|---| -| Compute FAIL | GPU 时钟、功耗策略、MIG/MPS、后台进程、PyTorch/CUDA 版本、benchmark 算法是否用到目标 Tensor Core 路径 | -| NCCL FAIL | `NCCL_DEBUG=INFO`、拓扑、NVSwitch/NVLink、NCCL 算法、消息大小、PCIe/NUMA、进程绑核 | -| Stress FAIL | 机箱风道、风扇、环境温度、功耗上限、`nvidia-smi -q -d POWER,CLOCK,TEMPERATURE` | -| RDMA FAIL | 端口速率、HCA 固件、线缆、交换机、PFC/ECN、NUMA、BIOS、跨节点 server/client 配置 | -| Training FAIL | torchrun、NCCL 环境变量、CUDA OOM、loss NaN/Inf、DDP 初始化、网络/共享内存 | - -## 一句话版 - -这套报告不是只看 GPU 能不能亮、训练能不能跑，而是同时验证：硬件识别、基础健康、显存和互联带宽、计算吞吐、多卡通信、长时间满载稳定性、IB/RDMA 网络、官方 DCGM 诊断和 8 卡训练业务路径。任何一个关键项 FAIL，按生产验收都应判整机不通过。 diff --git a/docs/multinode_nccl_concepts.md b/docs/multinode_nccl_concepts.md deleted file mode 100644 index 52d9b87..0000000 --- a/docs/multinode_nccl_concepts.md +++ /dev/null @@ -1,361 +0,0 @@ -# 多机多卡 NCCL 测试概念说明 - -本文先讲概念，不涉及脚本改造。目标是理解两台 8 卡 H100 服务器做多机多卡通信测试时，应该从哪些层次逐步验证，以及每一层到底在证明什么。 - -当前示例机器： - -| 别名 | 主机名 | 内网 IP | GPU | -|---|---|---|---| -| nccl-gpu-1 | aikubeworker0012 | 172.72.8.12 | 8 x H100 | -| nccl-gpu-2 | aikubeworker0016 | 172.72.8.16 | 8 x H100 | - -两台机器合起来就是 16 张 GPU。多机 NCCL 测试的核心问题是：这 16 张 GPU 是否能通过正确的 GPU、NVLink、PCIe、IB/RDMA 网络路径，高效且正确地完成集体通信。 - -## 1. 总体思路 - -多机多卡通信测试是一个自底向上的过程。越底层越接近硬件和链路，越上层越接近真实训练业务。 - -```mermaid -flowchart TD - L0["0. 物理与基础连通
电源 / GPU / 网卡 / 线缆 / 交换机 / SSH"] --> L1["1. 系统识别层
nvidia-smi / lspci / ibstat / ibdev2netdev"] - L1 --> L2["2. 单机 GPU 健康层
温度 / 功耗 / ECC / PCIe / Throttling / NVLink Topo"] - L2 --> L3["3. 单机 GPU 性能层
HBM 带宽 / H2D-D2H / FP32-TF32-FP16-BF16-FP8 算力"] - L3 --> L4["4. 单机多卡通信层
单节点 8 卡 NCCL over NVLink/NVSwitch"] - L4 --> L5["5. 跨机网络与 RDMA 层
IP 连通 / IB Active / RDMA 带宽 / RDMA 延迟"] - L5 --> L6["6. 跨机 NCCL 层
两机 16 卡 AllReduce / AllGather / ReduceScatter / Broadcast / AllToAll"] - L6 --> L7["7. 训练负载层
torchrun / Megatron / DeepSpeed / 业务训练压测"] -``` - -最重要的原则： - -**上层失败，不一定是上层问题。** - -比如两机 `all_reduce_perf` 失败，原因可能在 NCCL，也可能在 SSH、MPI、IB、GID、网卡选择、驱动版本、CUDA 版本、NCCL 版本或 GPU Direct RDMA。 - -所以排查顺序应该是： - -```text -基础连通 -> 单机健康 -> 单机性能 -> 单机 NCCL -> 跨机 RDMA -> 跨机 NCCL -> 训练业务 -``` - -## 2. 两机 16 卡通信路径 - -单机内部主要走 NVLink/NVSwitch；跨机器时，数据必须经过 GPU、PCIe/NVLink、网卡、交换机和对端网卡。 - -```mermaid -flowchart LR - subgraph A["aikubeworker0012 / 172.72.8.12"] - A0["GPU0"] --- ASW["NVSwitch / NVLink"] - A1["GPU1"] --- ASW - A2["..."] --- ASW - A7["GPU7"] --- ASW - ASW --> ANIC["IB/RDMA NIC(s)"] - end - - subgraph NET["InfiniBand / RoCE Fabric"] - SW["IB Switch"] - end - - subgraph B["aikubeworker0016 / 172.72.8.16"] - BNIC["IB/RDMA NIC(s)"] --> BSW["NVSwitch / NVLink"] - B0["GPU0"] --- BSW - B1["GPU1"] --- BSW - B2["..."] --- BSW - B7["GPU7"] --- BSW - end - - ANIC <--> SW - SW <--> BNIC -``` - -这里有两个不同的通信域： - -| 通信域 | 典型路径 | 主要测试 | -|---|---|---| -| 单机内 8 卡 | GPU -> NVLink/NVSwitch -> GPU | 单机 NCCL、NVLink topo、D2D | -| 跨机器 16 卡 | GPU -> NIC -> IB/RDMA 网络 -> NIC -> GPU | RDMA、跨机 NCCL | - -这两个域的性能阈值不能混用。单机 NVSwitch 很快，跨机 RDMA 一般慢一些，跨机 NCCL 的瓶颈通常在 IB/RDMA 网络。 - -## 3. 每一层要测什么 - -### 3.1 基础连通层 - -这一层只证明机器能访问、身份和地址正确。 - -要确认： - -| 检查项 | 目的 | -|---|---| -| SSH 互通 | MPI/NCCL 多机启动依赖远端拉起进程 | -| hostname 正确 | 避免登录错机器 | -| IP 正确 | 确认使用的是训练网络或 IB/RDMA 对应网络 | -| 时间同步 | 长时间训练日志和超时排查更可靠 | - -这一层不证明 GPU 或 RDMA 性能，只证明“机器能互相找到”。 - -### 3.2 系统识别层 - -这一层证明系统能看见 GPU 和网卡。 - -常见信息： - -| 工具 | 看什么 | -|---|---| -| `nvidia-smi` | GPU 数量、型号、驱动、CUDA、温度、功耗 | -| `nvidia-smi topo -m` | GPU、NIC、CPU NUMA、NVLink/NVSwitch 拓扑 | -| `ibstat` | IB 设备、端口状态、链路速率 | -| `ibdev2netdev` | mlx5 设备和网络接口的映射 | -| `/sys/class/infiniband` | 端口状态、link layer、rate、GID | - -这一层很关键，因为 NCCL 经常因为选错网卡而跑到 TCP 或错误的接口上。 - -### 3.3 单机 GPU 健康层 - -这一层证明每台机器自己是健康的。 - -```mermaid -flowchart LR - H["单机健康检查"] --> T["温度"] - H --> P["功耗"] - H --> E["ECC 错误"] - H --> PCIE["PCIe Gen/Width"] - H --> C["SM/Mem Clock"] - H --> TH["Throttling"] - H --> PM["Persistence Mode"] -``` - -如果某张卡温度过高、ECC double-bit、PCIe 降级或 throttling，后面的 NCCL 测试即使能跑，结果也不可信。 - -### 3.4 单机 GPU 性能层 - -这一层证明每台机器的 GPU 本身性能正常。 - -| 测试 | 证明什么 | -|---|---| -| HBM/D2D 带宽 | GPU 显存和设备间拷贝能力 | -| H2D/D2H 带宽 | CPU/Host 到 GPU 的 PCIe 路径 | -| FP32/TF32 | 基础矩阵计算能力 | -| FP16/BF16/FP8 | 训练常用 Tensor Core 能力 | - -这一步是单机验收。它不能证明两台机器之间通信正常，但可以排除“某台机器本身 GPU 算力或带宽异常”。 - -### 3.5 单机多卡 NCCL 层 - -这一层验证单台机器 8 卡之间的集体通信。 - -```mermaid -flowchart TD - S["单机 8 卡 NCCL"] --> AR["AllReduce"] - S --> AG["AllGather"] - S --> RS["ReduceScatter"] - S --> BC["Broadcast"] - S --> AT["AllToAll"] -``` - -单机 NCCL 主要看 NVLink/NVSwitch 通信路径是否正常。常见指标： - -| 指标 | 含义 | -|---|---| -| `algbw` | 算法视角的有效带宽 | -| `busbw` | 总线视角的带宽，更适合比较通信链路利用率 | -| `#wrong` | 结果错误数量，必须是 0 | - -单机测试通过后，只能说明单台服务器内部 8 卡通信正常。 - -### 3.6 跨机 RDMA 层 - -这一层验证两台机器之间的网络和 RDMA 能力，不涉及 NCCL。 - -```mermaid -sequenceDiagram - participant N1 as aikubeworker0012 - participant FAB as IB/RDMA Fabric - participant N2 as aikubeworker0016 - - N1->>N2: ping / ssh - N1->>FAB: ib_write_bw client - FAB->>N2: ib_write_bw server - N1->>FAB: ib_read_bw client - FAB->>N2: ib_read_bw server - N1->>N2: ib_write_lat / ib_read_lat -``` - -这一层要回答： - -| 问题 | 说明 | -|---|---| -| IB 端口是否 Active | 没 Active 就不用跑 NCCL | -| RDMA 带宽是否达标 | 证明网络数据面能跑起来 | -| RDMA 延迟是否正常 | 高延迟会影响小消息和训练同步 | -| 是否是 InfiniBand/RoCE | 两者环境变量和排障点不同 | - -如果 RDMA 层失败，跨机 NCCL 大概率也会失败或退化到 TCP。 - -### 3.7 跨机 NCCL 层 - -这一层才是真正的多机多卡 NCCL 测试。 - -两台 8 卡机器通常是： - -```text -2 nodes x 8 GPUs = 16 ranks -每个 rank 绑定 1 张 GPU -``` - -概念上是： - -```mermaid -flowchart LR - subgraph N1["Node 1: 172.72.8.12"] - R0["rank 0 / GPU0"] - R1["rank 1 / GPU1"] - R2["..."] - R7["rank 7 / GPU7"] - end - - subgraph N2["Node 2: 172.72.8.16"] - R8["rank 8 / GPU0"] - R9["rank 9 / GPU1"] - R10["..."] - R15["rank 15 / GPU7"] - end - - R0 <--> R8 - R1 <--> R9 - R7 <--> R15 - N1 <--> N2 -``` - -典型测试项： - -| NCCL 测试 | 训练里对应什么 | -|---|---| -| AllReduce | 数据并行梯度同步 | -| ReduceScatter | ZeRO/FSDP 梯度切分 | -| AllGather | ZeRO/FSDP 参数聚合 | -| Broadcast | 参数广播、初始化 | -| AllToAll | MoE、专家并行、部分并行策略 | -| SendRecv | 点对点通信、pipeline parallel | - -跨机 NCCL 要看： - -| 指标 | 判定 | -|---|---| -| 是否成功启动 16 rank | MPI/SSH/路径/环境是否正常 | -| `#wrong == 0` | 正确性必须过 | -| `busbw` | 跨节点通信链路利用率 | -| 是否走 IB/RDMA | 需要从 `NCCL_DEBUG=INFO` 确认 | -| 是否退化 TCP | 如果退化，性能会明显偏低 | - -## 4. NCCL 为什么要分单机和跨机 - -单机 8 卡通信和跨机 16 卡通信的瓶颈不同。 - -```mermaid -flowchart TD - A["NCCL 性能结果"] --> B{"测试范围"} - B --> C["单机 8 卡"] - B --> D["跨机 16 卡"] - - C --> C1["主要瓶颈：NVLink / NVSwitch"] - C --> C2["阈值可参考 GPU NVLink 能力"] - - D --> D1["主要瓶颈：IB/RDMA 网络"] - D --> D2["阈值应参考网卡数量、速率、拓扑和 rail 数"] -``` - -所以不能用单机 NVLink 的阈值直接判断跨机 NCCL。跨机要根据真实网络能力设阈值，例如： - -| 网络配置 | 理论上限理解 | -|---|---| -| 单张 400G 网卡 | 约 50 GB/s 单向原始带宽 | -| 8 张 400G 网卡 | 约 400 GB/s 原始聚合带宽 | -| 实测 NCCL busbw | 会受拓扑、GDR、rail、NUMA、交换机、NCCL 算法影响 | - -实际验收时，应该先知道每台机器有几张 IB/RDMA 网卡、每张速率多少、GPU 到 NIC 的拓扑关系，再定跨机 NCCL 阈值。 - -## 5. 常见失败位置 - -```mermaid -flowchart TD - F["跨机 NCCL 失败"] --> A["启动失败"] - F --> B["能启动但很慢"] - F --> C["运行中 timeout"] - F --> D["结果 #wrong 非 0"] - - A --> A1["SSH 不通"] - A --> A2["远端路径不存在"] - A --> A3["MPI 环境不一致"] - A --> A4["root 运行未允许"] - - B --> B1["NCCL_SOCKET_IFNAME 选错"] - B --> B2["没走 IB/RDMA，退化 TCP"] - B --> B3["NCCL_IB_HCA 没选对"] - B --> B4["GPU Direct RDMA 没生效"] - - C --> C1["IB 端口不稳定"] - C --> C2["交换机/PFC/ECN 问题"] - C --> C3["NCCL timeout 配置"] - C --> C4["驱动/CUDA/NCCL 版本不兼容"] - - D --> D1["通信正确性失败"] - D --> D2["必须 FAIL，不能只看带宽"] -``` - -## 6. 推荐验收顺序 - -下面是面向两台 8 卡机器的推荐顺序： - -```mermaid -flowchart TD - A["Step 1: 两台机器基础信息"] --> B["Step 2: 两台机器单机 GPU 健康"] - B --> C["Step 3: 两台机器单机 benchmark"] - C --> D["Step 4: 两台机器分别跑单机 8 卡 NCCL"] - D --> E["Step 5: 两台机器互测 RDMA bandwidth/latency"] - E --> F["Step 6: 两机 16 卡 NCCL correctness"] - F --> G["Step 7: 两机 16 卡 NCCL performance"] - G --> H["Step 8: 两机训练 demo 或业务压测"] -``` - -每一步的意义： - -| 步骤 | 目的 | -|---|---| -| Step 1 | 确认没有登录错机器，基础网络和环境存在 | -| Step 2 | 排除 GPU 健康问题 | -| Step 3 | 排除 GPU 单卡/单机性能问题 | -| Step 4 | 排除单机 NVLink/NVSwitch/NCCL 问题 | -| Step 5 | 排除跨机 RDMA 问题 | -| Step 6 | 先证明 NCCL 正确性 | -| Step 7 | 再证明 NCCL 性能 | -| Step 8 | 最后用真实训练形态验证稳定性 | - -## 7. 对当前脚本的映射 - -当前脚本已有模块和上面层次的关系： - -| 当前模块 | 覆盖层次 | 备注 | -|---|---|---| -| `gpu_info` | 系统识别层 | 单机 | -| `health` | 单机 GPU 健康层 | 单机 | -| `benchmark` | 单机 GPU 性能层 | 单机 | -| `nccl` | 单机多卡通信层 | 当前主要是单机 | -| `rdma` | RDMA 检查 | 当前偏本机检查，不是两机互测 | -| `stress` | 稳定性 | 单机 | -| `training` | 训练负载层 | 当前偏单机 | -| 建议新增 `multi_node_nccl` | 跨机 NCCL 层 | 专门处理 hostfile、mpirun、多节点环境、结果解析 | - -如果未来要扩展脚本，比较自然的方向是新增一个多机模块，而不是把所有逻辑塞进现有 `nccl` 模块。 - -## 8. 最小概念模型 - -记住这句话即可： - -```text -单机 NCCL 验证 GPU 之间的 NVLink/NVSwitch。 -跨机 RDMA 验证机器之间的网络。 -跨机 NCCL 验证 NCCL 是否能把 GPU 和网络组合起来，为真实训练提供高效通信。 -``` - -因此，多机多卡测试不是一个命令，而是一条验证链路。 diff --git a/docs/multinode_nccl_deep_diagnose_runbook.md b/docs/multinode_nccl_deep_diagnose_runbook.md deleted file mode 100644 index 433d1ce..0000000 --- a/docs/multinode_nccl_deep_diagnose_runbook.md +++ /dev/null @@ -1,219 +0,0 @@ -# 多机 NCCL 深度诊断 runbook - -本文档用于复现 2026-05-23 这轮 2 机 8 卡 NCCL 排查里的关键动作：counter 抓取、GRAPH/TUNING 日志、以及 PXN disabled 基线上的二次参数 sweep。 - -## 适用场景 - -当前默认参数面向： - -- `aikubeworker0012` / `172.72.8.12` -- `aikubeworker0016` / `172.72.8.16` -- 每节点 8 GPU -- 每节点 4 条 400G HCA：`mlx5_0,mlx5_1,mlx5_6,mlx5_7` -- NCCL 临时运行库：`/tmp/nccl-2.27.7-cuda12.4` -- nccl-tests：`/data/nccl-tests-latest/build` -- OpenMPI：`/usr/mpi/gcc/openmpi-4.1.9a1/bin/mpirun` - -脚本应在 coordinator 节点上执行，当前即 `aikubeworker0012`。 - -## 快速运行 - -```bash -cd /root/test_gpu_scripts -bash scripts/multinode_nccl_deep_diagnose.sh preflight -bash scripts/multinode_nccl_deep_diagnose.sh all -``` - -如果要按 PDF 参考矩阵跑正式多机多卡报告，使用： - -```bash -cd /root/test_gpu_scripts -bash scripts/run_multinode_nccl_pdf_matrix.sh -``` - -它会跑 2 机 x 1/2/4/8 GPU per node 的 `all_reduce_perf` 和 `alltoall_perf`，输出到 -`reports/multinode_nccl_pdf_matrix_YYYYMMDD_HHMMSS.md`。 - -同时会生成： - -```text -reports/multinode_nccl_pdf_matrix_YYYYMMDD_HHMMSS_artifacts/ -``` - -每个 case 保存完整 `*.cmd.txt`、`*.stdout.txt`、`*.stderr.txt` 和解析后的 `*.json`，用于复核原始 NCCL 输出。 - -默认输出目录为： - -```text -/tmp/nccl_deep_diagnose_YYYYMMDD_HHMMSS -``` - -只跑单项： - -```bash -# 轻量检查 SSH、mpirun、nccl-tests 和 HCA 路径 -bash scripts/multinode_nccl_deep_diagnose.sh preflight - -# allreduce counter 对照 -bash scripts/multinode_nccl_deep_diagnose.sh allreduce-counter - -# PXN disabled alltoall counter -bash scripts/multinode_nccl_deep_diagnose.sh alltoall-counter - -# NCCL GRAPH/TUNING/COLL 对照 -bash scripts/multinode_nccl_deep_diagnose.sh graph - -# PXN disabled 基线上的二次参数 sweep -bash scripts/multinode_nccl_deep_diagnose.sh pxn-sweep -``` - -## 常用参数覆盖 - -```bash -OUT_DIR=/tmp/my_nccl_diag \ -HOSTS=172.72.8.12:8,172.72.8.16:8 \ -PEER_HOST=172.72.8.16 \ -HCAS="mlx5_0 mlx5_1 mlx5_6 mlx5_7" \ -HCA_CSV=mlx5_0,mlx5_1,mlx5_6,mlx5_7 \ -bash scripts/multinode_nccl_deep_diagnose.sh all -``` - -如果 nccl-tests 或 NCCL 运行库路径变化： - -```bash -NCCL_TESTS_DIR=/data/nccl-tests-latest/build \ -NCCL_LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi-4.1.9a1/lib:/path/to/nccl/lib:/usr/local/cuda/lib64 \ -bash scripts/multinode_nccl_deep_diagnose.sh graph -``` - -## 输出解读 - -### preflight 模式 - -典型输出文件： - -```text -preflight.txt -``` - -该模式不跑 NCCL workload，只检查： - -- 本机和对端主机名。 -- OpenMPI `mpirun` 是否存在且可执行。 -- `all_reduce_perf` / `alltoall_perf` 是否存在且可执行。 -- 配置的 HCA 是否能在 `/sys/class/infiniband//ports/1` 下读到 state/rate。 -- 发起节点到 `PEER_HOST` 的 root SSH 是否可用。 - -如果这里出现 `MISSING`，先修环境；否则再跑 `all` 或单项诊断。 - -### counter 模式 - -典型输出文件： - -```text -allreduce_counter/ - allreduce.log - before.local - before.remote - after.local - after.remote - counter_delta.txt - -alltoall_pxn_counter/ - alltoall_pxn.log - before.local - before.remote - after.local - after.remote - counter_delta.txt -``` - -重点看 `counter_delta.txt`： - -- `port_xmit_data` / `port_rcv_data`：端口流量，单位为 4-byte words，脚本同时换算 GiB。 -- `port_xmit_wait`：发送等待或 credit/拥塞等待信号。注意它不是 alltoall 独有根因，因为高吞吐 allreduce 也会出现。 -- `port_xmit_discards`、`port_rcv_errors`、`symbol_error`、`roce_adp_retrans`、`packet_seq_err` 等：错误、丢包、重传、链路异常类信号。 - -当前已知基线： - -- allreduce 可到约 `354 GB/s busbw`，4 条 rail 均衡。 -- PXN disabled alltoall 通常在 `36-37 GB/s busbw` 附近，但有窗口波动。 -- alltoall PXN disabled 后 rail 均衡，且没有明显 error/retrans/slow restart。 - -### graph 模式 - -典型输出文件： - -```text -graph/ - allreduce.log - allreduce_summary.txt - alltoall_pxn.log - alltoall_pxn_summary.txt -``` - -重点看： - -- `nccl_version` -- `plugin_missing` -- `gdr_enabled_lines` -- `pattern_counts` -- `channel_summary` -- `NET/IB/*/GDRDMA` -- `P2P/CUMEM` -- `channel_edge_lines` - -当前已知对照： - -| 观察项 | allreduce | alltoall + `NCCL_PXN_DISABLE=1` | -|--------|-----------|----------------------------------| -| HCA / GDR | 4 HCA, GDR enabled | 4 HCA, GDR enabled | -| channels | `16 coll / 16 nvls / 16 p2p` | `16 coll / 16 nvls / 16 p2p` | -| `NET/IB/*/GDRDMA` channel edge lines | `256` | `512` | -| `P2P/CUMEM` channel edge lines | `0` | `224` | -| total NET/P2P channel edge lines | `256` | `736` | - -判断边界： - -- 如果 HCA/GDR/channel 基础状态一致，但 alltoall graph 明显更复杂，问题更偏向 NCCL collective graph、P2P/NET 组合方式、internal IB plugin 或交换网络策略。 -- 如果 GDR disabled、HCA 不完整、plugin 路径变化，则不能直接与当前报告结论对比。 - -### pxn-sweep 模式 - -典型输出： - -```text -pxn_sweep/ - baseline.log - nvls_off.log - qps4_split1.log - qps8_split1.log - qps4_split0.log - channels16.log - buff8m.log - p2pchunk4m.log - netpeer8.log - ar0.log - summary.txt -``` - -当前结论： - -- `NCCL_PXN_DISABLE=1` 是已发现的唯一稳定正向项。 -- 在 PXN disabled 基线上继续叠加 NVLS、P2P chunk、buffer、channel、QP/split、AR，没有稳定收益。 -- QP/split 和 `NCCL_NCHANNELS_PER_NET_PEER=8` 在当前环境下明显变差。 - -## 交接给网络/NCCL 环境侧的重点 - -1. 当前不是旧 NCCL/GDR disabled 问题：NCCL `2.27.7` 下 4 条 HCA 都是 GDR enabled。 -2. 当前不是 rail 完全打偏问题：`NCCL_PXN_DISABLE=1` 后 alltoall 的 4 条 rail 已均衡。 -3. 当前不是明显坏链路/重传问题：未看到 discard、symbol error、RoCE retrans、slow restart、packet sequence error 等增长。 -4. allreduce 已接近当前 4 x 400G rail 的物理可用带宽；PDF 8 卡 allreduce 目标反推需要超过当前 4 rail 单向理论带宽。 -5. alltoall 剩余差距更像 NCCL internal alltoall graph、P2P/NET 组合方式、缺少 NCCL net plugin/SHARP，或交换网络策略/ECMP/拥塞控制问题。 - -## 关联报告 - -- `reports_multinode_nccl_diagnosis_20260523.md` -- `reports_multinode_nccl_alltoall_tuning_20260523.md` -- `reports_multinode_nccl_counter_probe_20260523.md` -- `reports_multinode_nccl_pdf_matrix_nccl227.md` diff --git a/reports_all_aikubeworker0016.json b/reports_all_aikubeworker0016.json deleted file mode 100644 index d3db53f..0000000 --- a/reports_all_aikubeworker0016.json +++ /dev/null @@ -1,921 +0,0 @@ -{ - "timestamp": "2026-05-22T15:49:02.368516", - "gpu_info": { - "driver_version": "580.159.03", - "cuda_version": "13.0", - "gpu_count": 8, - "gpus": [ - { - "index": 0, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-dfbc9513-255d-4fe7-2b77-7b1ec3972e75", - "pci_bus_id": "00000000:18:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 4, - "vram_free_mb": 81076, - "power_draw": 69.98, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 21, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924016120", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 1, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-bb845ef7-d7b5-f011-9395-ea74274e2282", - "pci_bus_id": "00000000:2A:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 4, - "vram_free_mb": 81076, - "power_draw": 67.54, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 21, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924015483", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 2, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-3720cf13-2a34-be38-27be-0a7adc4addc4", - "pci_bus_id": "00000000:3A:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 4, - "vram_free_mb": 81076, - "power_draw": 66.82, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 22, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924025595", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 3, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-87080b2d-ac43-be0d-d574-c193078850ae", - "pci_bus_id": "00000000:5D:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 4, - "vram_free_mb": 81076, - "power_draw": 67.02, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 21, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924016862", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 4, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-599bd883-cc5c-a5dd-6c33-c15f7049da48", - "pci_bus_id": "00000000:9A:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 4, - "vram_free_mb": 81076, - "power_draw": 67.24, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 21, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924025670", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 5, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-a1c6bba4-61b0-e623-06c9-9c88635e26fe", - "pci_bus_id": "00000000:AB:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 4, - "vram_free_mb": 81076, - "power_draw": 69.31, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 23, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924027166", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 6, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-98745a0c-39bd-3e56-d6ca-54ba3647ab6d", - "pci_bus_id": "00000000:BA:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 4, - "vram_free_mb": 81076, - "power_draw": 67.84, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 21, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924026234", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 7, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-8c73bd8b-666b-357e-ac5d-c75ac7a759db", - "pci_bus_id": "00000000:DB:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 4, - "vram_free_mb": 81076, - "power_draw": 66.21, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 21, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924027255", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - } - ], - "topology": "\t\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tGPU4\tGPU5\tGPU6\tGPU7\tNIC0\tNIC1\tNIC2\tNIC3\tNIC4\tNIC5\tNIC6\tNIC7\tNIC8\tNIC9\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tPIX\tNODE\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU1\tNV18\t X \tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tNODE\tPIX\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU2\tNV18\tNV18\t X \tNV18\tNV18\tNV18\tNV18\tNV18\tNODE\tNODE\tPIX\tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU3\tNV18\tNV18\tNV18\t X \tNV18\tNV18\tNV18\tNV18\tNODE\tNODE\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU4\tNV18\tNV18\tNV18\tNV18\t X \tNV18\tNV18\tNV18\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tPIX\tNODE\tNODE\tNODE\t56-111,168-223\t1\t\tN/A\nGPU5\tNV18\tNV18\tNV18\tNV18\tNV18\t X \tNV18\tNV18\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tPIX\tNODE\tNODE\t56-111,168-223\t1\t\tN/A\nGPU6\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\t X \tNV18\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\tPIX\t56-111,168-223\t1\t\tN/A\nGPU7\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\t X \tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tNODE\tNODE\t56-111,168-223\t1\t\tN/A\nNIC0\tPIX\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t X \tNODE\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC1\tNODE\tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\t X \tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC2\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\t X \tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC3\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\t X \tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC4\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tNODE\tNODE\t X \tPIX\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC5\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tNODE\tNODE\tPIX\t X \tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC6\tSYS\tSYS\tSYS\tSYS\tPIX\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\t X \tNODE\tNODE\tNODE\t\t\t\t\nNIC7\tSYS\tSYS\tSYS\tSYS\tNODE\tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\t X \tNODE\tNODE\t\t\t\t\nNIC8\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\t X \tPIX\t\t\t\t\nNIC9\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\t X \t\t\t\t\n\nLegend:\n\n X = Self\n SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n PIX = Connection traversing at most a single PCIe bridge\n NV# = Connection traversing a bonded set of # NVLinks\n\nNIC Legend:\n\n NIC0: mlx5_0\n NIC1: mlx5_1\n NIC2: mlx5_2\n NIC3: mlx5_3\n NIC4: mlx5_4\n NIC5: mlx5_5\n NIC6: mlx5_6\n NIC7: mlx5_7\n NIC8: mlx5_8\n NIC9: mlx5_9\n\n", - "timestamp": "2026-05-22T15:49:09.197459", - "detected_gpu_type": "h100", - "gpu_label": "H100 SXM5" - }, - "health": { - "passed": true, - "gpu_health": [ - { - "index": 0, - "status": "WARN", - "checks": { - "temperature": { - "value": 21, - "status": "PASS", - "threshold": 75 - }, - "power": { - "value": 69.86, - "limit": 700.0, - "status": "PASS" - }, - "ecc_errors": { - "single": 0, - "double": 0, - "status": "PASS" - }, - "memory_errors": { - "status": "PASS" - }, - "pcie_link": { - "gen": 5, - "width": 16, - "status": "PASS" - }, - "clock_speed": { - "sm": 345, - "mem": 2619, - "status": "PASS" - }, - "throttling": { - "status": "PASS", - "reasons": [] - }, - "persistence_mode": { - "enabled": false, - "status": "WARN" - } - } - }, - { - "index": 1, - "status": "WARN", - "checks": { - "temperature": { - "value": 21, - "status": "PASS", - "threshold": 75 - }, - "power": { - "value": 67.48, - "limit": 700.0, - "status": "PASS" - }, - "ecc_errors": { - "single": 0, - "double": 0, - "status": "PASS" - }, - "memory_errors": { - "status": "PASS" - }, - "pcie_link": { - "gen": 5, - "width": 16, - "status": "PASS" - }, - "clock_speed": { - "sm": 345, - "mem": 2619, - "status": "PASS" - }, - "throttling": { - "status": "PASS", - "reasons": [] - }, - "persistence_mode": { - "enabled": false, - "status": "WARN" - } - } - }, - { - "index": 2, - "status": "WARN", - "checks": { - "temperature": { - "value": 22, - "status": "PASS", - "threshold": 75 - }, - "power": { - "value": 66.76, - "limit": 700.0, - "status": "PASS" - }, - "ecc_errors": { - "single": 0, - "double": 0, - "status": "PASS" - }, - "memory_errors": { - "status": "PASS" - }, - "pcie_link": { - "gen": 5, - "width": 16, - "status": "PASS" - }, - "clock_speed": { - "sm": 345, - "mem": 2619, - "status": "PASS" - }, - "throttling": { - "status": "PASS", - "reasons": [] - }, - "persistence_mode": { - "enabled": false, - "status": "WARN" - } - } - }, - { - "index": 3, - "status": "WARN", - "checks": { - "temperature": { - "value": 21, - "status": "PASS", - "threshold": 75 - }, - "power": { - "value": 67.06, - "limit": 700.0, - "status": "PASS" - }, - "ecc_errors": { - "single": 0, - "double": 0, - "status": "PASS" - }, - "memory_errors": { - "status": "PASS" - }, - "pcie_link": { - "gen": 5, - "width": 16, - "status": "PASS" - }, - "clock_speed": { - "sm": 345, - "mem": 2619, - "status": "PASS" - }, - "throttling": { - "status": "PASS", - "reasons": [] - }, - "persistence_mode": { - "enabled": false, - "status": "WARN" - } - } - }, - { - "index": 4, - "status": "WARN", - "checks": { - "temperature": { - "value": 21, - "status": "PASS", - "threshold": 75 - }, - "power": { - "value": 67.23, - "limit": 700.0, - "status": "PASS" - }, - "ecc_errors": { - "single": 0, - "double": 0, - "status": "PASS" - }, - "memory_errors": { - "status": "PASS" - }, - "pcie_link": { - "gen": 5, - "width": 16, - "status": "PASS" - }, - "clock_speed": { - "sm": 345, - "mem": 2619, - "status": "PASS" - }, - "throttling": { - "status": "PASS", - "reasons": [] - }, - "persistence_mode": { - "enabled": false, - "status": "WARN" - } - } - }, - { - "index": 5, - "status": "WARN", - "checks": { - "temperature": { - "value": 23, - "status": "PASS", - "threshold": 75 - }, - "power": { - "value": 69.27, - "limit": 700.0, - "status": "PASS" - }, - "ecc_errors": { - "single": 0, - "double": 0, - "status": "PASS" - }, - "memory_errors": { - "status": "PASS" - }, - "pcie_link": { - "gen": 5, - "width": 16, - "status": "PASS" - }, - "clock_speed": { - "sm": 345, - "mem": 2619, - "status": "PASS" - }, - "throttling": { - "status": "PASS", - "reasons": [] - }, - "persistence_mode": { - "enabled": false, - "status": "WARN" - } - } - }, - { - "index": 6, - "status": "WARN", - "checks": { - "temperature": { - "value": 21, - "status": "PASS", - "threshold": 75 - }, - "power": { - "value": 67.81, - "limit": 700.0, - "status": "PASS" - }, - "ecc_errors": { - "single": 0, - "double": 0, - "status": "PASS" - }, - "memory_errors": { - "status": "PASS" - }, - "pcie_link": { - "gen": 5, - "width": 16, - "status": "PASS" - }, - "clock_speed": { - "sm": 345, - "mem": 2619, - "status": "PASS" - }, - "throttling": { - "status": "PASS", - "reasons": [] - }, - "persistence_mode": { - "enabled": false, - "status": "WARN" - } - } - }, - { - "index": 7, - "status": "WARN", - "checks": { - "temperature": { - "value": 21, - "status": "PASS", - "threshold": 75 - }, - "power": { - "value": 66.3, - "limit": 700.0, - "status": "PASS" - }, - "ecc_errors": { - "single": 0, - "double": 0, - "status": "PASS" - }, - "memory_errors": { - "status": "PASS" - }, - "pcie_link": { - "gen": 5, - "width": 16, - "status": "PASS" - }, - "clock_speed": { - "sm": 345, - "mem": 2619, - "status": "PASS" - }, - "throttling": { - "status": "PASS", - "reasons": [] - }, - "persistence_mode": { - "enabled": false, - "status": "WARN" - } - } - } - ], - "system_health": { - "nvidia_persistenced": { - "installed": true, - "running": false - }, - "hugepages": { - "configured": false, - "count": 0 - }, - "swap": { - "enabled": true - }, - "transparent_hugepage": "madvise", - "file_descriptors": { - "soft": 1024, - "max": 1048576 - }, - "infiniband_devices": [ - "mlx5_4", - "mlx5_2", - "mlx5_0", - "mlx5_9", - "mlx5_7", - "mlx5_5", - "mlx5_3", - "mlx5_1", - "mlx5_8", - "mlx5_6" - ], - "rdma_devices": [ - "abi_version", - "uverbs4", - "uverbs2", - "uverbs0", - "uverbs9", - "uverbs7", - "uverbs5", - "uverbs3", - "uverbs1", - "uverbs8", - "uverbs6" - ], - "nccl_env_vars": {} - }, - "timestamp": "2026-05-22T15:49:11.294816", - "detected_gpu_type": "h100" - }, - "memory_bench": { - "memory": { - "source": "nvbandwidth", - "h2d_bandwidth_gbps": 55.5, - "d2h_bandwidth_gbps": 55.3, - "d2d_bandwidth_gbps": 486.5, - "h2d_peak_gbps": 64, - "d2h_peak_gbps": 64, - "d2d_peak_gbps": 450.0, - "h2d_efficiency_pct": 86.7, - "d2h_efficiency_pct": 86.4, - "d2d_efficiency_pct": 108.1, - "peak_bandwidth_gbps": 3400, - "efficiency_pct": 108.1, - "results_by_test": { - "h2d": 55.5, - "d2h": 55.3, - "d2d_write": 397.4, - "d2d_read": 395.1, - "d2d_bidir": 486.5 - }, - "per_gpu": [] - } - }, - "compute_bench": { - "compute": { - "per_dtype_tflops": { - "fp32": 51.9, - "tf32": 357.0, - "fp16": 664.0, - "bf16": 700.1, - "fp8": 1116.2 - }, - "peak_tflops": { - "fp32": 67, - "tf32": 495, - "fp16": 990, - "bf16": 990, - "fp8": 1979 - }, - "efficiency_pct": { - "fp32": 77.5, - "tf32": 72.1, - "fp16": 67.1, - "bf16": 70.7, - "fp8": 56.4 - }, - "pass_thresholds_tflops": { - "fp32": 54, - "tf32": 444, - "fp16": 734, - "bf16": 745, - "fp8": 1400 - }, - "per_gpu": [ - { - "index": 0, - "fp32": 51.9, - "tf32": 357.0, - "fp16": 664.0, - "bf16": 700.1, - "fp8": 1116.2 - }, - { - "index": 1, - "fp32": 51.9, - "tf32": 357.0, - "fp16": 664.0, - "bf16": 700.1, - "fp8": 1116.2 - }, - { - "index": 2, - "fp32": 51.9, - "tf32": 357.0, - "fp16": 664.0, - "bf16": 700.1, - "fp8": 1116.2 - }, - { - "index": 3, - "fp32": 51.9, - "tf32": 357.0, - "fp16": 664.0, - "bf16": 700.1, - "fp8": 1116.2 - }, - { - "index": 4, - "fp32": 51.9, - "tf32": 357.0, - "fp16": 664.0, - "bf16": 700.1, - "fp8": 1116.2 - }, - { - "index": 5, - "fp32": 51.9, - "tf32": 357.0, - "fp16": 664.0, - "bf16": 700.1, - "fp8": 1116.2 - }, - { - "index": 6, - "fp32": 51.9, - "tf32": 357.0, - "fp16": 664.0, - "bf16": 700.1, - "fp8": 1116.2 - }, - { - "index": 7, - "fp32": 51.9, - "tf32": 357.0, - "fp16": 664.0, - "bf16": 700.1, - "fp8": 1116.2 - } - ], - "matrix_size": 8192, - "warmup": 50, - "iterations": 500 - } - }, - "nccl": { - "passed": false, - "source": "torchrun_fallback", - "tests": { - "NCCL version 2.21.5+cuda12.4": { - "status": "FAIL", - "error": null - }, - "allreduce": { - "status": "PASS", - "error": null - }, - "broadcast": { - "status": "PASS", - "error": null - }, - "allgather": { - "status": "PASS", - "error": null - }, - "reducescatter": { - "status": "PASS", - "error": null - }, - "alltoall": { - "status": "PASS", - "error": null - } - }, - "gpu_count": 8 - }, - "stress": { - "source": "pytorch", - "passed": true, - "duration_sec": 60, - "elapsed_sec": 60.0, - "gpu_status": { - "0": "PASS", - "1": "PASS", - "2": "PASS", - "3": "PASS", - "4": "PASS", - "5": "PASS", - "6": "PASS", - "7": "PASS" - }, - "timestamp": "2026-05-22T15:51:56.803540" - }, - "rdma": { - "passed": false, - "devices": [ - { - "name": "mlx5_0", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:58a2:e103:0088:81e0" - } - ] - }, - { - "name": "mlx5_1", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:9c63:c003:0054:e00a" - } - ] - }, - { - "name": "mlx5_2", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:a02d:75ff:feae:2bcf" - } - ] - }, - { - "name": "mlx5_3", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "1: DOWN", - "phys_state": "3: Disabled", - "gid": "fe80:0000:0000:0000:c670:bdff:fefd:5bd9" - } - ] - }, - { - "name": "mlx5_4", - "ports": [ - { - "port": "1", - "rate": "100 Gb/sec (2X HDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:9c63:c003:005f:58ec" - } - ] - }, - { - "name": "mlx5_5", - "ports": [ - { - "port": "1", - "rate": "100 Gb/sec (2X HDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:9c63:c003:005f:58ed" - } - ] - }, - { - "name": "mlx5_6", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:9c63:c003:0055:0e56" - } - ] - }, - { - "name": "mlx5_7", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:a088:c203:00f0:286c" - } - ] - }, - { - "name": "mlx5_8", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:a02d:75ff:feae:2bcf" - } - ] - }, - { - "name": "mlx5_9", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "1: DOWN", - "phys_state": "3: Disabled", - "gid": "fe80:0000:0000:0000:c670:bdff:fefd:569d" - } - ] - } - ], - "bandwidth_tests": [ - { - "test": "ib_write_bw", - "status": "WARN", - "bandwidth_gbps": 0.13, - "min_required_gbps": 50 - }, - { - "test": "ib_read_bw", - "status": "WARN", - "bandwidth_gbps": 0.13, - "min_required_gbps": 50 - } - ], - "latency_tests": [ - { - "test": "ib_write_lat", - "status": "PASS", - "latency_us": 4.1, - "max_allowed_us": 10 - }, - { - "test": "ib_read_lat", - "status": "WARN", - "latency_us": 16.0, - "max_allowed_us": 10 - } - ], - "timestamp": "2026-05-22T15:52:03.507540" - }, - "training": { - "model": "synthetic_transformer", - "total_params_m": 1470.5, - "num_layers": 6, - "hidden_size": 4096, - "gpu_count": 8, - "dtype": "bfloat16", - "batch_size": 8, - "seq_length": 2048, - "num_steps": 50, - "avg_step_time_ms": 312.3, - "throughput_tokens_per_sec": 52471.0, - "throughput_samples_per_sec": 25.62, - "peak_memory_gb": 27.31, - "final_loss": 0.0041, - "timestamp": "2026-05-22T15:52:32.650522" - } -} \ No newline at end of file diff --git a/reports_all_aikubeworker0016.md b/reports_all_aikubeworker0016.md deleted file mode 100644 index 80dda75..0000000 --- a/reports_all_aikubeworker0016.md +++ /dev/null @@ -1,157 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T15:49:02.368516 -- **Host:** aikubeworker0016 -- **GPU:** NVIDIA H100 80GB HBM3 x8 -- **Driver:** 580.159.03 | **CUDA:** 13.0 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Failed or unverified items: -- Compute Throughput: FAIL (worst FP32 52 vs >= 54) -- NCCL: FAIL (no nccl-tests bus BW) -- RDMA: FAIL -- Training: UNVERIFIED (52471 tokens/sec; legacy result lacks explicit acceptance verdict) - -Missing required evidence: -- NVLink/NVSwitch -- DCGM - -## Summary - -| Test | Result | -|------|--------| -| GPU Info | PASS (8 GPUs detected) | -| Health Check | PASS | -| Memory Bandwidth | PASS (108.1%) | -| Compute Throughput | FAIL (worst FP32 52 vs >= 54) | -| NCCL | FAIL (no nccl-tests bus BW) | -| Stress Test | PASS | -| RDMA | FAIL | -| Training | UNVERIFIED (52471 tokens/sec; legacy result lacks explicit acceptance verdict) | - -## GPU Information - -| GPU | Model | VRAM | Temp | Power | SM Clock | -|-----|-------|------|------|-------|----------| -| 0 | NVIDIA H100 80GB HBM3 | 81559 MB | 21C | 70/700W | 345 MHz | -| 1 | NVIDIA H100 80GB HBM3 | 81559 MB | 21C | 68/700W | 345 MHz | -| 2 | NVIDIA H100 80GB HBM3 | 81559 MB | 22C | 67/700W | 345 MHz | -| 3 | NVIDIA H100 80GB HBM3 | 81559 MB | 21C | 67/700W | 345 MHz | -| 4 | NVIDIA H100 80GB HBM3 | 81559 MB | 21C | 67/700W | 345 MHz | -| 5 | NVIDIA H100 80GB HBM3 | 81559 MB | 23C | 69/700W | 345 MHz | -| 6 | NVIDIA H100 80GB HBM3 | 81559 MB | 21C | 68/700W | 345 MHz | -| 7 | NVIDIA H100 80GB HBM3 | 81559 MB | 21C | 66/700W | 345 MHz | - -## Health Check - -**Overall: PASS** - -| GPU | Temp | Power | ECC | PCIe | Throttle | Status | -|-----|------|-------|-----|------|----------|--------| -| 0 | 21C PASS | 70W PASS | S:0 D:0 | Gen5x16 | PASS | **WARN** | -| 1 | 21C PASS | 67W PASS | S:0 D:0 | Gen5x16 | PASS | **WARN** | -| 2 | 22C PASS | 67W PASS | S:0 D:0 | Gen5x16 | PASS | **WARN** | -| 3 | 21C PASS | 67W PASS | S:0 D:0 | Gen5x16 | PASS | **WARN** | -| 4 | 21C PASS | 67W PASS | S:0 D:0 | Gen5x16 | PASS | **WARN** | -| 5 | 23C PASS | 69W PASS | S:0 D:0 | Gen5x16 | PASS | **WARN** | -| 6 | 21C PASS | 68W PASS | S:0 D:0 | Gen5x16 | PASS | **WARN** | -| 7 | 21C PASS | 66W PASS | S:0 D:0 | Gen5x16 | PASS | **WARN** | - -## Memory Bandwidth - -Source: nvbandwidth - -| Metric | Value | Peak | Efficiency | -|--------|-------|------|------------| -| H2D (PCIe) | 55.5 GB/s | 64 GB/s | 86.7% | -| D2H (PCIe) | 55.3 GB/s | 64 GB/s | 86.4% | -| D2D (NVLink) | 486.5 GB/s | 450 GB/s | 108.1% | - -**Verdict: PASS** (D2D efficiency 108.1%) - -## Compute Throughput - -| DType | Achieved (TFLOPS) | Peak | Threshold | Status | -|-------|-------------------|------|------------|--------| -| FP32 | 51.9 | 67 | >= 54 | FAIL | -| TF32 | 357.0 | 495 | >= 444 | FAIL | -| FP16 | 664.0 | 990 | >= 734 | FAIL | -| BF16 | 700.1 | 990 | >= 745 | FAIL | -| FP8 | 1116.2 | 1979 | >= 1400 | FAIL | - -**Verdict: FAIL** (absolute TFLOPS thresholds; worst efficiency 56.4%) - -### Compute Per-GPU TFLOPS - -| GPU | FP32 | TF32 | FP16 | BF16 | FP8 | -|---|---|---|---|---|---| -| 0 | 51.9 | 357.0 | 664.0 | 700.1 | 1116.2 | -| 1 | 51.9 | 357.0 | 664.0 | 700.1 | 1116.2 | -| 2 | 51.9 | 357.0 | 664.0 | 700.1 | 1116.2 | -| 3 | 51.9 | 357.0 | 664.0 | 700.1 | 1116.2 | -| 4 | 51.9 | 357.0 | 664.0 | 700.1 | 1116.2 | -| 5 | 51.9 | 357.0 | 664.0 | 700.1 | 1116.2 | -| 6 | 51.9 | 357.0 | 664.0 | 700.1 | 1116.2 | -| 7 | 51.9 | 357.0 | 664.0 | 700.1 | 1116.2 | - -## NCCL Multi-GPU - -Source: torchrun_fallback | GPUs: 8 - -> Functional NCCL smoke only: nccl-tests bus bandwidth was not measured, so this does not satisfy production acceptance. - -| Operation | Bus BW (GB/s) | Threshold | Status | -|-----------|---------------|-----------|--------| -| NCCL version 2.21.5+cuda12.4 | 0.0 | >= 0 | FAIL | -| allreduce | 0.0 | >= 0 | PASS | -| broadcast | 0.0 | >= 0 | PASS | -| allgather | 0.0 | >= 0 | PASS | -| reducescatter | 0.0 | >= 0 | PASS | -| alltoall | 0.0 | >= 0 | PASS | - -**Overall: FAIL** - -## Stress Test - -- **Source:** pytorch -- **Duration:** 60s (requested 60s) -- **Result: PASS** - -## RDMA/InfiniBand - -> Legacy RDMA result re-evaluated with current PDF acceptance thresholds; old WARN statuses and old 50GB/s/10us limits are not used for verdict. - -| Test | Value | Threshold | Status | -|------|-------|-----------|--------| -| ib_write_bw | 0.1 GB/s | >= 47 GB/s | FAIL | -| ib_read_bw | 0.1 GB/s | >= 47 GB/s | FAIL | -| ib_write_lat | 4.10 us | <= 2 us | FAIL | -| ib_read_lat | 16.00 us | <= 3.5 us | FAIL | - -- **Failure reasons:** - - ib_write_bw bandwidth 0.13GB/s < 47GB/s - - ib_read_bw bandwidth 0.13GB/s < 47GB/s - - ib_write_lat latency 4.1us > 2us - - ib_read_lat latency 16.0us > 3.5us -**Overall: FAIL** - -## Training Simulation - -| Metric | Value | -|--------|-------| -| Model | synthetic_transformer | -| Params | 1470.5M | -| Throughput | 52471 tokens/sec | -| Avg Step Time | 312.3 ms | -| Peak Memory | 27.3 GB | -| Final Loss | 0.0041 | -| Step Jitter | N/A% | -| Distributed Mode | N/A | -| Acceptance Gaps | missing passed, step_jitter_pct, distributed_mode, loss_finite | -| Verdict | UNVERIFIED (52471 tokens/sec; legacy result lacks explicit acceptance verdict) | - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_cublaslt_fp8_crosscheck_20260524.md b/reports_cublaslt_fp8_crosscheck_20260524.md deleted file mode 100644 index 194a562..0000000 --- a/reports_cublaslt_fp8_crosscheck_20260524.md +++ /dev/null @@ -1,87 +0,0 @@ -# cuBLASLt FP8 GEMM Cross-Check Report - -Date: 2026-05-24 - -Scope: Validate whether the single-node FP8 compute FAIL is caused by hardware/platform limits or by the original PyTorch `_scaled_mm` benchmark path. - -## Method - -Added a direct cuBLASLt FP8 GEMM micro-benchmark: - -- Source: `scripts/cublaslt_fp8_gemm_bench.cu` -- Wrapper: `scripts/run_cublaslt_fp8_gemm.sh` -- Input dtype: `CUDA_R_8F_E4M3` -- Output dtype: `CUDA_R_16BF` -- Accumulate / compute type: `CUBLAS_COMPUTE_32F` -- Layout: cuBLASLt FP8-required TN format -- Matrix size: `8192` -- Warmup: `50` -- Iterations: `500` -- GPUs: single-node 8 GPUs, measured one GPU at a time - -NVIDIA cuBLASLt documentation states FP8 kernels require TN format, `CUBLAS_COMPUTE_32F`, and `CUDA_R_32F` scale type. The implemented benchmark follows those constraints. - -## Results - -### aikubeworker0012 / nccl-gpu-1 - -Raw report: `reports_cublaslt_fp8_gemm_aikubeworker0012_20260524_071148.json` - -| GPU | FP8 TFLOPS | -|---:|---:| -| 0 | 1615.6 | -| 1 | 1611.0 | -| 2 | 1599.0 | -| 3 | 1607.1 | -| 4 | 1614.0 | -| 5 | 1604.4 | -| 6 | 1608.4 | -| 7 | 1609.1 | - -Summary: - -- Mean: `1608.6 TFLOPS` -- Min / Max: `1599.0 / 1615.6 TFLOPS` -- Spread: `1.03%` -- FP8 absolute threshold: `>= 1400 TFLOPS` -- Verdict against FP8 absolute threshold: **PASS** -- Verdict against 8-GPU consistency threshold `<= 3%`: **PASS** - -### aikubeworker0016 / nccl-gpu-2 - -Raw report: `reports_cublaslt_fp8_gemm_aikubeworker0016_20260524_071200.json` - -| GPU | FP8 TFLOPS | -|---:|---:| -| 0 | 1602.3 | -| 1 | 1604.0 | -| 2 | 1616.9 | -| 3 | 1610.6 | -| 4 | 1620.5 | -| 5 | 1630.3 | -| 6 | 1605.1 | -| 7 | 1620.2 | - -Summary: - -- Mean: `1613.7 TFLOPS` -- Min / Max: `1602.3 / 1630.3 TFLOPS` -- Spread: `1.74%` -- FP8 absolute threshold: `>= 1400 TFLOPS` -- Verdict against FP8 absolute threshold: **PASS** -- Verdict against 8-GPU consistency threshold `<= 3%`: **PASS** - -## Comparison With Existing PyTorch `_scaled_mm` Result - -| Host | PyTorch `_scaled_mm` FP8 | cuBLASLt FP8 | Delta | -|---|---:|---:|---:| -| aikubeworker0012 | 1170.4 | 1608.6 | +438.2 | -| aikubeworker0016 | 1179.5 | 1613.7 | +434.2 | - -The cuBLASLt path passes the `>= 1400 TFLOPS` FP8 absolute threshold on both machines, while the original PyTorch `_scaled_mm` path remains around `1170-1180 TFLOPS`. - -## Conclusion - -The FP8 hardware path is capable of exceeding the configured H100 FP8 acceptance threshold on both machines. The earlier FP8 FAIL is therefore most likely a benchmark implementation issue in the current PyTorch `_scaled_mm` path, not a GPU hardware, power, clock, thermal, MIG, ECC, or Fabric Manager issue. - -Recommended next action: replace or augment the existing FP8 compute acceptance item with the cuBLASLt FP8 GEMM cross-check, while keeping the PyTorch `_scaled_mm` result as a secondary software-stack signal. diff --git a/reports_cublaslt_fp8_gemm_aikubeworker0012_20260524_071148.json b/reports_cublaslt_fp8_gemm_aikubeworker0012_20260524_071148.json deleted file mode 100644 index b61e641..0000000 --- a/reports_cublaslt_fp8_gemm_aikubeworker0012_20260524_071148.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "source": "cuBLASLt", - "dtype": "fp8_e4m3_inputs_bf16_output_fp32_accum", - "matrix_size": 8192, - "warmup": 50, - "iterations": 500, - "per_gpu": [ - {"index": 0, "fp8_tflops": 1615.6}, - {"index": 1, "fp8_tflops": 1611.0}, - {"index": 2, "fp8_tflops": 1599.0}, - {"index": 3, "fp8_tflops": 1607.1}, - {"index": 4, "fp8_tflops": 1614.0}, - {"index": 5, "fp8_tflops": 1604.4}, - {"index": 6, "fp8_tflops": 1608.4}, - {"index": 7, "fp8_tflops": 1609.1} - ], - "mean_tflops": 1608.6, - "min_tflops": 1599.0, - "max_tflops": 1615.6, - "spread_pct": 1.03 -} diff --git a/reports_cublaslt_fp8_gemm_aikubeworker0016_20260524_071200.json b/reports_cublaslt_fp8_gemm_aikubeworker0016_20260524_071200.json deleted file mode 100644 index 6808990..0000000 --- a/reports_cublaslt_fp8_gemm_aikubeworker0016_20260524_071200.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "source": "cuBLASLt", - "dtype": "fp8_e4m3_inputs_bf16_output_fp32_accum", - "matrix_size": 8192, - "warmup": 50, - "iterations": 500, - "per_gpu": [ - {"index": 0, "fp8_tflops": 1602.3}, - {"index": 1, "fp8_tflops": 1604.0}, - {"index": 2, "fp8_tflops": 1616.9}, - {"index": 3, "fp8_tflops": 1610.6}, - {"index": 4, "fp8_tflops": 1620.5}, - {"index": 5, "fp8_tflops": 1630.3}, - {"index": 6, "fp8_tflops": 1605.1}, - {"index": 7, "fp8_tflops": 1620.2} - ], - "mean_tflops": 1613.7, - "min_tflops": 1602.3, - "max_tflops": 1630.3, - "spread_pct": 1.74 -} diff --git a/reports_dcgm_r3_aikubeworker0012_20260522_200338.md b/reports_dcgm_r3_aikubeworker0012_20260522_200338.md deleted file mode 100644 index 1663b83..0000000 --- a/reports_dcgm_r3_aikubeworker0012_20260522_200338.md +++ /dev/null @@ -1,65 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T20:26:56.947796 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- Training - -## Summary - -| Test | Result | -|------|--------| -| DCGM | PASS | - -## DCGM Diagnostic - -**Overall: PASS** - -| Subtest | Status | -|---------|--------| -| Hardware/nvbandwidth/GPU6 | PASS | -| Hardware/nvbandwidth/GPU7 | PASS | -| Hardware/nvbandwidth/summary | PASS | -| Integration/pcie/GPU0 | PASS | -| Integration/pcie/GPU1 | PASS | -| Integration/pcie/GPU2 | PASS | -| Integration/pcie/GPU3 | PASS | -| Integration/pcie/GPU4 | PASS | -| Integration/pcie/GPU5 | PASS | -| Integration/pcie/GPU6 | PASS | -| Integration/pcie/GPU7 | PASS | -| Integration/pcie/summary | PASS | -| Stress/targeted_stress/GPU0 | PASS | -| Stress/targeted_stress/GPU1 | PASS | -| Stress/targeted_stress/GPU2 | PASS | -| Stress/targeted_stress/GPU3 | PASS | -| Stress/targeted_stress/GPU4 | PASS | -| Stress/targeted_stress/GPU5 | PASS | -| Stress/targeted_stress/GPU6 | PASS | -| Stress/targeted_stress/GPU7 | PASS | -| Stress/targeted_stress/summary | PASS | -| Stress/targeted_power/GPU0 | PASS | -| Stress/targeted_power/GPU1 | PASS | -| Stress/targeted_power/GPU2 | PASS | -| Stress/targeted_power/GPU3 | PASS | -| Stress/targeted_power/GPU4 | PASS | -| Stress/targeted_power/GPU5 | PASS | -| Stress/targeted_power/GPU6 | PASS | -| Stress/targeted_power/GPU7 | PASS | -| Stress/targeted_power/summary | PASS | - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_dcgm_r3_aikubeworker0016_20260522_200538.md b/reports_dcgm_r3_aikubeworker0016_20260522_200538.md deleted file mode 100644 index f51b5bf..0000000 --- a/reports_dcgm_r3_aikubeworker0016_20260522_200538.md +++ /dev/null @@ -1,65 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T20:28:58.716266 -- **Host:** aikubeworker0016 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- Training - -## Summary - -| Test | Result | -|------|--------| -| DCGM | PASS | - -## DCGM Diagnostic - -**Overall: PASS** - -| Subtest | Status | -|---------|--------| -| Hardware/nvbandwidth/GPU6 | PASS | -| Hardware/nvbandwidth/GPU7 | PASS | -| Hardware/nvbandwidth/summary | PASS | -| Integration/pcie/GPU0 | PASS | -| Integration/pcie/GPU1 | PASS | -| Integration/pcie/GPU2 | PASS | -| Integration/pcie/GPU3 | PASS | -| Integration/pcie/GPU4 | PASS | -| Integration/pcie/GPU5 | PASS | -| Integration/pcie/GPU6 | PASS | -| Integration/pcie/GPU7 | PASS | -| Integration/pcie/summary | PASS | -| Stress/targeted_stress/GPU0 | PASS | -| Stress/targeted_stress/GPU1 | PASS | -| Stress/targeted_stress/GPU2 | PASS | -| Stress/targeted_stress/GPU3 | PASS | -| Stress/targeted_stress/GPU4 | PASS | -| Stress/targeted_stress/GPU5 | PASS | -| Stress/targeted_stress/GPU6 | PASS | -| Stress/targeted_stress/GPU7 | PASS | -| Stress/targeted_stress/summary | PASS | -| Stress/targeted_power/GPU0 | PASS | -| Stress/targeted_power/GPU1 | PASS | -| Stress/targeted_power/GPU2 | PASS | -| Stress/targeted_power/GPU3 | PASS | -| Stress/targeted_power/GPU4 | PASS | -| Stress/targeted_power/GPU5 | PASS | -| Stress/targeted_power/GPU6 | PASS | -| Stress/targeted_power/GPU7 | PASS | -| Stress/targeted_power/summary | PASS | - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_fp8_path_comparison_20260525.md b/reports_fp8_path_comparison_20260525.md deleted file mode 100644 index 6c5d9cf..0000000 --- a/reports_fp8_path_comparison_20260525.md +++ /dev/null @@ -1,168 +0,0 @@ -# FP8 GEMM 路径对比测试报告 - -测试日期：2026-05-25 -测试节点：aikubeworker0012、aikubeworker0016 -测试 GPU：NVIDIA H100 80GB HBM3 -测试目标：对比同一 FP8 GEMM 规模下 PyTorch eager、CUDA Graph、Transformer Engine 和 direct cuBLASLt 的性能差异。 - -## 一、测试结论 - -本次 A-E 五条路径均已完成实测。 - -核心结论： - -1. direct cuBLASLt 是本组测试里最快路径，两台机器分别达到 1626.6 TFLOPS 和 1598.1 TFLOPS。 -2. PyTorch eager `_scaled_mm` 默认路径约为 1161.9-1186.1 TFLOPS。 -3. 打开 `use_fast_accum=True` 后，PyTorch eager 路径有稳定提升，约提升 5.0%-6.7%。 -4. CUDA Graph + `_scaled_mm(use_fast_accum=True)` 进一步提升到 1277.7-1322.2 TFLOPS，但仍低于 direct cuBLASLt。 -5. Transformer Engine 本次使用的是 `te.Linear` + `fp8_autocast` 路径，不是裸 GEMM，因此包含 TE module、cast、FP8 recipe 等额外开销，结果低于 direct cuBLASLt，也低于 CUDA Graph `_scaled_mm`。 - -这说明：当前 GPU 硬件和 cuBLASLt 裸 GEMM 能力本身没有问题；之前 PyTorch `_scaled_mm` 1170-1180 TFLOPS 左右的结果，主要反映的是 PyTorch eager 路径和当前 benchmark 方式下的端到端路径性能，而不是 GPU 算力极限。 - -## 二、测试方法 - -统一参数： - -| 参数 | 值 | -|---|---:| -| matrix_size | 8192 | -| M/N/K | 8192/8192/8192 | -| warmup | 50 | -| iterations | 500 | -| GPU index | 0 | -| PyTorch | 2.6.0+cu124 | -| CUDA | 12.4 | -| 输入 dtype | FP8 E4M3 | -| 输出 dtype | BF16 | -| accumulation | FP32 | -| scale_a / scale_b | 1.0 / 1.0 | - -测试路径定义： - -| 路径 | 名称 | 含义 | -|---|---|---| -| A | 当前 eager `_scaled_mm` | PyTorch 立即执行模式调用 `torch._scaled_mm`，默认 accumulation 参数 | -| B | `_scaled_mm(use_fast_accum=True)` | PyTorch eager 路径，但显式打开 fast accumulation | -| C | CUDA Graph + `_scaled_mm(use_fast_accum=True)` | 捕获并 replay 同一个 `_scaled_mm` 调用，降低 Python/PyTorch launch 间隙 | -| D | Transformer Engine FP8 GEMM | `te.Linear` 在 `fp8_autocast` 下执行，包含 TE 层封装和 FP8 recipe 开销 | -| E | direct cuBLASLt | C++/CUDA 直接调用 `cublasLtMatmul`，绕过 PyTorch eager | - -复现脚本： - -```bash -MATRIX_SIZE=8192 WARMUP=50 ITERATIONS=500 GPU_INDEX=0 WORKSPACE_MB=256 \ - /root/test_gpu_scripts/scripts/run_fp8_path_comparison.sh -``` - -## 三、实测结果 - -### aikubeworker0012 - -原始 JSON：`/Users/d-robotics/lab/test_gpu_scripts/reports_fp8_paths_combined_aikubeworker0012_20260525_045408.json` - -| 路径 | 状态 | TFLOPS | 单轮 CUDA event 时间 | -|---|---|---:|---:| -| A eager `_scaled_mm` default | OK | 1186.1 | 927.014 us | -| B eager `_scaled_mm` fast_accum | OK | 1266.0 | 868.481 us | -| C CUDA Graph + fast_accum | OK | 1322.2 | 831.573 us | -| D Transformer Engine FP8 Linear | OK | 1153.2 | 953.478 us | -| E direct cuBLASLt fast_accum | OK | 1626.6 | 未在 combined JSON 中记录 | - -相对 A 的提升： - -| 路径 | 相对 A | -|---|---:| -| B | +6.7% | -| C | +11.5% | -| D | -2.8% | -| E | +37.1% | - -E 路径 cuBLASLt 算法信息： - -| 字段 | 值 | -|---|---:| -| algo_id | 52 | -| tile_id | 23 | -| splitk | 1 | -| stages_id | 36 | -| inner_shape_id | 0 | -| cluster_shape_id | 3 | - -### aikubeworker0016 - -原始 JSON：`/Users/d-robotics/lab/test_gpu_scripts/reports_fp8_paths_combined_aikubeworker0016_20260525_050048.json` - -| 路径 | 状态 | TFLOPS | 单轮 CUDA event 时间 | -|---|---|---:|---:| -| A eager `_scaled_mm` default | OK | 1161.9 | 946.313 us | -| B eager `_scaled_mm` fast_accum | OK | 1220.4 | 900.960 us | -| C CUDA Graph + fast_accum | OK | 1277.7 | 860.543 us | -| D Transformer Engine FP8 Linear | OK | 1125.3 | 977.054 us | -| E direct cuBLASLt fast_accum | OK | 1598.1 | 未在 combined JSON 中记录 | - -相对 A 的提升： - -| 路径 | 相对 A | -|---|---:| -| B | +5.0% | -| C | +10.0% | -| D | -3.2% | -| E | +37.5% | - -E 路径 cuBLASLt 算法信息： - -| 字段 | 值 | -|---|---:| -| algo_id | 52 | -| tile_id | 23 | -| splitk | 1 | -| stages_id | 36 | -| inner_shape_id | 0 | -| cluster_shape_id | 3 | - -## 四、对 PyTorch FP8 能否“上去”的判断 - -从本次结果看，PyTorch FP8 路径可以通过两类方式上去： - -1. 打开更快的 math/accumulation 参数，例如 `use_fast_accum=True`。 -2. 使用 CUDA Graph replay，减少 eager 模式下每轮调度、enqueue 之间的间隙。 - -但在当前 `matrix_size=8192`、单个 `_scaled_mm`、PyTorch eager/Graph benchmark 的测试形态下，PyTorch 路径仍没有达到 direct cuBLASLt 的 1598-1626 TFLOPS。也就是说，direct cuBLASLt 证明硬件和底层库有能力跑得更高；PyTorch eager `_scaled_mm` 测到的是 PyTorch 当前封装路径在这个 shape 下的实际表现。 - -如果把目标定义为“让 PyTorch 代码路径更接近裸 cuBLASLt”，后续可以继续验证： - -1. 更大的 GEMM size，例如 16384。 -2. 固定 shape 后用 `torch.compile` 或 Inductor。 -3. CUDA Graph 覆盖更完整的 step，而不是只 replay 单个 op。 -4. 使用 Transformer Engine 的更底层 GEMM API 或官方 microbenchmark，而不是 `te.Linear` module forward。 -5. 对 `_scaled_mm` 做 Nsight Systems / Nsight Compute 抓取，确认实际 kernel、间隙和 cuBLASLt 算法选择。 - -## 五、术语说明 - -`eager` 指 PyTorch 立即执行模式。每次 Python 调用 `torch._scaled_mm`，PyTorch 都会经过 dispatcher、参数检查、Tensor 创建、准备 descriptor、调用 cuBLASLt heuristic，然后把 matmul enqueue 到 CUDA stream。 - -`cuBLAS` 是 NVIDIA 的基础矩阵乘库。`cuBLASLt` 是更灵活的矩阵乘接口，支持更多 layout、FP8、算法 heuristic、workspace、epilogue 等能力。 - -`direct cuBLASLt` 指我们自己写 C++/CUDA 直接调用 `cublasLtMatmul`，不经过 PyTorch eager，因此更接近裸 GEMM 峰值。 - -`CUDA Graph` 指把一次 CUDA work 提前捕获成图，后续直接 replay，减少 CPU 侧反复 launch/调度带来的间隙。 - -`Transformer Engine` 是 NVIDIA 面向 Transformer/FP8 训练优化的库。本次 D 路径使用的是 `te.Linear` module forward，不等同于裸 GEMM microbenchmark。 - -## 六、文件清单 - -本地脚本： - -| 文件 | 用途 | -|---|---| -| `/Users/d-robotics/lab/test_gpu_scripts/scripts/pytorch_fp8_path_bench.py` | A/B/C/D PyTorch 与 Transformer Engine 路径 | -| `/Users/d-robotics/lab/test_gpu_scripts/scripts/cublaslt_fp8_gemm_bench.cu` | E direct cuBLASLt 路径 | -| `/Users/d-robotics/lab/test_gpu_scripts/scripts/run_fp8_path_comparison.sh` | 统一运行并合并 A-E 结果 | - -本地结果： - -| 文件 | 用途 | -|---|---| -| `/Users/d-robotics/lab/test_gpu_scripts/reports_fp8_paths_combined_aikubeworker0012_20260525_045408.json` | aikubeworker0012 A-E 原始结果 | -| `/Users/d-robotics/lab/test_gpu_scripts/reports_fp8_paths_combined_aikubeworker0016_20260525_050048.json` | aikubeworker0016 A-E 原始结果 | -| `/Users/d-robotics/lab/test_gpu_scripts/reports_fp8_path_comparison_20260525.md` | 本中文汇总报告 | diff --git a/reports_fp8_paths_combined_aikubeworker0012_20260525_042347.json b/reports_fp8_paths_combined_aikubeworker0012_20260525_042347.json deleted file mode 100644 index 51a1540..0000000 --- a/reports_fp8_paths_combined_aikubeworker0012_20260525_042347.json +++ /dev/null @@ -1,142 +0,0 @@ -{ - "source": "fp8_path_comparison", - "host": null, - "matrix_size": 8192, - "gpu_index": 0, - "pytorch": { - "source": "pytorch_fp8_path_bench", - "torch": "2.6.0+cu124", - "cuda": "12.4", - "gpu_index": 0, - "gpu_name": "NVIDIA H100 80GB HBM3", - "matrix_size": 8192, - "warmup": 50, - "iterations": 500, - "results": [ - { - "name": "A_eager_scaled_mm_default", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 465.145, - "event_us_per_iter": 930.29, - "wall_ms_total": 465.21, - "tflops": 1181.9 - }, - { - "name": "B_eager_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 440.252, - "event_us_per_iter": 880.504, - "wall_ms_total": 440.289, - "tflops": 1248.7 - }, - { - "name": "C_cuda_graph_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 3, - "event_ms_total": 415.631, - "event_us_per_iter": 831.262, - "wall_ms_total": 415.664, - "tflops": 1322.7 - }, - { - "name": "D_transformer_engine_fp8_linear", - "status": "unavailable", - "reason": "ModuleNotFoundError: No module named 'transformer_engine'" - } - ], - "summary": { - "max_tflops": 1322.7, - "min_tflops": 1181.9, - "mean_tflops": 1251.1 - } - }, - "cublaslt": { - "source": "cuBLASLt", - "dtype": "fp8_e4m3_inputs_bf16_output_fp32_accum", - "matrix_size": 8192, - "warmup": 50, - "iterations": 500, - "fast_accum": 1, - "per_gpu": [ - { - "index": 0, - "fp8_tflops": 1615.4, - "algo_id": 52, - "tile_id": 23, - "splitk": 1, - "stages_id": 36, - "inner_shape_id": 0, - "cluster_shape_id": 3 - } - ], - "mean_tflops": 1615.4, - "min_tflops": 1615.4, - "max_tflops": 1615.4, - "spread_pct": 0.0 - }, - "results": [ - { - "name": "A_eager_scaled_mm_default", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 465.145, - "event_us_per_iter": 930.29, - "wall_ms_total": 465.21, - "tflops": 1181.9 - }, - { - "name": "B_eager_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 440.252, - "event_us_per_iter": 880.504, - "wall_ms_total": 440.289, - "tflops": 1248.7 - }, - { - "name": "C_cuda_graph_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 3, - "event_ms_total": 415.631, - "event_us_per_iter": 831.262, - "wall_ms_total": 415.664, - "tflops": 1322.7 - }, - { - "name": "D_transformer_engine_fp8_linear", - "status": "unavailable", - "reason": "ModuleNotFoundError: No module named 'transformer_engine'" - }, - { - "index": 0, - "algo_id": 52, - "tile_id": 23, - "splitk": 1, - "stages_id": 36, - "inner_shape_id": 0, - "cluster_shape_id": 3, - "name": "E_direct_cublaslt_fast_accum", - "status": "ok", - "tflops": 1615.4, - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "fast_accum": 1, - "note": "Direct cuBLASLt FP8 GEMM, bypasses PyTorch eager." - } - ] -} \ No newline at end of file diff --git a/reports_fp8_paths_combined_aikubeworker0012_20260525_045408.json b/reports_fp8_paths_combined_aikubeworker0012_20260525_045408.json deleted file mode 100644 index 56cbce5..0000000 --- a/reports_fp8_paths_combined_aikubeworker0012_20260525_045408.json +++ /dev/null @@ -1,156 +0,0 @@ -{ - "source": "fp8_path_comparison", - "host": null, - "matrix_size": 8192, - "gpu_index": 0, - "pytorch": { - "source": "pytorch_fp8_path_bench", - "torch": "2.6.0+cu124", - "cuda": "12.4", - "gpu_index": 0, - "gpu_name": "NVIDIA H100 80GB HBM3", - "matrix_size": 8192, - "warmup": 50, - "iterations": 500, - "results": [ - { - "name": "A_eager_scaled_mm_default", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 463.507, - "event_us_per_iter": 927.014, - "wall_ms_total": 463.573, - "tflops": 1186.1 - }, - { - "name": "B_eager_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 434.241, - "event_us_per_iter": 868.481, - "wall_ms_total": 434.492, - "tflops": 1266.0 - }, - { - "name": "C_cuda_graph_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 3, - "event_ms_total": 415.786, - "event_us_per_iter": 831.573, - "wall_ms_total": 415.825, - "tflops": 1322.2 - }, - { - "name": "D_transformer_engine_fp8_linear", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 476.739, - "event_us_per_iter": 953.478, - "wall_ms_total": 476.8, - "tflops": 1153.2, - "note": "Transformer Engine Linear forward under fp8_autocast; includes TE module/cast overhead." - } - ], - "summary": { - "max_tflops": 1322.2, - "min_tflops": 1153.2, - "mean_tflops": 1231.9 - } - }, - "cublaslt": { - "source": "cuBLASLt", - "dtype": "fp8_e4m3_inputs_bf16_output_fp32_accum", - "matrix_size": 8192, - "warmup": 50, - "iterations": 500, - "fast_accum": 1, - "per_gpu": [ - { - "index": 0, - "fp8_tflops": 1626.6, - "algo_id": 52, - "tile_id": 23, - "splitk": 1, - "stages_id": 36, - "inner_shape_id": 0, - "cluster_shape_id": 3 - } - ], - "mean_tflops": 1626.6, - "min_tflops": 1626.6, - "max_tflops": 1626.6, - "spread_pct": 0.0 - }, - "results": [ - { - "name": "A_eager_scaled_mm_default", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 463.507, - "event_us_per_iter": 927.014, - "wall_ms_total": 463.573, - "tflops": 1186.1 - }, - { - "name": "B_eager_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 434.241, - "event_us_per_iter": 868.481, - "wall_ms_total": 434.492, - "tflops": 1266.0 - }, - { - "name": "C_cuda_graph_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 3, - "event_ms_total": 415.786, - "event_us_per_iter": 831.573, - "wall_ms_total": 415.825, - "tflops": 1322.2 - }, - { - "name": "D_transformer_engine_fp8_linear", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 476.739, - "event_us_per_iter": 953.478, - "wall_ms_total": 476.8, - "tflops": 1153.2, - "note": "Transformer Engine Linear forward under fp8_autocast; includes TE module/cast overhead." - }, - { - "index": 0, - "algo_id": 52, - "tile_id": 23, - "splitk": 1, - "stages_id": 36, - "inner_shape_id": 0, - "cluster_shape_id": 3, - "name": "E_direct_cublaslt_fast_accum", - "status": "ok", - "tflops": 1626.6, - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "fast_accum": 1, - "note": "Direct cuBLASLt FP8 GEMM, bypasses PyTorch eager." - } - ] -} \ No newline at end of file diff --git a/reports_fp8_paths_combined_aikubeworker0016_20260525_042402.json b/reports_fp8_paths_combined_aikubeworker0016_20260525_042402.json deleted file mode 100644 index 6d6a3a2..0000000 --- a/reports_fp8_paths_combined_aikubeworker0016_20260525_042402.json +++ /dev/null @@ -1,142 +0,0 @@ -{ - "source": "fp8_path_comparison", - "host": null, - "matrix_size": 8192, - "gpu_index": 0, - "pytorch": { - "source": "pytorch_fp8_path_bench", - "torch": "2.6.0+cu124", - "cuda": "12.4", - "gpu_index": 0, - "gpu_name": "NVIDIA H100 80GB HBM3", - "matrix_size": 8192, - "warmup": 50, - "iterations": 500, - "results": [ - { - "name": "A_eager_scaled_mm_default", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 470.909, - "event_us_per_iter": 941.817, - "wall_ms_total": 470.974, - "tflops": 1167.4 - }, - { - "name": "B_eager_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 452.608, - "event_us_per_iter": 905.215, - "wall_ms_total": 452.647, - "tflops": 1214.6 - }, - { - "name": "C_cuda_graph_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 3, - "event_ms_total": 427.724, - "event_us_per_iter": 855.449, - "wall_ms_total": 427.768, - "tflops": 1285.3 - }, - { - "name": "D_transformer_engine_fp8_linear", - "status": "unavailable", - "reason": "ModuleNotFoundError: No module named 'transformer_engine'" - } - ], - "summary": { - "max_tflops": 1285.3, - "min_tflops": 1167.4, - "mean_tflops": 1222.4 - } - }, - "cublaslt": { - "source": "cuBLASLt", - "dtype": "fp8_e4m3_inputs_bf16_output_fp32_accum", - "matrix_size": 8192, - "warmup": 50, - "iterations": 500, - "fast_accum": 1, - "per_gpu": [ - { - "index": 0, - "fp8_tflops": 1594.3, - "algo_id": 52, - "tile_id": 23, - "splitk": 1, - "stages_id": 36, - "inner_shape_id": 0, - "cluster_shape_id": 3 - } - ], - "mean_tflops": 1594.3, - "min_tflops": 1594.3, - "max_tflops": 1594.3, - "spread_pct": 0.0 - }, - "results": [ - { - "name": "A_eager_scaled_mm_default", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 470.909, - "event_us_per_iter": 941.817, - "wall_ms_total": 470.974, - "tflops": 1167.4 - }, - { - "name": "B_eager_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 452.608, - "event_us_per_iter": 905.215, - "wall_ms_total": 452.647, - "tflops": 1214.6 - }, - { - "name": "C_cuda_graph_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 3, - "event_ms_total": 427.724, - "event_us_per_iter": 855.449, - "wall_ms_total": 427.768, - "tflops": 1285.3 - }, - { - "name": "D_transformer_engine_fp8_linear", - "status": "unavailable", - "reason": "ModuleNotFoundError: No module named 'transformer_engine'" - }, - { - "index": 0, - "algo_id": 52, - "tile_id": 23, - "splitk": 1, - "stages_id": 36, - "inner_shape_id": 0, - "cluster_shape_id": 3, - "name": "E_direct_cublaslt_fast_accum", - "status": "ok", - "tflops": 1594.3, - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "fast_accum": 1, - "note": "Direct cuBLASLt FP8 GEMM, bypasses PyTorch eager." - } - ] -} \ No newline at end of file diff --git a/reports_fp8_paths_combined_aikubeworker0016_20260525_050048.json b/reports_fp8_paths_combined_aikubeworker0016_20260525_050048.json deleted file mode 100644 index 7168c05..0000000 --- a/reports_fp8_paths_combined_aikubeworker0016_20260525_050048.json +++ /dev/null @@ -1,156 +0,0 @@ -{ - "source": "fp8_path_comparison", - "host": null, - "matrix_size": 8192, - "gpu_index": 0, - "pytorch": { - "source": "pytorch_fp8_path_bench", - "torch": "2.6.0+cu124", - "cuda": "12.4", - "gpu_index": 0, - "gpu_name": "NVIDIA H100 80GB HBM3", - "matrix_size": 8192, - "warmup": 50, - "iterations": 500, - "results": [ - { - "name": "A_eager_scaled_mm_default", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 473.156, - "event_us_per_iter": 946.313, - "wall_ms_total": 473.199, - "tflops": 1161.9 - }, - { - "name": "B_eager_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 450.48, - "event_us_per_iter": 900.96, - "wall_ms_total": 450.505, - "tflops": 1220.4 - }, - { - "name": "C_cuda_graph_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 3, - "event_ms_total": 430.272, - "event_us_per_iter": 860.543, - "wall_ms_total": 430.304, - "tflops": 1277.7 - }, - { - "name": "D_transformer_engine_fp8_linear", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 488.527, - "event_us_per_iter": 977.054, - "wall_ms_total": 488.576, - "tflops": 1125.3, - "note": "Transformer Engine Linear forward under fp8_autocast; includes TE module/cast overhead." - } - ], - "summary": { - "max_tflops": 1277.7, - "min_tflops": 1125.3, - "mean_tflops": 1196.3 - } - }, - "cublaslt": { - "source": "cuBLASLt", - "dtype": "fp8_e4m3_inputs_bf16_output_fp32_accum", - "matrix_size": 8192, - "warmup": 50, - "iterations": 500, - "fast_accum": 1, - "per_gpu": [ - { - "index": 0, - "fp8_tflops": 1598.1, - "algo_id": 52, - "tile_id": 23, - "splitk": 1, - "stages_id": 36, - "inner_shape_id": 0, - "cluster_shape_id": 3 - } - ], - "mean_tflops": 1598.1, - "min_tflops": 1598.1, - "max_tflops": 1598.1, - "spread_pct": 0.0 - }, - "results": [ - { - "name": "A_eager_scaled_mm_default", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 473.156, - "event_us_per_iter": 946.313, - "wall_ms_total": 473.199, - "tflops": 1161.9 - }, - { - "name": "B_eager_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 450.48, - "event_us_per_iter": 900.96, - "wall_ms_total": 450.505, - "tflops": 1220.4 - }, - { - "name": "C_cuda_graph_scaled_mm_fast_accum", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 3, - "event_ms_total": 430.272, - "event_us_per_iter": 860.543, - "wall_ms_total": 430.304, - "tflops": 1277.7 - }, - { - "name": "D_transformer_engine_fp8_linear", - "status": "ok", - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "event_ms_total": 488.527, - "event_us_per_iter": 977.054, - "wall_ms_total": 488.576, - "tflops": 1125.3, - "note": "Transformer Engine Linear forward under fp8_autocast; includes TE module/cast overhead." - }, - { - "index": 0, - "algo_id": 52, - "tile_id": 23, - "splitk": 1, - "stages_id": 36, - "inner_shape_id": 0, - "cluster_shape_id": 3, - "name": "E_direct_cublaslt_fast_accum", - "status": "ok", - "tflops": 1598.1, - "matrix_size": 8192, - "iterations": 500, - "warmup": 50, - "fast_accum": 1, - "note": "Direct cuBLASLt FP8 GEMM, bypasses PyTorch eager." - } - ] -} \ No newline at end of file diff --git a/reports_gpu_Test_combined_20260524.md b/reports_gpu_Test_combined_20260524.md deleted file mode 100644 index b4fff0a..0000000 --- a/reports_gpu_Test_combined_20260524.md +++ /dev/null @@ -1,152 +0,0 @@ -# GPU_Test 合并报告 - -- **日期:** 2026-05-24 -- **节点:** `aikubeworker0012 / 172.72.8.12`，`aikubeworker0016 / 172.72.8.16` -- **GPU:** NVIDIA H100 80GB HBM3 x8 / node -- **范围:** 单机单卡算力与多机多卡 NCCL 通信 -- **说明:** 本报告汇总既有原始测试结果，不重新启动额外压力测试。 - -## 总体结论 - -| 测试项 | 结论 | 说明 | -|---|---|---| -| 单机 GPU 识别 | PASS | 两台机器均识别 8 张 H100 80GB HBM3 | -| 单机单卡 FP8 硬件算力 | PASS | direct cuBLASLt FP8 GEMM 两台机器均超过 `>= 1400 TFLOPS` | -| PyTorch `_scaled_mm` FP8 路径 | FAIL / 软件栈信号 | 约 `1170-1180 TFLOPS`，低于阈值；已定位为 PyTorch eager / `_scaled_mm` benchmark 路径偏低，不作为硬件失败依据 | -| 多机多卡 NCCL 正确性 | PASS | return code `0`，`Wrong=0` / `Out of bounds values: 0 OK` | -| 多机多卡 NCCL 性能 | 符合当前 4x400Gbps 网络形态 | 2x8 allreduce / alltoall 低于 PDF 8x400Gbps 阈值，但该阈值不应直接硬套到当前 4x400Gbps 环境 | - -## 单机单卡 / 算力测试 - -### 机器信息 - -| Host | GPU | Driver | CUDA | GPU 数量 | -|---|---|---|---|---:| -| `aikubeworker0012` | NVIDIA H100 80GB HBM3 | 580.159.03 | 13.0 | 8 | -| `aikubeworker0016` | NVIDIA H100 80GB HBM3 | 580.159.03 | 13.0 | 8 | - -来源： - -- `reports_single_gpu_aikubeworker0012.md` -- `reports_single_gpu_aikubeworker0016.md` - -### 原始 PyTorch 单机算力结果 - -| Host | FP32 | TF32 | FP16 | BF16 | FP8 `_scaled_mm` | 原始 Verdict | -|---|---:|---:|---:|---:|---:|---| -| `aikubeworker0012` | 52.0 | 362.3 | 691.0 | 713.0 | 1148.8 | FAIL | -| `aikubeworker0016` | 51.9 | 357.8 | 667.2 | 699.1 | 1146.2 | FAIL | - -原始 PyTorch 路径使用 `torch._scaled_mm` 做 FP8 GEMM。后续复查显示，该路径会受到 PyTorch eager dispatch、输出 Tensor 创建、cuBLASLt heuristic 路径、默认 `use_fast_accum=False` 等因素影响，不能直接代表 H100 FP8 Tensor Core 硬件上限。 - -### direct cuBLASLt FP8 GEMM 交叉验证 - -测试参数： - -| 参数 | 值 | -|---|---| -| Benchmark | direct cuBLASLt FP8 GEMM | -| Source | `scripts/cublaslt_fp8_gemm_bench.cu` | -| Matrix | `8192 x 8192 x 8192` | -| A/B dtype | FP8 E4M3 | -| Output dtype | BF16 | -| Compute type | `CUBLAS_COMPUTE_32F` | -| Scale type | `CUDA_R_32F` | -| Scale A/B | `1.0` | -| Layout | TN | -| fast accumulation | enabled | -| Threshold | `>= 1400 TFLOPS` | - -结果： - -| Host | Mean FP8 TFLOPS | Min | Max | Spread | Threshold | Verdict | -|---|---:|---:|---:|---:|---:|---| -| `aikubeworker0012` | 1608.6 | 1599.0 | 1615.6 | 1.03% | >= 1400 | PASS | -| `aikubeworker0016` | 1613.7 | 1602.3 | 1630.3 | 1.74% | >= 1400 | PASS | - -单卡逐张结果： - -| Host | GPU0 | GPU1 | GPU2 | GPU3 | GPU4 | GPU5 | GPU6 | GPU7 | -|---|---:|---:|---:|---:|---:|---:|---:|---:| -| `aikubeworker0012` | 1615.6 | 1611.0 | 1599.0 | 1607.1 | 1614.0 | 1604.4 | 1608.4 | 1609.1 | -| `aikubeworker0016` | 1602.3 | 1604.0 | 1616.9 | 1610.6 | 1620.5 | 1630.3 | 1605.1 | 1620.2 | - -结论：direct cuBLASLt FP8 GEMM 已通过 `>= 1400 TFLOPS` 阈值，说明两台机器的 FP8 硬件计算路径具备达标能力。PyTorch `_scaled_mm` 的 FAIL 更适合作为软件栈 benchmark 路径问题记录，而不是 GPU 硬件失败结论。 - -来源： - -- `reports_cublaslt_fp8_crosscheck_20260524.md` -- `reports_cublaslt_fp8_gemm_aikubeworker0012_20260524_071148.json` -- `reports_cublaslt_fp8_gemm_aikubeworker0016_20260524_071200.json` - -## 多机多卡 NCCL 测试 - -### 测试环境 - -| 项目 | 结果 | -|---|---| -| Hosts | `nccl-gpu-1(172.72.8.12)`，`nccl-gpu-2(172.72.8.16)` | -| Topology | 2 nodes x 8 GPUs，合计 16 GPUs | -| NCCL source | `nccl-tests-mpirun` | -| NCCL network | IB | -| GPU Direct RDMA | ENABLED | -| Active HCA rails | `mlx5_0, mlx5_1, mlx5_6, mlx5_7` | -| HCA speed | 4 条 `400 Gb/sec (4X NDR)` ACTIVE | - -注意：NCCL 表里的 `GB/s` 是大 B，即 Bytes/s。IB 网卡口径 `400 Gb/s` 是小 b，即 bits/s。 - -### 2x8 全集合通信结果 - -| Operation | Peak Bus BW | Avg Bus BW | PDF 8x400Gbps Threshold | Correctness | 当前 4x400Gbps 口径 | -|---|---:|---:|---:|---|---| -| allreduce | 354.27 GB/s | 354.45 GB/s | >= 491.84 GB/s | PASS | 符合当前硬件形态，低于 PDF 8 rail 阈值 | -| alltoall | 37.00 GB/s | 37.14 GB/s | >= 76.54 GB/s | PASS | 符合当前硬件形态，低于 PDF 8 rail 阈值 | -| broadcast | 191.65 GB/s | 190.25 GB/s | 未配置 PDF 阈值 | PASS | PASS / 仅记录 | -| reducescatter | 192.75 GB/s | 192.74 GB/s | 未配置 PDF 阈值 | PASS | PASS / 仅记录 | -| allgather | 192.14 GB/s | 192.47 GB/s | 未配置 PDF 阈值 | PASS | PASS / 仅记录 | -| sendrecv | 26.98 GB/s | 26.97 GB/s | 未配置 PDF 阈值 | PASS | PASS / 仅记录 | - -结论：2x8 全集合通信测试中，NCCL 正确性通过。allreduce 和 alltoall 低于 PDF 8x400Gbps 参考阈值，但当前机器确认参与 NCCL 的是 4 条 400Gbps rail，因此该差距不应直接判定为当前 4x400Gbps 环境不合格。 - -来源： - -- `reports_multinode_nccl_all_collectives_20260523_120144.md` -- `reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md` - -### PDF Matrix allreduce / alltoall 结果 - -AllReduce（PDF 8x400Gbps 阈值对比，仅作参考）: - -| Topology | Peak Bus BW | Avg Bus BW | PDF 8x400Gbps Threshold | Gap | 当前解释 | -|---|---:|---:|---:|---:|---| -| 2 nodes x 1 GPU | 47.29 GB/s | 47.26 GB/s | >= 48.90 GB/s | -1.61 GB/s | 接近 PDF 阈值 | -| 2 nodes x 2 GPUs | 137.16 GB/s | 137.13 GB/s | >= 136.93 GB/s | +0.23 GB/s | 达到 PDF 阈值 | -| 2 nodes x 4 GPUs | 335.07 GB/s | 335.02 GB/s | >= 335.48 GB/s | -0.41 GB/s | 接近 PDF 阈值 | -| 2 nodes x 8 GPUs | 353.85 GB/s | 353.85 GB/s | >= 491.84 GB/s | -137.99 GB/s | 低于 PDF 8 rail 阈值；当前为 4 rail 环境，不直接判不合格 | - -AllToAll（PDF 8x400Gbps 阈值对比，仅作参考）: - -| Topology | Peak Bus BW | Avg Bus BW | PDF 8x400Gbps Threshold | Gap | 当前解释 | -|---|---:|---:|---:|---:|---| -| 2 nodes x 1 GPU | 24.85 GB/s | 24.90 GB/s | >= 27.25 GB/s | -2.40 GB/s | 接近 PDF 阈值 | -| 2 nodes x 2 GPUs | 47.76 GB/s | 47.98 GB/s | >= 54.41 GB/s | -6.65 GB/s | 低于 PDF 8 rail 阈值 | -| 2 nodes x 4 GPUs | 72.74 GB/s | 72.80 GB/s | >= 73.73 GB/s | -0.99 GB/s | 接近 PDF 阈值 | -| 2 nodes x 8 GPUs | 36.83 GB/s | 36.85 GB/s | >= 76.54 GB/s | -39.71 GB/s | 低于 PDF 8 rail 阈值；当前为 4 rail 环境，不直接判不合格 | - -来源： - -- `reports_multinode_nccl_pdf_matrix_run_20260523.md` -- `reports_multinode_nccl_pdf_matrix_20260523_113803.md` - -## 风险与判断 - -1. 单机 FP8 硬件能力通过 direct cuBLASLt 验证，当前不支持将 PyTorch `_scaled_mm` FAIL 直接判定为 GPU 硬件故障。 -2. 多机 NCCL 正确性通过，性能结果应按当前 4x400Gbps rail 环境解释。 -3. 当前多机环境确认参与 NCCL 的是 4 条 400G IB rail；PDF 参考环境为 8x400G 计算管理网络，因此 2x8 阈值与当前硬件形态不等价。 -4. 2x8 allreduce 和 alltoall 低于 PDF 8 rail 阈值，建议作为“与 PDF 参考环境差异”记录，而不是作为当前 4 rail 环境不合格结论。 - -## 建议 - -1. 单机 FP8 验收以 direct cuBLASLt 或 Transformer Engine GEMM benchmark 为主，PyTorch `_scaled_mm` 作为软件栈参考项保留。 -2. 多机 NCCL 后续若要按 PDF 阈值验收，需要先对齐 PDF 参考环境的 8x400Gbps rail 数量、NCCL net plugin / SHARP、跨 Leaf 交换策略、ECMP / 拥塞控制配置。 -3. 对外报告建议明确区分 `GB/s` 与 `Gb/s`：NCCL bus bandwidth 是大 B，IB 端口速率是小 b。 diff --git a/reports_gpu_Test_formal_20260524.md b/reports_gpu_Test_formal_20260524.md deleted file mode 100644 index 49e2695..0000000 --- a/reports_gpu_Test_formal_20260524.md +++ /dev/null @@ -1,122 +0,0 @@ -# GPU_Test 双节点测试报告 - -- **测试日期:** 2026-05-24 -- **测试节点:** `aikubeworker0012 / 172.72.8.12`，`aikubeworker0016 / 172.72.8.16` -- **节点配置:** 每节点 8 张 NVIDIA H100 80GB HBM3 GPU -- **测试范围:** 单机算力、单机 8 卡通信、多机 2x8 GPU 通信 -- **网络形态:** 当前参与 NCCL 的计算网络为 4 条 400Gbps IB rail - -## 结论摘要 - -| 项目 | 结果摘要 | -|---|---| -| GPU 识别 | 两台节点均识别 8 张 H100 80GB HBM3 GPU | -| 单机 FP8 GEMM | 两台节点 direct cuBLASLt FP8 GEMM 均超过 1600 TFLOPS | -| 单机 8 卡 NCCL | 两台节点单机 8 卡 NCCL 集合通信均可正常完成，主要大包通信带宽稳定 | -| 多机 2x8 NCCL | 两节点 16 GPU NCCL 正确性通过，所有测试 `Wrong=0` / return code `0` | -| 多机网络口径 | 当前为 4x400Gbps IB rail 环境，结果按该硬件形态解释 | - -## 测试环境 - -| Host | GPU | Driver | CUDA | GPU 数量 | -|---|---|---|---|---:| -| `aikubeworker0012` | NVIDIA H100 80GB HBM3 | 580.159.03 | 13.0 | 8 | -| `aikubeworker0016` | NVIDIA H100 80GB HBM3 | 580.159.03 | 13.0 | 8 | - -## 单机算力测试 - -### FP8 GEMM 硬件路径验证 - -本项使用 direct cuBLASLt FP8 GEMM benchmark，绕过 PyTorch eager 调度路径，直接验证 GPU FP8 Tensor Core 与 cuBLASLt GEMM 能力。 - -| 参数 | 配置 | -|---|---| -| GEMM shape | `8192 x 8192 x 8192` | -| 输入类型 | FP8 E4M3 | -| 输出类型 | BF16 | -| 累加类型 | FP32 compute | -| Layout | TN | -| Scale | `scale_a = 1.0`，`scale_b = 1.0` | -| fast accumulation | enabled | -| 测试 GPU | 每节点 8 张 GPU 逐张测试 | - -| Host | Mean FP8 TFLOPS | Min | Max | Spread | -|---|---:|---:|---:|---:| -| `aikubeworker0012` | 1608.6 | 1599.0 | 1615.6 | 1.03% | -| `aikubeworker0016` | 1613.7 | 1602.3 | 1630.3 | 1.74% | - -| Host | GPU0 | GPU1 | GPU2 | GPU3 | GPU4 | GPU5 | GPU6 | GPU7 | -|---|---:|---:|---:|---:|---:|---:|---:|---:| -| `aikubeworker0012` | 1615.6 | 1611.0 | 1599.0 | 1607.1 | 1614.0 | 1604.4 | 1608.4 | 1609.1 | -| `aikubeworker0016` | 1602.3 | 1604.0 | 1616.9 | 1610.6 | 1620.5 | 1630.3 | 1605.1 | 1620.2 | - -**说明:** PyTorch `_scaled_mm` eager benchmark 结果约为 1170-1180 TFLOPS，该结果反映 PyTorch 软件路径与调度开销，不作为本报告的硬件算力结论。 - -## 单机 8 卡 NCCL 通信测试 - -本项在单个节点内使用 8 张 GPU 进行 NCCL 集合通信测试，结果单位为 `GB/s`，即 Bytes/s。 - -| Operation | `aikubeworker0012` Bus BW | `aikubeworker0016` Bus BW | -|---|---:|---:| -| allreduce | 472.3 GB/s | 472.4 GB/s | -| alltoall | 343.3 GB/s | 344.3 GB/s | -| broadcast | 364.1 GB/s | 363.6 GB/s | -| reducescatter | 352.8 GB/s | 353.1 GB/s | -| allgather | 366.4 GB/s | 366.4 GB/s | -| sendrecv | 369.0 GB/s | 368.9 GB/s | - -**说明:** 单机 8 卡通信主要依赖节点内 GPU 互联与 NCCL collective 实现。两台节点的同类 operation 结果接近，节点间差异较小。 - -## 多机 2x8 NCCL 通信测试 - -本项使用两台节点，每台 8 张 GPU，共 16 张 GPU 进行跨节点 NCCL 集合通信测试。 - -### 网络环境 - -| 项目 | 配置 | -|---|---| -| Host A | `aikubeworker0012 / 172.72.8.12` | -| Host B | `aikubeworker0016 / 172.72.8.16` | -| 拓扑 | 2 nodes x 8 GPUs | -| NCCL network | IB | -| GPU Direct RDMA | ENABLED | -| Active rails | `mlx5_0, mlx5_1, mlx5_6, mlx5_7` | -| Rail 速率 | 4 条 `400 Gb/sec (4X NDR)` ACTIVE | - -### 跨节点 NCCL 结果 - -| Operation | Peak Bus BW | Avg Bus BW | Correctness | -|---|---:|---:|---| -| allreduce | 354.27 GB/s | 354.45 GB/s | PASS | -| alltoall | 37.00 GB/s | 37.14 GB/s | PASS | -| broadcast | 191.65 GB/s | 190.25 GB/s | PASS | -| reducescatter | 192.75 GB/s | 192.74 GB/s | PASS | -| allgather | 192.14 GB/s | 192.47 GB/s | PASS | -| sendrecv | 26.98 GB/s | 26.97 GB/s | PASS | - -**正确性:** 本轮多机 NCCL 测试 return code 为 `0`，`Wrong=0`，未发现数据正确性错误。 - -## 单位说明 - -| 写法 | 含义 | 说明 | -|---|---|---| -| `GB/s` | Gigabytes per second | 大 B，字节每秒，NCCL bus bandwidth 使用此单位 | -| `Gbps` / `Gb/s` | Gigabits per second | 小 b，比特每秒，IB 端口速率通常使用此单位 | - -换算关系： - -```text -1 Byte = 8 bits -400 Gb/s = 50 GB/s -4 x 400 Gb/s = 1600 Gb/s = 200 GB/s 物理链路字节带宽 -``` - -NCCL 的 `busbw` 是 collective 通信的逻辑折算带宽，不等同于单条物理链路的线速。 - -## 结果说明 - -1. 两台节点 GPU 识别正常，均为 8 张 H100 80GB HBM3。 -2. direct cuBLASLt FP8 GEMM 显示两台节点单卡 FP8 算力均超过 1600 TFLOPS，GPU FP8 硬件计算路径正常。 -3. 单机 8 卡 NCCL 通信在两台节点上结果接近，未观察到明显节点间异常差异。 -4. 多机 2x8 NCCL 正确性通过，跨节点通信功能正常。 -5. 当前多机通信结果应按 4x400Gbps IB rail 环境解释；若后续需要对齐 8x400Gbps 环境，应先确认 rail 数量、NCCL net plugin / SHARP、交换网络策略等配置一致。 diff --git a/reports_gpu_Test_pdf.css b/reports_gpu_Test_pdf.css deleted file mode 100644 index 9a44015..0000000 --- a/reports_gpu_Test_pdf.css +++ /dev/null @@ -1,101 +0,0 @@ -@page { - size: A4 landscape; - margin: 13mm; -} - -body { - color: #111827; - font-family: "PingFang SC", "Heiti SC", "Arial Unicode MS", sans-serif; - font-size: 11px; - line-height: 1.45; -} - -h1 { - color: #0f172a; - font-size: 24px; - margin: 0 0 14px; -} - -h2 { - border-bottom: 1px solid #cbd5e1; - color: #0f172a; - font-size: 17px; - margin: 24px 0 10px; - padding-bottom: 4px; -} - -h3 { - color: #1f2937; - font-size: 13px; - margin: 16px 0 8px; -} - -p { - margin: 7px 0; -} - -code { - background: #f1f5f9; - border-radius: 3px; - color: #0f172a; - font-family: Menlo, Consolas, monospace; - font-size: 10px; - padding: 1px 3px; -} - -pre { - background: #f8fafc; - border: 1px solid #e2e8f0; - border-radius: 4px; - padding: 8px; - white-space: pre-wrap; -} - -table { - border-collapse: collapse; - margin: 8px 0 14px; - page-break-inside: auto; - width: 100%; -} - -thead { - display: table-header-group; -} - -tr { - page-break-inside: avoid; -} - -th, -td { - border: 1px solid #cbd5e1; - padding: 5px 6px; - text-align: left; - vertical-align: middle; - word-break: break-word; -} - -th { - background: #e2e8f0; - color: #0f172a; - font-weight: 700; -} - -tbody tr:nth-child(even) td { - background: #f8fafc; -} - -a { - color: #2563eb; - text-decoration: none; -} - -ul, -ol { - margin: 6px 0 10px 20px; - padding: 0; -} - -li { - margin: 3px 0; -} diff --git a/reports_h100_acceptance_closure_checklist_20260523.md b/reports_h100_acceptance_closure_checklist_20260523.md deleted file mode 100644 index 6b0264f..0000000 --- a/reports_h100_acceptance_closure_checklist_20260523.md +++ /dev/null @@ -1,105 +0,0 @@ -# H100 验收收尾检查清单 2026-05-23 - -## 结论 - -当前项目已经可以进入“阶段性交付/问题转交”状态，但不能进入“生产验收通过”状态。 - -原因不是测试没跑完，而是当前证据明确显示多个验收门禁仍为 `FAIL`。要真正收尾，必须满足下面两种路径之一： - -1. **通过路径：** 修复硬件/网络/软件环境后复跑，单节点、跨节点 RDMA、多节点 NCCL 均达到 PDF/配置阈值。 -2. **例外路径：** 硬件/网络/环境侧书面确认当前机器与 PDF 参考环境不等价，并给出新的验收阈值或豁免口径，再按新口径复核。 - -在这两条路径完成前，本项目只能交付“已测证据 + 阻塞定位 + 复跑入口”，不能判定 H100 节点生产验收通过。 - -## 当前可关闭的工作 - -| 工作项 | 状态 | 证据 | -|---|---|---| -| 单节点 `test all` 入口 | 完成 | `scripts/run_h100_single_node_all.sh` | -| 单节点中文原始汇总 | 完成 | `reports_test_all_latest_summary_cn_20260523.md` | -| 跨节点 RDMA 单 rail 证据 | 完成 | `reports_rdma_cross_node_mlx5_0_20260523.md` | -| 多节点 NCCL PDF matrix | 完成 | `scripts/run_multinode_nccl_pdf_matrix.sh`，`reports_multinode_nccl_pdf_matrix_run_20260523.md` | -| 多节点 2x8 六项 collective | 完成 | `scripts/run_multinode_nccl_all_collectives.sh`，`reports_multinode_nccl_all_collectives_run_20260523.md` | -| NCCL artifacts / checksum | 完成 | `reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md`，`reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md` | -| 环境等价性分析 | 完成 | `reports_multinode_nccl_environment_gap_20260523.md` | -| 交付包 manifest | 完成 | `reports_h100_acceptance_delivery_manifest_20260523.md` | -| 网络/硬件/环境闭环请求 | 完成 | `reports_h100_network_hardware_escalation_request_20260523.md` | -| 接手 runbook / README 入口 | 完成 | `README.md`，`reports_multinode_nccl_handoff_plan_20260523.md` | - -这些工作可以作为当前阶段交付物归档。 - -## 不能关闭的验收门禁 - -| 门禁 | 当前结果 | 现有证据 | 关闭条件 | -|---|---|---|---| -| 单节点 Compute | FAIL | 两台机器多 dtype 绝对 TFLOPS 未达阈值，部分 GPU spread 超 3% | 复核阈值/测试实现后重跑通过，或更新阈值口径 | -| 单节点 NCCL | FAIL | 多 op/size 未达阈值，尤其小包和部分 2G case | 按 PDF/config 逐 size 通过，或明确小包/阈值豁免 | -| 单节点 Stress | FAIL | 30 分钟可跑满，但温差和 `sw_power_cap` throttle 触发 FAIL | 调整散热/功耗策略或阈值后重跑通过 | -| 单节点 RDMA | FAIL | read BW 未达 47 GB/s，`mlx5_4/5` 只有 100G | perftest read/write/latency 和端口速率满足验收口径 | -| 跨节点 RDMA | FAIL | `mlx5_0` 写带宽 PASS，但读带宽和读写 latency FAIL | 双向 write/read BW/latency 全部达标 | -| 多节点 NCCL allreduce | FAIL | 2x8 `353.85 GB/s`，目标 `491.84 GB/s` | 环境等价后达到 PDF 阈值，或按 4 x 400G rail 重定标 | -| 多节点 NCCL alltoall | FAIL | 2x8 `36.83 GB/s`，目标 `76.54 GB/s` | 网络/plugin/SHARP/路径修复后达到阈值，或明确新口径 | -| PDF 环境等价性 | 未证明 | 当前每节点只有 4 条 400G rail，缺外部 NCCL net plugin / SHARP | 确认参考环境 rail/plugin/SHARP/交换策略，并补齐或重定标 | - -## 最短收尾路径 - -### 路径 A：按原 PDF 阈值验收 - -必须先完成环境补齐： - -1. 确认每节点是否应有 8 条 400G IB rail；如果是，修复 `mlx5_4/5`、`mlx5_2/8`、`mlx5_3/9` 的速率/模式/状态。 -2. 如 PDF 参考环境使用 SHARP、HCOLL、UCX plugin 或 NCCL net plugin，则在两台节点补齐同等组件。 -3. 让网络侧确认跨 Leaf ECMP / adaptive routing / congestion control / credit wait 配置。 -4. 复跑： - -```bash -cd /root/test_gpu_scripts -bash scripts/run_h100_single_node_all.sh -bash scripts/run_multinode_nccl_pdf_matrix.sh -bash scripts/run_multinode_nccl_all_collectives.sh -``` - -关闭标准：`reports_h100_acceptance_current_status_*.md` 中所有必测项不再有 `FAIL`。 - -### 路径 B：承认当前环境与 PDF 不等价 - -必须拿到新的验收口径： - -1. 硬件/网络侧确认当前机器实际有效 400G IB rail 数量。 -2. 明确是否允许按 4 x 400G rail 的物理上限重定 allreduce 阈值。 -3. 明确 2x8 alltoall 的合理目标，或要求安装 plugin/SHARP 后再判。 -4. 明确单节点 Compute、Stress、RDMA 的阈值是否沿用 PDF 原口径。 -5. 用新口径更新配置后复跑并生成新报告。 - -关闭标准：新口径必须写进配置或报告，不能只口头说明。 - -## 下一步优先级 - -| 优先级 | 动作 | 负责人建议 | 为什么 | -|---:|---|---|---| -| P0 | 确认 PDF 参考环境 rail/plugin/SHARP 状态 | 硬件/网络/环境侧 | 不确认等价性，2x8 allreduce 阈值是否合理无法判断 | -| P0 | 查跨 Leaf alltoall 网络路径 | 网络侧 | alltoall 低于目标过多，且参数 sweep 无稳定收益 | -| P1 | 复核单节点 Compute 阈值和测试 dtype 路径 | 测试/平台侧 | 两台机器多 dtype 绝对阈值均失败，需要确认是不是口径问题 | -| P1 | 处理 Stress `sw_power_cap` 和温差 | 机房/硬件侧 | 压测能跑满，但 telemetry 门禁未过 | -| P1 | 处理 RDMA read BW/latency | 网络/OFED/固件侧 | 单节点和跨节点 RDMA 都有 read/latency 缺口 | -| P2 | 启用 plugin/SHARP 后复跑 NCCL graph | 平台侧 | 用于验证 `plugin_missing` 是否消失、图策略是否变化 | - -## 当前交付物入口 - -优先读： - -1. `reports_h100_acceptance_current_status_20260523.md` -2. `reports_h100_acceptance_closure_checklist_20260523.md` -3. `reports_h100_acceptance_delivery_manifest_20260523.md` -4. `reports_h100_network_hardware_escalation_request_20260523.md` -5. `reports_multinode_nccl_handoff_plan_20260523.md` -6. `reports_multinode_nccl_environment_gap_20260523.md` -7. `reports_multinode_nccl_latest_index_20260523.md` - -当前项目可以向外汇报为： - -```text -测试脚本、复跑入口、原始 artifacts、checksum 和中文报告已经齐备； -但当前 H100 生产验收未通过，剩余问题集中在单节点 Compute/NCCL/Stress/RDMA、 -跨节点 RDMA read/latency、多节点 NCCL 2x8 allreduce/alltoall 性能，以及 PDF 环境等价性。 -``` diff --git a/reports_h100_acceptance_current_status_20260523.md b/reports_h100_acceptance_current_status_20260523.md deleted file mode 100644 index 0686918..0000000 --- a/reports_h100_acceptance_current_status_20260523.md +++ /dev/null @@ -1,164 +0,0 @@ -# H100 验收当前状态总览 2026-05-23 - -## 一句话结论 - -当前脚本能力和证据链已经基本补齐：单节点 `test all`、多机多卡 PDF matrix、2x8 六项 collective、跨节点 RDMA、NCCL artifacts、环境快照和 checksum 都已经有可复跑入口和原始证据。但按当前 PDF/配置口径，两台 H100 节点仍不能判定生产验收通过，主要阻塞不是脚本没跑，而是多项实测指标低于阈值，以及当前硬件/软件环境无法证明与 PDF 参考环境等价。 - -## 当前总状态 - -| 范围 | 当前证据 | 结论 | 主要阻塞 | -|---|---|---|---| -| 单节点 `test all` | `reports_test_all_latest_summary_cn_20260523.md` | 两台均 FAIL | Compute、NCCL、Stress、RDMA | -| 跨节点 RDMA | `reports_rdma_cross_node_mlx5_0_20260523.md` | FAIL | read BW、write/read latency 未达阈值 | -| 多机多卡 PDF matrix | `reports_multinode_nccl_pdf_matrix_run_20260523.md` | FAIL | 2x8 allreduce/alltoall 差距大，1/4 GPU 档位部分小差距 | -| 多机多卡 2x8 六项 collective | `reports_multinode_nccl_all_collectives_run_20260523.md` | FAIL / evidence complete | 6 项正确性通过；allreduce/alltoall 按 PDF 阈值 FAIL | -| NCCL artifacts 信号 | `reports_multinode_nccl_artifact_signal_analysis_20260523.md` | 基础链路正常 | IB/GDRDMA/HCA 均正常；无 SHARP/CollNet/外部 net plugin | -| 环境等价性 | `reports_multinode_nccl_environment_gap_20260523.md` | 未证明等价 | 每节点只有 4 条 400G rail，缺 NCCL net plugin / SHARP | -| 收尾检查 | `reports_h100_acceptance_closure_checklist_20260523.md` | 可阶段性交付 | 生产验收门禁仍未关闭 | -| 交付包 manifest | `reports_h100_acceptance_delivery_manifest_20260523.md` | 已形成 | 入口、脚本、远端 artifacts、checksum 已汇总 | -| 网络/硬件/环境闭环 | `reports_h100_network_hardware_escalation_request_20260523.md` | 已形成请求 | 等待 rail/plugin/SHARP/交换策略/阈值口径回填 | - -## 已完成的能力 - -| 能力 | 当前状态 | -|---|---| -| 单节点 H100 all 验收入口 | `scripts/run_h100_single_node_all.sh` 已可用，默认带环境快照 | -| 多机 PDF matrix 入口 | `scripts/run_multinode_nccl_pdf_matrix.sh` 已可用，自动归档每个 case 的 `cmd/stdout/stderr/json` | -| 多机 2x8 六项 collective 入口 | `scripts/run_multinode_nccl_all_collectives.sh` 已可用，覆盖 `allreduce/alltoall/broadcast/reducescatter/allgather/sendrecv` | -| NCCL 深度诊断入口 | `scripts/multinode_nccl_deep_diagnose.sh` 已可用，覆盖 preflight、counter、graph、PXN sweep | -| 环境等价性快照 | `scripts/nccl_environment_snapshot.sh` 已可用 | -| 原始证据归档 | PDF matrix 和六项 collective artifacts 均已 tar + checksum | -| 中文解释文档 | 指标说明、NCCL/RDMA 概念、handoff、environment gap、artifact signal analysis 均已生成 | - -## 单节点验收状态 - -两台机器的单节点 `test all` 当前都是： - -```text -Suite: 6/10 PASS -PDF acceptance: FAIL -``` - -通过项： - -- GPU Info -- Health -- Memory Bandwidth -- NVLink/NVSwitch -- DCGM diag -r 3 -- Training Simulation - -失败项： - -| 项目 | 当前现象 | 备注 | -|---|---|---| -| Compute | 多 dtype 绝对 TFLOPS 阈值未达，部分 GPU 间 spread 超 3% | 需要复核 H100 阈值口径和具体 dtype 路径 | -| NCCL 单机 | 真实 `nccl-tests` 已可测，但多 op/size 未达阈值 | 主要是 1M 小包，以及 reducescatter/allgather 的 2G | -| Stress | 30 分钟可跑满，但温差和 `sw_power_cap` throttle 导致 FAIL | 更像散热/功耗策略或阈值口径问题 | -| RDMA 单机 | read BW 未达标，部分端口速率低于 400G | 单机 local-loopback 不能替代跨节点 RDMA | - -## 跨节点 RDMA 状态 - -跨节点 `mlx5_0` 单 rail perftest 结果： - -| Direction | Test | Value | Threshold | Status | -|---|---|---:|---:|---| -| 0016 -> 0012 | ib_write_bw | 49.35 GB/s | >= 47 GB/s | PASS | -| 0016 -> 0012 | ib_read_bw | 44.36 GB/s | >= 47 GB/s | FAIL | -| 0016 -> 0012 | ib_write_lat avg | 2.17 us | <= 2.0 us | FAIL | -| 0016 -> 0012 | ib_read_lat avg | 4.05 us | <= 3.5 us | FAIL | -| 0012 -> 0016 | ib_write_bw | 48.38 GB/s | >= 47 GB/s | PASS | -| 0012 -> 0016 | ib_read_bw | 44.37 GB/s | >= 47 GB/s | FAIL | -| 0012 -> 0016 | ib_write_lat avg | 2.13 us | <= 2.0 us | FAIL | -| 0012 -> 0016 | ib_read_lat avg | 4.08 us | <= 3.5 us | FAIL | - -判断：链路连通、ibping 正常、PFC/ECN/CNP/congestion counter 干净；但 read bandwidth 和 latency 仍低于阈值，需要网络/OFED/BIOS/firmware 或 perftest 参数侧继续确认。 - -## 多机多卡 NCCL 状态 - -### PDF Matrix - -| Topology | AllReduce | Target | Status | AllToAll | Target | Status | -|---|---:|---:|---|---:|---:|---| -| 2 nodes x 1 GPU | 47.29 | 48.90 | FAIL | 24.85 | 27.25 | FAIL | -| 2 nodes x 2 GPUs | 137.16 | 136.93 | PASS | 47.76 | 54.41 | FAIL | -| 2 nodes x 4 GPUs | 335.07 | 335.48 | FAIL | 72.74 | 73.73 | FAIL | -| 2 nodes x 8 GPUs | 353.85 | 491.84 | FAIL | 36.83 | 76.54 | FAIL | - -所有 case 均 `returncode=0`、`wrong=0`，所以 FAIL 来自性能阈值，不是功能错误。 - -### 2x8 六项 Collective 补测 - -| Operation | Peak Bus BW | Threshold | Correctness | Network | Status | -|---|---:|---:|---|---|---| -| allreduce | 354.27 | >= 491.84 | wrong=0 | IB/GDRDMA | FAIL | -| alltoall | 37.00 | >= 76.54 | wrong=0 | IB/GDRDMA | FAIL | -| broadcast | 191.65 | 未配置 | wrong=0 | IB/GDRDMA | PASS evidence | -| reducescatter | 192.75 | 未配置 | wrong=0 | IB/GDRDMA | PASS evidence | -| allgather | 192.14 | 未配置 | wrong=0 | IB/GDRDMA | PASS evidence | -| sendrecv | 26.98 | 未配置 | wrong=0 | IB/GDRDMA | PASS evidence | - -这说明多机多卡 collective 覆盖面已经补齐，但生产性能是否达标仍取决于 PDF 是否有对应跨节点阈值，以及当前环境是否与 PDF 等价。 - -## 当前最关键阻塞 - -### 1. PDF 参考环境等价性未确认 - -当前两台节点每节点只有 4 条可用于 NCCL 的 400G IB rail： - -```text -mlx5_0, mlx5_1, mlx5_6, mlx5_7 -``` - -其他 HCA： - -```text -mlx5_4, mlx5_5: 100G InfiniBand -mlx5_2, mlx5_8: 25G Ethernet -mlx5_3, mlx5_9: DOWN -``` - -PDF 2x8 allreduce 目标 `491.84 GB/s busbw` 反推 algbw 为 `262.31 GB/s`，高于当前 4 x 400G rail 的理论单向原始带宽 `200 GB/s`。如果 PDF 参考环境有更多 400G rail 或 SHARP/plugin，当前硬件/软件栈不等价。 - -### 2. 缺少 NCCL net plugin / SHARP - -当前没有发现： - -```text -libnccl-net*.so* -libsharp*.so* -SHARP / HCOLL package -``` - -NCCL 日志中没有 SHARP/CollNet 迹象，当前走 internal IB plugin。 - -### 3. alltoall 仍是独立问题 - -`NCCL_PXN_DISABLE=1` 后 alltoall rail 更均衡，但 2x8 仍只有约 `36-37 GB/s`。已有 sweep 没找到稳定正收益，下一步应该交给网络路径、ECMP/adaptive routing、拥塞控制、plugin/SHARP 等方向，而不是继续盲调 NCCL 小参数。 - -### 4. 单节点 Compute/Stress/RDMA 也未过 - -即使多机 NCCL 后续解决，两台机器按当前 PDF `test all` 仍因 Compute、Stress、RDMA 项失败，不能直接判整机生产验收通过。 - -## 建议下一步 - -1. **硬件/网络侧先确认 PDF 等价性。** 确认参考环境每节点到底是 4 条还是 8 条 400G rail，是否启用 SHARP/NCCL net plugin，交换网络是否同一策略。 -2. **环境侧补齐或明确排除 SHARP/plugin。** 如果 PDF 环境有，当前必须补齐后重跑 `scripts/run_multinode_nccl_pdf_matrix.sh` 和 `scripts/run_multinode_nccl_all_collectives.sh`。 -3. **网络侧排查 alltoall。** 重点看跨 Leaf ECMP/adaptive routing/拥塞控制/credit wait，而不是只看链路是否 up。 -4. **单节点继续分项收敛。** Compute 阈值、Stress 温差/功耗 cap、RDMA read/latency 需要分别确认是机器问题、配置问题还是阈值口径问题。 -5. **如果硬件不等价，调整验收阈值或换等价节点复测。** 当前证据不支持把 4 rail 环境直接按疑似更高规格 PDF 阈值判定。 - -## 当前最值得先读的文件 - -| 顺序 | 文件 | 用途 | -|---:|---|---| -| 1 | `reports_h100_acceptance_current_status_20260523.md` | 当前总览和阻塞清单 | -| 2 | `reports_h100_acceptance_closure_checklist_20260523.md` | 收尾检查清单和关闭条件 | -| 3 | `reports_h100_acceptance_delivery_manifest_20260523.md` | 交付包 manifest 和 checksum | -| 4 | `reports_h100_network_hardware_escalation_request_20260523.md` | 给网络/硬件/环境侧的闭环请求 | -| 5 | `reports_multinode_nccl_handoff_plan_20260523.md` | 给网络/硬件/环境侧的交接计划 | -| 6 | `reports_multinode_nccl_environment_gap_20260523.md` | PDF 环境等价性缺口 | -| 7 | `reports_multinode_nccl_artifact_signal_analysis_20260523.md` | NCCL artifacts 信号分析 | -| 8 | `reports_multinode_nccl_all_collectives_run_20260523.md` | 多机 2x8 六项 collective 补测摘要 | -| 9 | `reports_test_all_latest_summary_cn_20260523.md` | 单节点 test all 中文汇总 | -| 10 | `reports_rdma_cross_node_mlx5_0_20260523.md` | 跨节点 RDMA 单 rail 证据 | diff --git a/reports_h100_acceptance_delivery_manifest_20260523.md b/reports_h100_acceptance_delivery_manifest_20260523.md deleted file mode 100644 index 735b5ea..0000000 --- a/reports_h100_acceptance_delivery_manifest_20260523.md +++ /dev/null @@ -1,152 +0,0 @@ -# H100 验收交付包 Manifest 2026-05-23 - -## 交付结论 - -当前分支：`h100-acceptance-current` - -最新 commit：以 `git log -1 --oneline` 为准。 - -当前状态：**测试侧阶段性交付完成，生产验收未通过。** - -本交付包已经覆盖单节点 `test all`、跨节点 RDMA、多节点 NCCL PDF matrix、多节点 2x8 六项 collective、环境等价性分析、网络/硬件/环境闭环请求、复跑脚本和 artifacts checksum。剩余工作需要网络/硬件/环境侧确认后才能继续往最终验收推进。 - -## 主入口 - -按下面顺序阅读： - -| 顺序 | 文件 | 用途 | -|---:|---|---| -| 1 | `README.md` | 仓库入口和 H100 当前验收入口 | -| 2 | `reports_h100_acceptance_current_status_20260523.md` | 当前总状态和阻塞项 | -| 3 | `reports_h100_acceptance_closure_checklist_20260523.md` | 可交付项、未关闭门禁、收尾路径 | -| 4 | `reports_h100_acceptance_pr_summary_20260523.md` | PR/审阅摘要 | -| 5 | `reports_h100_network_hardware_escalation_request_20260523.md` | 给网络/硬件/环境侧的回填请求 | -| 6 | `reports_multinode_nccl_latest_index_20260523.md` | 多节点 NCCL 报告索引 | - -## 核心报告 - -| 分类 | 文件 | 当前结论 | -|---|---|---| -| 总览 | `reports_h100_acceptance_current_status_20260523.md` | FAIL，证据链完整但门禁未过 | -| 收尾 | `reports_h100_acceptance_closure_checklist_20260523.md` | 可阶段性交付，不能判生产通过 | -| PR 摘要 | `reports_h100_acceptance_pr_summary_20260523.md` | 给代码审阅和合并说明使用 | -| 闭环请求 | `reports_h100_network_hardware_escalation_request_20260523.md` | 等待网络/硬件/环境侧回填 | -| 单节点 | `reports_test_all_latest_summary_cn_20260523.md` | 两台均 `6/10 PASS`，整体 FAIL | -| 跨节点 RDMA | `reports_rdma_cross_node_mlx5_0_20260523.md` | write BW PASS，read BW/latency FAIL | -| 多节点 NCCL PDF matrix | `reports_multinode_nccl_pdf_matrix_run_20260523.md` | 8 个 case 仅 1 个性能 PASS；正确性均 OK | -| 多节点 NCCL 六项 collective | `reports_multinode_nccl_all_collectives_run_20260523.md` | 6 项正确性 OK；allreduce/alltoall 按 PDF 阈值 FAIL | -| 环境等价性 | `reports_multinode_nccl_environment_gap_20260523.md` | 当前不能证明与 PDF 等价 | -| NCCL artifact 信号 | `reports_multinode_nccl_artifact_signal_analysis_20260523.md` | IB/GDRDMA 正常；缺外部 plugin/SHARP | -| 接手计划 | `reports_multinode_nccl_handoff_plan_20260523.md` | 给继续定位和复跑的人使用 | - -## 可复跑入口 - -| 脚本 | 用途 | 建议执行位置 | -|---|---|---| -| `scripts/run_h100_single_node_all.sh` | 单节点 H100 全量验收 | 两台节点分别执行 | -| `scripts/run_multinode_nccl_pdf_matrix.sh` | 多节点 NCCL PDF matrix | `nccl-gpu-1` | -| `scripts/run_multinode_nccl_all_collectives.sh` | 多节点 2x8 六项 collective | `nccl-gpu-1` | -| `scripts/multinode_nccl_deep_diagnose.sh` | 多节点 NCCL 深度诊断 | `nccl-gpu-1` | -| `scripts/nccl_environment_snapshot.sh` | 单节点 HCA/plugin/topo 快照 | 两台节点分别执行 | - -推荐复跑顺序： - -```bash -cd /root/test_gpu_scripts -bash scripts/multinode_nccl_deep_diagnose.sh preflight -bash scripts/run_multinode_nccl_pdf_matrix.sh -bash scripts/run_multinode_nccl_all_collectives.sh -``` - -如果网络/硬件/环境侧调整了单节点条件，还需要分别在两台节点执行： - -```bash -cd /root/test_gpu_scripts -bash scripts/run_h100_single_node_all.sh -``` - -## 远端位置 - -两台远端默认路径： - -```text -nccl-gpu-1: /root/test_gpu_scripts -nccl-gpu-2: /root/test_gpu_scripts -``` - -最新多节点 NCCL 原始 artifacts 位于 `nccl-gpu-1`： - -| 类型 | 路径 | -|---|---| -| PDF matrix raw report | `/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803.md` | -| PDF matrix artifacts dir | `/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts` | -| PDF matrix artifacts tar | `/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts.tar.gz` | -| 六项 collective raw report | `/root/test_gpu_scripts/reports/multinode_nccl_all_collectives_20260523_120144.md` | -| 六项 collective artifacts dir | `/root/test_gpu_scripts/reports/multinode_nccl_all_collectives_20260523_120144_artifacts` | -| 六项 collective artifacts tar | `/root/test_gpu_scripts/reports/multinode_nccl_all_collectives_20260523_120144_artifacts.tar.gz` | - -## Artifact 校验 - -PDF matrix bundle checksum： - -```text -682ac637460472d464a0d56ccc0f3335ed7f79a270157a403ebec23b8d9feceb reports/multinode_nccl_pdf_matrix_20260523_113803.md -7371fcaf7269f92eb1544e5e63573ebf77f4ae38f668b5b22169ca86e6d603ee reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts.tar.gz -``` - -六项 collective bundle checksum： - -```text -06c565281813c4260da9cfee8f0b0289b61b3be95c01dd670c71fa1a441133e3 reports/multinode_nccl_all_collectives_20260523_120144.md -fa5961d47a5905da6ebc6c726421d73ddc2314a316a8f578683d31fe69c256e5 reports/multinode_nccl_all_collectives_20260523_120144_artifacts.tar.gz -``` - -逐文件 checksum： - -| 文件 | 用途 | -|---|---| -| `reports_multinode_nccl_all_collectives_20260523_120144_bundle.sha256` | 六项 collective raw report + tar checksum | -| `reports_multinode_nccl_all_collectives_20260523_120144_artifacts.sha256` | 六项 collective artifacts 逐文件 checksum | -| `reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md` | PDF matrix case summary + bundle checksum | -| `reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md` | 六项 collective case summary + bundle/per-file checksum | - -## 入口文件 SHA256 - -以下 hash 用于确认本地与两台远端入口文件一致。本 manifest 本身不做自引用 hash。 - -```text -e2faf6cbd968924727c669827d7e838d5165ee961133c8e55e8993134b5e7b63 README.md -846c3da4ac655a0b3ad072e4c4475d91b55e2bdc9d8aedb9c5f9d800608fb64c reports_h100_acceptance_current_status_20260523.md -4a0ee9f456acc1284bf3a42df5bd338affb831471c27ca4b6584201acd72fd52 reports_h100_acceptance_closure_checklist_20260523.md -0c71f36b9b1a6c5a73bd32337a56a702d3faa37c02640b93cb5d00b9b80c362f reports_h100_acceptance_pr_summary_20260523.md -45438db9204ceef5f65019a6594c016f3183799ed3b89dcf40f383a34f9e3466 reports_h100_network_hardware_escalation_request_20260523.md -d982d6f3698e8860b8505d65105f6056c11f1f72758401a4613ae8315b6f92d0 reports_multinode_nccl_latest_index_20260523.md -8fca70e703961745d5bdacaa3fccb814709c426c0fa7713d0df2d1f2fb26a3f4 reports_multinode_nccl_handoff_plan_20260523.md -b0d0d1fa9b1aa0d8cbdd2672508df5c7bafffc91b607b35b199e624352147e75 reports_multinode_nccl_environment_gap_20260523.md -a7bc27c630fb97c0b83a3427ed4017a16a21e1285f4be5a2cc21f653921fab2b reports_multinode_nccl_pdf_matrix_run_20260523.md -60bdb85e087e796d59c6f0cb7e79c7e60b4147b5fff8c6b60606f6c1f53b1b58 reports_multinode_nccl_all_collectives_run_20260523.md -6affec63694d31dc2d7f097210794e7821e931b8c8b9ac8f451c6f7948bf138a reports_test_all_latest_summary_cn_20260523.md -3895cdf040220aa13093c3377c301580120f04eb9958dbb7c3df3d7285c2d733 reports_rdma_cross_node_mlx5_0_20260523.md -``` - -## 还不能关闭的事项 - -| 项目 | 当前阻塞 | -|---|---| -| 单节点 Compute | 多 dtype 绝对 TFLOPS 阈值未达，部分 GPU spread 超 3% | -| 单节点 NCCL | 多 op/size 未达阈值，小包和部分 2G case 明显 | -| 单节点 Stress | 30 分钟可跑满，但温差和 `sw_power_cap` throttle 触发 FAIL | -| 单节点 RDMA | read BW 未达 47 GB/s，部分端口不是 400G | -| 跨节点 RDMA | read BW 和 write/read latency 未达阈值 | -| 多节点 NCCL allreduce | 2x8 `353.85 GB/s`，PDF 目标 `491.84 GB/s` | -| 多节点 NCCL alltoall | 2x8 `36.83 GB/s`，PDF 目标 `76.54 GB/s` | -| PDF 环境等价性 | 当前只有 4 条 400G rail，缺 NCCL net plugin / SHARP 证据 | - -## 下一步闭环条件 - -网络/硬件/环境侧需要给出以下任一结论： - -1. 当前两台机器已修复到 PDF 参考环境等价状态，测试侧复跑。 -2. 当前机器与 PDF 参考环境不等价，但可以接受新的阈值或豁免口径。 -3. 当前硬件/网络不满足交付规格，需要先修复。 -4. PDF 阈值不适用于当前跨 Leaf/4 rail/plugin 缺失场景，需要更新验收标准。 diff --git a/reports_h100_acceptance_pr_summary_20260523.md b/reports_h100_acceptance_pr_summary_20260523.md deleted file mode 100644 index 27b6436..0000000 --- a/reports_h100_acceptance_pr_summary_20260523.md +++ /dev/null @@ -1,144 +0,0 @@ -# H100 验收分支 PR 摘要 2026-05-23 - -## 建议 PR 标题 - -```text -Add H100 acceptance evidence, multinode NCCL runs, and handoff reports -``` - -## PR 结论 - -本 PR 完成 H100 验收测试侧的阶段性交付：脚本、单节点报告、多节点 NCCL 报告、RDMA 证据、artifacts、checksum、中文说明和交接文档已经齐备。 - -但本 PR **不表示生产验收通过**。当前两台 H100 节点按现有 PDF/配置口径仍为 `FAIL`，需要网络/硬件/环境侧完成回填或修复后再复跑。 - -## 变更范围 - -### 测试入口 - -- 新增/完善单节点 H100 `test all` 入口。 -- 新增多节点 NCCL PDF matrix 复跑入口。 -- 新增多节点 2x8 六项 collective 复跑入口。 -- 新增 NCCL 深度诊断和环境快照入口。 - -### 配置 - -- 固定 NCCL 2.27.7 / nccl-tests 路径的多节点 PDF matrix 配置。 -- 新增 2x8 六项 collective 配置。 -- `allreduce/alltoall` 保留已知 PDF 2x8 阈值；新增的 `broadcast/reducescatter/allgather/sendrecv` 暂按证据采集处理。 - -### 报告和证据 - -- 单节点 `test all` 中文汇总。 -- 跨节点 RDMA `mlx5_0` 双向证据。 -- 多节点 NCCL PDF matrix 中文摘要、原始报告、artifacts manifest。 -- 多节点 2x8 六项 collective 中文摘要、原始报告、artifacts manifest。 -- NCCL artifact 信号分析、环境等价性分析、handoff 计划、收尾清单。 -- 网络/硬件/环境侧闭环请求和交付包 manifest。 - -## 当前验收状态 - -| 范围 | 结论 | 说明 | -|---|---|---| -| 单节点 `test all` | FAIL | 两台均 `6/10 PASS`；Compute、NCCL、Stress、RDMA 未过 | -| 跨节点 RDMA | FAIL | write BW PASS；read BW 和 latency 未达阈值 | -| 多节点 NCCL PDF matrix | FAIL | 8 个 case 仅 2x2 allreduce 性能 PASS；所有 case 正确性 OK | -| 多节点 2x8 六项 collective | FAIL / evidence complete | 6 项正确性 OK；allreduce/alltoall 按 PDF 阈值 FAIL | -| 环境等价性 | 未证明 | 当前每节点只有 4 条 400G rail，缺外部 NCCL net plugin / SHARP 证据 | - -## 关键结果 - -### 单节点 - -```text -aikubeworker0012: 6/10 PASS, PDF acceptance FAIL -aikubeworker0016: 6/10 PASS, PDF acceptance FAIL -``` - -### 跨节点 RDMA - -```text -ib_write_bw: 48.38-49.35 GB/s, PASS -ib_read_bw: 44.36-44.37 GB/s, FAIL -ib_write_lat avg: 2.13-2.17 us, FAIL -ib_read_lat avg: 4.05-4.08 us, FAIL -``` - -### 多节点 NCCL PDF matrix - -| Topology | AllReduce | Target | Status | AllToAll | Target | Status | -|---|---:|---:|---|---:|---:|---| -| 2 nodes x 1 GPU | 47.29 | 48.90 | FAIL | 24.85 | 27.25 | FAIL | -| 2 nodes x 2 GPUs | 137.16 | 136.93 | PASS | 47.76 | 54.41 | FAIL | -| 2 nodes x 4 GPUs | 335.07 | 335.48 | FAIL | 72.74 | 73.73 | FAIL | -| 2 nodes x 8 GPUs | 353.85 | 491.84 | FAIL | 36.83 | 76.54 | FAIL | - -所有 NCCL case 均 `returncode=0`、`wrong=0`，当前失败来自性能阈值，不是功能错误。 - -## 主要风险 - -1. **不能把本 PR 合并理解为验收通过。** - 当前结果明确是 `FAIL`，本 PR 交付的是证据链和复跑能力。 - -2. **PDF 2x8 allreduce 阈值可能要求比当前环境更强的 rail/plugin 能力。** - 当前每节点仅 4 条 400G IB rail；PDF 2x8 allreduce 目标 `491.84 GB/s busbw` 反推 algbw `262.31 GB/s`，高于 4 x 400G rail 的理论单向原始带宽 `200 GB/s`。 - -3. **alltoall 需要网络侧继续定位。** - `NCCL_PXN_DISABLE=1` 后 rail 更均衡，但 2x8 alltoall 仍只有 `36-37 GB/s`。 - -4. **单节点门禁也仍未过。** - 即使多节点 NCCL 后续解决，Compute、Stress、RDMA 单节点项仍需闭环。 - -## 验证方式 - -已完成： - -- `git diff --check` -- 本地与两台远端入口文件 sha256 核对 -- 多节点 NCCL PDF matrix 复跑并归档 artifacts -- 多节点 2x8 六项 collective 复跑并归档 artifacts -- 跨节点 RDMA 单 rail 双向测试 -- 单节点 `test all` 汇总 - -远端同步路径： - -```text -nccl-gpu-1: /root/test_gpu_scripts -nccl-gpu-2: /root/test_gpu_scripts -``` - -## 复跑命令 - -```bash -cd /root/test_gpu_scripts -bash scripts/multinode_nccl_deep_diagnose.sh preflight -bash scripts/run_multinode_nccl_pdf_matrix.sh -bash scripts/run_multinode_nccl_all_collectives.sh -``` - -单节点复跑： - -```bash -cd /root/test_gpu_scripts -bash scripts/run_h100_single_node_all.sh -``` - -## Reviewer 重点看 - -| 文件 | 为什么要看 | -|---|---| -| `reports_h100_acceptance_current_status_20260523.md` | 当前总览和失败项 | -| `reports_h100_acceptance_delivery_manifest_20260523.md` | 交付包入口、远端 artifacts、checksum | -| `reports_h100_network_hardware_escalation_request_20260523.md` | 需要网络/硬件/环境侧回填的问题 | -| `reports_multinode_nccl_environment_gap_20260523.md` | 为什么当前环境不能证明与 PDF 等价 | -| `reports_multinode_nccl_pdf_matrix_run_20260523.md` | 多节点 PDF matrix 结果 | -| `reports_multinode_nccl_all_collectives_run_20260523.md` | 六项 collective 补测结果 | - -## 合并建议 - -可以合并为测试侧交付分支，但合并说明中必须保留： - -```text -当前 H100 生产验收未通过；本分支交付测试证据、复跑脚本和闭环请求。 -最终验收需等待网络/硬件/环境侧确认或修复后复跑。 -``` diff --git a/reports_h100_network_hardware_escalation_request_20260523.md b/reports_h100_network_hardware_escalation_request_20260523.md deleted file mode 100644 index f4a82d5..0000000 --- a/reports_h100_network_hardware_escalation_request_20260523.md +++ /dev/null @@ -1,193 +0,0 @@ -# H100 网络/硬件/环境侧闭环请求 2026-05-23 - -## 用途 - -这份文档用于转交给网络、硬件、机房、环境维护同事，目标是把当前 H100 验收剩余 `FAIL` 从“测试侧已复现”推进到“责任侧确认并闭环”。 - -当前测试侧已经完成单节点 `test all`、跨节点 RDMA、多节点 NCCL PDF matrix、2x8 六项 collective、NCCL artifacts、checksum 和中文报告。当前不能判生产验收通过，剩余问题需要网络/硬件/环境侧确认。 - -## 需要对方先读的结论 - -当前两台机器： - -| 角色 | 主机名 | 地址 | -|---|---|---| -| nccl-gpu-1 | `aikubeworker0012` | `172.72.8.12` | -| nccl-gpu-2 | `aikubeworker0016` | `172.72.8.16` | - -当前主要阻塞： - -| 阻塞 | 当前证据 | 需要确认 | -|---|---|---| -| 每节点有效 400G IB rail 只有 4 条 | `mlx5_0,mlx5_1,mlx5_6,mlx5_7` | 这是否符合采购/布线/验收预期 | -| 其他 HCA 不等价 | `mlx5_4/5` 为 100G IB，`mlx5_2/8` 为 25G Ethernet，`mlx5_3/9` DOWN | 是配置问题、线缆/模块问题、交换端口问题，还是设计如此 | -| 缺外部 NCCL 网络组件 | 未找到 `libnccl-net*.so*`、`libsharp*.so*`，未见 SHARP/HCOLL 包 | PDF 参考环境是否启用这些组件 | -| 跨节点 RDMA read/latency 未过 | `ib_read_bw` 约 44.36 GB/s，目标 >= 47 GB/s；latency 也未达阈值 | OFED/固件/BIOS/交换网络/perftest 参数是否需要调整 | -| 2x8 NCCL allreduce 未达 PDF | `353.85 GB/s` vs `491.84 GB/s` | PDF 目标是否要求更多 rail 或 plugin/SHARP | -| 2x8 NCCL alltoall 未达 PDF | `36.83 GB/s` vs `76.54 GB/s` | 跨 Leaf ECMP/adaptive routing/congestion control 是否影响多点流量 | - -## 请对方必须回填的问题 - -### 1. Rail / 端口 / HCA - -请逐项回答： - -| 问题 | 回答 | -|---|---| -| 这两台机器是否设计为每节点 8 条 400G InfiniBand rail？ | | -| 如果是，为什么当前只有 `mlx5_0,mlx5_1,mlx5_6,mlx5_7` 是 400G IB ACTIVE？ | | -| `mlx5_4`、`mlx5_5` 为什么只有 100G IB？ | | -| `mlx5_2`、`mlx5_8` 为什么是 25G Ethernet？ | | -| `mlx5_3`、`mlx5_9` 为什么 DOWN？ | | -| 当前 HCA 状态是否符合这批机器的采购/交付规格？ | | -| 如果不符合，修复动作和预计完成时间是什么？ | | - -建议在两台节点分别执行并回填输出： - -```bash -hostname -for d in /sys/class/infiniband/mlx5_*; do - dev=$(basename "$d") - printf "%s state=%s rate=%s link_layer=%s\n" \ - "$dev" \ - "$(cat "$d/ports/1/state" 2>/dev/null)" \ - "$(cat "$d/ports/1/rate" 2>/dev/null)" \ - "$(cat "$d/ports/1/link_layer" 2>/dev/null)" -done -nvidia-smi topo -m -``` - -### 2. PDF 参考环境等价性 - -请确认 PDF 参考环境到底是什么形态： - -| 问题 | 回答 | -|---|---| -| PDF 参考环境每节点实际参与 NCCL 的 400G rail 数量是多少？ | | -| PDF 参考环境的 HCA 列表是否全部为 400G IB ACTIVE？ | | -| PDF 是否是在同一 Leaf、跨 Leaf，还是不同交换路径下测得？ | | -| PDF 是否启用了 adaptive routing / ECMP / congestion control 特定策略？ | | -| PDF 是否使用了外部 NCCL net plugin / SHARP / HCOLL / UCX plugin？ | | -| 如果当前环境与 PDF 不等价，是否仍要求按 PDF 阈值验收？ | | - -测试侧当前判断：如果 PDF 2x8 allreduce 目标 `491.84 GB/s busbw` 是硬阈值，则其反推 algbw 为： - -```text -491.84 / 1.875 = 262.31 GB/s -``` - -当前每节点 4 条 400G rail 的理论单向原始带宽约： - -```text -4 * 400Gb/s / 8 = 200 GB/s -``` - -因此请明确：当前 4 rail 形态是否允许按 PDF 2x8 allreduce 目标验收。 - -### 3. NCCL net plugin / SHARP / HCOLL - -请逐项回答： - -| 问题 | 回答 | -|---|---| -| 当前生产验收标准是否要求安装 NCCL net plugin？ | | -| 当前生产验收标准是否要求启用 SHARP 或 HCOLL？ | | -| 如果要求，安装包来源、版本、安装路径是什么？ | | -| 安装后是否需要设置 `LD_LIBRARY_PATH`、`NCCL_NET_PLUGIN`、`NCCL_COLLNET_ENABLE` 等变量？ | | -| 如果不要求，是否确认 internal IB plugin 即为验收参考环境？ | | - -建议在两台节点分别执行并回填输出： - -```bash -hostname -find /usr /opt /root /data -name 'libnccl-net*.so*' -o -name 'libsharp*.so*' 2>/dev/null -dpkg -l | egrep -i 'sharp|hcoll|nccl|ucx|ofed|doca' || true -ldconfig -p | egrep -i 'nccl-net|sharp|hcoll|ucx' || true -``` - -### 4. 跨节点 RDMA read/latency - -当前测试侧证据： - -| Direction | Test | Value | Threshold | Status | -|---|---|---:|---:|---| -| 0016 -> 0012 | `ib_write_bw` | 49.35 GB/s | >= 47 GB/s | PASS | -| 0016 -> 0012 | `ib_read_bw` | 44.36 GB/s | >= 47 GB/s | FAIL | -| 0016 -> 0012 | `ib_write_lat` avg | 2.17 us | <= 2.0 us | FAIL | -| 0016 -> 0012 | `ib_read_lat` avg | 4.05 us | <= 3.5 us | FAIL | -| 0012 -> 0016 | `ib_write_bw` | 48.38 GB/s | >= 47 GB/s | PASS | -| 0012 -> 0016 | `ib_read_bw` | 44.37 GB/s | >= 47 GB/s | FAIL | -| 0012 -> 0016 | `ib_write_lat` avg | 2.13 us | <= 2.0 us | FAIL | -| 0012 -> 0016 | `ib_read_lat` avg | 4.08 us | <= 3.5 us | FAIL | - -请确认： - -| 问题 | 回答 | -|---|---| -| 当前 OFED / firmware / BIOS 设置是否符合 400G IB perftest 验收推荐？ | | -| read BW 明显低于 write BW 是否符合预期？ | | -| 当前 latency 阈值是否适用于跨 Leaf 场景？ | | -| 是否需要指定 GID index、MTU、SL、traffic class、PCI relaxed ordering 或其他参数？ | | -| 是否能提供网络侧 port counter / credit wait / congestion 证据？ | | - -### 5. alltoall 跨 Leaf 路径 - -当前测试侧已经做过 NCCL 参数 sweep，`NCCL_PXN_DISABLE=1` 后 rail 更均衡，但 2x8 alltoall 仍只有 `36-37 GB/s`。继续盲调 NCCL 小参数没有明显收益。 - -请网络侧确认： - -| 问题 | 回答 | -|---|---| -| 两台机器是否跨 Leaf？ | | -| 当前跨 Leaf ECMP hash 是否适合 alltoall 多点到多点流量？ | | -| adaptive routing 是否开启？ | | -| 是否存在 credit wait、PFC pause、拥塞控制、buffer 或 QoS 策略限制？ | | -| 是否能提供 alltoall 运行窗口内的交换机端口 counter？ | | - -## 测试侧可配合复跑的命令 - -如果网络/硬件/环境侧完成调整，请在 `nccl-gpu-1` 上复跑： - -```bash -cd /root/test_gpu_scripts -bash scripts/multinode_nccl_deep_diagnose.sh preflight -bash scripts/run_multinode_nccl_pdf_matrix.sh -bash scripts/run_multinode_nccl_all_collectives.sh -``` - -如果调整了 SHARP/plugin，请额外跑： - -```bash -cd /root/test_gpu_scripts -OUT_DIR=/root/test_gpu_scripts/reports/nccl_deep_diag_plugin_check_$(date +%Y%m%d_%H%M%S) \ - bash scripts/multinode_nccl_deep_diagnose.sh graph -``` - -如果调整了单节点环境，请分别在两台节点跑： - -```bash -cd /root/test_gpu_scripts -bash scripts/run_h100_single_node_all.sh -``` - -## 测试侧当前交付物 - -| 文件 | 用途 | -|---|---| -| `reports_h100_acceptance_current_status_20260523.md` | 当前总览 | -| `reports_h100_acceptance_closure_checklist_20260523.md` | 收尾检查清单和关闭条件 | -| `reports_h100_network_hardware_escalation_request_20260523.md` | 本闭环请求 | -| `reports_multinode_nccl_environment_gap_20260523.md` | PDF 环境等价性缺口 | -| `reports_multinode_nccl_handoff_plan_20260523.md` | 复跑和接手计划 | -| `reports_multinode_nccl_pdf_matrix_run_20260523.md` | 多节点 NCCL PDF matrix 摘要 | -| `reports_multinode_nccl_all_collectives_run_20260523.md` | 多节点 2x8 六项 collective 摘要 | -| `reports_rdma_cross_node_mlx5_0_20260523.md` | 跨节点 RDMA 单 rail 证据 | - -## 闭环判定 - -网络/硬件/环境侧需要输出以下任一结论，测试侧才能继续往最终验收推进： - -1. **环境修复完成：** 当前两台机器已达到 PDF 参考环境等价状态，请测试侧复跑。 -2. **环境不等价但可接受：** 当前机器规格与 PDF 不同，请按新的阈值/豁免口径复跑；新口径需写入配置或报告。 -3. **硬件/网络异常：** 当前机器或网络不满足交付规格，需要先修复硬件/布线/交换配置。 -4. **参考标准有误：** PDF 阈值不适用于当前场景，需要更新验收标准。 diff --git a/reports_multinode_nccl_16g_2x8_nccl227.md b/reports_multinode_nccl_16g_2x8_nccl227.md deleted file mode 100644 index 394f191..0000000 --- a/reports_multinode_nccl_16g_2x8_nccl227.md +++ /dev/null @@ -1,66 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T07:56:26.791384 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- DCGM -- Training - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: large-message-nccl-2.27.7 - -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS - -### Multi-node NCCL allreduce - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs NCCL 2.27.7 16G | 237.86 GB/s | 16G | 238.56 GB/s | >= 480 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 16G | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 16G | 0 | aikubeworker0016:1019342:1020412 [4] NCCL INFO comm 0x559f14871c30 rank 12 nranks 16 cudaDev 4 busId 9a000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 238.555 # # Collective test concluded: all_reduce_perf # | - -### Multi-node NCCL alltoall - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs NCCL 2.27.7 16G | 28.62 GB/s | 16G | 28.62 GB/s | >= 75 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 16G | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 16G | 0 | E aikubeworker0016:1020609:1021756 [5] NCCL INFO comm 0x55f920e55d90 rank 13 nranks 16 cudaDev 5 busId ab000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 28.6222 # # Collective test concluded: alltoall_perf # | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_multinode_nccl_16g_2x8_nccl227_auto.md b/reports_multinode_nccl_16g_2x8_nccl227_auto.md deleted file mode 100644 index 0481813..0000000 --- a/reports_multinode_nccl_16g_2x8_nccl227_auto.md +++ /dev/null @@ -1,66 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T08:09:56.340954 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- DCGM -- Training - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: large-message-nccl-2.27.7-auto - -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS - -### Multi-node NCCL allreduce - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs NCCL 2.27.7 auto 16G | 354.60 GB/s | 16G | 354.57 GB/s | >= 480 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 auto 16G | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 auto 16G | 0 | 0012:2149404:2149572 [7] NCCL INFO comm 0x560bd3541a30 rank 7 nranks 16 cudaDev 7 busId db000 - Destroy COMPLETE aikubeworker0016:1066162:1066981 [5] NCCL INFO comm 0x55e73208e200 rank 13 nranks 16 cudaDev 5 busId ab000 - Destroy COMPLETE | - -### Multi-node NCCL alltoall - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs NCCL 2.27.7 auto 16G | 30.01 GB/s | 16G | 30.02 GB/s | >= 75 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 auto 16G | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 auto 16G | 0 | r0012:2149589:2149764 [7] NCCL INFO comm 0x55fef234b7c0 rank 7 nranks 16 cudaDev 7 busId db000 - Destroy COMPLETE aikubeworker0012:2149588:2149765 [6] NCCL INFO comm 0x5637718f1dd0 rank 6 nranks 16 cudaDev 6 busId ba000 - Destroy COMPLETE | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_multinode_nccl_all_collectives_20260523_120144.md b/reports_multinode_nccl_all_collectives_20260523_120144.md deleted file mode 100644 index 2b1d604..0000000 --- a/reports_multinode_nccl_all_collectives_20260523_120144.md +++ /dev/null @@ -1,98 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T12:04:48.257734 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Failed or unverified items: -- Multi-node NCCL: FAIL - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: cross-leaf-all-collectives-nccl-2.27.7 - -- **Artifacts:** `/root/test_gpu_scripts/reports/multinode_nccl_all_collectives_20260523_120144_artifacts` -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS - -### Multi-node NCCL allreduce - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs (all collectives evidence run) | - | 354.27 GB/s | 16G | 354.45 GB/s | >= 491.84 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs (all collectives evidence run) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs (all collectives evidence run) | 0 | nks 16 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0012:2208791:2208941 [0] NCCL INFO comm 0x557970d9f5f0 rank 0 nranks 16 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 354.452 # | - -### Multi-node NCCL alltoall - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs (all collectives evidence run) | - | 37.00 GB/s | 16G | 37.14 GB/s | >= 76.54 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs (all collectives evidence run) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs (all collectives evidence run) | 0 | r0012:2208962:2209141 [5] NCCL INFO comm 0x564c4f9c4a30 rank 5 nranks 16 cudaDev 5 busId ab000 - Destroy COMPLETE aikubeworker0012:2208963:2209143 [6] NCCL INFO comm 0x56328e52f270 rank 6 nranks 16 cudaDev 6 busId ba000 - Destroy COMPLETE | - -### Multi-node NCCL broadcast - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs (all collectives evidence run) | - | 191.65 GB/s | 16G | 190.25 GB/s | - | PASS | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs (all collectives evidence run) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -### Multi-node NCCL reducescatter - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs (all collectives evidence run) | - | 192.75 GB/s | 16G | 192.74 GB/s | - | PASS | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs (all collectives evidence run) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -### Multi-node NCCL allgather - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs (all collectives evidence run) | - | 192.14 GB/s | 16G | 192.47 GB/s | - | PASS | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs (all collectives evidence run) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -### Multi-node NCCL sendrecv - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs (all collectives evidence run) | - | 26.98 GB/s | 16G | 26.97 GB/s | - | PASS | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs (all collectives evidence run) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_multinode_nccl_all_collectives_20260523_120144_artifacts.sha256 b/reports_multinode_nccl_all_collectives_20260523_120144_artifacts.sha256 deleted file mode 100644 index 0264ba3..0000000 --- a/reports_multinode_nccl_all_collectives_20260523_120144_artifacts.sha256 +++ /dev/null @@ -1,24 +0,0 @@ -efa4a915bdf4943aef5d88c402c24eb2c60848e5f440f58058a1e99217b07e0d reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allgather_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.cmd.txt -020eb35ddc5933da78b5c00c1b6fc25b11b23c4505300276d9736fbe8a35519b reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allgather_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allgather_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stderr.txt -903772b675d9a9f7b04e061a25a90f97bf7844dddb5f3809bc9c501f4d6c783d reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allgather_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stdout.txt -b7ea7350b3703d4b31389d92b375562bd04a50b40fe16a6c8d037b134a51dbd5 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allreduce_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.cmd.txt -47f68b7510df3b472e7ac0ec2fb53dcefbe687bb4de0c889f8947cc652d09e61 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allreduce_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allreduce_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stderr.txt -6889180431d639e414e188e1dbc586157565e8506255731b7b38d221d0f72919 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allreduce_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stdout.txt -6ecbd8473d987d2a7839135029902bd629403eb407a7873502a49be26fa1c947 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/alltoall_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.cmd.txt -fa2828cdfcb86e6715a17c8bf45de10ce421c12f0877efff9bafb218b2f00df3 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/alltoall_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/alltoall_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stderr.txt -2eae24183754f8d084945d9857b84033ebccf1a2e606931b4f4fc19c5e2e876f reports/multinode_nccl_all_collectives_20260523_120144_artifacts/alltoall_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stdout.txt -277e900dc1efa8f036616226dbc30cb616ba97337e929ad8b1a14c12484867b3 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/broadcast_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.cmd.txt -077fec1bf498fd202e2866f1cf6fb4502ac8d1bafba156f213453b21f6a6df2b reports/multinode_nccl_all_collectives_20260523_120144_artifacts/broadcast_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/broadcast_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stderr.txt -727c69ad6111b891c25360bd9e97ce15f2e7a36d5ff61ae88a7577ecb61c895f reports/multinode_nccl_all_collectives_20260523_120144_artifacts/broadcast_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stdout.txt -8bec99a952eeb26fa3c6d89cbf2331393923fd4f0fae153b8efe3da239c0a09f reports/multinode_nccl_all_collectives_20260523_120144_artifacts/reducescatter_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.cmd.txt -be24943eb4b63e304cee41831adeb23ffbbc0e890ff19b067e06d6a4b48b2d90 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/reducescatter_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/reducescatter_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stderr.txt -a8220b6a4fe3ae037837919a181452e0fc735f58f27fafff07ea431b09b905de reports/multinode_nccl_all_collectives_20260523_120144_artifacts/reducescatter_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stdout.txt -ead794f19e1d2d780cf1840c124b6e0955c70c8b157feb47c4826599d5643b39 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/sendrecv_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.cmd.txt -4560364922a85d21827357b906491aae8283c6148ff1c0e0f0dc379a68307fdd reports/multinode_nccl_all_collectives_20260523_120144_artifacts/sendrecv_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/sendrecv_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stderr.txt -ade548ee5fdbe2d1fce461237b5b713cc2af24e6c2857bbbd73837f28551af27 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/sendrecv_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.stdout.txt diff --git a/reports_multinode_nccl_all_collectives_20260523_120144_bundle.sha256 b/reports_multinode_nccl_all_collectives_20260523_120144_bundle.sha256 deleted file mode 100644 index 3097f81..0000000 --- a/reports_multinode_nccl_all_collectives_20260523_120144_bundle.sha256 +++ /dev/null @@ -1,2 +0,0 @@ -06c565281813c4260da9cfee8f0b0289b61b3be95c01dd670c71fa1a441133e3 reports/multinode_nccl_all_collectives_20260523_120144.md -fa5961d47a5905da6ebc6c726421d73ddc2314a316a8f578683d31fe69c256e5 reports/multinode_nccl_all_collectives_20260523_120144_artifacts.tar.gz diff --git a/reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md b/reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md deleted file mode 100644 index b1fc9b5..0000000 --- a/reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md +++ /dev/null @@ -1,46 +0,0 @@ -# 多机多卡 NCCL 六项 Collective Artifacts Manifest 2026-05-23 - -- Remote report: `reports/multinode_nccl_all_collectives_20260523_120144.md` -- Remote artifact dir: `reports/multinode_nccl_all_collectives_20260523_120144_artifacts` -- Remote artifact tar: `reports/multinode_nccl_all_collectives_20260523_120144_artifacts.tar.gz` -- Remote bundle checksum: `reports/multinode_nccl_all_collectives_20260523_120144_bundle.sha256` -- Remote per-file checksum: `reports/multinode_nccl_all_collectives_20260523_120144_artifacts.sha256` -- Local report copy: `reports_multinode_nccl_all_collectives_20260523_120144.md` -- Local artifact tar copy: `/private/tmp/multinode_nccl_all_collectives_20260523_120144_artifacts.tar.gz` -- Case count: `6` -- Artifact files: `24` - -## Case Summary - -| Case | Peak Bus BW | Avg Bus BW | Threshold | Wrong | Return Code | Status | -|---|---:|---:|---:|---:|---:|---| -| `allreduce_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run` | 354.27 | 354.45 | 491.84 | 0 | 0 | FAIL | -| `alltoall_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run` | 37.00 | 37.14 | 76.54 | 0 | 0 | FAIL | -| `broadcast_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run` | 191.65 | 190.25 | 0.00 | 0 | 0 | PASS | -| `reducescatter_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run` | 192.75 | 192.74 | 0.00 | 0 | 0 | PASS | -| `allgather_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run` | 192.14 | 192.47 | 0.00 | 0 | 0 | PASS | -| `sendrecv_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run` | 26.98 | 26.97 | 0.00 | 0 | 0 | PASS | - -## Bundle Checksums - -```text -06c565281813c4260da9cfee8f0b0289b61b3be95c01dd670c71fa1a441133e3 reports/multinode_nccl_all_collectives_20260523_120144.md -fa5961d47a5905da6ebc6c726421d73ddc2314a316a8f578683d31fe69c256e5 reports/multinode_nccl_all_collectives_20260523_120144_artifacts.tar.gz -``` - -## Per-file Checksums - -```text -020eb35ddc5933da78b5c00c1b6fc25b11b23c4505300276d9736fbe8a35519b reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allgather_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -47f68b7510df3b472e7ac0ec2fb53dcefbe687bb4de0c889f8947cc652d09e61 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allreduce_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -fa2828cdfcb86e6715a17c8bf45de10ce421c12f0877efff9bafb218b2f00df3 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/alltoall_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -077fec1bf498fd202e2866f1cf6fb4502ac8d1bafba156f213453b21f6a6df2b reports/multinode_nccl_all_collectives_20260523_120144_artifacts/broadcast_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -be24943eb4b63e304cee41831adeb23ffbbc0e890ff19b067e06d6a4b48b2d90 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/reducescatter_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -4560364922a85d21827357b906491aae8283c6148ff1c0e0f0dc379a68307fdd reports/multinode_nccl_all_collectives_20260523_120144_artifacts/sendrecv_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -``` - -完整逐文件 checksum 已保存为： - -```text -reports_multinode_nccl_all_collectives_20260523_120144_artifacts.sha256 -``` diff --git a/reports_multinode_nccl_all_collectives_run_20260523.md b/reports_multinode_nccl_all_collectives_run_20260523.md deleted file mode 100644 index 9468190..0000000 --- a/reports_multinode_nccl_all_collectives_run_20260523.md +++ /dev/null @@ -1,49 +0,0 @@ -# 多机多卡 NCCL 六项 Collective 补测结果 2026-05-23 - -## 测试对象 - -- 节点：`nccl-gpu-1(172.72.8.12)` + `nccl-gpu-2(172.72.8.16)` -- 拓扑：`2 nodes x 8 GPUs` -- NCCL：`2.27.7` -- nccl-tests：`/data/nccl-tests-latest/build` -- 配置：`configs/multinode_nccl_nccl227_all_collectives_2x8.yaml` -- 入口：`scripts/run_multinode_nccl_all_collectives.sh` -- 远端报告：`/root/test_gpu_scripts/reports/multinode_nccl_all_collectives_20260523_120144.md` -- 远端 artifacts：`/root/test_gpu_scripts/reports/multinode_nccl_all_collectives_20260523_120144_artifacts` -- 本地报告：`reports_multinode_nccl_all_collectives_20260523_120144.md` - -## 一句话结论 - -这次补测已经把单机 `test all` 中的 6 个 NCCL collective 扩展到了多机 2x8 场景：`allreduce/alltoall/broadcast/reducescatter/allgather/sendrecv` 都能跑通，`returncode=0`、`wrong_count=0`，并且都走 `IB + GDRDMA`。按已知 PDF 2x8 阈值，`allreduce` 和 `alltoall` 仍 FAIL；新增的 4 项目前没有 PDF 跨节点阈值，因此只作为证据采集项，不判生产验收性能。 - -## 结果表 - -| Operation | Peak Bus BW | Threshold | Correctness | Network | Status | -|---|---:|---:|---|---|---| -| allreduce | `354.27 GB/s` | `>= 491.84 GB/s` | `wrong=0` | `IB/GDRDMA` | FAIL | -| alltoall | `37.00 GB/s` | `>= 76.54 GB/s` | `wrong=0` | `IB/GDRDMA` | FAIL | -| broadcast | `191.65 GB/s` | 未配置 | `wrong=0` | `IB/GDRDMA` | PASS evidence | -| reducescatter | `192.75 GB/s` | 未配置 | `wrong=0` | `IB/GDRDMA` | PASS evidence | -| allgather | `192.14 GB/s` | 未配置 | `wrong=0` | `IB/GDRDMA` | PASS evidence | -| sendrecv | `26.98 GB/s` | 未配置 | `wrong=0` | `IB/GDRDMA` | PASS evidence | - -## 怎么解读 - -1. 这次不是替代 PDF matrix，而是补齐多机多卡 collective 覆盖面。 -2. `allreduce/alltoall` 继续沿用已知 PDF 2x8 阈值，所以报告整体是 `FAIL`。 -3. `broadcast/reducescatter/allgather/sendrecv` 当前只能证明“多机 2x8 能跑、正确性为 0 wrong、走 IB/GDRDMA”，还不能证明生产性能达标，因为手头 PDF matrix 没给这 4 项跨节点阈值。 -4. 新增 4 项的带宽大致呈现两个层次： - - `broadcast/reducescatter/allgather` 在 `191-193 GB/s`，接近当前 4 x 400G rail 的单向原始上限。 - - `sendrecv` 只有 `26.98 GB/s`，需要结合 sendrecv 的 traffic pattern 单独解读，不能直接和 allreduce busbw 混比。 - -## 校验信息 - -```text -06c565281813c4260da9cfee8f0b0289b61b3be95c01dd670c71fa1a441133e3 reports/multinode_nccl_all_collectives_20260523_120144.md -020eb35ddc5933da78b5c00c1b6fc25b11b23c4505300276d9736fbe8a35519b reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allgather_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -47f68b7510df3b472e7ac0ec2fb53dcefbe687bb4de0c889f8947cc652d09e61 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/allreduce_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -fa2828cdfcb86e6715a17c8bf45de10ce421c12f0877efff9bafb218b2f00df3 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/alltoall_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -077fec1bf498fd202e2866f1cf6fb4502ac8d1bafba156f213453b21f6a6df2b reports/multinode_nccl_all_collectives_20260523_120144_artifacts/broadcast_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -be24943eb4b63e304cee41831adeb23ffbbc0e890ff19b067e06d6a4b48b2d90 reports/multinode_nccl_all_collectives_20260523_120144_artifacts/reducescatter_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -4560364922a85d21827357b906491aae8283c6148ff1c0e0f0dc379a68307fdd reports/multinode_nccl_all_collectives_20260523_120144_artifacts/sendrecv_2x8_2_nodes_x_8_GPUs_all_collectives_evidence_run.json -``` diff --git a/reports_multinode_nccl_alltoall_tuning_20260523.md b/reports_multinode_nccl_alltoall_tuning_20260523.md deleted file mode 100644 index dcf75c4..0000000 --- a/reports_multinode_nccl_alltoall_tuning_20260523.md +++ /dev/null @@ -1,160 +0,0 @@ -# 多机 NCCL 8 卡 alltoall 网络参数 sweep - -- 日期：2026-05-23 -- 主机：`aikubeworker0012` / `172.72.8.12`，`aikubeworker0016` / `172.72.8.16` -- NCCL：临时 `2.27.7+cuda12.4` -- 测试：2 nodes x 8 GPUs，`alltoall_perf -b 16G -e 16G` -- HCA：`mlx5_0,mlx5_1,mlx5_6,mlx5_7` - -## 结论 - -`NCCL_PXN_DISABLE=1` 是本轮唯一有效正向参数，可以把 8 卡 alltoall 从约 `30.06 GB/s` 提升到约 `37.24 GB/s`。纳入正式 PDF 矩阵配置后，8 卡 alltoall 原始报告结果为 `36.70 GB/s peak` / `36.74 GB/s avg`。 - -补充计数器探测显示，`NCCL_PXN_DISABLE=1` 的实际作用是把 alltoall 流量重新均匀分配到 4 条 400G rail 上。baseline 下 `mlx5_0/6` 与 `mlx5_1/7` 的流量约为 3:1；禁用 PXN 后四条 HCA 均衡。但每条 rail 的实际吞吐仍只有约 `19-20 GB/s`，没有打满 400G rail。 - -复测错误/拥塞 counter 后，没有看到 discard、链路错误、RoCE 重传、slow restart 或 packet sequence error 增长；主要非零异常是部分端口 `port_xmit_wait`。不过 allreduce 对照在 `354 GB/s busbw` 时也会出现同类 `port_xmit_wait`，所以当前不支持“链路坏包/重传导致慢”的判断，也不能只用 `port_xmit_wait` 解释 alltoall 低吞吐。更可能的方向是 NCCL internal alltoall 通信模式效率、交换侧调度/拥塞控制，或缺少 NCCL net plugin/SHARP。 - -这个提升有实际价值，但仍远低于 PDF 参考 `76.54 GB/s`。在 `NCCL_PXN_DISABLE=1` 之前做过一轮参数 sweep，其他参数没有改善，部分明显变差： - -| Case | Avg Bus BW | 结论 | -|------|------------|------| -| baseline | `30.0633 GB/s` | 基线 | -| `NCCL_PXN_DISABLE=1` | `37.2421 GB/s` | 有效提升 | -| `NCCL_P2P_PXN_LEVEL=0` | `20.1205 GB/s` | 明显变差 | -| `NCCL_P2P_PXN_LEVEL=1` | `30.0588 GB/s` | 无改善 | -| `NCCL_P2P_PXN_LEVEL=2` | `30.0437 GB/s` | 无改善 | -| `NCCL_NET_SHARED_COMMS=0` | `27.3889 GB/s` | 变差 | -| `NCCL_NET_SHARED_BUFFERS=0` | `28.2389 GB/s` | 变差 | -| `NCCL_NET_SHARED_COMMS=0 NCCL_NET_SHARED_BUFFERS=0` | `28.2279 GB/s` | 变差 | -| `NCCL_NCHANNELS_PER_NET_PEER=2` | `30.0281 GB/s` | 无改善 | -| `NCCL_NCHANNELS_PER_NET_PEER=4` | `29.9802 GB/s` | 无改善 | -| `NCCL_IB_ADAPTIVE_ROUTING=1 NCCL_IB_AR_THRESHOLD=0` | `30.0526 GB/s` | 无改善 | -| `NCCL_IB_ADAPTIVE_ROUTING=0` | `30.0535 GB/s` | 无改善 | -| `NCCL_IB_PCI_RELAXED_ORDERING=0` | 未完成 | 明显异常，不建议 | - -在 `NCCL_PXN_DISABLE=1` 作为基线后又补跑了一轮叠加参数 sweep。短测窗口里 `NVLS_ENABLE=0`、`P2P_NET_CHUNKSIZE=4M` 有小幅波动式提升，但更长 `-w 10 -n 10` 复测没有复现，不能作为稳定优化项。 - -| Case | Avg Bus BW | 结论 | -|------|------------|------| -| `NCCL_PXN_DISABLE=1` | `37.0069 GB/s` | 短测基线 | -| `+ NCCL_NVLS_ENABLE=0` | `37.2217 GB/s` | 小幅波动，不稳定 | -| `+ NCCL_P2P_NET_CHUNKSIZE=4194304` | `37.2522 GB/s` | 小幅波动，不稳定 | -| `+ NCCL_BUFFSIZE=8388608` | `37.0911 GB/s` | 无实质改善 | -| `+ NCCL_MIN_NCHANNELS=16 NCCL_MAX_NCHANNELS=16` | `37.0189 GB/s` | 无实质改善 | -| `+ NCCL_IB_AR_THRESHOLD=0` | `37.0843 GB/s` | 无实质改善 | -| `+ NCCL_IB_QPS_PER_CONNECTION=4 NCCL_IB_SPLIT_DATA_ON_QPS=0` | `35.9847 GB/s` | 变差 | -| `+ NCCL_IB_QPS_PER_CONNECTION=4 NCCL_IB_SPLIT_DATA_ON_QPS=1` | `29.8406 GB/s` | 明显变差 | -| `+ NCCL_IB_QPS_PER_CONNECTION=8 NCCL_IB_SPLIT_DATA_ON_QPS=1` | `24.1183 GB/s` | 明显变差 | -| `+ NCCL_NCHANNELS_PER_NET_PEER=8` | `29.8904 GB/s` | 明显变差 | - -长测复核： - -| Case | Avg Bus BW | 结论 | -|------|------------|------| -| `NCCL_PXN_DISABLE=1` | `32.7280 GB/s` | 当前窗口基线下滑 | -| `+ NCCL_P2P_NET_CHUNKSIZE=4194304` | `31.9340 GB/s` | 未复现短测提升 | -| `+ NCCL_NVLS_ENABLE=0 NCCL_P2P_NET_CHUNKSIZE=4194304` | `27.6585 GB/s` | 明显变差 | - -补充 ENV/INIT/NET 日志确认，性能波动时仍是 NCCL `2.27.7+cuda12.4`、4 条 400G HCA、GDR enabled、internal IB plugin；不是退回旧 NCCL、HCA 选择错误或 GDR 失效。 - -## NCCL GRAPH/TUNING 对照 - -为避免只看带宽结果，补抓了 allreduce 与 PXN disabled alltoall 的 `NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING,COLL` 日志。该日志采样使用短迭代，只用于看 NCCL 图和通道选择，不作为性能结论。 - -共同点： - -| 观察项 | allreduce | alltoall + `NCCL_PXN_DISABLE=1` | -|--------|-----------|----------------------------------| -| NCCL version | `2.27.7+cuda12.4` | `2.27.7+cuda12.4` | -| HCA | `mlx5_0,mlx5_1,mlx5_6,mlx5_7` | `mlx5_0,mlx5_1,mlx5_6,mlx5_7` | -| GDR | enabled | enabled | -| external net plugin | missing, internal IB | missing, internal IB | -| channels | `16 coll / 16 nvls / 16 p2p` | `16 coll / 16 nvls / 16 p2p` | -| p2p channels per peer | `2` | `2` | -| P2P chunk | `131072` | `131072` | - -差异： - -| 观察项 | allreduce | alltoall + `NCCL_PXN_DISABLE=1` | -|--------|-----------|----------------------------------| -| Pattern 4 | `crossNic 0`, `type NVL/PXN`, `nChannels 8` | `crossNic 2`, `type NVL/PIX`, `nChannels 8` | -| `NET/IB/*/GDRDMA` channel edge lines | `256` | `512` | -| `P2P/CUMEM` channel edge lines | `0` | `224` | -| total NET/P2P channel edge lines | `256` | `736` | - -判断：PXN disabled 后 4 条 IB/GDRDMA rail 都仍被使用，且通道数没有少；但 alltoall 的 NCCL graph 明显更复杂，并混入大量本机 `P2P/CUMEM` 路径。这个结果进一步支持：剩余差距不是 HCA/GDR 基础环境没有生效，而是 alltoall collective graph、P2P/NET 组合方式、internal IB plugin 能力或交换网络策略的问题。 - -## PXN disabled 端口计数器 - -`NCCL_PXN_DISABLE=1` 后，8 卡 alltoall 输出： - -| Metric | Value | -|--------|-------| -| `algbw` | `39.37 / 39.46 GB/s` | -| `busbw` | `36.91 / 37.00 GB/s` | -| `Avg bus bandwidth` | `36.9518 GB/s` | - -端口计数器： - -| Host | HCA | Xmit GB | Recv GB | Xmit GB/s | Recv GB/s | -|------|-----|---------|---------|-----------|-----------| -| 172.72.8.12 | `mlx5_0` | `590.98` | `590.91` | `19.82` | `19.82` | -| 172.72.8.12 | `mlx5_1` | `590.98` | `590.98` | `19.82` | `19.82` | -| 172.72.8.12 | `mlx5_6` | `590.98` | `590.90` | `19.82` | `19.82` | -| 172.72.8.12 | `mlx5_7` | `590.98` | `590.98` | `19.82` | `19.82` | -| 172.72.8.16 | `mlx5_0` | `590.94` | `590.98` | `19.82` | `19.82` | -| 172.72.8.16 | `mlx5_1` | `590.94` | `590.98` | `19.82` | `19.82` | -| 172.72.8.16 | `mlx5_6` | `590.94` | `590.98` | `19.82` | `19.82` | -| 172.72.8.16 | `mlx5_7` | `590.94` | `590.98` | `19.82` | `19.82` | - -对比 baseline： - -| Case | Rail 分布 | Avg Bus BW | -|------|-----------|------------| -| baseline | `mlx5_0/6` 约 `885 GB`，`mlx5_1/7` 约 `295 GB` | `30.04 GB/s` | -| `NCCL_PXN_DISABLE=1` | 四条 HCA 均约 `591 GB` | `36.95 GB/s` | - -### 错误/等待 counter 复测 - -PXN disabled 复测结果： - -| 观察项 | 结果 | -|--------|------| -| `Avg bus bandwidth` | `36.4512 GB/s` | -| 每条 HCA 流量 | 约 `712.18-712.28 GiB`，四条 rail 均衡 | -| discard / rcv error / symbol error / link down / link recovery | `0` 增量 | -| RoCE retrans / slow restart / packet sequence error / out of sequence | `0` 增量 | -| `port_xmit_wait` | `mlx5_1`、`mlx5_7` 有增长，约 `15.65M-23.49M` | - -allreduce 对照： - -| 观察项 | 结果 | -|--------|------| -| `Avg bus bandwidth` | `354.366 GB/s` | -| 每条 HCA 流量 | 约 `178.03-178.07 GiB`，四条 rail 均衡 | -| 错误/重传类 counter | `0` 增量 | -| `port_xmit_wait` | `mlx5_1`、`mlx5_7` 有增长，约 `6.11M-6.59M` | - -## 正式配置更新 - -`configs/multinode_nccl_nccl227_pdf_matrix.yaml` 已对 2 nodes x 8 GPUs 的 alltoall 增加： - -```yaml -op_env: - alltoall: - NCCL_PXN_DISABLE: 1 -``` - -正式矩阵报告：`reports_multinode_nccl_pdf_matrix_nccl227.md` - -| Topology | alltoall Peak Bus BW | alltoall Avg Bus BW | PDF Reference | Status | -|----------|----------------------|---------------------|---------------|--------| -| 2 nodes x 8 GPUs | `36.70 GB/s` | `36.74 GB/s` | `76.54 GB/s` | FAIL | - -## 判断 - -1. PXN 在当前拓扑下对 8 卡 alltoall 有负面影响，禁用后有约 `22-24%` 提升。 -2. 禁用 PXN 可以修复 rail 分布不均衡，但无法打满每条 400G rail。 -3. PXN disabled 基线上继续叠加 NVLS、P2P chunk、buffer、channel、QP/split、AR 等参数，没有稳定收益；QP/split 和 `NCCL_NCHANNELS_PER_NET_PEER=8` 反而明显变差。 -4. 禁用 PXN 后仍只有 PDF 目标的一半左右，剩余差距不是单一 NCCL 环境变量可以补齐。 -5. 后续重点仍应放在 NCCL net plugin/SHARP、交换网络策略和 NCCL internal alltoall 实现效率；`port_xmit_wait` 需要结合 allreduce 对照解读，不能单独作为 alltoall 根因。 diff --git a/reports_multinode_nccl_artifact_signal_analysis_20260523.md b/reports_multinode_nccl_artifact_signal_analysis_20260523.md deleted file mode 100644 index 1d8bc64..0000000 --- a/reports_multinode_nccl_artifact_signal_analysis_20260523.md +++ /dev/null @@ -1,141 +0,0 @@ -# 多机多卡 NCCL Artifacts 信号分析 2026-05-23 - -## 分析对象 - -- 本地 artifacts 解包目录：`/private/tmp/nccl_artifacts_113803/multinode_nccl_pdf_matrix_20260523_113803_artifacts` -- 远端原始报告：`/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803.md` -- 远端 artifacts：`/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts` -- 远端 artifacts tar：`/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts.tar.gz` -- 本地 manifest：`reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md` - -这份文档只看最新正式 PDF matrix 复跑产生的原始 `cmd/stdout/stderr/json`，目的是回答：当前多机多卡 NCCL 是否真的走了 IB/GDRDMA，是否用到了正确 HCA，是否有 SHARP/外部 NCCL net plugin 信号，以及 2x8 失败更像卡在哪一层。 - -## 一句话结论 - -最新 artifacts 证明本轮多机多卡测试不是 launch 失败、不是回退 TCP、不是 GDRDMA 没开，也不是 HCA 名字选错；所有 case 都走 `IB`，都识别并启用了 `mlx5_0,mlx5_1,mlx5_6,mlx5_7` 这 4 条 400G rail，NCCL 正确性 `wrong=0`。当前主要缺口仍然是：环境没有外部 NCCL net plugin / SHARP 证据，且 2x8 档位的 PDF 阈值明显高于当前 4 rail 环境可解释能力，alltoall 还存在独立的跨 Leaf 多点通信效率问题。 - -## Artifacts 信号表 - -| Case | Peak | Threshold | Status | Plugin missing | NET/IB using | Using network IB | HCA set | GDR HCA set | GDRDMA edges | P2P/CUMEM | SHARP/CollNet | stdout KB | -|---|---:|---:|---|---:|---:|---:|---|---|---:|---:|---:|---:| -| allreduce_2x1 1_GPU | 47.29 | 48.90 | FAIL | 2 | 2 | 2 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | 16 | 0 | 0 | 24 | -| allreduce_2x2 2_GPUs | 137.16 | 136.93 | PASS | 4 | 4 | 4 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | 32 | 32 | 0 | 68 | -| allreduce_2x4 4_GPUs | 335.07 | 335.48 | FAIL | 8 | 8 | 8 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | 256 | 0 | 0 | 259 | -| allreduce_2x8 8_GPUs | 353.85 | 491.84 | FAIL | 16 | 16 | 16 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | 256 | 0 | 0 | 410 | -| alltoall_2x1 1_GPU | 24.85 | 27.25 | FAIL | 2 | 2 | 2 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | 8 | 0 | 0 | 19 | -| alltoall_2x2 2_GPUs | 47.76 | 54.41 | FAIL | 4 | 4 | 4 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | 24 | 8 | 0 | 52 | -| alltoall_2x4 4_GPUs | 72.74 | 73.73 | FAIL | 8 | 8 | 8 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | 80 | 48 | 0 | 200 | -| alltoall_2x8 8_GPUs | 36.83 | 76.54 | FAIL | 16 | 16 | 16 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | mlx5_0,mlx5_1,mlx5_6,mlx5_7 | 512 | 224 | 0 | 603 | - -字段解释： - -- `Plugin missing`：日志里的 `NET/Plugin: Could not find: none libnccl-net-none.so.` 次数。当前命令显式设置了 `NCCL_NET_PLUGIN=none`，所以这个信号表示没有使用外部 NCCL net plugin，而不是 NCCL 没有网络。 -- `NET/IB using`：日志里的 `NET/IB : Using ...` 次数，说明每个 rank 初始化时看到的 IB HCA 列表。 -- `Using network IB`：NCCL 最终选择了 `IB` 网络。 -- `GDR HCA set`：出现 `GPU Direct RDMA Enabled for HCA ...` 的 HCA 集合。 -- `GDRDMA edges`：NCCL graph/connection 中经由 `NET/IB/*/GDRDMA` 的跨节点边数量。 -- `P2P/CUMEM`：节点内 GPU 间路径信号，不是跨节点 IB。 -- `SHARP/CollNet`：日志中 `SHARP`、`CollNet`、`HCOLL` 相关信号计数。当前为 0。 - -## 已排除的问题 - -### 1. 不是 TCP 回退 - -所有 8 个 case 都有 `Using network IB`，且每个 rank 均有 `NET/IB : Using ...`。这说明 NCCL 通信路径不是 socket/TCP 回退。 - -### 2. 不是 HCA 名字选错 - -所有 case 的 HCA 集合都一致： - -```text -mlx5_0, mlx5_1, mlx5_6, mlx5_7 -``` - -这与当前配置里的 `NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_6,mlx5_7` 一致，也与前面环境快照中确认的 4 条 400G IB rail 一致。 - -### 3. 不是 GDRDMA 没开 - -所有 case 都出现 `GPU Direct RDMA Enabled for HCA ...`，并且跨节点连接里有 `NET/IB/*/GDRDMA` 边。2x8 alltoall 甚至有 512 条 `GDRDMA/Shared` 边，所以不能简单判断为 GDRDMA 被关掉。 - -### 4. 不是 NCCL 正确性失败 - -最新 manifest 中 8 个 case 全部： - -```text -returncode = 0 -wrong_count = 0 -``` - -因此当前 FAIL 是严格 PDF 性能阈值失败，不是结果错误。 - -## 仍然成立的缺口 - -### 1. 外部 NCCL net plugin / SHARP 仍缺证据 - -当前命令中显式设置： - -```text -NCCL_NET_PLUGIN=none -``` - -所有 case 均出现 `NET/Plugin: Could not find: none libnccl-net-none.so.`，同时 `SHARP/CollNet` 信号计数为 0。结合前面的环境检查没有找到 `libnccl-net*.so*` / `libsharp*.so*`，当前环境不能证明与 PDF 参考环境的软件栈等价。 - -### 2. 2x8 allreduce 更像被 4 rail 物理能力卡住 - -2x8 allreduce： - -```text -当前 busbw = 353.85 GB/s -PDF 阈值 = 491.84 GB/s -``` - -16 rank allreduce 的换算关系是： - -```text -busbw = algbw * 1.875 -``` - -当前实测反推： - -```text -353.85 / 1.875 = 188.72 GB/s algbw -``` - -当前每节点 4 条 400G rail 的理论单向原始带宽约： - -```text -4 * 400 Gb/s / 8 = 200 GB/s -``` - -所以 allreduce 已经接近 4 rail 的可解释上限；如果 PDF 阈值来自更多 400G rail 或带 SHARP/plugin 的环境，当前节点不应直接按该阈值判死。 - -### 3. 2x8 alltoall 是独立重点问题 - -2x8 alltoall： - -```text -当前 busbw = 36.83 GB/s -PDF 阈值 = 76.54 GB/s -``` - -alltoall 和 allreduce 使用同一组 HCA，同样走 IB/GDRDMA，但 2x8 alltoall 下降明显。这个现象更像多点到多点流量在当前跨 Leaf 网络、ECMP/adaptive routing、拥塞控制或 NCCL graph 策略下效率不够，而不是单纯 HCA 没起来。 - -## 下一步建议 - -1. 先不要继续盲扫 NCCL 小参数。已有 artifacts 说明基础链路已经起来，继续微调环境变量的收益大概率很低。 -2. 向硬件/网络侧确认 PDF 参考环境每节点是否有 8 条 400G rail，以及是否启用了 SHARP、HCOLL 或外部 NCCL net plugin。 -3. 如果验收坚持 PDF 原阈值，应先补齐 plugin/SHARP 或换等价 8 rail 节点复测。 -4. 如果当前硬件形态就是 4 条 400G rail，则 allreduce 阈值应重新定标；alltoall 单独作为跨 Leaf 多点通信效率问题继续排查。 -5. 补齐 plugin/SHARP 后，优先复跑： - -```bash -cd /root/test_gpu_scripts -bash scripts/run_multinode_nccl_pdf_matrix.sh -``` - -并对比新旧 artifacts 中： - -- `Plugin missing` 是否消失。 -- 是否出现外部 net plugin、SHARP 或 CollNet 信号。 -- 2x8 allreduce 是否突破当前 `353-354 GB/s` 平台。 -- 2x8 alltoall 是否突破当前 `36-37 GB/s` 平台。 diff --git a/reports_multinode_nccl_counter_probe_20260523.md b/reports_multinode_nccl_counter_probe_20260523.md deleted file mode 100644 index 9e42251..0000000 --- a/reports_multinode_nccl_counter_probe_20260523.md +++ /dev/null @@ -1,209 +0,0 @@ -# 多机 NCCL 8 卡链路计数器探测 - -- 日期：2026-05-23 -- 主机：`aikubeworker0012` / `172.72.8.12`，`aikubeworker0016` / `172.72.8.16` -- NCCL：临时 `2.27.7+cuda12.4` -- HCA：`mlx5_0,mlx5_1,mlx5_6,mlx5_7` -- HCA 速率：每节点 4 x 400Gb/s NDR，理论单向合计约 `200 GB/s` - -## 结论 - -8 卡 allreduce 的 NCCL `algbw` 已经到 `189 GB/s` 左右，接近当前每节点 4 条 400G rail 的理论单向合计 `200 GB/s`。因此 PDF 参考的 `491.84 GB/s busbw` 对应 `262 GB/s algbw`，在当前 4 x 400G rail 形态下不太可能达到，除非实际可用跨节点 rail 数量或网络能力高于当前节点暴露的 4 条 400G。 - -裸 RDMA 并发 perftest 也验证了这 4 条 400G rail 本身可以同时工作：4 个 HCA 并发 `ib_write_bw` 合计 `1476.95 Gb/s`，即 `184.62 GB/s`。这与 NCCL 8 卡 allreduce 换算出的 `189 GB/s algbw` 一致，说明 allreduce 已经接近裸网络可用带宽。 - -8 卡 alltoall 仍只有 `30 GB/s busbw`，不是 HCA 顺序导致。HCA 顺序 sweep 都稳定在 `30.02-30.07 GB/s`。计数器显示 alltoall 流量主要压在 `mlx5_0` 和 `mlx5_6` 上，`mlx5_1` 和 `mlx5_7` 只有约三分之一流量，说明剩余问题更像 NCCL alltoall rail 分布、路由、拥塞、NCCL net plugin/SHARP 或网络侧策略问题。 - -补充测试显示，`NCCL_PXN_DISABLE=1` 可以把 alltoall 流量均匀分配到四条 HCA，并将 busbw 提升到约 `36.5-37.0 GB/s`。不过每条 400G rail 仍只有约 `19-20 GB/s`，没有达到裸 RDMA 单 rail 能力。 - -进一步抓 `counters`/`hw_counters` 后，未看到 discard、CRC/符号错误、packet sequence error、RoCE retrans、slow restart 等错误类计数增长；只看到部分端口 `port_xmit_wait` 增长。对照 allreduce 后发现，allreduce 在 `354 GB/s busbw` 时也会出现同类 `port_xmit_wait`，因此 `port_xmit_wait` 不是 alltoall 低吞吐的充分解释，只能说明发送侧存在等待。剩余问题更像 NCCL internal alltoall 通信模式、交换网络调度/拥塞控制、或缺少 NCCL net plugin/SHARP 能力。 - -## 裸 RDMA 4 rail 并发 - -命令类型： - -```bash -ib_write_bw -d -i 1 -p -s 4194304 -n 5000 -F --report_gbits -``` - -结果： - -| HCA | BW average | -|-----|------------| -| `mlx5_0` | `387.16 Gb/s` | -| `mlx5_1` | `387.07 Gb/s` | -| `mlx5_6` | `355.02 Gb/s` | -| `mlx5_7` | `347.70 Gb/s` | -| Total | `1476.95 Gb/s` / `184.62 GB/s` | - -## 8 卡 allreduce - -NCCL 输出： - -| Metric | Value | -|--------|-------| -| `algbw` | `189.16 / 189.07 GB/s` | -| `busbw` | `354.68 / 354.52 GB/s` | -| `Avg bus bandwidth` | `354.597 GB/s` | - -allreduce busbw 换算关系约为： - -```text -busbw = algbw * 2 * (nranks - 1) / nranks - = algbw * 1.875 # nranks=16 -``` - -因此： - -| 项 | busbw | 换算 algbw | -|----|-------|------------| -| 当前测试 | `354.60 GB/s` | `189.12 GB/s` | -| PDF 参考 | `491.84 GB/s` | `262.31 GB/s` | - -当前 `189.12 GB/s algbw` 已接近 `4 x 400Gb/s = 200 GB/s` 理论单向总带宽。 - -### allreduce counter 对照 - -对同样 2 nodes x 8 GPUs、同样 4 条 HCA 的 16G allreduce 复测 counter： - -| Metric | Value | -|--------|-------| -| `algbw` | `189.22 / 188.77 GB/s` | -| `busbw` | `354.79 / 353.94 GB/s` | -| `Avg bus bandwidth` | `354.366 GB/s` | - -流量分布： - -| Host | HCA | Xmit GiB | Recv GiB | -|------|-----|----------|----------| -| aikubeworker0012 | `mlx5_0` | `178.07` | `178.03` | -| aikubeworker0012 | `mlx5_1` | `178.07` | `178.07` | -| aikubeworker0012 | `mlx5_6` | `178.07` | `178.03` | -| aikubeworker0012 | `mlx5_7` | `178.07` | `178.07` | -| aikubeworker0016 | `mlx5_0` | `178.03` | `178.07` | -| aikubeworker0016 | `mlx5_1` | `178.07` | `178.07` | -| aikubeworker0016 | `mlx5_6` | `178.03` | `178.07` | -| aikubeworker0016 | `mlx5_7` | `178.07` | `178.07` | - -错误类 counter 增量同样为 `0`，非零等待类 counter 为： - -| Host | HCA | `port_xmit_wait` delta | -|------|-----|------------------------| -| aikubeworker0012 | `mlx5_1` | `6,555,518` | -| aikubeworker0012 | `mlx5_7` | `6,325,059` | -| aikubeworker0016 | `mlx5_1` | `6,585,965` | -| aikubeworker0016 | `mlx5_7` | `6,112,874` | - -判断：allreduce 在达到当前 4 x 400G rail 物理上限附近时也会出现 `port_xmit_wait`，所以这个 counter 不能单独解释 alltoall 只有 `36-37 GB/s`。alltoall 的问题更偏向通信模式效率或网络调度策略，而不是简单链路错误。 - -## 8 卡 alltoall - -NCCL 输出： - -| Metric | Value | -|--------|-------| -| `algbw` | `32.04 / 32.05 GB/s` | -| `busbw` | `30.03 / 30.04 GB/s` | -| `Avg bus bandwidth` | `30.0389 GB/s` | - -同一测试窗口内，端口计数器增量显示流量不均衡： - -| Host | HCA | Xmit GB | Recv GB | -|------|-----|---------|---------| -| 172.72.8.12 | `mlx5_0` | `885.54` | `885.51` | -| 172.72.8.12 | `mlx5_1` | `295.19` | `295.19` | -| 172.72.8.12 | `mlx5_6` | `885.53` | `885.51` | -| 172.72.8.12 | `mlx5_7` | `295.19` | `295.19` | -| 172.72.8.16 | `mlx5_0` | `885.51` | `885.54` | -| 172.72.8.16 | `mlx5_1` | `295.19` | `295.19` | -| 172.72.8.16 | `mlx5_6` | `885.51` | `885.53` | -| 172.72.8.16 | `mlx5_7` | `295.19` | `295.19` | - -## HCA 顺序 sweep - -8 卡 alltoall 对 HCA 顺序不敏感： - -| `NCCL_IB_HCA` | Avg Bus BW | -|---------------|------------| -| `mlx5_0,mlx5_1,mlx5_6,mlx5_7` | `30.0367 GB/s` | -| `mlx5_0,mlx5_6,mlx5_1,mlx5_7` | `30.0696 GB/s` | -| `mlx5_0,mlx5_7,mlx5_1,mlx5_6` | `30.0397 GB/s` | -| `mlx5_1,mlx5_0,mlx5_7,mlx5_6` | `30.0413 GB/s` | -| `mlx5_6,mlx5_7,mlx5_0,mlx5_1` | `30.0230 GB/s` | - -## PXN disabled alltoall 计数器 - -`NCCL_PXN_DISABLE=1` 后： - -| Metric | Value | -|--------|-------| -| `Avg bus bandwidth` | `36.9518 GB/s` | -| 每条 HCA 流量 | 约 `590.94-590.98 GB` | -| 每条 HCA 吞吐 | 约 `19.82 GB/s` | -| 每节点 4 HCA 合计吞吐 | 约 `79.29 GB/s` | - -判断：禁用 PXN 可以修复 rail 分布不均衡，但不能让 alltoall 打满当前 4 条 400G rail。 - -### PXN disabled 错误/拥塞 counter 复测 - -复测命令仍为 2 nodes x 8 GPUs，`alltoall_perf -b 16G -e 16G -w 10 -n 10`，并使用： - -```bash -NCCL_PXN_DISABLE=1 -NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_6,mlx5_7 -NCCL_NET_PLUGIN=none -NCCL_NET_GDR_LEVEL=5 -NCCL_NET_GDR_READ=1 -NCCL_DMABUF_ENABLE=0 -``` - -NCCL 输出： - -| Metric | Value | -|--------|-------| -| `algbw` | `39.04 / 38.72 GB/s` | -| `busbw` | `36.60 / 36.30 GB/s` | -| `Avg bus bandwidth` | `36.4512 GB/s` | - -流量分布保持均衡： - -| Host | HCA | Xmit GiB | Recv GiB | -|------|-----|----------|----------| -| aikubeworker0012 | `mlx5_0` | `712.28` | `712.19` | -| aikubeworker0012 | `mlx5_1` | `712.27` | `712.27` | -| aikubeworker0012 | `mlx5_6` | `712.28` | `712.18` | -| aikubeworker0012 | `mlx5_7` | `712.27` | `712.27` | -| aikubeworker0016 | `mlx5_0` | `712.23` | `712.27` | -| aikubeworker0016 | `mlx5_1` | `712.23` | `712.27` | -| aikubeworker0016 | `mlx5_6` | `712.23` | `712.27` | -| aikubeworker0016 | `mlx5_7` | `712.23` | `712.27` | - -错误类 counter 增量： - -| Counter group | Result | -|---------------|--------| -| `port_xmit_discards`, `port_rcv_errors`, `port_rcv_remote_physical_errors`, `port_rcv_switch_relay_errors` | `0` | -| `symbol_error`, `link_error_recovery`, `link_downed`, `local_link_integrity_errors`, `excessive_buffer_overrun_errors` | `0` | -| `roce_adp_retrans`, `roce_adp_retrans_to`, `roce_slow_restart*` | `0` | -| `packet_seq_err`, `out_of_sequence`, `out_of_buffer`, `duplicate_request`, `implied_nak_seq_err` | `0` | -| `local_ack_timeout_err`, `req_transport_retries_exceeded`, `rnr_nak_retry_err` | `0` | - -非零等待类 counter： - -| Host | HCA | `port_xmit_wait` delta | -|------|-----|------------------------| -| aikubeworker0012 | `mlx5_1` | `23,492,853` | -| aikubeworker0012 | `mlx5_7` | `17,420,720` | -| aikubeworker0016 | `mlx5_1` | `20,428,901` | -| aikubeworker0016 | `mlx5_7` | `15,650,027` | - -判断：PXN disabled 后 alltoall 没有明显链路错误、重传或丢包证据。结合 allreduce 对照，`port_xmit_wait` 只能作为发送等待信号，不能单独解释 alltoall 低吞吐；剩余性能缺口更偏向 NCCL internal alltoall 在当前拓扑下的通信模式效率、交换网络调度/拥塞控制，或外部 NCCL net plugin/SHARP 缺失。 - -## 判断 - -1. 裸 RDMA 4 rail 可以并发跑到约 `184.62 GB/s`，网络基础带宽不是单 rail 瓶颈。 -2. 8 卡 allreduce 当前不是软件参数小调能解决的问题，性能已经贴近当前 4 条 400G rail 的物理带宽上限。 -3. 8 卡 alltoall 仍明显异常，且不是 HCA 顺序问题；PXN disabled 后 rail 已均衡，`port_xmit_wait` 不是 alltoall 独有，需要继续从 NCCL alltoall 模式、交换机侧策略、NCCL net plugin/SHARP 排查。 -4. `NCCL_PXN_DISABLE=1` 可改善 8 卡 alltoall 的 rail 均衡性和性能，但无法补齐到 PDF 目标。 -5. 如果验收必须达到 PDF 的 2 机 16 卡 `491.84/76.54 GB/s`，需要确认当前两台机器是否具备与 PDF 参考环境同等的有效跨节点 rail 数量和交换网络能力。 -6. 两台机器当前均未发现 `libnccl-net.so` 或 SHARP/HCOLL 包，NCCL 使用 internal IB plugin；如果目标值依赖 NCCL net plugin/SHARP，需要先补齐对应运行环境。 diff --git a/reports_multinode_nccl_deep_diagnose_run_20260523.md b/reports_multinode_nccl_deep_diagnose_run_20260523.md deleted file mode 100644 index a96c20d..0000000 --- a/reports_multinode_nccl_deep_diagnose_run_20260523.md +++ /dev/null @@ -1,125 +0,0 @@ -# 多节点 NCCL 深度诊断复跑报告 2026-05-23 - -## 执行信息 - -- 发起节点：`aikubeworker0012` -- 对端节点：`aikubeworker0016` -- 测试规模：2 节点 x 8 GPU -- NCCL：`2.27.7+cuda12.4` -- nccl-tests：`/data/nccl-tests-latest/build` -- OpenMPI：`/usr/mpi/gcc/openmpi-4.1.9a1/bin/mpirun` -- 远端产物目录：`/root/test_gpu_scripts/reports/nccl_deep_diag_20260523_103932` -- 诊断脚本：`scripts/multinode_nccl_deep_diagnose.sh all` - -## Preflight - -两台机器均通过轻量环境检查： - -| 项目 | aikubeworker0012 | aikubeworker0016 | -|---|---:|---:| -| OpenMPI | `4.1.9a1` | `4.1.9a1` | -| `all_reduce_perf` | OK | OK | -| `alltoall_perf` | OK | OK | -| `mlx5_0` | 400 Gb/sec ACTIVE | 400 Gb/sec ACTIVE | -| `mlx5_1` | 400 Gb/sec ACTIVE | 400 Gb/sec ACTIVE | -| `mlx5_6` | 400 Gb/sec ACTIVE | 400 Gb/sec ACTIVE | -| `mlx5_7` | 400 Gb/sec ACTIVE | 400 Gb/sec ACTIVE | - -## 16G 核心结果 - -| 测试 | 配置 | Avg Bus BW | 结论 | -|---|---|---:|---| -| allreduce | 自动参数 | `354.025 GB/s` | 稳定复现当前高位基线 | -| alltoall | `NCCL_PXN_DISABLE=1` | `36.9377 GB/s` | 稳定复现当前瓶颈基线 | -| graph allreduce | `NCCL_DEBUG=INFO` | `354.224 GB/s` | 与 counter run 一致 | -| graph alltoall | `NCCL_PXN_DISABLE=1`, `NCCL_DEBUG=INFO` | `37.14 GB/s` | 与 counter run 一致 | - -对 PDF 目标的含义： - -- 2x8 allreduce 仍明显低于 PDF 2 机 16 GPU 目标 `491.84 GB/s`。 -- 2x8 alltoall 仍明显低于 PDF 2 机 16 GPU 目标 `76.54 GB/s`。 -- 本轮没有发现能把 8 卡 alltoall 推出 `36-37 GB/s` 平台的参数。 - -## Counter 观察 - -### Rail 流量 - -allreduce 每条 rail 发送流量约 `178.03-178.07 GiB`，alltoall + PXN disabled 每条 rail 发送流量约 `712.23-712.28 GiB`。四条 400G rail 在两类测试中都均衡。 - -### 错误/拥塞类计数 - -本轮未看到 discard、symbol error、RoCE retrans、slow restart、packet sequence error 等硬错误增长。 - -有增长的是 `port_xmit_wait`： - -| 测试 | 计数增长 | -|---|---| -| allreduce | `aikubeworker0016 mlx5_1 +6725565`, `mlx5_7 +6103180` | -| alltoall + PXN disabled | `aikubeworker0016 mlx5_1 +20988680`, `mlx5_7 +16271960` | - -这说明 `port_xmit_wait` 不是 alltoall 独有现象；高吞吐 allreduce 也会出现。它可以作为交换网络/credit 等待的信号继续给网络侧看，但不能单独解释 alltoall 低带宽。 - -## GRAPH/TUNING 对照 - -| 观察项 | allreduce | alltoall + `NCCL_PXN_DISABLE=1` | -|---|---:|---:| -| `avg_busbw` | `354.224` | `37.14` | -| `plugin_missing` | `16` | `16` | -| GDR enabled lines | `1344` | `704` | -| channel summary | `16 coll / 16 nvls / 16 p2p` | `16 coll / 16 nvls / 16 p2p` | -| Pattern 4 | `crossNic 0`, `NVL/PXN` | `crossNic 2`, `NVL/PIX` | -| `NET/IB/*/GDRDMA` lines | `256` | `512` | -| `P2P/CUMEM` lines | `0` | `224` | -| total NET/P2P edge lines | `256` | `736` | - -解释： - -- HCA、GDR、NCCL 版本和基础 channel 数量不是差异根因。 -- alltoall 的通信图明显更复杂，引入更多 NET/P2P 边，且 Pattern 4 从 allreduce 的 `NVL/PXN` 变成 `NVL/PIX`。 -- 这继续支持问题偏向 NCCL alltoall 图策略、internal IB plugin、缺少外部 `libnccl-net.so`/SHARP，或交换网络策略，而不是单纯链路坏、HCA 不通、GDR 没开。 - -## PXN Disabled Sweep - -基线均为 `NCCL_PXN_DISABLE=1`，16G，2x8 GPU。 - -| Case | 额外参数 | Avg Bus BW | -|---|---|---:| -| baseline | 无 | `36.8024` | -| nvls_off | `NCCL_NVLS_ENABLE=0` | `36.8095` | -| qps4_split1 | `NCCL_IB_QPS_PER_CONNECTION=4 NCCL_IB_SPLIT_DATA_ON_QPS=1` | `30.5464` | -| qps8_split1 | `NCCL_IB_QPS_PER_CONNECTION=8 NCCL_IB_SPLIT_DATA_ON_QPS=1` | `23.9345` | -| qps4_split0 | `NCCL_IB_QPS_PER_CONNECTION=4 NCCL_IB_SPLIT_DATA_ON_QPS=0` | `35.8679` | -| channels16 | `NCCL_MIN_NCHANNELS=16 NCCL_MAX_NCHANNELS=16` | `37.1776` | -| buff8m | `NCCL_BUFFSIZE=8388608` | `37.0265` | -| p2pchunk4m | `NCCL_P2P_NET_CHUNKSIZE=4194304` | `37.0188` | -| netpeer8 | `NCCL_NCHANNELS_PER_NET_PEER=8` | `31.103` | -| ar0 | `NCCL_IB_AR_THRESHOLD=0` | `36.9965` | - -结论： - -- `channels16`、`buff8m`、`p2pchunk4m`、`ar0` 只有 0.2-1.0% 左右波动，不能视为有效优化。 -- `qps4_split1`、`qps8_split1`、`netpeer8` 明显负向。 -- 当前 8 卡 alltoall 不建议套用 PDF 固定 QP/split 参数。 - -## 脚本修正验证 - -复跑后发现脚本在 GRAPH 模式后会把 `NCCL_DEBUG=INFO` 继承到 sweep，导致 sweep 日志过大；同时 OpenMPI 会对未设置的 `-x` 变量打印 warning。 - -已修正： - -- `set_common_env` 每个 case 重置到默认 `NCCL_DEBUG=WARN`。 -- `mpi_xargs` 只导出已经设置的环境变量。 - -验证方式： - -- 本地 `bash -n scripts/multinode_nccl_deep_diagnose.sh` 通过。 -- 远端 1M tiny `all` 冒烟测试通过。 -- tiny 产物中 `could not find environment variable` 计数为 `0`。 - -## 当前判断 - -1. allreduce 的高位基线稳定，2x8 仍在 `354 GB/s` 左右。 -2. alltoall 即使 PXN disabled 并且 rail 均衡，也只能稳定在 `36-37 GB/s`。 -3. 未发现明显坏链路、重传、丢包、HCA 不通或 GDR disabled。 -4. 当前 4 条 400G rail 的硬件形态与 PDF 目标疑似不等价；PDF 2x8 allreduce 目标 `491.84 GB/s` 反推需要超过当前 4 rail 单向理论上限。 -5. alltoall 还需要从 NCCL net plugin/SHARP、交换机路径/ECMP/拥塞控制、以及 NCCL alltoall 图策略侧继续排。 diff --git a/reports_multinode_nccl_diagnosis_20260523.md b/reports_multinode_nccl_diagnosis_20260523.md deleted file mode 100644 index 6e769b5..0000000 --- a/reports_multinode_nccl_diagnosis_20260523.md +++ /dev/null @@ -1,500 +0,0 @@ -# 多机多卡 NCCL 诊断报告 - -- 日期：2026-05-23 -- 测试入口：`nccl-gpu-1` / `aikubeworker0012` / `172.72.8.12` -- 对端节点：`nccl-gpu-2` / `aikubeworker0016` / `172.72.8.16` -- 诊断配置：`configs/multinode_nccl_nccl227_auto_16g.yaml` -- 当前最佳原始脚本报告：`reports_multinode_nccl_16g_2x8_nccl227_auto.md` - -## 当前结论 - -这不是单纯 “IB 不通” 的问题。底层 CUDA RDMA perftest 可以跑到接近单端口 400Gb/s 的水平；最初使用 pip 包里的 NCCL 2.21.5 时，NCCL 在实际 2 节点通信中把 GPU Direct RDMA 禁用了，导致带宽显著偏低。 - -后续临时切换到 apt 包解压出的 NCCL 2.27.7+cuda12.4 后，NCCL GDR 已经恢复启用，2 节点 x 8 GPU allreduce 从 `67.42 GB/s` 提升到 `237.86 GB/s`，alltoall 从 `9.56 GB/s` 提升到 `28.62 GB/s`。 - -继续 tuning 后发现，配置里固定的 `NCCL_MIN_NCHANNELS=4`、`NCCL_IB_QPS_PER_CONNECTION=4`、`NCCL_IB_SPLIT_DATA_ON_QPS=1` 会明显压低 16G allreduce。去掉这些固定参数、让 NCCL 2.27.7 自动选择后，正式脚本报告中 2 节点 x 8 GPU allreduce 提升到 `354.60 GB/s`，alltoall 小幅提升到 `30.01 GB/s`。当前剩余问题不再是 GDR disabled，而是 GDR enabled 且 NCCL 自动调参后，仍低于当前配置里的验收阈值。 - -按 `sx算力节点跨Leaf NCCL测试报告.pdf` 的矩阵继续对齐后，发现 2 机 4 卡档位的核心问题是默认 GPU 选择不符合 GPU-NIC 亲和性。显式选择 `CUDA_VISIBLE_DEVICES=0,1,4,5` 后，2 机 4 卡 allreduce 可以恢复到 `333-335 GB/s` 区间，接近 PDF 的 `335.48 GB/s`；alltoall 配合 PDF 固定 NCCL 参数可到 `72.93 GB/s`，接近 PDF 的 `73.73 GB/s`。但 2 机 8 卡档位仍只有 allreduce `354.02 GB/s`、alltoall `30.04 GB/s`，与 PDF 的 `491.84/76.54 GB/s` 差距明显。 - -进一步 sweep 8 卡 alltoall 网络参数后，`NCCL_PXN_DISABLE=1` 是唯一有效正向项。正式矩阵配置已对 2 机 8 GPU 的 alltoall 单独加入该变量，8 卡 alltoall 从约 `30.04 GB/s` 提升到 `36.70 GB/s` peak / `36.74 GB/s` avg，但仍低于 PDF 参考 `76.54 GB/s`。复测端口 counter 后，PXN disabled 下 4 条 rail 的流量已均衡，且没有明显链路错误、丢包、RoCE 重传或 slow restart；同类 `port_xmit_wait` 在高吞吐 allreduce 中也会出现，因此它不是 alltoall 低吞吐的充分解释。继续在 PXN disabled 基线上叠加 NVLS、P2P chunk、buffer、channel、QP/split、AR 等参数，没有稳定收益。NCCL GRAPH/TUNING 日志显示 alltoall 的 channel graph 比 allreduce 复杂很多，且混入大量本机 `P2P/CUMEM` 路径，但 HCA/GDR/channel 基础状态一致。剩余差距更像 NCCL internal alltoall 通信模式效率、交换网络策略，或缺少 NCCL net plugin/SHARP 能力。 - -同时，`nccl-gpu-2` 的 SSH 入口曾因未认证连接过多触发 `MaxStartups` 随机拒绝，导致 `mpirun` 拉起远端 rank 失败。已经做了临时 SSHD 缓解并拿到有效的 2 节点 x 8 GPU allreduce/alltoall 报告。 - -## 已完成的修正 - -1. 修正 `mpirun` 使用路径，避开系统 `/usr/bin/mpirun` 与 DOCA OpenMPI 动态库混用导致的崩溃。 -2. 补充 `LD_LIBRARY_PATH`，确保 `mpirun`、CUDA、pip 安装的 NCCL 动态库可同时解析。 -3. 将 NCCL HCA 限定到 400Gb/s 活跃端口：`mlx5_0,mlx5_1,mlx5_6,mlx5_7`。 -4. 在脚本中加入 multi-node NCCL 网络诊断解析，报告会展示 `NCCL Network`、`GPU Direct RDMA`、`GDR Disabled HCAs`。 -5. 增加 `multinode_nccl.extra_env`，可以在配置里快速试 NCCL 环境变量，不需要改代码。 -6. 增加诊断配置 `configs/multinode_nccl_diagnostic.yaml`，固定跑 2 节点 x 8 GPU、256M、`NCCL_DEBUG=INFO` 和 `NCCL_DEBUG_SUBSYS=INIT,NET`。 -7. 在 `nccl-gpu-2` 上临时提高 SSHD `MaxStartups` 并缩短 `LoginGraceTime`，缓解未认证连接过多导致的 SSH 随机拒绝。 -8. 将 OpenMPI OOB TCP 控制通道固定到 `bond0`，并加入 `plm_rsh_args`，减少 `mpirun` 远端启动受 SSH/host key/接口选择影响的概率。 -9. 从 NVIDIA apt 源下载但不安装 `libnccl2=2.27.7-1+cuda12.4`，解压到两台机器 `/tmp/nccl-2.27.7-cuda12.4`，用 `LD_LIBRARY_PATH` 临时覆盖 NCCL 运行库验证。 -10. 增强报告解析，能够区分 `GPU Direct RDMA ENABLED` 和 `DISABLED`，并列出 enabled/disabled HCA。 -11. 将 multi-node NCCL 配置中的 `qps_per_connection`、`min_nchannels`、`split_data_on_qps` 改为 `null`，避免默认导出会压低大包 allreduce 的固定 NCCL 参数。 -12. 增加 topology 级 `cuda_visible_devices`、`env`、`op_env` 配置能力，支持按 GPU/NIC 亲和性和不同 NCCL op 分别设置环境变量。 -13. 生成 PDF 矩阵式原始报告 `reports_multinode_nccl_pdf_matrix_nccl227.md`，覆盖 2 机 1/2/4/8 GPU per node。 -14. 对 8 卡 alltoall 做 NCCL 网络参数 sweep，并将有效项 `NCCL_PXN_DISABLE=1` 固化到 PDF 矩阵配置。 -15. 对 PXN disabled 后的 8 卡 alltoall 抓取 `counters`/`hw_counters` 增量，确认 rail 已均衡且无明显错误/重传。 -16. 对同样 2x8 allreduce 抓 counter 对照，确认高吞吐 allreduce 也会出现 `port_xmit_wait`，因此该 counter 不是 alltoall 低吞吐的唯一根因。 -17. 在 PXN disabled 基线上继续 sweep NVLS、P2P chunk、buffer、channel、QP/split、AR 等参数，确认没有稳定收益，部分参数明显变差。 -18. 抓取 allreduce 与 PXN disabled alltoall 的 `GRAPH/TUNING/COLL` 日志，确认两者 HCA/GDR/channel 基础状态一致，但 alltoall graph 明显更复杂。 - -## 关键证据 - -### 1. CUDA RDMA perftest 通过 - -命令类型： - -```bash -CUDA_VISIBLE_DEVICES=0 ib_write_bw -d mlx5_0 -i 1 --use_cuda=0 -s 4194304 -F --report_gbits 172.72.8.16 -``` - -结果： - -| 测试 | 设备 | GPU | 平均带宽 | 结论 | -|------|------|-----|----------|------| -| `ib_write_bw --use_cuda` | `mlx5_0` | GPU0 | `387.16 Gb/s` | PASS | - -解释：GPU 内存参与 RDMA 写带宽测试可以接近 400Gb/s，说明 `nvidia_peermem`/经典 GPUDirect RDMA 路径并非完全不可用。 - -### 2. CUDA DMA-BUF 路径不可用 - -命令类型： - -```bash -CUDA_VISIBLE_DEVICES=0 ib_write_bw -d mlx5_0 -i 1 --use_cuda=0 --use_cuda_dmabuf -s 4194304 -F --report_gbits 172.72.8.16 -``` - -结果： - -| 测试 | 输出 | 结论 | -|------|------|------| -| `ib_write_bw --use_cuda_dmabuf` | `DMA-BUF is not supported on this GPU` | FAIL | - -解释：当前环境不能走 CUDA DMA-BUF RDMA。后续 NCCL 应优先确认是否能稳定走经典 `nvidia_peermem` 路径。 - -### 3. NCCL 单卡跨节点仍禁用 GDR - -使用 pip NCCL 2.21.5 时， - -已经尝试： - -- `NCCL_NET_GDR_LEVEL=SYS` -- `NCCL_NET_GDR_LEVEL=5` -- `NCCL_NET_GDR_READ=1` -- `NCCL_DMABUF_ENABLE=0` -- `NCCL_IB_CUDA_SUPPORT=1` -- `NCCL_IB_HCA=mlx5_0` - -结果仍显示： - -```text -NCCL INFO Using network IB -NCCL INFO NET/IB : GPU Direct RDMA Disabled for HCA 0 'mlx5_0' -``` - -256M allreduce 约 `13.4 GB/s`，明显低于 400Gb/s IB 端口能力。 - -### 3.1 NCCL 2.27.7 恢复 GDR - -临时使用： - -```bash -LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi-4.1.9a1/lib:/tmp/nccl-2.27.7-cuda12.4/usr/lib/x86_64-linux-gnu:/usr/local/cuda-12.4/targets/x86_64-linux/lib -``` - -2 节点 x 1 GPU 日志显示： - -```text -NCCL version 2.27.7+cuda12.4 -NET/IB : GPU Direct RDMA Enabled for HCA 0 'mlx5_0' -Channel ... via NET/IB/0/GDRDMA -``` - -256M allreduce 从 NCCL 2.21.5 的约 `13.4 GB/s` 提升到 `45.2 GB/s`。判断：NCCL 2.21.5 与当前 driver/OFED/H100 组合存在 GDR 判定或注册路径兼容问题；升级 NCCL 是有效修复方向。 - -### 4. 脚本 2 节点 x 8 GPU 诊断结果 - -原始报告：`reports_multinode_nccl_diagnostic_2x8_sshfix.md`，使用 pip NCCL 2.21.5。 - -| Operation | Topology | Peak Bus BW | Threshold | Status | NCCL Network | GPU Direct RDMA | -|-----------|----------|-------------|-----------|--------|--------------|-----------------| -| allreduce | 2 nodes x 8 GPUs | `67.42 GB/s` | `>= 480 GB/s` | FAIL | IB | DISABLED | -| alltoall | 2 nodes x 8 GPUs | `9.56 GB/s` | `>= 75 GB/s` | FAIL | IB | DISABLED | - -allreduce 失败原因是带宽不达标，且报告捕获到 GDR 被 NCCL 禁用： - -| GDR Disabled HCAs | -|-------------------| -| `mlx5_0, mlx5_1, mlx5_6, mlx5_7` | - -allreduce 和 alltoall 本轮均正常完成，`returncode=0`、`wrong=0`，失败原因是带宽低于阈值，不是正确性失败。 - -### 4.1 NCCL 2.27.7 诊断结果 - -256M 诊断报告：`reports_multinode_nccl_diagnostic_2x8_nccl227_v2.md` - -| Operation | Topology | Peak Bus BW | Threshold | Status | NCCL Network | GPU Direct RDMA | -|-----------|----------|-------------|-----------|--------|--------------|-----------------| -| allreduce | 2 nodes x 8 GPUs | `212.19 GB/s` | `>= 480 GB/s` | FAIL | IB | ENABLED | -| alltoall | 2 nodes x 8 GPUs | `28.37 GB/s` | `>= 75 GB/s` | FAIL | IB | ENABLED | - -1M 到 4G sweep 报告：`reports_multinode_nccl_sweep_2x8_nccl227.md` - -| Operation | Peak Bus BW | Peak Size | Threshold | Status | GPU Direct RDMA | -|-----------|-------------|-----------|-----------|--------|-----------------| -| allreduce | `237.26 GB/s` | `4G` | `>= 480 GB/s` | FAIL | ENABLED | -| alltoall | `28.78 GB/s` | `1G` | `>= 75 GB/s` | FAIL | ENABLED | - -16G 大包报告：`reports_multinode_nccl_16g_2x8_nccl227.md` - -| Operation | Peak Bus BW | Peak Size | Threshold | Status | GPU Direct RDMA | -|-----------|-------------|-----------|-----------|--------|-----------------| -| allreduce | `237.86 GB/s` | `16G` | `>= 480 GB/s` | FAIL | ENABLED | -| alltoall | `28.62 GB/s` | `16G` | `>= 75 GB/s` | FAIL | ENABLED | - -解释：NCCL 2.27.7 已经修复 GDR 禁用问题，且性能提升明显；但在固定 `min_nchannels=4/qps=4/split=1` 的配置下仍不达标。allreduce 约稳定在 `238 GB/s`，alltoall 约稳定在 `28-29 GB/s`。 - -### 4.2 NCCL 2.27.7 自动通道/QP 参数结果 - -进一步对 16G 大包做 tuning，发现默认配置里锁定的参数会压低 allreduce： - -| 配置 | allreduce Avg Bus BW | alltoall Avg Bus BW | 结论 | -|------|----------------------|---------------------|------| -| NCCL 2.27.7 + 固定 `min_nchannels=4/qps=4/split=1` | `238.56 GB/s` | `28.62 GB/s` | GDR 已启用，但 allreduce 被压低 | -| NCCL 2.27.7 + NCCL 自动选择 channel/QP | `354.57 GB/s` | `30.02 GB/s` | 当前最佳脚本结果 | - -正式脚本报告：`reports_multinode_nccl_16g_2x8_nccl227_auto.md` - -| Operation | Peak Bus BW | Avg Bus BW | Peak Size | Threshold | Status | GPU Direct RDMA | -|-----------|-------------|------------|-----------|-----------|--------|-----------------| -| allreduce | `354.60 GB/s` | `354.57 GB/s` | `16G` | `>= 480 GB/s` | FAIL | ENABLED | -| alltoall | `30.01 GB/s` | `30.02 GB/s` | `16G` | `>= 75 GB/s` | FAIL | ENABLED | - -对比临时 tuning 命令： - -| 变量组合 | allreduce Avg Bus BW | alltoall Avg Bus BW | -|----------|----------------------|---------------------| -| baseline auto | `353.63 GB/s` | `30.05 GB/s` | -| `NCCL_IB_MERGE_NICS=1` | `352.73 GB/s` | `30.07 GB/s` | -| `NCCL_CROSS_NIC=1` | `354.68 GB/s` | `30.05 GB/s` | -| `NCCL_IB_QPS_PER_CONNECTION=8` + `NCCL_IB_SPLIT_DATA_ON_QPS=0` | `350.91 GB/s` | `29.41 GB/s` | -| `NCCL_MIN_NCHANNELS=16` + `NCCL_MAX_NCHANNELS=16` | `354.32 GB/s` | `30.06 GB/s` | - -解释：allreduce 的主要提升来自取消不合适的固定参数，而不是 `MERGE_NICS` 或 `CROSS_NIC`。alltoall 对这些参数不敏感，当前基本稳定在 `30 GB/s` 左右。 - -### 5. SSHD MaxStartups 阻塞已临时缓解 - -`nccl-gpu-2` 曾显示： - -```text -sshd: /usr/sbin/sshd -D [listener] 52 of 10-100 startups -maxstartups 10:30:100 -``` - -同时存在大量 `sshd: unknown [priv]` / `sshd: unknown [net]` 未认证连接，来源主要是 `172.239.10.85`。这会触发 OpenSSH `MaxStartups` 随机拒绝，直接表现为： - -```text -kex_exchange_identification: Connection closed by remote host -``` - -先临时改为： - -```text -MaxStartups 120:30:240 -LoginGraceTime 20 -``` - -后续外部未认证连接继续上涨到 `110 of 120-240 startups`，测试窗口进一步临时改为： - -```text -MaxStartups 500:30:1000 -LoginGraceTime 5 -``` - -改完后从 0012 连续 SSH 0016 5 次成功，2 节点 `mpirun hostname` 成功，2 节点 x 8 GPU allreduce/alltoall 也都能跑出有效结果。 - -### 6. `nvidia_peermem` legacy 模式实验无效 - -两台机器默认参数一致： - -| 参数 | 值 | -|------|----| -| `nvidia_peermem` version | `580.159.03` | -| `peerdirect_support` | `0` | -| `persistent_api_support` | `1` | -| OFED | `OFED-internal-26.01-1.0.0` | - -临时切换两台机器到 `peerdirect_support=1` 后，2 节点 x 1 GPU NCCL 仍显示： - -```text -NET/IB : GPU Direct RDMA Disabled for HCA 0 'mlx5_0' -``` - -带宽仍约 `13.4 GB/s`。测试后已经恢复默认 `peerdirect_support=0,persistent_api_support=1`。 - -### 7. PDF 矩阵对齐与 GPU-NIC 亲和性 - -参考 PDF 的跨 Leaf 命令覆盖 2 机 2/4/8/16 卡矩阵，并使用： - -- `NCCL_IB_GID_INDEX=3` -- `NCCL_IB_SL=5` -- `NCCL_IB_TC=136` -- `NCCL_SOCKET_IFNAME=bond0` -- `NCCL_IB_TIMEOUT=22` -- `NCCL_NET_PLUGIN=none` -- `NCCL_NVLS_ENABLE=1` - -本环境与 PDF 参考机器有一个关键硬件差异：当前两台机器只有 `mlx5_0,mlx5_1,mlx5_6,mlx5_7` 是 400Gb/s NDR；`mlx5_4,mlx5_5` 是 100Gb/s HDR；`mlx5_2,mlx5_8` 是 25Gb/s；`mlx5_3,mlx5_9` 为 DOWN。参考 PDF 的命令列出了更多 HCA，但当前节点不能等价使用为 8 条 400G rail。 - -`nvidia-smi topo -m` 显示： - -| GPU | 最近的 400G HCA | -|-----|-----------------| -| GPU0 | `mlx5_0` | -| GPU1 | `mlx5_1` | -| GPU4 | `mlx5_6` | -| GPU5 | `mlx5_7` | - -默认 2 机 4 卡会选择 GPU0/1/2/3，其中 GPU2 最近的是 25G/down 端口，GPU3 没有直接对应 400G rail。因此 2 机 4 卡默认 allreduce 只有约 `168 GB/s`。显式设置 `CUDA_VISIBLE_DEVICES=0,1,4,5` 后： - -| 场景 | allreduce | alltoall | 说明 | -|------|-----------|----------|------| -| 默认 GPU0/1/2/3 | `167.89 GB/s` | `39.68 GB/s` | GPU/NIC 亲和性错误 | -| `CUDA_VISIBLE_DEVICES=0,1,4,5` + auto NCCL | `335.34 GB/s` | `63.90 GB/s` | allreduce 接近 PDF | -| `CUDA_VISIBLE_DEVICES=0,1,4,5` + PDF 固定参数 | `225.29 GB/s` | `73.10 GB/s` | alltoall 接近 PDF，但 allreduce 被压低 | - -因此当前脚本支持按 op 配环境变量：4 卡 allreduce 用 auto，4 卡 alltoall 用 PDF 固定参数。 - -矩阵式正式报告：`reports_multinode_nccl_pdf_matrix_nccl227.md` - -| Topology | allreduce | PDF Reference | Status | alltoall | PDF Reference | Status | -|----------|-----------|---------------|--------|----------|---------------|--------| -| 2 nodes x 1 GPU | `47.26 GB/s` | `48.90 GB/s` | FAIL | `24.87 GB/s` | `27.25 GB/s` | FAIL | -| 2 nodes x 2 GPUs | `136.36 GB/s` | `136.93 GB/s` | FAIL | `47.69 GB/s` | `54.41 GB/s` | FAIL | -| 2 nodes x 4 GPUs | `333.23 GB/s` | `335.48 GB/s` | FAIL | `72.82 GB/s` | `73.73 GB/s` | FAIL | -| 2 nodes x 8 GPUs | `353.47 GB/s` | `491.84 GB/s` | FAIL | `36.70 GB/s` | `76.54 GB/s` | FAIL | - -解释：2 机 4 卡档位已经基本定位并修复到接近 PDF；2 机 8 卡档位不是简单 GPU 顺序问题。尝试调整 8 卡 `CUDA_VISIBLE_DEVICES` 顺序、加入 100G/25G active HCA、以及套 PDF 固定参数都没有改善；固定参数反而会把 8 卡 allreduce 从约 `354 GB/s` 压到约 `239 GB/s`。 - -8 卡 alltoall 目前的最佳软件侧改动是 `NCCL_PXN_DISABLE=1`： - -| Case | 8 卡 alltoall Avg Bus BW | -|------|--------------------------| -| baseline | `30.06 GB/s` | -| `NCCL_PXN_DISABLE=1` | `37.24 GB/s` | -| 正式矩阵报告 | `36.74 GB/s` | - -其他变量如 `NCCL_P2P_PXN_LEVEL`、`NCCL_NET_SHARED_COMMS`、`NCCL_NET_SHARED_BUFFERS`、`NCCL_NCHANNELS_PER_NET_PEER`、`NCCL_IB_ADAPTIVE_ROUTING` 均无改善或变差。 - -PXN disabled 计数器显示该参数确实修复了 rail 分布： - -| Case | Rail 分布 | Avg Bus BW | -|------|-----------|------------| -| baseline | `mlx5_0/6` 约 `885 GB`，`mlx5_1/7` 约 `295 GB` | `30.04 GB/s` | -| `NCCL_PXN_DISABLE=1` | 四条 HCA 均约 `591 GB` | `36.95 GB/s` | - -但禁用 PXN 后每条 400G rail 仍只有约 `19-20 GB/s`，没有接近裸 RDMA 单 rail 的 `347-387 Gb/s`。因此它解决的是 rail 分布不均衡的一部分，不是全部 alltoall 性能问题。 - -复测 PXN disabled alltoall 时继续抓 `counters`/`hw_counters`： - -| 观察项 | 结果 | -|--------|------| -| alltoall `Avg bus bandwidth` | `36.4512 GB/s` | -| 每条 HCA 流量 | 约 `712.18-712.28 GiB`，四条 rail 均衡 | -| discard / rcv error / symbol error / link down / link recovery | `0` 增量 | -| RoCE retrans / slow restart / packet sequence error / out of sequence | `0` 增量 | -| `port_xmit_wait` | `mlx5_1`、`mlx5_7` 有增长，约 `15.65M-23.49M` | - -判断：当前没有明显坏链路、丢包或重传证据；`port_xmit_wait` 更像发送侧等待 credit/拥塞控制/交换侧调度，或者 NCCL internal alltoall 在当前拓扑下没有把 rail 吞吐打起来。 - -同样 2 nodes x 8 GPUs、同样 4 条 HCA 的 16G allreduce 对照： - -| 观察项 | 结果 | -|--------|------| -| allreduce `Avg bus bandwidth` | `354.366 GB/s` | -| 每条 HCA 流量 | 约 `178.03-178.07 GiB`，四条 rail 均衡 | -| 错误/重传类 counter | `0` 增量 | -| `port_xmit_wait` | `mlx5_1`、`mlx5_7` 有增长，约 `6.11M-6.59M` | - -判断：allreduce 在接近物理上限时也会出现 `port_xmit_wait`，所以 alltoall 的核心问题不能只归因于该 counter。现在更应关注 NCCL alltoall 通信模式、交换网络策略、以及 NCCL net plugin/SHARP 能力差异。 - -PXN disabled 基线上的二次参数 sweep： - -| Case | Avg Bus BW | 结论 | -|------|------------|------| -| `NCCL_PXN_DISABLE=1` | `37.0069 GB/s` | 短测基线 | -| `+ NCCL_NVLS_ENABLE=0` | `37.2217 GB/s` | 小幅波动，不稳定 | -| `+ NCCL_P2P_NET_CHUNKSIZE=4194304` | `37.2522 GB/s` | 小幅波动，不稳定 | -| `+ NCCL_BUFFSIZE=8388608` | `37.0911 GB/s` | 无实质改善 | -| `+ NCCL_MIN_NCHANNELS=16 NCCL_MAX_NCHANNELS=16` | `37.0189 GB/s` | 无实质改善 | -| `+ NCCL_IB_AR_THRESHOLD=0` | `37.0843 GB/s` | 无实质改善 | -| `+ NCCL_IB_QPS_PER_CONNECTION=4 NCCL_IB_SPLIT_DATA_ON_QPS=0` | `35.9847 GB/s` | 变差 | -| `+ NCCL_IB_QPS_PER_CONNECTION=4 NCCL_IB_SPLIT_DATA_ON_QPS=1` | `29.8406 GB/s` | 明显变差 | -| `+ NCCL_IB_QPS_PER_CONNECTION=8 NCCL_IB_SPLIT_DATA_ON_QPS=1` | `24.1183 GB/s` | 明显变差 | -| `+ NCCL_NCHANNELS_PER_NET_PEER=8` | `29.8904 GB/s` | 明显变差 | - -长测复核没有复现 `NVLS/P2P chunk` 的短测小涨：同一环境确认仍为 NCCL `2.27.7+cuda12.4`、4 条 400G HCA、GDR enabled、internal IB plugin，但 baseline 窗口下滑到 `32.7280 GB/s`，`P2P_NET_CHUNKSIZE=4M` 为 `31.9340 GB/s`，`NVLS_ENABLE=0 + P2P_NET_CHUNKSIZE=4M` 为 `27.6585 GB/s`。因此这些参数不应固化到正式配置。 - -`GRAPH/TUNING/COLL` 日志对照： - -| 观察项 | allreduce | alltoall + `NCCL_PXN_DISABLE=1` | -|--------|-----------|----------------------------------| -| NCCL version | `2.27.7+cuda12.4` | `2.27.7+cuda12.4` | -| HCA / GDR | 4 HCA, GDR enabled | 4 HCA, GDR enabled | -| external net plugin | missing, internal IB | missing, internal IB | -| channels | `16 coll / 16 nvls / 16 p2p` | `16 coll / 16 nvls / 16 p2p` | -| Pattern 4 | `crossNic 0`, `type NVL/PXN`, `nChannels 8` | `crossNic 2`, `type NVL/PIX`, `nChannels 8` | -| `NET/IB/*/GDRDMA` channel edge lines | `256` | `512` | -| `P2P/CUMEM` channel edge lines | `0` | `224` | -| total NET/P2P channel edge lines | `256` | `736` | - -判断：PXN disabled 后 4 条 IB/GDRDMA rail 和 16 个 p2p/coll/nvls channels 都仍在；但 alltoall graph 明显比 allreduce 复杂，并包含大量本机 P2P/CUMEM 边。这进一步说明问题不在 HCA/GDR 没生效，而在 alltoall collective graph、P2P/NET 组合方式、internal IB plugin 或交换网络策略。 - -### 8. 8 卡链路计数器与物理上限判断 - -计数器探测报告：`reports_multinode_nccl_counter_probe_20260523.md` - -当前 2 机 8 GPU allreduce 输出： - -| Metric | Value | -|--------|-------| -| `algbw` | `189.16 / 189.07 GB/s` | -| `busbw` | `354.68 / 354.52 GB/s` | -| `Avg bus bandwidth` | `354.597 GB/s` | - -allreduce 在 16 ranks 下的换算关系约为： - -```text -busbw = algbw * 2 * (nranks - 1) / nranks = algbw * 1.875 -``` - -因此 PDF 参考 `491.84 GB/s busbw` 对应约 `262.31 GB/s algbw`。但当前节点可用的 400G HCA 是 `mlx5_0,mlx5_1,mlx5_6,mlx5_7`，每节点 4 条 400Gb/s，理论单向合计约 `200 GB/s`。当前 allreduce `189 GB/s algbw` 已经接近这个物理上限，所以 8 卡 allreduce 剩余差距基本不能靠 NCCL 参数小调解决。 - -裸 RDMA 4 rail 并发 `ib_write_bw` 也验证了底层 4 条 400G rail 可以同时工作： - -| HCA | BW average | -|-----|------------| -| `mlx5_0` | `387.16 Gb/s` | -| `mlx5_1` | `387.07 Gb/s` | -| `mlx5_6` | `355.02 Gb/s` | -| `mlx5_7` | `347.70 Gb/s` | -| Total | `1476.95 Gb/s` / `184.62 GB/s` | - -这个裸 RDMA 总带宽与 NCCL 8 卡 allreduce 的 `189 GB/s algbw` 接近，进一步说明 allreduce 已经贴近当前网络形态可提供的实际带宽。 - -8 卡 alltoall 当前仍只有： - -| Metric | Value | -|--------|-------| -| `algbw` | `32.04 / 32.05 GB/s` | -| `busbw` | `30.03 / 30.04 GB/s` | -| `Avg bus bandwidth` | `30.0389 GB/s` | - -同一测试窗口内端口计数器显示 alltoall 流量分布不均衡：`mlx5_0` 和 `mlx5_6` 的流量约 `885 GB`，`mlx5_1` 和 `mlx5_7` 约 `295 GB`，约为三倍差距。继续调换 `NCCL_IB_HCA` 顺序后，8 卡 alltoall 仍稳定在 `30.02-30.07 GB/s`，说明不是简单 HCA 列表顺序问题。 - -`NCCL_PXN_DISABLE=1` 后，端口流量变为四条 HCA 均约 `591 GB`，alltoall `Avg bus bandwidth` 提升到 `36.9518 GB/s`，但每条 rail 吞吐仍只有约 `19.82 GB/s`。 - -### 9. NCCL net plugin / SHARP 状态 - -两台机器上均未找到： - -- `libnccl-net.so` -- `libsharp*` -- SHARP/HCOLL 相关 deb 包 - -当前仅看到 UCX 包： - -```text -ucx 1.20.0-1.20260211.d9a4f352d.2601100 -``` - -apt 源里与 NCCL 直接相关的包只有： - -```text -libnccl2 -libnccl-dev -``` - -因此当前 NCCL 日志里的 `Could not find: libnccl-net.so` 是真实环境缺失，不是脚本漏配路径。当前运行走的是 NCCL internal IB plugin；如果要继续追 8 卡 alltoall 或 PDF 2 机 16 卡参考值，需要补齐匹配当前 OFED/driver/CUDA/NCCL 的 NCCL net plugin/SHARP 环境，或由网络侧确认该集群不依赖这些组件也能达到目标值。 - -## 当前阻塞 - -### 阻塞 1：当前生产 NCCL 版本过旧，GDR 被禁用 - -现象： - -- pip NCCL 2.21.5：`GPU Direct RDMA Disabled`，2x8 allreduce `67.42 GB/s` -- 临时 NCCL 2.27.7：`GPU Direct RDMA Enabled`，2x8 allreduce `237.86 GB/s` -- 因此，生产测试环境应避免继续使用 pip NCCL 2.21.5 作为多机 NCCL 验收运行库 - -判断：底层 RDMA 能力存在，GDR 禁用主要由旧 NCCL 版本触发。建议正式安装并固定 NCCL 2.27.7+cuda12.4 或更新的已验证版本。 - -### 阻塞 2：2 机 8 GPU 档位仍低于 PDF 参考值 - -现象： - -- 2x8 16G allreduce：`354.02 GB/s`，PDF 参考 `491.84 GB/s` -- 2x8 16G alltoall：`30.04 GB/s`，PDF 参考 `76.54 GB/s` -- 已使用 4 个 400Gb/s HCA：`mlx5_0, mlx5_1, mlx5_6, mlx5_7` -- 加入 `mlx5_4,mlx5_5` 100G HCA 或 `mlx5_2,mlx5_8` 25G HCA 基本无收益 -- 调整 8 卡 `CUDA_VISIBLE_DEVICES` 顺序基本无收益 -- 套 PDF 固定参数会让 8 卡 allreduce 明显变差 - -判断：2 机 8 GPU 档位的剩余差距更像硬件 rail 数量/交换网络/路由/拥塞/NCCL net plugin 能力问题，不再是旧 NCCL GDR disabled 或 4 卡 GPU 选择问题。 - -补充证据： - -- 8 卡 allreduce `algbw ~= 189 GB/s`，接近当前 4 x 400G HCA 的理论单向合计 `200 GB/s` -- 裸 RDMA 4 rail 并发 `ib_write_bw` 合计 `1476.95 Gb/s` / `184.62 GB/s` -- PDF 8 卡 allreduce `491.84 GB/s busbw` 反推需要约 `262 GB/s algbw`，超过当前 4 x 400G 的物理单向总带宽 -- 8 卡 alltoall baseline 端口计数器显示 rail 分布不均，且 HCA 顺序 sweep 无改善 -- 当前环境缺失 NCCL net plugin/SHARP，NCCL 只能使用 internal IB plugin -- `NCCL_PXN_DISABLE=1` 可将 8 卡 alltoall 提升到约 `36.7 GB/s`，并修复 rail 分布不均，但仍不到 PDF 参考值的一半 -- PXN disabled 复测没有看到 discard、链路错误、RoCE 重传、slow restart、packet sequence error 等错误类 counter 增长 -- allreduce 对照同样出现 `port_xmit_wait` 但能跑到 `354.366 GB/s`，说明 `port_xmit_wait` 不是 alltoall 低吞吐的唯一根因 -- PXN disabled 基线上继续叠加 NVLS、P2P chunk、buffer、channel、QP/split、AR 等参数没有稳定收益；QP/split 和 `NCCL_NCHANNELS_PER_NET_PEER=8` 明显变差 -- NCCL GRAPH/TUNING 对照显示 alltoall 与 allreduce 的 HCA/GDR/channel 基础状态一致，但 alltoall channel edge 更多，并混入大量 `P2P/CUMEM` 本地路径 - -### 阻塞 3：`nccl-gpu-2` SSH 存在外部连接压力 - -现象： - -- 多次出现过：`kex_exchange_identification: Connection closed by remote host` -- 根因是未认证连接过多触发 `MaxStartups` -- 当前已经通过临时 SSHD 配置缓解，并拿到了有效 2x8 报告 -- 但如果外部连接压力持续，仍建议从网络侧或安全策略侧处理来源连接 - -判断：这不再阻塞当前报告产出，但属于环境稳定性风险。 - -## 建议下一步 - -1. 从网络/安全侧处理 `172.239.10.85` 等来源的 SSH 未认证连接压力，或者保留更高的 `MaxStartups` 配置作为测试窗口临时策略。 -2. 正式安装并固定已验证的 NCCL 2.27.7+cuda12.4 或更新版本，不要依赖 pip NCCL 2.21.5；当前 `/tmp/nccl-2.27.7-cuda12.4` 只是临时解压验证。 -3. 4 卡 per node 测试应显式使用 `CUDA_VISIBLE_DEVICES=0,1,4,5`，避免默认 GPU0/1/2/3 落到错误 GPU/NIC 亲和性。 -4. 4 卡 allreduce 建议继续让 NCCL 自动选择 channel/QP；4 卡 alltoall 如果要贴近 PDF，可单独套 `NCCL_IB_QPS_PER_CONNECTION=4`、`NCCL_MIN_NCHANNELS=4`、`NCCL_IB_SPLIT_DATA_ON_QPS=1`。 -5. 8 卡 per node 不建议套上述固定参数，会降低 allreduce；继续用 auto。 -6. 尝试安装或启用匹配当前 OFED/driver 的 NCCL net plugin/SHARP；当前日志显示 `Could not find: libnccl-net.so`，NCCL 使用的是 internal IB plugin。 -7. 核对跨 Leaf 链路的 rail mapping、交换机端口速率、路由、credit/拥塞等待与交换机侧队列计数；同时用 allreduce 对照避免把 `port_xmit_wait` 误判为 alltoall 独有根因。 -8. 确认当前 PDF 的 `491.84/76.54 GB/s` 是否要求当前这两台节点在只有 4 条 400G rail 的形态下也达到；如果要求一致，需要网络/硬件侧继续介入。 -9. 8 卡 alltoall 当前不建议继续盲调 NCCL 环境变量；重点查 SHARP/NCCL net plugin、NCCL internal alltoall 行为、交换机 ECMP/自适应路由和拥塞/credit 等待；`NCCL_IB_HCA` 顺序与 rail 分布本身已经不是当前主问题。 - -## 当前可交付物 - -- `configs/multinode_nccl_diagnostic.yaml`：多机多卡诊断配置 -- `configs/multinode_nccl_nccl227_diagnostic.yaml`：NCCL 2.27.7 256M 诊断配置 -- `configs/multinode_nccl_nccl227_sweep.yaml`：NCCL 2.27.7 1M 到 4G sweep 配置 -- `configs/multinode_nccl_nccl227_16g.yaml`：NCCL 2.27.7 16G 大包配置 -- `configs/multinode_nccl_nccl227_auto_16g.yaml`：NCCL 2.27.7 16G 自动 channel/QP 配置 -- `configs/multinode_nccl_nccl227_pdf_matrix.yaml`：按 PDF 矩阵和 GPU 亲和性优化后的跨 Leaf 配置 -- `reports_multinode_nccl_diagnostic_2x8_sshfix.md`：脚本生成的原始 2x8 诊断报告 -- `reports_multinode_nccl_diagnostic_2x8_nccl227_v2.md`：NCCL 2.27.7 256M 诊断报告 -- `reports_multinode_nccl_sweep_2x8_nccl227.md`：NCCL 2.27.7 1M 到 4G sweep 报告 -- `reports_multinode_nccl_16g_2x8_nccl227.md`：NCCL 2.27.7 16G 大包报告 -- `reports_multinode_nccl_16g_2x8_nccl227_auto.md`：NCCL 2.27.7 16G 自动 channel/QP 原始报告 -- `reports_multinode_nccl_pdf_matrix_nccl227.md`：NCCL 2.27.7 PDF 矩阵式原始报告 -- `reports_multinode_nccl_counter_probe_20260523.md`：8 卡链路计数器与 HCA 顺序 sweep 报告 -- `reports_multinode_nccl_alltoall_tuning_20260523.md`：8 卡 alltoall NCCL 网络参数 sweep 报告 -- `reports_multinode_nccl_diagnosis_20260523.md`：本中文诊断总结 diff --git a/reports_multinode_nccl_diagnostic_2x8_debug_v2.md b/reports_multinode_nccl_diagnostic_2x8_debug_v2.md deleted file mode 100644 index 2076245..0000000 --- a/reports_multinode_nccl_diagnostic_2x8_debug_v2.md +++ /dev/null @@ -1,66 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T07:37:41.426792 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- DCGM -- Training - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: diagnostic - -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS (1 warnings) - -### Multi-node NCCL allreduce - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs diagnostic | 68.69 GB/s | 256M | 68.21 GB/s | >= 480 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Disabled HCAs | -|----------|--------------|-----------------|-------------------| -| 2 nodes x 8 GPUs diagnostic | IB | DISABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs diagnostic | 0 | aikubeworker0012:2139504:2139504 [0] NCCL INFO comm 0x55646d15f590 rank 0 nranks 16 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 68.2135 # # Collective test concluded: all_reduce_perf # | - -### Multi-node NCCL alltoall - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs diagnostic | 0.00 GB/s | | 0.00 GB/s | >= 75 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Disabled HCAs | -|----------|--------------|-----------------|-------------------| -| 2 nodes x 8 GPUs diagnostic | unknown | UNKNOWN | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs diagnostic | 255 | lack of common network interfaces and/or no route found between them. Please check network connectivity (including firewalls and network routing requirements). -------------------------------------------------------------------------- | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_multinode_nccl_diagnostic_2x8_nccl227_v2.md b/reports_multinode_nccl_diagnostic_2x8_nccl227_v2.md deleted file mode 100644 index 1b188d5..0000000 --- a/reports_multinode_nccl_diagnostic_2x8_nccl227_v2.md +++ /dev/null @@ -1,66 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T07:53:24.460277 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- DCGM -- Training - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: diagnostic-nccl-2.27.7 - -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS - -### Multi-node NCCL allreduce - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs NCCL 2.27.7 | 212.19 GB/s | 256M | 211.75 GB/s | >= 480 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 | 0 | 0016:1009332:1009965 [2] NCCL INFO comm 0x56388eec2e40 rank 10 nranks 16 cudaDev 2 busId 3a000 - Destroy COMPLETE aikubeworker0012:2144366:2144531 [5] NCCL INFO comm 0x556e4fcf5280 rank 5 nranks 16 cudaDev 5 busId ab000 - Destroy COMPLETE | - -### Multi-node NCCL alltoall - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs NCCL 2.27.7 | 28.37 GB/s | 256M | 28.32 GB/s | >= 75 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 | 0 | 0012:2144547:2144713 [4] NCCL INFO comm 0x55896a1dae20 rank 4 nranks 16 cudaDev 4 busId 9a000 - Destroy COMPLETE aikubeworker0016:1010164:1010881 [2] NCCL INFO comm 0x565344db7790 rank 10 nranks 16 cudaDev 2 busId 3a000 - Destroy COMPLETE | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_multinode_nccl_diagnostic_2x8_sshfix.md b/reports_multinode_nccl_diagnostic_2x8_sshfix.md deleted file mode 100644 index 1872c50..0000000 --- a/reports_multinode_nccl_diagnostic_2x8_sshfix.md +++ /dev/null @@ -1,66 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T07:46:11.464439 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- DCGM -- Training - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: diagnostic - -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS - -### Multi-node NCCL allreduce - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs diagnostic | 67.42 GB/s | 256M | 67.50 GB/s | >= 480 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Disabled HCAs | -|----------|--------------|-----------------|-------------------| -| 2 nodes x 8 GPUs diagnostic | IB | DISABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs diagnostic | 0 | orker0016:986293:986293 [1] NCCL INFO comm 0x563abe94c350 rank 9 nranks 16 cudaDev 1 busId 2a000 - Destroy COMPLETE aikubeworker0016:986292:986292 [0] NCCL INFO comm 0x560ffac51160 rank 8 nranks 16 cudaDev 0 busId 18000 - Destroy COMPLETE | - -### Multi-node NCCL alltoall - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs diagnostic | 9.56 GB/s | 256M | 9.55 GB/s | >= 75 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Disabled HCAs | -|----------|--------------|-----------------|-------------------| -| 2 nodes x 8 GPUs diagnostic | IB | DISABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs diagnostic | 0 | TE aikubeworker0012:2141982:2141982 [4] NCCL INFO comm 0x55d0bf9c6a00 rank 4 nranks 16 cudaDev 4 busId 9a000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 9.55234 # # Collective test concluded: alltoall_perf # | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_multinode_nccl_environment_gap_20260523.md b/reports_multinode_nccl_environment_gap_20260523.md deleted file mode 100644 index c4a65a5..0000000 --- a/reports_multinode_nccl_environment_gap_20260523.md +++ /dev/null @@ -1,168 +0,0 @@ -# 多节点 NCCL 环境等价性缺口说明 2026-05-23 - -## 目的 - -这份文档用于回答一个核心问题：当前 `aikubeworker0012` / `aikubeworker0016` 是否具备与参考 PDF 的 2 机 16 GPU NCCL 目标相同的硬件和 NCCL 网络软件环境。 - -结论先行：**当前环境不能证明与 PDF 参考环境等价**。主要差异有两类： - -1. 当前每节点只有 4 条可用于 NCCL 的 400G InfiniBand rail。 -2. 当前没有外部 NCCL net plugin / SHARP / HCOLL 组件，NCCL 使用 internal IB plugin。 - -## 采集时间和节点 - -采集时间：`2026-05-23T10:53:18+00:00` 至 `2026-05-23T10:53:21+00:00` - -| 节点 | SSH alias | 内网地址 | kernel | -|---|---|---|---| -| `aikubeworker0012` | `nccl-gpu-1` | `172.72.8.12` | `5.15.0-119-generic` | -| `aikubeworker0016` | `nccl-gpu-2` | `172.72.8.16` | `5.15.0-119-generic` | - -## HCA / Rail 现状 - -两台机器的 `/sys/class/infiniband/mlx5_*/ports/1` 结果一致： - -| HCA | State | Rate | Link layer | 对 NCCL 跨节点验收的含义 | -|---|---|---:|---|---| -| `mlx5_0` | ACTIVE | `400 Gb/sec (4X NDR)` | InfiniBand | 可作为 400G rail | -| `mlx5_1` | ACTIVE | `400 Gb/sec (4X NDR)` | InfiniBand | 可作为 400G rail | -| `mlx5_2` | ACTIVE | `25 Gb/sec (1X EDR)` | Ethernet | 不是 400G IB rail | -| `mlx5_3` | DOWN | `25 Gb/sec (1X EDR)` | Ethernet | 不可用 | -| `mlx5_4` | ACTIVE | `100 Gb/sec (2X HDR)` | InfiniBand | 不是 400G rail | -| `mlx5_5` | ACTIVE | `100 Gb/sec (2X HDR)` | InfiniBand | 不是 400G rail | -| `mlx5_6` | ACTIVE | `400 Gb/sec (4X NDR)` | InfiniBand | 可作为 400G rail | -| `mlx5_7` | ACTIVE | `400 Gb/sec (4X NDR)` | InfiniBand | 可作为 400G rail | -| `mlx5_8` | ACTIVE | `25 Gb/sec (1X EDR)` | Ethernet | 不是 400G IB rail | -| `mlx5_9` | DOWN | `25 Gb/sec (1X EDR)` | Ethernet | 不可用 | - -因此当前推荐并实际使用的 HCA 列表是： - -```text -NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_6,mlx5_7 -``` - -这代表每节点 `4 x 400Gb/s`，理论单向原始带宽约： - -```text -4 * 400Gb/s / 8 = 200 GB/s -``` - -## 与 PDF 目标的物理带宽关系 - -参考 PDF 的 2 机 16 GPU 目标： - -| Operation | PDF Bus BW | -|---|---:| -| AllReduce | `491.84 GB/s` | -| AllToAll | `76.54 GB/s` | - -NCCL allreduce 在 16 ranks 下，`busbw = algbw * 2 * (n - 1) / n = algbw * 1.875`。 - -因此 PDF 的 allreduce `491.84 GB/s busbw` 反推： - -```text -491.84 / 1.875 = 262.31 GB/s algbw -``` - -但当前 4 条 400G rail 的理论单向原始带宽约 `200 GB/s`。本项目实测 2x8 allreduce： - -| 测试 | Bus BW | 反推 Alg BW | -|---|---:|---:| -| 本轮深度诊断 allreduce | `354.025 GB/s` | `188.81 GB/s` | -| 本轮 GRAPH allreduce | `354.224 GB/s` | `188.92 GB/s` | - -这已经接近当前 4 x 400G rail 的物理单向上限。除非 PDF 参考环境具备更多有效 400G rail、更高交换网络能力，或使用了当前缺失的网络加速组件，否则当前 2x8 allreduce 很难靠 NCCL 环境变量小调达到 `491.84 GB/s`。 - -## GPU-NIC 亲和性影响 - -`nvidia-smi topo -m` 显示的 NIC legend 两台一致： - -| NIC | HCA | -|---|---| -| NIC0 | `mlx5_0` | -| NIC1 | `mlx5_1` | -| NIC2 | `mlx5_2` | -| NIC3 | `mlx5_3` | -| NIC4 | `mlx5_4` | -| NIC5 | `mlx5_5` | -| NIC6 | `mlx5_6` | -| NIC7 | `mlx5_7` | -| NIC8 | `mlx5_8` | -| NIC9 | `mlx5_9` | - -关键亲和关系： - -| GPU | 最近的有效 400G HCA | -|---|---| -| GPU0 | `mlx5_0` | -| GPU1 | `mlx5_1` | -| GPU4 | `mlx5_6` | -| GPU5 | `mlx5_7` | - -这解释了为什么 2 机 4 GPU 档位需要使用： - -```text -CUDA_VISIBLE_DEVICES=0,1,4,5 -``` - -默认 GPU0/1/2/3 会把 GPU2/GPU3 放到非理想 NIC 亲和路径上，其中 GPU2 最近的 `mlx5_2/3` 不是可用 400G IB rail。 - -## NCCL Net Plugin / SHARP 状态 - -在两台节点上搜索： - -```text -find /usr /opt /tmp /root -name 'libnccl-net*.so*' -o -name 'libsharp*.so*' -``` - -结果为空。 - -两台节点包列表中能看到： - -| 包 | 版本/说明 | -|---|---| -| `doca-ofed` | `3.3.0-088000` | -| `mlnx-ofed-kernel-dkms` | `26.01.OFED.26.01.1.0.0.1-1` | -| `ucx` | `1.20.0-1.20260211...` | - -未看到： - -- `libnccl-net.so` -- `libsharp*.so` -- SHARP packages -- HCOLL packages - -本轮 NCCL GRAPH 日志也显示 `plugin_missing=16`，说明 NCCL 只能走 internal IB plugin。 - -## 当前 2x8 结果归因边界 - -已经基本排除： - -- 不是 SSH / mpirun launch 问题：preflight 已通过。 -- 不是 HCA 完全不可用：4 条 400G rail 都 ACTIVE，allreduce 能跑到约 `354 GB/s busbw`。 -- 不是 GDR disabled：NCCL `2.27.7` 日志中 GDR enabled。 -- 不是 rail 完全打偏：`NCCL_PXN_DISABLE=1` 后 alltoall 四条 rail 流量均衡。 -- 不是明显坏链路/重传：counter 未见 discard、RoCE retrans、slow restart、packet sequence error 等增长。 - -仍然成立的缺口： - -1. **2x8 allreduce 的 PDF 目标疑似超过当前 4 x 400G rail 物理能力。** -2. **2x8 alltoall 即使 rail 均衡仍只有 `36-37 GB/s`，更像 NCCL alltoall 图策略、internal IB plugin 能力、缺少 SHARP/NCCL net plugin 或交换网络策略问题。** - -## 给网络/环境侧的确认清单 - -请网络/环境侧确认以下问题： - -1. PDF 参考环境每节点实际参与 NCCL 的 400G rail 数量是多少？是否为 8 条 400G，而不是当前的 4 条 400G？ -2. PDF 命令中列出的 HCA 列表是否在参考环境中全部为 400G InfiniBand ACTIVE？ -3. PDF 参考环境是否启用了 NCCL net plugin、SHARP、HCOLL、UCX plugin 或交换机侧 SHARP aggregation？ -4. 当前交换网络是否开启 adaptive routing / ECMP / congestion control，是否存在跨 Leaf 场景下对 alltoall pattern 不友好的 hash 或路径限制？ -5. 当前 `mlx5_4/5` 为什么只有 100G，`mlx5_2/8` 为什么是 Ethernet 25G，`mlx5_3/9` 为什么 DOWN；这些是否符合机器采购和验收预期？ -6. 如果验收必须按 PDF 的 `491.84/76.54 GB/s`，是否需要更换到与 PDF 等价的 rail 数量/交换网络/软件栈再测。 - -## 建议下一步 - -1. 暂停继续盲调 NCCL 小参数；已有 sweep 显示收益不稳定或负向。 -2. 先让硬件/网络侧确认 rail 数量和速率是否与 PDF 等价。 -3. 如果确认硬件等价，再补齐 NCCL net plugin / SHARP 环境，并用 `scripts/multinode_nccl_deep_diagnose.sh graph` 复查 plugin 和 graph 变化。 -4. 如果硬件不等价，应调整验收阈值或改用与 PDF 等价的节点组合复测。 diff --git a/reports_multinode_nccl_handoff_plan_20260523.md b/reports_multinode_nccl_handoff_plan_20260523.md deleted file mode 100644 index d70ea8b..0000000 --- a/reports_multinode_nccl_handoff_plan_20260523.md +++ /dev/null @@ -1,213 +0,0 @@ -# 多节点 NCCL 交接计划 2026-05-23 - -## 当前一句话结论 - -当前 2 机 8 卡 NCCL 已经排除旧 NCCL、GDR disabled、HCA 选择错误、SSH/mpirun launch、明显链路错误等问题；剩余差距集中在 **硬件 rail 数量是否与 PDF 等价**、**NCCL net plugin / SHARP 是否缺失**、以及 **alltoall 在当前跨 Leaf 网络下的图策略/交换路径效率**。 - -全局验收状态先看 `reports_h100_acceptance_current_status_20260523.md`；该文件把单节点 `test all`、跨节点 RDMA、多机 NCCL 和阻塞项汇总到一张总表。 - -## 已经验证的事实 - -| 事实 | 当前证据 | -|---|---| -| 两台机器可用于 NCCL 的 400G IB rail 是 4 条 | `mlx5_0,mlx5_1,mlx5_6,mlx5_7` 均为 `400 Gb/sec (4X NDR)` | -| 其他 HCA 不等价 | `mlx5_4/5` 为 100G IB，`mlx5_2/8` 为 25G Ethernet，`mlx5_3/9` DOWN | -| NCCL 2.27.7 GDR 可用 | GRAPH/NET 日志中 GDR enabled | -| allreduce 已接近当前 4 rail 物理上限 | 最新 PDF matrix 2x8 为 `353.85 GB/s busbw`，反推 `188.72 GB/s algbw`，接近 4 x 400G 的 `200 GB/s` 单向原始带宽 | -| alltoall PXN disabled 后 rail 均衡但仍低 | 最新 PDF matrix 2x8 为 `36.83 GB/s busbw`，每条 rail 约 `19-20 GB/s` | -| 正式 PDF matrix 已复跑 | `reports_multinode_nccl_pdf_matrix_20260523_113803.md`，所有 case 正确性通过；除 2x2 allreduce 外，性能阈值仍 FAIL | -| 原始 artifacts 已归档 | `/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts`，每个 case 有完整 `cmd/stdout/stderr/json` | -| artifacts 信号已分析 | `reports_multinode_nccl_artifact_signal_analysis_20260523.md`，确认所有 case 都走 IB/GDRDMA 和 4 条 400G HCA，未见 SHARP/CollNet | -| 多机六项 collective 已补测 | `reports_multinode_nccl_all_collectives_run_20260523.md`，2x8 下 6 项均正确性通过，allreduce/alltoall 按 PDF 阈值仍 FAIL | -| 六项 collective artifacts 已归档 | `reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md`，远端 tar 为 `reports/multinode_nccl_all_collectives_20260523_120144_artifacts.tar.gz` | -| 没看到硬错误 | 未见 discard、RoCE retrans、slow restart、packet sequence error 等增长 | -| 当前缺外部 NCCL 网络组件 | 未找到 `libnccl-net*.so*` / `libsharp*.so*`，未见 SHARP/HCOLL 包 | - -## PDF 目标与当前物理能力的冲突 - -PDF 2 机 16 GPU allreduce 目标是： - -```text -491.84 GB/s busbw -``` - -16 ranks allreduce 换算关系： - -```text -busbw = algbw * 1.875 -``` - -因此 PDF 目标反推： - -```text -491.84 / 1.875 = 262.31 GB/s algbw -``` - -当前每节点 4 条 400G rail 的理论单向原始带宽： - -```text -4 * 400Gb/s / 8 = 200 GB/s -``` - -所以如果 PDF 环境有更多有效 400G rail，或启用了 SHARP/NCCL net plugin，而当前环境没有，则当前节点不应直接按 PDF 2x8 目标判定。 - -## 决策树 - -### A. 如果验收坚持 PDF 原始阈值 - -必须先证明当前环境与 PDF 等价： - -1. 每节点是否有 8 条 400G IB rail 可用？ -2. PDF 命令中的 HCA 在参考环境里是否全部是 400G IB ACTIVE？ -3. PDF 环境是否启用了 SHARP / NCCL net plugin / HCOLL / UCX plugin？ -4. 当前跨 Leaf 交换网络策略是否与 PDF 环境一致？ - -如果任一答案是否定或未知，应先补齐硬件/软件/网络环境再复测，不应继续靠 NCCL 小参数追 `491.84/76.54 GB/s`。 - -### B. 如果验收按当前硬件形态重新定标 - -建议把当前 2x8 allreduce 的可解释目标按 4 x 400G rail 物理能力重新评估： - -- allreduce 当前 `353.85 GB/s busbw`，反推 `188.72 GB/s algbw`，接近 `200 GB/s` 单向原始上限。 -- alltoall 当前 `36.83 GB/s` 仍偏低，需要作为独立问题继续排查。 - -## 最新 PDF matrix 结果 - -| Topology | AllReduce | AllReduce Target | AllToAll | AllToAll Target | -|---|---:|---:|---:|---:| -| 2 nodes x 1 GPU | `47.29` | `48.90` | `24.85` | `27.25` | -| 2 nodes x 2 GPUs | `137.16` | `136.93` | `47.76` | `54.41` | -| 2 nodes x 4 GPUs | `335.07` | `335.48` | `72.74` | `73.73` | -| 2 nodes x 8 GPUs | `353.85` | `491.84` | `36.83` | `76.54` | - -所有 case 的 return code 为 `0`，NCCL `Out of bounds values` 为 `0 OK`。因此本轮 FAIL 是性能阈值失败，不是 NCCL 正确性或启动链路失败。 - -### C. 如果要继续优化 alltoall - -不要继续盲扫以下参数： - -- `NCCL_IB_QPS_PER_CONNECTION` -- `NCCL_IB_SPLIT_DATA_ON_QPS` -- `NCCL_NCHANNELS_PER_NET_PEER` -- `NCCL_BUFFSIZE` -- `NCCL_P2P_NET_CHUNKSIZE` -- `NCCL_IB_AR_THRESHOLD` - -已有 sweep 表明它们没有稳定正收益，部分明显负向。 - -优先做： - -1. 补齐并验证 `libnccl-net.so` / SHARP 环境。 -2. 让网络侧查跨 Leaf ECMP / adaptive routing / congestion control / credit wait。 -3. 用 `scripts/multinode_nccl_deep_diagnose.sh graph` 对比启用 plugin 前后的 NCCL graph。 -4. 如有等价 8 rail 节点，迁移同一脚本复测，确认 allreduce 物理上限是否抬升。 - -## 给网络/硬件/环境侧的问题 - -请直接确认下面这些问题： - -1. 这两台机器是否本来应该有 8 条 400G IB rail？如果是，为什么当前只有 4 条？ -2. `mlx5_4/5` 当前只有 100G，是配置、线缆、模块、交换机端口还是硬件限制？ -3. `mlx5_2/8` 为什么是 Ethernet 25G？是否预期不参与 IB NCCL？ -4. `mlx5_3/9` DOWN 是否符合预期？ -5. PDF 参考环境是否安装了 SHARP、HCOLL 或 NCCL net plugin？ -6. 当前交换机是否开启 adaptive routing，并且对 alltoall 这种多点到多点流量友好？ -7. 当前跨 Leaf 路径是否存在 ECMP hash 不均、PFC/credit wait、拥塞控制参数差异？ - -## 后续复跑命令 - -### 轻量检查 - -```bash -cd /root/test_gpu_scripts -bash scripts/multinode_nccl_deep_diagnose.sh preflight -``` - -### 单节点环境等价性快照 - -```bash -cd /root/test_gpu_scripts -bash scripts/nccl_environment_snapshot.sh reports/nccl_environment_snapshot_$(hostname)_$(date +%Y%m%d_%H%M%S).md -``` - -### 单节点 H100 原始 all 报告 - -```bash -cd /root/test_gpu_scripts -bash scripts/run_h100_single_node_all.sh -``` - -### 多机多卡 PDF 矩阵 - -```bash -cd /root/test_gpu_scripts -bash scripts/run_multinode_nccl_pdf_matrix.sh -``` - -### 多机多卡 2x8 六项 collective 补测 - -```bash -cd /root/test_gpu_scripts -bash scripts/run_multinode_nccl_all_collectives.sh -``` - -说明：这个入口用于补齐单机 `test all` 中已有、但多机 PDF matrix 还没覆盖的 NCCL collective。已知 PDF 2x8 阈值仍用于 `allreduce/alltoall`；新增的 `broadcast/reducescatter/allgather/sendrecv` 暂作为证据采集项，不强行套 PDF allreduce/alltoall 阈值。 - -### 完整深度诊断 - -```bash -cd /root/test_gpu_scripts -OUT_DIR=/root/test_gpu_scripts/reports/nccl_deep_diag_$(date +%Y%m%d_%H%M%S) \ - bash scripts/multinode_nccl_deep_diagnose.sh all -``` - -### 启用新 NCCL plugin / SHARP 后的最小复核 - -```bash -cd /root/test_gpu_scripts -OUT_DIR=/root/test_gpu_scripts/reports/nccl_deep_diag_plugin_check_$(date +%Y%m%d_%H%M%S) \ - bash scripts/multinode_nccl_deep_diagnose.sh graph -``` - -复核重点： - -- `plugin_missing` 是否消失或明显减少。 -- NCCL 日志是否出现外部 net plugin。 -- alltoall graph 中 `P2P/CUMEM`、`NET/IB/*/GDRDMA`、`channel_edge_lines` 是否变化。 -- alltoall busbw 是否突破 `36-37 GB/s` 平台。 - -## 关键文件 - -| 文件 | 用途 | -|---|---| -| `reports_h100_acceptance_current_status_20260523.md` | 当前 H100 验收总览，汇总单节点、多机 NCCL、跨节点 RDMA 和阻塞项 | -| `reports_multinode_nccl_diagnosis_20260523.md` | 总诊断报告 | -| `reports_multinode_nccl_pdf_matrix_20260523_112247.md` | 上一次多机多卡 PDF matrix 原始报告 | -| `reports_multinode_nccl_pdf_matrix_20260523_113803.md` | 最新带 artifacts 的多机多卡 PDF matrix 原始报告 | -| `reports_multinode_nccl_pdf_matrix_run_20260523.md` | 最新多机多卡 PDF matrix 中文摘要 | -| `reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md` | 最新 artifacts manifest 和 checksum | -| `reports_multinode_nccl_artifact_signal_analysis_20260523.md` | 最新 artifacts 的 IB/GDRDMA/HCA/plugin/SHARP 信号分析 | -| `reports_multinode_nccl_all_collectives_20260523_120144.md` | 最新多机多卡 2x8 六项 collective 原始报告 | -| `reports_multinode_nccl_all_collectives_run_20260523.md` | 最新多机多卡 2x8 六项 collective 中文摘要 | -| `reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md` | 最新多机多卡 2x8 六项 collective artifacts manifest 和 checksum | -| `reports_multinode_nccl_deep_diagnose_run_20260523.md` | 本轮深度复跑结果 | -| `reports_multinode_nccl_environment_gap_20260523.md` | 硬件/软件环境等价性缺口 | -| `reports_multinode_nccl_counter_probe_20260523.md` | RDMA rail/counter 证据 | -| `reports_multinode_nccl_alltoall_tuning_20260523.md` | alltoall 参数 sweep 和结论 | -| `docs/multinode_nccl_deep_diagnose_runbook.md` | 诊断脚本 runbook | -| `scripts/multinode_nccl_deep_diagnose.sh` | 可复跑诊断脚本 | -| `scripts/nccl_environment_snapshot.sh` | 单节点 HCA/plugin/topo 快照脚本 | -| `scripts/run_h100_single_node_all.sh` | 单节点原始 `test all` 报告入口 | -| `scripts/run_multinode_nccl_pdf_matrix.sh` | 多机多卡 PDF 矩阵报告入口；复跑时额外归档每个 case 的完整 `cmd/stdout/stderr/json` | -| `scripts/run_multinode_nccl_all_collectives.sh` | 多机多卡 2x8 六项 collective 补测入口；复跑时额外归档每个 case 的完整 `cmd/stdout/stderr/json` | -| `configs/multinode_nccl_nccl227_pdf_matrix.yaml` | 多机多卡 PDF 矩阵配置 | -| `configs/multinode_nccl_nccl227_all_collectives_2x8.yaml` | 多机多卡 2x8 六项 collective 补测配置 | - -## 当前建议 - -当前不建议继续把精力放在 NCCL 环境变量微调上。更高价值的动作是： - -1. 确认 PDF 参考环境的 rail 数量、速率和 SHARP/plugin 状态。 -2. 补齐或明确排除 NCCL net plugin / SHARP。 -3. 让网络侧针对 alltoall 多点通信模式查跨 Leaf 路径和拥塞策略。 -4. 如果硬件不等价，调整验收阈值或换等价节点重测。 diff --git a/reports_multinode_nccl_latest_index_20260523.md b/reports_multinode_nccl_latest_index_20260523.md deleted file mode 100644 index 129b50d..0000000 --- a/reports_multinode_nccl_latest_index_20260523.md +++ /dev/null @@ -1,265 +0,0 @@ -# 多节点 NCCL 最新索引 2026-05-23 - -## 当前状态 - -当前工作分支：`h100-acceptance-current` - -当前结论： - -- 2026-05-23 `11:38` 已完成带 artifacts 的正式多机多卡 PDF matrix 复跑，原始报告为 `reports_multinode_nccl_pdf_matrix_20260523_113803.md`，中文结论为 `reports_multinode_nccl_pdf_matrix_run_20260523.md`，artifact manifest 为 `reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md`。 -- 已补充 artifacts 信号分析：`reports_multinode_nccl_artifact_signal_analysis_20260523.md`。结论是所有 case 都走 `IB`，都使用 `mlx5_0,mlx5_1,mlx5_6,mlx5_7`，都有 GDRDMA 信号，但没有 SHARP/CollNet/外部 NCCL net plugin 证据。 -- 已补充并实跑多机多卡 2x8 六项 collective：`reports_multinode_nccl_all_collectives_run_20260523.md`。新增 `broadcast/reducescatter/allgather/sendrecv` 均 `returncode=0`、`wrong=0`、走 `IB/GDRDMA`；已知 PDF 阈值项 `allreduce/alltoall` 仍 FAIL。 -- 六项 collective 的完整 artifacts 已归档：`reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md`，远端 tar 为 `reports/multinode_nccl_all_collectives_20260523_120144_artifacts.tar.gz`。 -- 已补充当前验收状态总览：`reports_h100_acceptance_current_status_20260523.md`，把单节点、多机 NCCL、跨节点 RDMA、环境等价性和阻塞项合并到一份中文总表。 -- 已补充收尾检查清单：`reports_h100_acceptance_closure_checklist_20260523.md`，明确哪些工作可以阶段性交付、哪些验收门禁仍不能关闭。 -- 已补充网络/硬件/环境侧闭环请求：`reports_h100_network_hardware_escalation_request_20260523.md`，用于让责任侧回填 rail、plugin/SHARP、跨 Leaf 和新阈值口径。 -- 已补充交付包 manifest：`reports_h100_acceptance_delivery_manifest_20260523.md`，汇总主入口、脚本、远端 artifacts 和 checksum。 -- 2 机 1/2/4 GPU per node 档位已接近 PDF 参考值，但严格按阈值仍 FAIL。 -- 2 机 8 GPU 档位仍未达到 PDF 参考值： - - allreduce 实测 `353.85 GB/s busbw`，PDF 目标 `491.84 GB/s`。 - - alltoall 实测 `36.83 GB/s busbw`，PDF 目标 `76.54 GB/s`。 -- 当前 2 机 8 GPU 剩余差距不再像是旧 NCCL、GDR disabled、HCA 顺序、SSH/mpirun 或明显坏链路问题。 -- 当前更像是硬件 rail 数量与 PDF 不等价、NCCL net plugin / SHARP 缺失、或跨 Leaf alltoall 网络/图策略问题。 - -## 先看这三份 - -| 顺序 | 文件 | 用途 | -|---:|---|---| -| 1 | `reports_h100_acceptance_current_status_20260523.md` | 当前 H100 验收总览，汇总单节点、多机 NCCL、跨节点 RDMA 和阻塞项 | -| 2 | `reports_h100_acceptance_closure_checklist_20260523.md` | 收尾检查清单：可交付项、未关闭门禁、最短收尾路径 | -| 3 | `reports_h100_acceptance_delivery_manifest_20260523.md` | 交付包 manifest：入口、脚本、远端 artifacts、checksum | -| 4 | `reports_h100_network_hardware_escalation_request_20260523.md` | 给网络/硬件/环境侧的闭环请求和回填表 | -| 5 | `reports_multinode_nccl_handoff_plan_20260523.md` | 给网络/硬件/环境侧的交接计划，包含决策树、要问的问题和复跑命令 | -| 6 | `reports_multinode_nccl_environment_gap_20260523.md` | 说明当前环境为什么不能证明与 PDF 等价，重点是 4 x 400G rail 和缺少 NCCL net plugin / SHARP | -| 7 | `reports_multinode_nccl_artifact_signal_analysis_20260523.md` | 最新 artifacts 信号分析，确认 IB/GDRDMA/HCA 使用情况和 plugin/SHARP 缺口 | -| 8 | `reports_multinode_nccl_all_collectives_run_20260523.md` | 多机多卡 2x8 六项 collective 补测结果，补齐单机 test all 的 NCCL 覆盖面 | -| 9 | `reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md` | 多机多卡 2x8 六项 collective artifacts manifest 和 checksum | -| 10 | `reports_multinode_nccl_pdf_matrix_run_20260523.md` | 最新正式多机多卡 PDF matrix 结果摘要 | -| 11 | `reports_multinode_nccl_deep_diagnose_run_20260523.md` | 本轮完整深度诊断复跑结果，包含 counter、GRAPH、PXN sweep | - -## 关键脚本 - -| 文件 | 用途 | -|---|---| -| `scripts/multinode_nccl_deep_diagnose.sh` | 可复跑的多节点 NCCL 深度诊断脚本 | -| `scripts/nccl_environment_snapshot.sh` | 单节点 NCCL/RDMA 环境等价性快照脚本，不启动 NCCL workload | -| `scripts/run_h100_single_node_all.sh` | 单节点 H100 `test all` 原始报告入口，默认同时采环境快照 | -| `scripts/run_multinode_nccl_pdf_matrix.sh` | 多机多卡 PDF 矩阵入口，跑 2 机 x 1/2/4/8 GPU per node 的 allreduce/alltoall，并归档每个 case 的 command/stdout/stderr/parsed JSON | -| `scripts/run_multinode_nccl_all_collectives.sh` | 多机多卡 2x8 六项 collective 补测入口，跑 allreduce/alltoall/broadcast/reducescatter/allgather/sendrecv，并归档每个 case | -| `configs/multinode_nccl_nccl227_pdf_matrix.yaml` | 多机多卡 PDF 矩阵配置，固定 NCCL 2.27.7 和 `/data/nccl-tests-latest/build` | -| `configs/multinode_nccl_nccl227_all_collectives_2x8.yaml` | 多机多卡 2x8 六项 collective 补测配置，allreduce/alltoall 保留 PDF 阈值，新增 4 项暂按证据采集 | -| `docs/multinode_nccl_deep_diagnose_runbook.md` | 诊断脚本中文 runbook | - -多机多卡 PDF 矩阵： - -```bash -cd /root/test_gpu_scripts -bash scripts/run_multinode_nccl_pdf_matrix.sh -``` - -多机多卡 2x8 六项 collective 补测： - -```bash -cd /root/test_gpu_scripts -bash scripts/run_multinode_nccl_all_collectives.sh -``` - -单节点 H100 原始 all 报告： - -```bash -cd /root/test_gpu_scripts -bash scripts/run_h100_single_node_all.sh -``` - -推荐先跑轻量检查： - -```bash -cd /root/test_gpu_scripts -bash scripts/multinode_nccl_deep_diagnose.sh preflight -``` - -采集单节点环境快照： - -```bash -cd /root/test_gpu_scripts -bash scripts/nccl_environment_snapshot.sh reports/nccl_environment_snapshot_$(hostname)_$(date +%Y%m%d_%H%M%S).md -``` - -完整复跑： - -```bash -cd /root/test_gpu_scripts -OUT_DIR=/root/test_gpu_scripts/reports/nccl_deep_diag_$(date +%Y%m%d_%H%M%S) \ - bash scripts/multinode_nccl_deep_diagnose.sh all -``` - -启用 NCCL plugin / SHARP 后的最小复核： - -```bash -cd /root/test_gpu_scripts -OUT_DIR=/root/test_gpu_scripts/reports/nccl_deep_diag_plugin_check_$(date +%Y%m%d_%H%M%S) \ - bash scripts/multinode_nccl_deep_diagnose.sh graph -``` - -## 远端机器上的最新同步文件 - -三份关键报告已经同步到两台节点： - -```text -/root/test_gpu_scripts/reports_multinode_nccl_handoff_plan_20260523.md -/root/test_gpu_scripts/reports_h100_acceptance_current_status_20260523.md -/root/test_gpu_scripts/reports_h100_acceptance_closure_checklist_20260523.md -/root/test_gpu_scripts/reports_h100_acceptance_delivery_manifest_20260523.md -/root/test_gpu_scripts/reports_h100_network_hardware_escalation_request_20260523.md -/root/test_gpu_scripts/reports_multinode_nccl_environment_gap_20260523.md -/root/test_gpu_scripts/reports_multinode_nccl_artifact_signal_analysis_20260523.md -/root/test_gpu_scripts/reports_multinode_nccl_all_collectives_run_20260523.md -/root/test_gpu_scripts/reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md -/root/test_gpu_scripts/reports_multinode_nccl_deep_diagnose_run_20260523.md -``` - -最新完整诊断产物目录在 `aikubeworker0012`： - -```text -/root/test_gpu_scripts/reports/nccl_deep_diag_20260523_103932 -``` - -该目录包含： - -- `preflight.txt` -- `allreduce_counter/` -- `alltoall_pxn_counter/` -- `graph/` -- `pxn_sweep/` - -最新单节点环境快照： - -```text -aikubeworker0012: /root/test_gpu_scripts/reports/nccl_environment_snapshot_aikubeworker0012_20260523_111142.md -aikubeworker0016: /root/test_gpu_scripts/reports/nccl_environment_snapshot_aikubeworker0016_20260523_111143.md -``` - -最新多机多卡 PDF matrix： - -```text -aikubeworker0012: /root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803.md -artifacts: /root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts -artifacts tar: /root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts.tar.gz -local copy: reports_multinode_nccl_pdf_matrix_20260523_113803.md -summary: reports_multinode_nccl_pdf_matrix_run_20260523.md -manifest: reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md -``` - -最新多机多卡 2x8 六项 collective 补测： - -```text -aikubeworker0012: /root/test_gpu_scripts/reports/multinode_nccl_all_collectives_20260523_120144.md -artifacts: /root/test_gpu_scripts/reports/multinode_nccl_all_collectives_20260523_120144_artifacts -artifacts tar: /root/test_gpu_scripts/reports/multinode_nccl_all_collectives_20260523_120144_artifacts.tar.gz -local copy: reports_multinode_nccl_all_collectives_20260523_120144.md -summary: reports_multinode_nccl_all_collectives_run_20260523.md -manifest: reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md -``` - -下一次用 `scripts/run_multinode_nccl_pdf_matrix.sh` 复跑时，还会生成： - -```text -/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_YYYYMMDD_HHMMSS_artifacts/ -``` - -目录内按 case 保存完整 `cmd/stdout/stderr/json`，用于给网络/硬件侧复核原始 NCCL 输出。 - -下一次用 `scripts/run_multinode_nccl_all_collectives.sh` 补测时，还会生成： - -```text -/root/test_gpu_scripts/reports/multinode_nccl_all_collectives_YYYYMMDD_HHMMSS_artifacts/ -``` - -目录内按 6 个 collective 保存完整 `cmd/stdout/stderr/json`。该入口用于补齐单节点 `test all` 中已有、但多机 PDF matrix 未覆盖的 `broadcast/reducescatter/allgather/sendrecv` 证据；已知 PDF 2x8 阈值仍用于 `allreduce/alltoall`。 - -## 当前证据摘要 - -### HCA / rail - -两台节点当前有效 400G IB rail 一致： - -```text -mlx5_0, mlx5_1, mlx5_6, mlx5_7 -``` - -非等价 HCA： - -```text -mlx5_4, mlx5_5: 100G InfiniBand -mlx5_2, mlx5_8: 25G Ethernet -mlx5_3, mlx5_9: DOWN -``` - -因此当前每节点可用于 NCCL 的 400G rail 是 4 条，理论单向原始带宽约 `200 GB/s`。 - -PDF allreduce 目标 `491.84 GB/s busbw` 反推 `262.31 GB/s algbw`，超过当前 4 x 400G rail 的理论单向带宽。 - -### NCCL / plugin - -当前两台节点没有找到： - -```text -libnccl-net*.so* -libsharp*.so* -``` - -也没有看到 SHARP/HCOLL 包。NCCL GRAPH 日志显示 `plugin_missing=16`，当前走 internal IB plugin。 - -### 深度诊断 - -正式 PDF matrix 复跑： - -| Topology | AllReduce | AllReduce Target | AllToAll | AllToAll Target | -|---|---:|---:|---:|---:| -| 2 nodes x 1 GPU | `47.29` | `48.90` | `24.85` | `27.25` | -| 2 nodes x 2 GPUs | `137.16` | `136.93` | `47.76` | `54.41` | -| 2 nodes x 4 GPUs | `335.07` | `335.48` | `72.74` | `73.73` | -| 2 nodes x 8 GPUs | `353.85` | `491.84` | `36.83` | `76.54` | - -本轮完整复跑： - -| 项目 | 结果 | -|---|---:| -| allreduce 16G | `354.025 GB/s` | -| graph allreduce 16G | `354.224 GB/s` | -| alltoall + PXN disabled 16G | `36.9377 GB/s` | -| graph alltoall + PXN disabled 16G | `37.14 GB/s` | - -PXN disabled sweep 未发现有效参数： - -- `channels16`、`buff8m`、`p2pchunk4m`、`ar0` 只有小幅噪声级波动。 -- `qps4_split1`、`qps8_split1`、`netpeer8` 明显负向。 - -## 历史/支撑报告 - -| 文件 | 说明 | -|---|---| -| `reports_multinode_nccl_diagnosis_20260523.md` | 长版总诊断，包含从旧 NCCL/GDR disabled 到 PDF 矩阵对齐的全过程 | -| `reports_h100_acceptance_current_status_20260523.md` | 当前 H100 验收总览，汇总单节点、多机 NCCL、跨节点 RDMA 和阻塞项 | -| `reports_multinode_nccl_pdf_matrix_nccl227.md` | 按 PDF 矩阵跑出的正式 raw report | -| `reports_multinode_nccl_pdf_matrix_20260523_112247.md` | 上一次正式 PDF matrix 原始报告 | -| `reports_multinode_nccl_pdf_matrix_20260523_113803.md` | 最新带 artifacts 的正式 PDF matrix 原始报告 | -| `reports_multinode_nccl_pdf_matrix_run_20260523.md` | 最新正式 PDF matrix 中文摘要 | -| `reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md` | 最新 artifacts manifest 和 checksum | -| `reports_multinode_nccl_artifact_signal_analysis_20260523.md` | 最新 artifacts 的 IB/GDRDMA/HCA/plugin/SHARP 信号分析 | -| `reports_multinode_nccl_all_collectives_20260523_120144.md` | 最新多机多卡 2x8 六项 collective 原始报告 | -| `reports_multinode_nccl_all_collectives_run_20260523.md` | 最新多机多卡 2x8 六项 collective 中文摘要 | -| `reports_multinode_nccl_all_collectives_artifacts_manifest_20260523_120144.md` | 最新多机多卡 2x8 六项 collective artifacts manifest 和 checksum | -| `reports_multinode_nccl_counter_probe_20260523.md` | RDMA rail 和 counter 证据 | -| `reports_multinode_nccl_alltoall_tuning_20260523.md` | alltoall PXN 和参数 sweep 结论 | -| `reports_rdma_single_node_summary.md` | 单节点 RDMA/HCA 速率摘要 | -| `docs/multinode_nccl_concepts.md` | NCCL/RDMA 概念解释 | - -## 给下一位接手人的路线 - -1. 先读 `reports_h100_acceptance_current_status_20260523.md`。 -2. 再读 `reports_multinode_nccl_handoff_plan_20260523.md`。 -3. 用 `reports_multinode_nccl_environment_gap_20260523.md` 和硬件/网络侧确认当前节点是否应具备 8 条 400G rail。 -4. 如果硬件不等价，调整验收口径或换等价节点复测。 -5. 如果硬件确认等价，先补齐 NCCL net plugin / SHARP，再跑 `scripts/multinode_nccl_deep_diagnose.sh graph` 对比 plugin 前后。 -6. alltoall 继续排查时优先找网络路径/ECMP/adaptive routing/拥塞策略，不建议继续盲扫 NCCL 小参数。 diff --git a/reports_multinode_nccl_pdf_matrix_20260523_112247.md b/reports_multinode_nccl_pdf_matrix_20260523_112247.md deleted file mode 100644 index 8d07aef..0000000 --- a/reports_multinode_nccl_pdf_matrix_20260523_112247.md +++ /dev/null @@ -1,75 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T11:26:21.306224 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Failed or unverified items: -- Multi-node NCCL: FAIL - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: cross-leaf-pdf-matrix-nccl-2.27.7 - -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS - -### Multi-node NCCL allreduce - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | - | 47.15 GB/s | 16G | 47.18 GB/s | >= 48.90 GB/s | FAIL | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | - | 136.62 GB/s | 16G | 136.67 GB/s | >= 136.93 GB/s | FAIL | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0,1,4,5 | 335.19 GB/s | 16G | 334.85 GB/s | >= 335.48 GB/s | FAIL | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | - | 354.56 GB/s | 16G | 354.21 GB/s | >= 491.84 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | 0 | ranks 2 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0016:1321368:1321509 [0] NCCL INFO comm 0x56428b645570 rank 1 nranks 2 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 47.1841 # | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | 0 | ranks 4 cudaDev 1 busId 2a000 - Destroy COMPLETE aikubeworker0012:2199872:2199936 [0] NCCL INFO comm 0x561da4512280 rank 0 nranks 4 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 136.668 # | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0 | ranks 8 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0016:1321707:1321805 [0] NCCL INFO comm 0x562bad8777a0 rank 4 nranks 8 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 334.846 # | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | 0 | nks 16 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0016:1321873:1322056 [0] NCCL INFO comm 0x55ba6708f500 rank 8 nranks 16 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 354.211 # | - -### Multi-node NCCL alltoall - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | - | 24.85 GB/s | 16G | 24.92 GB/s | >= 27.25 GB/s | FAIL | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | - | 47.71 GB/s | 16G | 47.93 GB/s | >= 54.41 GB/s | FAIL | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0,1,4,5 | 72.63 GB/s | 16G | 72.67 GB/s | >= 73.73 GB/s | FAIL | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | - | 36.82 GB/s | 16G | 36.86 GB/s | >= 76.54 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | 0 | nranks 2 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0016:1322113:1322193 [0] NCCL INFO comm 0x55b760411150 rank 1 nranks 2 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 24.917 # | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | 0 | ker0012:2200344:2200469 [1] NCCL INFO comm 0x55efef439da0 rank 1 nranks 4 cudaDev 1 busId 2a000 - Destroy COMPLETE aikubeworker0016:1322250:1322338 [1] NCCL INFO comm 0x558ecf546380 rank 3 nranks 4 cudaDev 1 busId 2a000 - Destroy COMPLETE | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0 | ranks 8 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0012:2200479:2200573 [0] NCCL INFO comm 0x55db60daef30 rank 0 nranks 8 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 72.6664 # | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | 0 | r0012:2200587:2200767 [5] NCCL INFO comm 0x5556a6f71620 rank 5 nranks 16 cudaDev 5 busId ab000 - Destroy COMPLETE aikubeworker0012:2200588:2200772 [6] NCCL INFO comm 0x5585a1623170 rank 6 nranks 16 cudaDev 6 busId ba000 - Destroy COMPLETE | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* diff --git a/reports_multinode_nccl_pdf_matrix_20260523_113803.md b/reports_multinode_nccl_pdf_matrix_20260523_113803.md deleted file mode 100644 index 06b509e..0000000 --- a/reports_multinode_nccl_pdf_matrix_20260523_113803.md +++ /dev/null @@ -1,75 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T11:41:35.567886 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Failed or unverified items: -- Multi-node NCCL: FAIL - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: cross-leaf-pdf-matrix-nccl-2.27.7 - -- **Artifacts:** `/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts` -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS - -### Multi-node NCCL allreduce - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | - | 47.29 GB/s | 16G | 47.26 GB/s | >= 48.90 GB/s | FAIL | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | - | 137.16 GB/s | 16G | 137.13 GB/s | >= 136.93 GB/s | PASS | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0,1,4,5 | 335.07 GB/s | 16G | 335.02 GB/s | >= 335.48 GB/s | FAIL | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | - | 353.85 GB/s | 16G | 353.85 GB/s | >= 491.84 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | 0 | ranks 2 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0012:2203142:2203200 [0] NCCL INFO comm 0x55e463572510 rank 0 nranks 2 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 47.2628 # | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0 | ranks 8 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0012:2203280:2203363 [0] NCCL INFO comm 0x55e2f3808c60 rank 0 nranks 8 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 335.021 # | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | 0 | nks 16 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0012:2203376:2203528 [0] NCCL INFO comm 0x55a5166a30c0 rank 0 nranks 16 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 353.854 # | - -### Multi-node NCCL alltoall - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | - | 24.85 GB/s | 16G | 24.90 GB/s | >= 27.25 GB/s | FAIL | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | - | 47.76 GB/s | 16G | 47.98 GB/s | >= 54.41 GB/s | FAIL | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0,1,4,5 | 72.74 GB/s | 16G | 72.80 GB/s | >= 73.73 GB/s | FAIL | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | - | 36.83 GB/s | 16G | 36.85 GB/s | >= 76.54 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | 0 | ranks 2 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0012:2203543:2203602 [0] NCCL INFO comm 0x55af2a804ba0 rank 0 nranks 2 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 24.9006 # | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | 0 | ker0012:2203610:2203792 [1] NCCL INFO comm 0x55e99a564500 rank 1 nranks 4 cudaDev 1 busId 2a000 - Destroy COMPLETE aikubeworker0016:1325607:1325696 [0] NCCL INFO comm 0x55eaaa7389c0 rank 2 nranks 4 cudaDev 0 busId 18000 - Destroy COMPLETE | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0 | ranks 8 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0016:1325765:1325869 [3] NCCL INFO comm 0x55cb0f1c9c10 rank 7 nranks 8 cudaDev 3 busId ab000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 72.7968 # | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | 0 | 0016:1325927:1326140 [2] NCCL INFO comm 0x5627d2adee20 rank 10 nranks 16 cudaDev 2 busId 3a000 - Destroy COMPLETE aikubeworker0016:1325926:1326135 [1] NCCL INFO comm 0x55c00c344ea0 rank 9 nranks 16 cudaDev 1 busId 2a000 - Destroy COMPLETE | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md b/reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md deleted file mode 100644 index a398123..0000000 --- a/reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md +++ /dev/null @@ -1,33 +0,0 @@ -# 多机多卡 NCCL PDF Matrix Artifacts Manifest 2026-05-23 - -- Remote report: `reports/multinode_nccl_pdf_matrix_20260523_113803.md` -- Remote artifact dir: `reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts` -- Remote artifact tar: `reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts.tar.gz` -- Case count: `8` -- Artifact files: `32` - -## Case Summary - -| Case | Peak Bus BW | Avg Bus BW | Threshold | Wrong | Return Code | Status | -|---|---:|---:|---:|---:|---:|---| -| `allreduce_2x1_2_nodes_x_1_GPU_PDF_2_machines_2_GPUs` | 47.29 | 47.26 | 48.90 | 0 | 0 | FAIL | -| `allreduce_2x2_2_nodes_x_2_GPUs_PDF_2_machines_4_GPUs` | 137.16 | 137.13 | 136.93 | 0 | 0 | PASS | -| `allreduce_2x4_2_nodes_x_4_GPUs_PDF_2_machines_8_GPUs` | 335.07 | 335.02 | 335.48 | 0 | 0 | FAIL | -| `allreduce_2x8_2_nodes_x_8_GPUs_PDF_2_machines_16_GPUs` | 353.85 | 353.85 | 491.84 | 0 | 0 | FAIL | -| `alltoall_2x1_2_nodes_x_1_GPU_PDF_2_machines_2_GPUs` | 24.85 | 24.90 | 27.25 | 0 | 0 | FAIL | -| `alltoall_2x2_2_nodes_x_2_GPUs_PDF_2_machines_4_GPUs` | 47.76 | 47.98 | 54.41 | 0 | 0 | FAIL | -| `alltoall_2x4_2_nodes_x_4_GPUs_PDF_2_machines_8_GPUs` | 72.74 | 72.80 | 73.73 | 0 | 0 | FAIL | -| `alltoall_2x8_2_nodes_x_8_GPUs_PDF_2_machines_16_GPUs` | 36.83 | 36.85 | 76.54 | 0 | 0 | FAIL | - -## Checksums - -```text -682ac637460472d464a0d56ccc0f3335ed7f79a270157a403ebec23b8d9feceb reports/multinode_nccl_pdf_matrix_20260523_113803.md -7371fcaf7269f92eb1544e5e63573ebf77f4ae38f668b5b22169ca86e6d603ee reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts.tar.gz -``` - -Per-file artifact checksums are on the remote node at: - -```text -reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts.sha256 -``` diff --git a/reports_multinode_nccl_pdf_matrix_nccl227.md b/reports_multinode_nccl_pdf_matrix_nccl227.md deleted file mode 100644 index c04d023..0000000 --- a/reports_multinode_nccl_pdf_matrix_nccl227.md +++ /dev/null @@ -1,84 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T08:58:19.911230 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- DCGM -- Training - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: cross-leaf-pdf-matrix-nccl-2.27.7 - -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS - -### Multi-node NCCL allreduce - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | - | 47.26 GB/s | 16G | 47.19 GB/s | >= 49 GB/s | FAIL | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | - | 136.36 GB/s | 16G | 136.69 GB/s | >= 137 GB/s | FAIL | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0,1,4,5 | 333.23 GB/s | 16G | 333.45 GB/s | >= 335 GB/s | FAIL | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | - | 353.47 GB/s | 16G | 353.86 GB/s | >= 492 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | 0 | TE aikubeworker0012:2165982:2166060 [0] NCCL INFO comm 0x55d452f2df80 rank 0 nranks 2 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 47.189 # # Collective test concluded: all_reduce_perf # | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | 0 | ker0016:1221425:1222411 [0] NCCL INFO comm 0x56437384f040 rank 2 nranks 4 cudaDev 0 busId 18000 - Destroy COMPLETE aikubeworker0016:1221427:1222412 [1] NCCL INFO comm 0x55ab9313f950 rank 3 nranks 4 cudaDev 1 busId 2a000 - Destroy COMPLETE | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0 | E aikubeworker0012:2166160:2166257 [0] NCCL INFO comm 0x557243829d50 rank 0 nranks 8 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 333.449 # # Collective test concluded: all_reduce_perf # | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | 0 | r0012:2166272:2166442 [5] NCCL INFO comm 0x55721e270960 rank 5 nranks 16 cudaDev 5 busId ab000 - Destroy COMPLETE aikubeworker0012:2166268:2166447 [1] NCCL INFO comm 0x5644fafd24e0 rank 1 nranks 16 cudaDev 1 busId 2a000 - Destroy COMPLETE | - -### Multi-node NCCL alltoall - -| Topology | CUDA Visible Devices | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|----------------------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | - | 24.87 GB/s | 16G | 24.93 GB/s | >= 27 GB/s | FAIL | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | - | 47.69 GB/s | 16G | 47.93 GB/s | >= 54 GB/s | FAIL | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0,1,4,5 | 72.82 GB/s | 16G | 72.87 GB/s | >= 74 GB/s | FAIL | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | - | 36.70 GB/s | 16G | 36.74 GB/s | >= 77 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 1 GPU (PDF 2 machines 2 GPUs) | 0 | ETE aikubeworker0012:2166458:2166534 [0] NCCL INFO comm 0x5603baefb150 rank 0 nranks 2 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 24.9304 # # Collective test concluded: alltoall_perf # | -| 2 nodes x 2 GPUs (PDF 2 machines 4 GPUs) | 0 | ETE aikubeworker0012:2166543:2166743 [0] NCCL INFO comm 0x5569d31d4f50 rank 0 nranks 4 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 47.9258 # # Collective test concluded: alltoall_perf # | -| 2 nodes x 4 GPUs (PDF 2 machines 8 GPUs) | 0 | ker0016:1227342:1228382 [1] NCCL INFO comm 0x55cdec231780 rank 5 nranks 8 cudaDev 1 busId 2a000 - Destroy COMPLETE aikubeworker0016:1227344:1228381 [3] NCCL INFO comm 0x563c7ed39680 rank 7 nranks 8 cudaDev 3 busId ab000 - Destroy COMPLETE | -| 2 nodes x 8 GPUs (PDF 2 machines 16 GPUs) | 0 | TE aikubeworker0012:2166925:2167127 [7] NCCL INFO comm 0x560553b91250 rank 7 nranks 16 cudaDev 7 busId db000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 36.7382 # # Collective test concluded: alltoall_perf # | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_multinode_nccl_pdf_matrix_run_20260523.md b/reports_multinode_nccl_pdf_matrix_run_20260523.md deleted file mode 100644 index 0006ea7..0000000 --- a/reports_multinode_nccl_pdf_matrix_run_20260523.md +++ /dev/null @@ -1,67 +0,0 @@ -# 多机多卡 NCCL PDF 矩阵实测 2026-05-23 - -执行节点：`aikubeworker0012` - -对端节点：`aikubeworker0016` - -原始报告：`reports_multinode_nccl_pdf_matrix_20260523_113803.md` - -远端报告：`/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803.md` - -远端 artifacts：`/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts` - -远端 artifacts tar：`/root/test_gpu_scripts/reports/multinode_nccl_pdf_matrix_20260523_113803_artifacts.tar.gz` - -Artifacts manifest：`reports_multinode_nccl_pdf_matrix_artifacts_manifest_20260523_113803.md` - -执行命令： - -```bash -cd /root/test_gpu_scripts -bash scripts/run_multinode_nccl_pdf_matrix.sh -``` - -## 结论 - -本轮正式矩阵已跑通，`mpirun`、SSH、`nccl-tests`、GDRDMA、4 条 400G HCA 都可用；失败不是启动失败或功能错误，而是 bus bandwidth 未达到 PDF 阈值。 - -所有 case 的 return code 都是 `0`，`Out of bounds values` 为 `0 OK`，说明 NCCL 正确性没有报错。FAIL 来自性能阈值。 - -## Preflight - -| 项目 | 结果 | -|---|---| -| OpenMPI | PASS，`/usr/mpi/gcc/openmpi-4.1.9a1/bin/mpirun` | -| all_reduce_perf | PASS，`/data/nccl-tests-latest/build/all_reduce_perf` | -| alltoall_perf | PASS，`/data/nccl-tests-latest/build/alltoall_perf` | -| SSH 172.72.8.12 | PASS | -| SSH 172.72.8.16 | PASS | -| HCA | 两端 `mlx5_0,mlx5_1,mlx5_6,mlx5_7` 均为 `400 Gb/sec (4X NDR)` ACTIVE | -| NCCL network | IB | -| GPU Direct RDMA | ENABLED | - -## AllReduce - -| Topology | Peak Bus BW | Avg Bus BW | PDF Threshold | Gap | Status | -|---|---:|---:|---:|---:|---| -| 2 nodes x 1 GPU | 47.29 GB/s | 47.26 GB/s | >= 48.90 GB/s | -1.61 GB/s | FAIL | -| 2 nodes x 2 GPUs | 137.16 GB/s | 137.13 GB/s | >= 136.93 GB/s | +0.23 GB/s | PASS | -| 2 nodes x 4 GPUs | 335.07 GB/s | 335.02 GB/s | >= 335.48 GB/s | -0.41 GB/s | FAIL | -| 2 nodes x 8 GPUs | 353.85 GB/s | 353.85 GB/s | >= 491.84 GB/s | -137.99 GB/s | FAIL | - -## AllToAll - -| Topology | Peak Bus BW | Avg Bus BW | PDF Threshold | Gap | Status | -|---|---:|---:|---:|---:|---| -| 2 nodes x 1 GPU | 24.85 GB/s | 24.90 GB/s | >= 27.25 GB/s | -2.40 GB/s | FAIL | -| 2 nodes x 2 GPUs | 47.76 GB/s | 47.98 GB/s | >= 54.41 GB/s | -6.65 GB/s | FAIL | -| 2 nodes x 4 GPUs | 72.74 GB/s | 72.80 GB/s | >= 73.73 GB/s | -0.99 GB/s | FAIL | -| 2 nodes x 8 GPUs | 36.83 GB/s | 36.85 GB/s | >= 76.54 GB/s | -39.71 GB/s | FAIL | - -## 判断 - -1. 2x2 的 AllReduce 本次过线，2x4 的 AllReduce 非常接近 PDF 阈值，差 `0.41 GB/s`。 -2. 2x4 的 AllToAll 也接近阈值，差 `0.99 GB/s`。 -3. 2x8 是主要问题：AllReduce 只有 `353.85 / 491.84`，AllToAll 只有 `36.83 / 76.54`。 -4. 当前环境已经确认只有 4 条 400G IB rail 参与 NCCL，且没有发现外部 NCCL net plugin / SHARP；这仍是解释 2x8 目标不可达或严重掉速的最强证据。 -5. 本轮没有看到 GDR disabled 或 HCA 不可用，所以下一步不应继续纠结 SSH/mpirun/nccl-tests 启动链路，而应对齐 PDF 参考环境的 rail 数量、net plugin/SHARP、交换机跨 Leaf 策略。 diff --git a/reports_multinode_nccl_smoke_256m_aikubeworker0012.json b/reports_multinode_nccl_smoke_256m_aikubeworker0012.json deleted file mode 100644 index 72c30ce..0000000 --- a/reports_multinode_nccl_smoke_256m_aikubeworker0012.json +++ /dev/null @@ -1,439 +0,0 @@ -{ - "multinode_nccl": { - "passed": false, - "source": "nccl-tests-mpirun", - "mode": "sweep", - "hosts": [ - { - "name": "nccl-gpu-1", - "addr": "172.72.8.12", - "slots": 8 - }, - { - "name": "nccl-gpu-2", - "addr": "172.72.8.16", - "slots": 8 - } - ], - "preflight": { - "checks": [ - { - "name": "mpirun", - "status": "PASS", - "detail": "/usr/mpi/gcc/openmpi-4.1.9a1/bin/mpirun" - }, - { - "name": "hosts", - "status": "PASS", - "detail": "2 configured" - }, - { - "name": "all_reduce_perf", - "status": "PASS", - "detail": "/opt/gpu-test-tools/nccl-tests/build/all_reduce_perf" - }, - { - "name": "alltoall_perf", - "status": "PASS", - "detail": "/opt/gpu-test-tools/nccl-tests/build/alltoall_perf" - }, - { - "name": "ssh 172.72.8.12", - "status": "WARN", - "detail": "Host key verification failed." - }, - { - "name": "ssh 172.72.8.16", - "status": "PASS", - "detail": "aikubeworker0016" - } - ], - "passed": true - }, - "tests": { - "allreduce": { - "binary": "/opt/gpu-test-tools/nccl-tests/build/all_reduce_perf", - "topologies": [ - { - "label": "2 nodes x 8 GPUs", - "nodes": 2, - "gpus_per_node": 8, - "ranks": 16, - "hosts": [ - { - "name": "nccl-gpu-1", - "addr": "172.72.8.12", - "slots": 8 - }, - { - "name": "nccl-gpu-2", - "addr": "172.72.8.16", - "slots": 8 - } - ], - "command": "/usr/mpi/gcc/openmpi-4.1.9a1/bin/mpirun --allow-run-as-root --mca btl_openib_warn_no_device_params_found 0 --mca btl_tcp_if_include bond0 -H 172.72.8.12:8,172.72.8.16:8 --map-by ppr:8:node -np 16 -x NCCL_DEBUG=WARN -x NCCL_SOCKET_IFNAME=bond0 -x NCCL_IB_GID_INDEX=3 -x NCCL_IB_SL=5 -x NCCL_IB_TC=136 -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 -x NCCL_IB_TIMEOUT=22 -x NCCL_IB_QPS_PER_CONNECTION=4 -x NCCL_MIN_NCHANNELS=4 -x NCCL_NET_PLUGIN=none -x NCCL_NVLS_ENABLE=1 -x NCCL_IB_SPLIT_DATA_ON_QPS=1 -x LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi-4.1.9a1/lib:/root/gpu-test-venv/lib/python3.10/site-packages/nvidia/nccl/lib:/usr/local/cuda-12.4/targets/x86_64-linux/lib /opt/gpu-test-tools/nccl-tests/build/all_reduce_perf -b 1k -e 256M -g 1 -f 2 -w 2", - "returncode": 0, - "status": "FAIL", - "peak_busbw_gbps": 39.32, - "peak_algbw_gbps": 20.97, - "peak_size": "4M", - "avg_busbw_gbps": 9.1, - "min_required_gbps": 100.0, - "wrong_count": 0, - "by_size": [ - { - "size_bytes": 1024, - "size": "1K", - "time_us": 80.32, - "algbw_gbps": 0.01, - "busbw_gbps": 0.02, - "wrong": 0 - }, - { - "size_bytes": 2048, - "size": "2K", - "time_us": 35.79, - "algbw_gbps": 0.06, - "busbw_gbps": 0.11, - "wrong": 0 - }, - { - "size_bytes": 4096, - "size": "4K", - "time_us": 37.49, - "algbw_gbps": 0.11, - "busbw_gbps": 0.2, - "wrong": 0 - }, - { - "size_bytes": 8192, - "size": "8K", - "time_us": 40.32, - "algbw_gbps": 0.2, - "busbw_gbps": 0.38, - "wrong": 0 - }, - { - "size_bytes": 16384, - "size": "16K", - "time_us": 43.04, - "algbw_gbps": 0.38, - "busbw_gbps": 0.71, - "wrong": 0 - }, - { - "size_bytes": 32768, - "size": "32K", - "time_us": 43.32, - "algbw_gbps": 0.76, - "busbw_gbps": 1.42, - "wrong": 0 - }, - { - "size_bytes": 65536, - "size": "64K", - "time_us": 47.45, - "algbw_gbps": 1.38, - "busbw_gbps": 2.59, - "wrong": 0 - }, - { - "size_bytes": 131072, - "size": "128K", - "time_us": 89.3, - "algbw_gbps": 1.47, - "busbw_gbps": 2.75, - "wrong": 0 - }, - { - "size_bytes": 262144, - "size": "256K", - "time_us": 165.38, - "algbw_gbps": 1.59, - "busbw_gbps": 2.97, - "wrong": 0 - }, - { - "size_bytes": 524288, - "size": "512K", - "time_us": 4292.69, - "algbw_gbps": 0.12, - "busbw_gbps": 0.23, - "wrong": 0 - }, - { - "size_bytes": 1048576, - "size": "1M", - "time_us": 139.29, - "algbw_gbps": 7.53, - "busbw_gbps": 14.12, - "wrong": 0 - }, - { - "size_bytes": 2097152, - "size": "2M", - "time_us": 4195.12, - "algbw_gbps": 0.5, - "busbw_gbps": 0.94, - "wrong": 0 - }, - { - "size_bytes": 4194304, - "size": "4M", - "time_us": 199.99, - "algbw_gbps": 20.97, - "busbw_gbps": 39.32, - "wrong": 0 - }, - { - "size_bytes": 8388608, - "size": "8M", - "time_us": 6159.0, - "algbw_gbps": 1.36, - "busbw_gbps": 2.55, - "wrong": 0 - }, - { - "size_bytes": 16777216, - "size": "16M", - "time_us": 6336.73, - "algbw_gbps": 2.65, - "busbw_gbps": 4.96, - "wrong": 0 - }, - { - "size_bytes": 33554432, - "size": "32M", - "time_us": 12623.3, - "algbw_gbps": 2.66, - "busbw_gbps": 4.98, - "wrong": 0 - }, - { - "size_bytes": 67108864, - "size": "64M", - "time_us": 17005.6, - "algbw_gbps": 3.95, - "busbw_gbps": 7.4, - "wrong": 0 - }, - { - "size_bytes": 134217728, - "size": "128M", - "time_us": 23826.7, - "algbw_gbps": 5.63, - "busbw_gbps": 10.56, - "wrong": 0 - }, - { - "size_bytes": 268435456, - "size": "256M", - "time_us": 47356.5, - "algbw_gbps": 5.67, - "busbw_gbps": 10.63, - "wrong": 0 - } - ], - "stderr_tail": "", - "stdout_tail": " 6.25 0\n 1048576 262144 float sum -1 139.29 7.53 14.12 0 3552.34 0.30 0.55 0\n 2097152 524288 float sum -1 4195.12 0.50 0.94 0 158.81 13.21 24.76 0\n 4194304 1048576 float sum -1 199.99 20.97 39.32 0 3623.39 1.16 2.17 0\n 8388608 2097152 float sum -1 6159.00 1.36 2.55 0 324.45 25.85 48.48 0\n 16777216 4194304 float sum -1 6336.73 2.65 4.96 0 600.96 27.92 52.35 0\n 33554432 8388608 float sum -1 12623.3 2.66 4.98 0 949.39 35.34 66.27 0\n 67108864 16777216 float sum -1 17005.6 3.95 7.40 0 17175.5 3.91 7.33 0\n 134217728 33554432 float sum -1 23826.7 5.63 10.56 0 25793.0 5.20 9.76 0\n 268435456 67108864 float sum -1 47356.5 5.67 10.63 0 43195.8 6.21 11.65 0\n# Out of bounds values : 0 OK\n# Avg bus bandwidth : 9.0956 \n#\n# Collective test concluded: all_reduce_perf\n#\n\n", - "started_at": "2026-05-23T04:59:28.584786", - "finished_at": "2026-05-23T04:59:54.886123" - } - ] - }, - "alltoall": { - "binary": "/opt/gpu-test-tools/nccl-tests/build/alltoall_perf", - "topologies": [ - { - "label": "2 nodes x 8 GPUs", - "nodes": 2, - "gpus_per_node": 8, - "ranks": 16, - "hosts": [ - { - "name": "nccl-gpu-1", - "addr": "172.72.8.12", - "slots": 8 - }, - { - "name": "nccl-gpu-2", - "addr": "172.72.8.16", - "slots": 8 - } - ], - "command": "/usr/mpi/gcc/openmpi-4.1.9a1/bin/mpirun --allow-run-as-root --mca btl_openib_warn_no_device_params_found 0 --mca btl_tcp_if_include bond0 -H 172.72.8.12:8,172.72.8.16:8 --map-by ppr:8:node -np 16 -x NCCL_DEBUG=WARN -x NCCL_SOCKET_IFNAME=bond0 -x NCCL_IB_GID_INDEX=3 -x NCCL_IB_SL=5 -x NCCL_IB_TC=136 -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 -x NCCL_IB_TIMEOUT=22 -x NCCL_IB_QPS_PER_CONNECTION=4 -x NCCL_MIN_NCHANNELS=4 -x NCCL_NET_PLUGIN=none -x NCCL_NVLS_ENABLE=1 -x NCCL_IB_SPLIT_DATA_ON_QPS=1 -x LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi-4.1.9a1/lib:/root/gpu-test-venv/lib/python3.10/site-packages/nvidia/nccl/lib:/usr/local/cuda-12.4/targets/x86_64-linux/lib /opt/gpu-test-tools/nccl-tests/build/alltoall_perf -b 1k -e 256M -g 1 -f 2 -w 2", - "returncode": 0, - "status": "FAIL", - "peak_busbw_gbps": 8.64, - "peak_algbw_gbps": 9.21, - "peak_size": "2M", - "avg_busbw_gbps": 2.19, - "min_required_gbps": 20.0, - "wrong_count": 0, - "by_size": [ - { - "size_bytes": 1024, - "size": "1K", - "time_us": 58.44, - "algbw_gbps": 0.02, - "busbw_gbps": 0.02, - "wrong": 0 - }, - { - "size_bytes": 2048, - "size": "2K", - "time_us": 47.2, - "algbw_gbps": 0.04, - "busbw_gbps": 0.04, - "wrong": 0 - }, - { - "size_bytes": 4096, - "size": "4K", - "time_us": 47.68, - "algbw_gbps": 0.09, - "busbw_gbps": 0.08, - "wrong": 0 - }, - { - "size_bytes": 8192, - "size": "8K", - "time_us": 48.78, - "algbw_gbps": 0.17, - "busbw_gbps": 0.16, - "wrong": 0 - }, - { - "size_bytes": 16384, - "size": "16K", - "time_us": 79.34, - "algbw_gbps": 0.21, - "busbw_gbps": 0.19, - "wrong": 0 - }, - { - "size_bytes": 32768, - "size": "32K", - "time_us": 68.8, - "algbw_gbps": 0.48, - "busbw_gbps": 0.45, - "wrong": 0 - }, - { - "size_bytes": 65536, - "size": "64K", - "time_us": 49.86, - "algbw_gbps": 1.31, - "busbw_gbps": 1.23, - "wrong": 0 - }, - { - "size_bytes": 131072, - "size": "128K", - "time_us": 52.89, - "algbw_gbps": 2.48, - "busbw_gbps": 2.32, - "wrong": 0 - }, - { - "size_bytes": 262144, - "size": "256K", - "time_us": 3861.98, - "algbw_gbps": 0.07, - "busbw_gbps": 0.06, - "wrong": 0 - }, - { - "size_bytes": 524288, - "size": "512K", - "time_us": 83.38, - "algbw_gbps": 6.29, - "busbw_gbps": 5.89, - "wrong": 0 - }, - { - "size_bytes": 1048576, - "size": "1M", - "time_us": 182.32, - "algbw_gbps": 5.75, - "busbw_gbps": 5.39, - "wrong": 0 - }, - { - "size_bytes": 2097152, - "size": "2M", - "time_us": 227.67, - "algbw_gbps": 9.21, - "busbw_gbps": 8.64, - "wrong": 0 - }, - { - "size_bytes": 4194304, - "size": "4M", - "time_us": 6482.39, - "algbw_gbps": 0.65, - "busbw_gbps": 0.61, - "wrong": 0 - }, - { - "size_bytes": 8388608, - "size": "8M", - "time_us": 10348.9, - "algbw_gbps": 0.81, - "busbw_gbps": 0.76, - "wrong": 0 - }, - { - "size_bytes": 16777216, - "size": "16M", - "time_us": 18616.5, - "algbw_gbps": 0.9, - "busbw_gbps": 0.84, - "wrong": 0 - }, - { - "size_bytes": 33554432, - "size": "32M", - "time_us": 17170.7, - "algbw_gbps": 1.95, - "busbw_gbps": 1.83, - "wrong": 0 - }, - { - "size_bytes": 67108864, - "size": "64M", - "time_us": 35735.6, - "algbw_gbps": 1.88, - "busbw_gbps": 1.76, - "wrong": 0 - }, - { - "size_bytes": 134217728, - "size": "128M", - "time_us": 69388.5, - "algbw_gbps": 1.93, - "busbw_gbps": 1.81, - "wrong": 0 - }, - { - "size_bytes": 268435456, - "size": "256M", - "time_us": 96873.9, - "algbw_gbps": 2.77, - "busbw_gbps": 2.6, - "wrong": 0 - } - ], - "stderr_tail": "", - "stdout_tail": "56 6.85 6.42 N/A\n 1048576 16384 float none -1 182.32 5.75 5.39 0 169.19 6.20 5.81 N/A\n 2097152 32768 float none -1 227.67 9.21 8.64 0 3664.15 0.57 0.54 N/A\n 4194304 65536 float none -1 6482.39 0.65 0.61 0 553.24 7.58 7.11 N/A\n 8388608 131072 float none -1 10348.9 0.81 0.76 0 803.01 10.45 9.79 N/A\n 16777216 262144 float none -1 18616.5 0.90 0.84 0 4237.22 3.96 3.71 N/A\n 33554432 524288 float none -1 17170.7 1.95 1.83 0 20849.4 1.61 1.51 N/A\n 67108864 1048576 float none -1 35735.6 1.88 1.76 0 34524.7 1.94 1.82 N/A\n 134217728 2097152 float none -1 69388.5 1.93 1.81 0 63535.3 2.11 1.98 N/A\n 268435456 4194304 float none -1 96873.9 2.77 2.60 0 100742 2.66 2.50 N/A\n# Out of bounds values : 0 OK\n# Avg bus bandwidth : 2.19061 \n#\n# Collective test concluded: alltoall_perf\n#\n\n", - "started_at": "2026-05-23T04:59:54.886310", - "finished_at": "2026-05-23T05:00:28.796555" - } - ] - } - }, - "timestamp": "2026-05-23T05:00:28.796580" - }, - "timestamp": "2026-05-23T05:00:28.807561", - "hostname": "aikubeworker0012" -} \ No newline at end of file diff --git a/reports_multinode_nccl_smoke_256m_aikubeworker0012.md b/reports_multinode_nccl_smoke_256m_aikubeworker0012.md deleted file mode 100644 index 57fea2a..0000000 --- a/reports_multinode_nccl_smoke_256m_aikubeworker0012.md +++ /dev/null @@ -1,50 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T05:00:28.807561 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- DCGM -- Training - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: sweep - -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS (1 warnings) - -### Multi-node NCCL allreduce - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs | 39.32 GB/s | 4M | 9.10 GB/s | >= 100 GB/s | FAIL | - -### Multi-node NCCL alltoall - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs | 8.64 GB/s | 2M | 2.19 GB/s | >= 20 GB/s | FAIL | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_multinode_nccl_sweep_2x8_nccl227.md b/reports_multinode_nccl_sweep_2x8_nccl227.md deleted file mode 100644 index 701492b..0000000 --- a/reports_multinode_nccl_sweep_2x8_nccl227.md +++ /dev/null @@ -1,66 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-23T07:54:48.990378 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- DCGM -- Training - -## Summary - -| Test | Result | -|------|--------| -| Multi-node NCCL | FAIL | - -## Multi-node NCCL / Cross Leaf - -Source: nccl-tests-mpirun | Mode: sweep-nccl-2.27.7 - -- **Hosts:** nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16) -- **Preflight:** PASS - -### Multi-node NCCL allreduce - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs NCCL 2.27.7 sweep | 237.26 GB/s | 4G | 150.62 GB/s | >= 480 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 sweep | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 sweep | 0 | aikubeworker0012:2145024:2145189 [0] NCCL INFO comm 0x561f7dc1f780 rank 0 nranks 16 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 150.624 # # Collective test concluded: all_reduce_perf # | - -### Multi-node NCCL alltoall - -| Topology | Peak Bus BW | Peak Size | Avg Bus BW | Threshold | Status | -|----------|-------------|-----------|------------|-----------|--------| -| 2 nodes x 8 GPUs NCCL 2.27.7 sweep | 28.78 GB/s | 1G | 23.57 GB/s | >= 75 GB/s | FAIL | - -| Topology | NCCL Network | GPU Direct RDMA | GDR Enabled HCAs | GDR Disabled HCAs | -|----------|--------------|-----------------|------------------|-------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 sweep | IB | ENABLED | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | - | - -| Topology | Return Code | Error / Output Tail | -|----------|-------------|---------------------| -| 2 nodes x 8 GPUs NCCL 2.27.7 sweep | 0 | r0012:2145213:2145384 [7] NCCL INFO comm 0x558d54228110 rank 7 nranks 16 cudaDev 7 busId db000 - Destroy COMPLETE aikubeworker0016:1014703:1015544 [0] NCCL INFO comm 0x55ed6d99d8e0 rank 8 nranks 16 cudaDev 0 busId 18000 - Destroy COMPLETE | - -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_nvbandwidth_aikubeworker0012.json b/reports_nvbandwidth_aikubeworker0012.json deleted file mode 100644 index 05a0587..0000000 --- a/reports_nvbandwidth_aikubeworker0012.json +++ /dev/null @@ -1,70 +0,0 @@ -{ - "benchmark": { - "memory": { - "source": "nvbandwidth", - "h2d_bandwidth_gbps": 55.5, - "d2h_bandwidth_gbps": 54.8, - "d2d_bandwidth_gbps": 0.0, - "h2d_peak_gbps": 64, - "d2h_peak_gbps": 64, - "d2d_peak_gbps": 450.0, - "h2d_efficiency_pct": 86.7, - "d2h_efficiency_pct": 85.6, - "d2d_efficiency_pct": null, - "peak_bandwidth_gbps": 3400, - "efficiency_pct": null, - "results_by_test": { - "h2d": 55.5, - "d2h": 54.8, - "d2d_write": 0.0, - "d2d_read": 0.0, - "d2d_bidir": 0.0 - }, - "per_gpu": [] - }, - "compute": { - "per_dtype_tflops": { - "fp32": 52.2, - "tf32": 360.7, - "fp16": 680.0, - "bf16": 707.6, - "fp8": 1142.4 - }, - "peak_tflops": { - "fp32": 67, - "tf32": 495, - "fp16": 990, - "bf16": 990, - "fp8": 1979 - }, - "efficiency_pct": { - "fp32": 77.9, - "tf32": 72.9, - "fp16": 68.7, - "bf16": 71.5, - "fp8": 57.7 - }, - "pass_thresholds_tflops": { - "fp32": 54, - "tf32": 444, - "fp16": 734, - "bf16": 745, - "fp8": 1400 - }, - "per_gpu": [ - { - "index": 0, - "fp32": 52.2, - "tf32": 360.7, - "fp16": 680.0, - "bf16": 707.6, - "fp8": 1142.4 - } - ], - "matrix_size": 8192, - "warmup": 50, - "iterations": 500 - } - }, - "timestamp": "2026-05-22T15:35:16.675924" -} \ No newline at end of file diff --git a/reports_nvbandwidth_aikubeworker0012.md b/reports_nvbandwidth_aikubeworker0012.md deleted file mode 100644 index bf571ab..0000000 --- a/reports_nvbandwidth_aikubeworker0012.md +++ /dev/null @@ -1,38 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22 15:37:12 -- **Host:** aikubeworker0012 - -## Summary - -| Test | Result | -|------|--------| -| Memory Bandwidth | FAIL (0.0%) | -| Compute Throughput | FAIL (worst TF32 361 vs >= 444) | - -## Memory Bandwidth - -Source: nvbandwidth - -| Metric | Value | Peak | Efficiency | -|--------|-------|------|------------| -| H2D (PCIe) | 55.5 GB/s | 64 GB/s | 86.7% | -| D2H (PCIe) | 54.8 GB/s | 64 GB/s | 85.6% | -| D2D (NVLink) | 0.0 GB/s | 450 GB/s | 0.0% | - -**Verdict: FAIL** (D2D efficiency 0.0%) - -## Compute Throughput - -| DType | Achieved (TFLOPS) | Peak | Threshold | Status | -|-------|-------------------|------|------------|--------| -| FP32 | 52.2 | 67 | >= 54 | WARN | -| TF32 | 360.7 | 495 | >= 444 | FAIL | -| FP16 | 680.0 | 990 | >= 734 | WARN | -| BF16 | 707.6 | 990 | >= 745 | WARN | -| FP8 | 1142.4 | 1979 | >= 1400 | FAIL | - -**Verdict: FAIL** (absolute TFLOPS thresholds; worst efficiency 57.7%) - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_nvbandwidth_aikubeworker0016.json b/reports_nvbandwidth_aikubeworker0016.json deleted file mode 100644 index 34ac61c..0000000 --- a/reports_nvbandwidth_aikubeworker0016.json +++ /dev/null @@ -1,70 +0,0 @@ -{ - "benchmark": { - "memory": { - "source": "nvbandwidth", - "h2d_bandwidth_gbps": 55.5, - "d2h_bandwidth_gbps": 55.0, - "d2d_bandwidth_gbps": 0.0, - "h2d_peak_gbps": 64, - "d2h_peak_gbps": 64, - "d2d_peak_gbps": 450.0, - "h2d_efficiency_pct": 86.7, - "d2h_efficiency_pct": 85.9, - "d2d_efficiency_pct": null, - "peak_bandwidth_gbps": 3400, - "efficiency_pct": null, - "results_by_test": { - "h2d": 55.5, - "d2h": 55.0, - "d2d_write": 0.0, - "d2d_read": 0.0, - "d2d_bidir": 0.0 - }, - "per_gpu": [] - }, - "compute": { - "per_dtype_tflops": { - "fp32": 52.2, - "tf32": 357.5, - "fp16": 665.3, - "bf16": 697.1, - "fp8": 1138.8 - }, - "peak_tflops": { - "fp32": 67, - "tf32": 495, - "fp16": 990, - "bf16": 990, - "fp8": 1979 - }, - "efficiency_pct": { - "fp32": 77.9, - "tf32": 72.2, - "fp16": 67.2, - "bf16": 70.4, - "fp8": 57.5 - }, - "pass_thresholds_tflops": { - "fp32": 54, - "tf32": 444, - "fp16": 734, - "bf16": 745, - "fp8": 1400 - }, - "per_gpu": [ - { - "index": 0, - "fp32": 52.2, - "tf32": 357.5, - "fp16": 665.3, - "bf16": 697.1, - "fp8": 1138.8 - } - ], - "matrix_size": 8192, - "warmup": 50, - "iterations": 500 - } - }, - "timestamp": "2026-05-22T15:35:19.219299" -} \ No newline at end of file diff --git a/reports_nvbandwidth_aikubeworker0016.md b/reports_nvbandwidth_aikubeworker0016.md deleted file mode 100644 index 01320cf..0000000 --- a/reports_nvbandwidth_aikubeworker0016.md +++ /dev/null @@ -1,38 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22 15:37:18 -- **Host:** aikubeworker0016 - -## Summary - -| Test | Result | -|------|--------| -| Memory Bandwidth | FAIL (0.0%) | -| Compute Throughput | FAIL (worst TF32 358 vs >= 444) | - -## Memory Bandwidth - -Source: nvbandwidth - -| Metric | Value | Peak | Efficiency | -|--------|-------|------|------------| -| H2D (PCIe) | 55.5 GB/s | 64 GB/s | 86.7% | -| D2H (PCIe) | 55.0 GB/s | 64 GB/s | 85.9% | -| D2D (NVLink) | 0.0 GB/s | 450 GB/s | 0.0% | - -**Verdict: FAIL** (D2D efficiency 0.0%) - -## Compute Throughput - -| DType | Achieved (TFLOPS) | Peak | Threshold | Status | -|-------|-------------------|------|------------|--------| -| FP32 | 52.2 | 67 | >= 54 | WARN | -| TF32 | 357.5 | 495 | >= 444 | FAIL | -| FP16 | 665.3 | 990 | >= 734 | WARN | -| BF16 | 697.1 | 990 | >= 745 | WARN | -| FP8 | 1138.8 | 1979 | >= 1400 | FAIL | - -**Verdict: FAIL** (absolute TFLOPS thresholds; worst efficiency 57.5%) - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_rdma_aikubeworker0012.json b/reports_rdma_aikubeworker0012.json deleted file mode 100644 index 93d7644..0000000 --- a/reports_rdma_aikubeworker0012.json +++ /dev/null @@ -1,157 +0,0 @@ -{ - "rdma": { - "passed": false, - "devices": [ - { - "name": "mlx5_0", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:58a2:e103:0093:3898" - } - ] - }, - { - "name": "mlx5_1", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:58a2:e103:0093:3db0" - } - ] - }, - { - "name": "mlx5_2", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:5c3f:b8ff:fe5e:7832" - } - ] - }, - { - "name": "mlx5_3", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "1: DOWN", - "phys_state": "3: Disabled", - "gid": "fe80:0000:0000:0000:5e25:73ff:fe4e:eac1" - } - ] - }, - { - "name": "mlx5_4", - "ports": [ - { - "port": "1", - "rate": "100 Gb/sec (2X HDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:9c63:c003:005f:63cc" - } - ] - }, - { - "name": "mlx5_5", - "ports": [ - { - "port": "1", - "rate": "100 Gb/sec (2X HDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:9c63:c003:005f:63cd" - } - ] - }, - { - "name": "mlx5_6", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:58a2:e103:0093:3bf4" - } - ] - }, - { - "name": "mlx5_7", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:58a2:e103:0093:3e28" - } - ] - }, - { - "name": "mlx5_8", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:5c3f:b8ff:fe5e:7832" - } - ] - }, - { - "name": "mlx5_9", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "1: DOWN", - "phys_state": "3: Disabled", - "gid": "fe80:0000:0000:0000:5e25:73ff:fe63:1717" - } - ] - } - ], - "bandwidth_tests": [ - { - "test": "ib_write_bw", - "status": "WARN", - "bandwidth_gbps": 0.13, - "min_required_gbps": 50 - }, - { - "test": "ib_read_bw", - "status": "WARN", - "bandwidth_gbps": 0.13, - "min_required_gbps": 50 - } - ], - "latency_tests": [ - { - "test": "ib_write_lat", - "status": "PASS", - "latency_us": 4.53, - "max_allowed_us": 10 - }, - { - "test": "ib_read_lat", - "status": "WARN", - "latency_us": 16.0, - "max_allowed_us": 10 - } - ], - "timestamp": "2026-05-22T15:41:20.534115" - }, - "timestamp": "2026-05-22T15:41:20.544589" -} \ No newline at end of file diff --git a/reports_rdma_aikubeworker0016.json b/reports_rdma_aikubeworker0016.json deleted file mode 100644 index 5e98f8a..0000000 --- a/reports_rdma_aikubeworker0016.json +++ /dev/null @@ -1,157 +0,0 @@ -{ - "rdma": { - "passed": false, - "devices": [ - { - "name": "mlx5_0", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:58a2:e103:0088:81e0" - } - ] - }, - { - "name": "mlx5_1", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:9c63:c003:0054:e00a" - } - ] - }, - { - "name": "mlx5_2", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:a02d:75ff:feae:2bcf" - } - ] - }, - { - "name": "mlx5_3", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "1: DOWN", - "phys_state": "3: Disabled", - "gid": "fe80:0000:0000:0000:c670:bdff:fefd:5bd9" - } - ] - }, - { - "name": "mlx5_4", - "ports": [ - { - "port": "1", - "rate": "100 Gb/sec (2X HDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:9c63:c003:005f:58ec" - } - ] - }, - { - "name": "mlx5_5", - "ports": [ - { - "port": "1", - "rate": "100 Gb/sec (2X HDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:9c63:c003:005f:58ed" - } - ] - }, - { - "name": "mlx5_6", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:9c63:c003:0055:0e56" - } - ] - }, - { - "name": "mlx5_7", - "ports": [ - { - "port": "1", - "rate": "400 Gb/sec (4X NDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:a088:c203:00f0:286c" - } - ] - }, - { - "name": "mlx5_8", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "4: ACTIVE", - "phys_state": "5: LinkUp", - "gid": "fe80:0000:0000:0000:a02d:75ff:feae:2bcf" - } - ] - }, - { - "name": "mlx5_9", - "ports": [ - { - "port": "1", - "rate": "25 Gb/sec (1X EDR)", - "state": "1: DOWN", - "phys_state": "3: Disabled", - "gid": "fe80:0000:0000:0000:c670:bdff:fefd:569d" - } - ] - } - ], - "bandwidth_tests": [ - { - "test": "ib_write_bw", - "status": "WARN", - "bandwidth_gbps": 0.13, - "min_required_gbps": 50 - }, - { - "test": "ib_read_bw", - "status": "WARN", - "bandwidth_gbps": 0.13, - "min_required_gbps": 50 - } - ], - "latency_tests": [ - { - "test": "ib_write_lat", - "status": "PASS", - "latency_us": 4.22, - "max_allowed_us": 10 - }, - { - "test": "ib_read_lat", - "status": "WARN", - "latency_us": 16.0, - "max_allowed_us": 10 - } - ], - "timestamp": "2026-05-22T15:41:07.851101" - }, - "timestamp": "2026-05-22T15:41:07.861558" -} \ No newline at end of file diff --git a/reports_rdma_counter_aikubeworker0012_20260522_194808.md b/reports_rdma_counter_aikubeworker0012_20260522_194808.md deleted file mode 100644 index f254bef..0000000 --- a/reports_rdma_counter_aikubeworker0012_20260522_194808.md +++ /dev/null @@ -1,62 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T19:48:26.622179 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Failed or unverified items: -- RDMA: FAIL - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- DCGM -- Training - -## Summary - -| Test | Result | -|------|--------| -| RDMA | FAIL | - -## RDMA/InfiniBand - -### RDMA Port Checks - -| Device | Port | State | Rate | Required | Status | -|--------|------|-------|------|----------|--------| -| mlx5_0 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_1 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_4 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_5 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_6 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_7 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | - -| Test | Value | Threshold | Status | -|------|-------|-----------|--------| -| ib_write_bw | 49.3 GB/s | >= 47 GB/s | PASS | -| ib_read_bw | 39.2 GB/s | >= 47 GB/s | FAIL | -| ib_write_lat | 4.49 us | <= 2 us | FAIL | -| ib_read_lat | 16.00 us | <= 3.5 us | FAIL | -| ibping | target=0x58 count=5 | 0% packet loss | PASS | - -- **PFC/ECN/CNP/congestion counters checked:** 146 -- **PFC/ECN/CNP/congestion non-zero:** no -- **Failure reasons:** - - mlx5_4 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - mlx5_5 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - ib_read_bw bandwidth 39.21GB/s < 47GB/s - - ib_write_lat latency 4.49us > 2.0us - - ib_read_lat latency 16.0us > 3.5us -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_rdma_counter_aikubeworker0016_20260522_194828.md b/reports_rdma_counter_aikubeworker0016_20260522_194828.md deleted file mode 100644 index a72f917..0000000 --- a/reports_rdma_counter_aikubeworker0016_20260522_194828.md +++ /dev/null @@ -1,62 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T19:48:45.899570 -- **Host:** aikubeworker0016 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Failed or unverified items: -- RDMA: FAIL - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- DCGM -- Training - -## Summary - -| Test | Result | -|------|--------| -| RDMA | FAIL | - -## RDMA/InfiniBand - -### RDMA Port Checks - -| Device | Port | State | Rate | Required | Status | -|--------|------|-------|------|----------|--------| -| mlx5_0 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_1 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_4 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_5 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_6 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_7 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | - -| Test | Value | Threshold | Status | -|------|-------|-----------|--------| -| ib_write_bw | 48.1 GB/s | >= 47 GB/s | PASS | -| ib_read_bw | 40.3 GB/s | >= 47 GB/s | FAIL | -| ib_write_lat | 4.28 us | <= 2 us | FAIL | -| ib_read_lat | 16.00 us | <= 3.5 us | FAIL | -| ibping | target=0x4b count=5 | 0% packet loss | PASS | - -- **PFC/ECN/CNP/congestion counters checked:** 146 -- **PFC/ECN/CNP/congestion non-zero:** no -- **Failure reasons:** - - mlx5_4 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - mlx5_5 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - ib_read_bw bandwidth 40.3GB/s < 47GB/s - - ib_write_lat latency 4.28us > 2.0us - - ib_read_lat latency 16.0us > 3.5us -**Overall: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_rdma_cross_node_mlx5_0_20260523.md b/reports_rdma_cross_node_mlx5_0_20260523.md deleted file mode 100644 index dfdfb8a..0000000 --- a/reports_rdma_cross_node_mlx5_0_20260523.md +++ /dev/null @@ -1,50 +0,0 @@ -# RDMA Cross-node Evidence Report - -- **Date:** 2026-05-23 Asia/Shanghai -- **Scope:** `aikubeworker0012` <-> `aikubeworker0016`, single rail `mlx5_0`, port 1 -- **Client/server bootstrap IPs:** `172.72.8.12` and `172.72.8.16` -- **Bandwidth message size:** 4MB -- **Latency message size:** 8B -- **Iterations:** 1000 - -## Port Evidence - -| Host | Device | State | Rate | Link | LID | -|---|---|---|---|---|---| -| aikubeworker0012 | mlx5_0/1 | ACTIVE | 400 Gb/sec (4X NDR) | InfiniBand | 0x58 | -| aikubeworker0016 | mlx5_0/1 | ACTIVE | 400 Gb/sec (4X NDR) | InfiniBand | 0x4b | - -## Cross-node Perftest Results - -| Direction | Test | Value | PDF Threshold | Status | -|---|---|---:|---:|---| -| 0016 -> 0012 | ib_write_bw | 49.35 GB/s | >= 47 GB/s | PASS | -| 0016 -> 0012 | ib_read_bw | 44.36 GB/s | >= 47 GB/s | FAIL | -| 0016 -> 0012 | ib_write_lat avg | 2.17 us | <= 2.0 us | FAIL | -| 0016 -> 0012 | ib_read_lat avg | 4.05 us | <= 3.5 us | FAIL | -| 0012 -> 0016 | ib_write_bw | 48.38 GB/s | >= 47 GB/s | PASS | -| 0012 -> 0016 | ib_read_bw | 44.37 GB/s | >= 47 GB/s | FAIL | -| 0012 -> 0016 | ib_write_lat avg | 2.13 us | <= 2.0 us | FAIL | -| 0012 -> 0016 | ib_read_lat avg | 4.08 us | <= 3.5 us | FAIL | - -## Bidirectional ibping - -| Direction | Target LID | Result | -|---|---|---| -| 0016 -> 0012 | 0x58 | 5 transmitted, 5 received, 0% packet loss; avg 0.005 ms | -| 0012 -> 0016 | 0x4b | 5 transmitted, 5 received, 0% packet loss; avg 0.005 ms | - -## Fabric Counters - -| Host | PFC/ECN/CNP/congestion Counters Checked | Non-zero Counters | Status | -|---|---:|---:|---| -| aikubeworker0012 | 146 | 0 | PASS | -| aikubeworker0016 | 146 | 0 | PASS | - -## Verdict - -**RDMA cross-node verdict: FAIL** - -Reason: bidirectional connectivity is good, PFC/ECN/CNP/congestion counters are clean, and write bandwidth passes. However read bandwidth is below 47 GB/s in both directions, write latency is slightly above 2.0 us in both directions, and read latency is above 3.5 us in both directions. - -Note: `modules/rdma_test.py` was corrected on 2026-05-23 to parse `ib_write_lat` / `ib_read_lat` `t_avg[usec]` rather than the 99.9 percentile column. Older reports that show `read_lat` around 16 us are therefore not the current parser output. diff --git a/reports_rdma_single_node_summary.md b/reports_rdma_single_node_summary.md deleted file mode 100644 index c1c95de..0000000 --- a/reports_rdma_single_node_summary.md +++ /dev/null @@ -1,73 +0,0 @@ -# Single-node RDMA/IB Report - -Generated: 2026-05-22 23:41 Asia/Shanghai - -Scope: project CLI `gpu_tester.py --test rdma --report --format json`, run separately on each host. - -Important note: the current repository RDMA test is single-node only. In `modules/rdma_test.py`, the perftest client connects to `localhost`, so this report validates local IB device discovery and local perftest behavior. It does not validate cross-node RDMA bandwidth between `aikubeworker0012` and `aikubeworker0016`. - -## Summary - -| Host | Devices Found | Active 400G Ports | Active 100G Ports | Down Ports | Overall | -| --- | ---: | --- | --- | --- | --- | -| aikubeworker0012 / 172.72.8.12 | 10 | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | mlx5_4, mlx5_5 | mlx5_3, mlx5_9 | WARN | -| aikubeworker0016 / 172.72.8.16 | 10 | mlx5_0, mlx5_1, mlx5_6, mlx5_7 | mlx5_4, mlx5_5 | mlx5_3, mlx5_9 | WARN | - -## Bandwidth - -The bandwidth numbers below are from the repo's local `localhost` RDMA perftest path. - -| Host | ib_write_bw | Threshold | Status | ib_read_bw | Threshold | Status | -| --- | ---: | ---: | --- | ---: | ---: | --- | -| aikubeworker0012 | 0.13 GB/s | 50 GB/s | WARN | 0.13 GB/s | 50 GB/s | WARN | -| aikubeworker0016 | 0.13 GB/s | 50 GB/s | WARN | 0.13 GB/s | 50 GB/s | WARN | - -## Latency - -| Host | ib_write_lat | Limit | Status | ib_read_lat | Limit | Status | -| --- | ---: | ---: | --- | ---: | ---: | --- | -| aikubeworker0012 | 4.53 us | 10 us | PASS | 16.00 us | 10 us | WARN | -| aikubeworker0016 | 4.22 us | 10 us | PASS | 16.00 us | 10 us | WARN | - -## Device Inventory - -### aikubeworker0012 - -| Device | Port | State | Physical State | Rate | -| --- | --- | --- | --- | --- | -| mlx5_0 | 1 | ACTIVE | LinkUp | 400 Gb/sec (4X NDR) | -| mlx5_1 | 1 | ACTIVE | LinkUp | 400 Gb/sec (4X NDR) | -| mlx5_2 | 1 | ACTIVE | LinkUp | 25 Gb/sec (1X EDR) | -| mlx5_3 | 1 | DOWN | Disabled | 25 Gb/sec (1X EDR) | -| mlx5_4 | 1 | ACTIVE | LinkUp | 100 Gb/sec (2X HDR) | -| mlx5_5 | 1 | ACTIVE | LinkUp | 100 Gb/sec (2X HDR) | -| mlx5_6 | 1 | ACTIVE | LinkUp | 400 Gb/sec (4X NDR) | -| mlx5_7 | 1 | ACTIVE | LinkUp | 400 Gb/sec (4X NDR) | -| mlx5_8 | 1 | ACTIVE | LinkUp | 25 Gb/sec (1X EDR) | -| mlx5_9 | 1 | DOWN | Disabled | 25 Gb/sec (1X EDR) | - -### aikubeworker0016 - -| Device | Port | State | Physical State | Rate | -| --- | --- | --- | --- | --- | -| mlx5_0 | 1 | ACTIVE | LinkUp | 400 Gb/sec (4X NDR) | -| mlx5_1 | 1 | ACTIVE | LinkUp | 400 Gb/sec (4X NDR) | -| mlx5_2 | 1 | ACTIVE | LinkUp | 25 Gb/sec (1X EDR) | -| mlx5_3 | 1 | DOWN | Disabled | 25 Gb/sec (1X EDR) | -| mlx5_4 | 1 | ACTIVE | LinkUp | 100 Gb/sec (2X HDR) | -| mlx5_5 | 1 | ACTIVE | LinkUp | 100 Gb/sec (2X HDR) | -| mlx5_6 | 1 | ACTIVE | LinkUp | 400 Gb/sec (4X NDR) | -| mlx5_7 | 1 | ACTIVE | LinkUp | 400 Gb/sec (4X NDR) | -| mlx5_8 | 1 | ACTIVE | LinkUp | 25 Gb/sec (1X EDR) | -| mlx5_9 | 1 | DOWN | Disabled | 25 Gb/sec (1X EDR) | - -## Files - -Raw JSON: - -- `reports_rdma_aikubeworker0012.json` -- `reports_rdma_aikubeworker0016.json` - -Markdown summary: - -- `reports_rdma_single_node_summary.md` diff --git a/reports_single_gpu_aikubeworker0012.json b/reports_single_gpu_aikubeworker0012.json deleted file mode 100644 index 6cc5a37..0000000 --- a/reports_single_gpu_aikubeworker0012.json +++ /dev/null @@ -1,292 +0,0 @@ -{ - "timestamp": "2026-05-22T15:26:26.973586", - "gpu_info": { - "driver_version": "580.159.03", - "cuda_version": "13.0", - "gpu_count": 8, - "gpus": [ - { - "index": 0, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-7658c03c-7659-9886-041e-545c21d53e12", - "pci_bus_id": "00000000:18:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 4, - "vram_free_mb": 81076, - "power_draw": 69.72, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 25, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1654923030411", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 1, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-6392d40b-893b-9fc2-4284-a3f1d8c4d7f1", - "pci_bus_id": "00000000:2A:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 73.17, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 25, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1654724063165", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 2, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-2ae38735-10de-fb0b-fb20-9d1b5b434558", - "pci_bus_id": "00000000:3A:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 68.71, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 26, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1654823036530", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 3, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-ec62123f-0c48-6dbd-49e4-8b231b3fed0e", - "pci_bus_id": "00000000:5D:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 69.73, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 25, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1654923021638", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 4, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-b64fc270-109e-1543-fb0c-be7feecf14f1", - "pci_bus_id": "00000000:9A:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 68.84, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 24, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1655023033179", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 5, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-15ab7baf-9010-7cf3-5462-eeb09f8dbe65", - "pci_bus_id": "00000000:AB:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 69.94, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 27, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1655023034225", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 6, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-225f6f3c-6fef-d1e2-5428-d90f665fb3d3", - "pci_bus_id": "00000000:BA:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 70.46, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 25, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1654923078278", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 7, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-79aeb6a8-c00c-6edb-956f-779ef56950a3", - "pci_bus_id": "00000000:DB:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 71.76, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 24, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1654024031464", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - } - ], - "topology": "\t\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tGPU4\tGPU5\tGPU6\tGPU7\tNIC0\tNIC1\tNIC2\tNIC3\tNIC4\tNIC5\tNIC6\tNIC7\tNIC8\tNIC9\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tPIX\tNODE\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU1\tNV18\t X \tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tNODE\tPIX\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU2\tNV18\tNV18\t X \tNV18\tNV18\tNV18\tNV18\tNV18\tNODE\tNODE\tPIX\tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU3\tNV18\tNV18\tNV18\t X \tNV18\tNV18\tNV18\tNV18\tNODE\tNODE\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU4\tNV18\tNV18\tNV18\tNV18\t X \tNV18\tNV18\tNV18\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tPIX\tNODE\tNODE\tNODE\t56-111,168-223\t1\t\tN/A\nGPU5\tNV18\tNV18\tNV18\tNV18\tNV18\t X \tNV18\tNV18\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tPIX\tNODE\tNODE\t56-111,168-223\t1\t\tN/A\nGPU6\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\t X \tNV18\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\tPIX\t56-111,168-223\t1\t\tN/A\nGPU7\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\t X \tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tNODE\tNODE\t56-111,168-223\t1\t\tN/A\nNIC0\tPIX\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t X \tNODE\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC1\tNODE\tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\t X \tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC2\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\t X \tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC3\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\t X \tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC4\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tNODE\tNODE\t X \tPIX\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC5\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tNODE\tNODE\tPIX\t X \tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC6\tSYS\tSYS\tSYS\tSYS\tPIX\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\t X \tNODE\tNODE\tNODE\t\t\t\t\nNIC7\tSYS\tSYS\tSYS\tSYS\tNODE\tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\t X \tNODE\tNODE\t\t\t\t\nNIC8\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\t X \tPIX\t\t\t\t\nNIC9\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\t X \t\t\t\t\n\nLegend:\n\n X = Self\n SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n PIX = Connection traversing at most a single PCIe bridge\n NV# = Connection traversing a bonded set of # NVLinks\n\nNIC Legend:\n\n NIC0: mlx5_0\n NIC1: mlx5_1\n NIC2: mlx5_2\n NIC3: mlx5_3\n NIC4: mlx5_4\n NIC5: mlx5_5\n NIC6: mlx5_6\n NIC7: mlx5_7\n NIC8: mlx5_8\n NIC9: mlx5_9\n\n", - "timestamp": "2026-05-22T15:26:34.187409", - "detected_gpu_type": "h100", - "gpu_label": "H100 SXM5" - }, - "memory_bench": { - "memory": { - "source": "pytorch", - "h2d_bandwidth_gbps": 11.8, - "d2h_bandwidth_gbps": 9.9, - "d2d_bandwidth_gbps": 829.1, - "peak_bandwidth_gbps": 3400, - "efficiency_pct": 24.4, - "test_sizes_mb": [ - 1, - 4, - 16, - 64, - 256, - 1024, - 4096 - ], - "bandwidth_by_size": { - "1": { - "h2d_gbps": 3.8, - "d2h_gbps": 1.4, - "d2d_gbps": 40.6 - }, - "4": { - "h2d_gbps": 7.6, - "d2h_gbps": 9.9, - "d2d_gbps": 141.5 - }, - "16": { - "h2d_gbps": 11.0, - "d2h_gbps": 1.9, - "d2d_gbps": 450.3 - }, - "64": { - "h2d_gbps": 11.8, - "d2h_gbps": 1.4, - "d2d_gbps": 726.5 - }, - "256": { - "h2d_gbps": 9.0, - "d2h_gbps": 1.4, - "d2d_gbps": 793.8 - }, - "1024": { - "h2d_gbps": 5.5, - "d2h_gbps": 1.4, - "d2d_gbps": 821.2 - }, - "4096": { - "h2d_gbps": 5.9, - "d2h_gbps": 1.4, - "d2d_gbps": 829.1 - } - }, - "per_gpu": [] - } - }, - "compute_bench": { - "compute": { - "per_dtype_tflops": { - "fp32": 52.0, - "tf32": 362.3, - "fp16": 691.0, - "bf16": 713.0, - "fp8": 1148.8 - }, - "peak_tflops": { - "fp32": 67, - "tf32": 495, - "fp16": 990, - "bf16": 990, - "fp8": 1979 - }, - "efficiency_pct": { - "fp32": 77.6, - "tf32": 73.2, - "fp16": 69.8, - "bf16": 72.0, - "fp8": 58.0 - }, - "pass_thresholds_tflops": { - "fp32": 54, - "tf32": 444, - "fp16": 734, - "bf16": 745, - "fp8": 1400 - }, - "per_gpu": [ - { - "index": 0, - "fp32": 52.0, - "tf32": 362.3, - "fp16": 691.0, - "bf16": 713.0, - "fp8": 1148.8 - } - ], - "matrix_size": 8192, - "warmup": 50, - "iterations": 500 - } - } -} \ No newline at end of file diff --git a/reports_single_gpu_aikubeworker0012.md b/reports_single_gpu_aikubeworker0012.md deleted file mode 100644 index 3a6c3c9..0000000 --- a/reports_single_gpu_aikubeworker0012.md +++ /dev/null @@ -1,54 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22 15:27:51 -- **Host:** aikubeworker0012 -- **GPU:** NVIDIA H100 80GB HBM3 x8 -- **Driver:** 580.159.03 | **CUDA:** 13.0 - -## Summary - -| Test | Result | -|------|--------| -| GPU Info | PASS (8 GPUs detected) | -| Memory Bandwidth | WARN (829 GB/s via PyTorch fallback) | -| Compute Throughput | FAIL (worst TF32 362 vs >= 444) | - -## GPU Information - -| GPU | Model | VRAM | Temp | Power | SM Clock | -|-----|-------|------|------|-------|----------| -| 0 | NVIDIA H100 80GB HBM3 | 81559 MB | 25C | 70/700W | 345 MHz | -| 1 | NVIDIA H100 80GB HBM3 | 81559 MB | 25C | 73/700W | 345 MHz | -| 2 | NVIDIA H100 80GB HBM3 | 81559 MB | 26C | 69/700W | 345 MHz | -| 3 | NVIDIA H100 80GB HBM3 | 81559 MB | 25C | 70/700W | 345 MHz | -| 4 | NVIDIA H100 80GB HBM3 | 81559 MB | 24C | 69/700W | 345 MHz | -| 5 | NVIDIA H100 80GB HBM3 | 81559 MB | 27C | 70/700W | 345 MHz | -| 6 | NVIDIA H100 80GB HBM3 | 81559 MB | 25C | 70/700W | 345 MHz | -| 7 | NVIDIA H100 80GB HBM3 | 81559 MB | 24C | 72/700W | 345 MHz | - -## Memory Bandwidth - -Source: pytorch - -| Metric | Value | Peak | Efficiency | -|--------|-------|------|------------| -| H2D (PCIe) | 11.8 GB/s | 0 GB/s | 0.0% | -| D2H (PCIe) | 9.9 GB/s | 0 GB/s | 0.0% | -| D2D (NVLink) | 829.1 GB/s | 3400 GB/s | 24.4% | - -**Verdict: WARN** (D2D 829.1 GB/s via PyTorch fallback; nvbandwidth unavailable — figure is indicative only, not a true HBM peak) - -## Compute Throughput - -| DType | Achieved (TFLOPS) | Peak | Threshold | Status | -|-------|-------------------|------|------------|--------| -| FP32 | 52.0 | 67 | >= 54 | WARN | -| TF32 | 362.3 | 495 | >= 444 | FAIL | -| FP16 | 691.0 | 990 | >= 734 | WARN | -| BF16 | 713.0 | 990 | >= 745 | WARN | -| FP8 | 1148.8 | 1979 | >= 1400 | FAIL | - -**Verdict: FAIL** (absolute TFLOPS thresholds; worst efficiency 58.0%) - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_single_gpu_aikubeworker0016.json b/reports_single_gpu_aikubeworker0016.json deleted file mode 100644 index 4b3c442..0000000 --- a/reports_single_gpu_aikubeworker0016.json +++ /dev/null @@ -1,292 +0,0 @@ -{ - "timestamp": "2026-05-22T15:26:29.511252", - "gpu_info": { - "driver_version": "580.159.03", - "cuda_version": "13.0", - "gpu_count": 8, - "gpus": [ - { - "index": 0, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-dfbc9513-255d-4fe7-2b77-7b1ec3972e75", - "pci_bus_id": "00000000:18:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 4, - "vram_free_mb": 81076, - "power_draw": 69.81, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 20, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924016120", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 1, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-bb845ef7-d7b5-f011-9395-ea74274e2282", - "pci_bus_id": "00000000:2A:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 67.45, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 20, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924015483", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 2, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-3720cf13-2a34-be38-27be-0a7adc4addc4", - "pci_bus_id": "00000000:3A:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 66.69, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 21, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924025595", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 3, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-87080b2d-ac43-be0d-d574-c193078850ae", - "pci_bus_id": "00000000:5D:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 66.86, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 20, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924016862", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 4, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-599bd883-cc5c-a5dd-6c33-c15f7049da48", - "pci_bus_id": "00000000:9A:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 67.07, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 20, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924025670", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 5, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-a1c6bba4-61b0-e623-06c9-9c88635e26fe", - "pci_bus_id": "00000000:AB:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 69.12, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 22, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924027166", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 6, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-98745a0c-39bd-3e56-d6ca-54ba3647ab6d", - "pci_bus_id": "00000000:BA:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 67.61, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 20, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924026234", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - }, - { - "index": 7, - "name": "NVIDIA H100 80GB HBM3", - "uuid": "GPU-8c73bd8b-666b-357e-ac5d-c75ac7a759db", - "pci_bus_id": "00000000:DB:00.0", - "pcie_link_gen": 5, - "pcie_link_width": 16, - "vram_total_mb": 81559, - "vram_used_mb": 0, - "vram_free_mb": 81079, - "power_draw": 66.19, - "power_limit": 700.0, - "clock_sm": 345, - "clock_mem": 2619, - "temperature": 20, - "fan_speed": 0, - "persistence_mode": false, - "compute_mode": "Default", - "serial_number": "1651924027255", - "ecc_errors_single": 0, - "ecc_errors_double": 0 - } - ], - "topology": "\t\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tGPU4\tGPU5\tGPU6\tGPU7\tNIC0\tNIC1\tNIC2\tNIC3\tNIC4\tNIC5\tNIC6\tNIC7\tNIC8\tNIC9\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tPIX\tNODE\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU1\tNV18\t X \tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tNODE\tPIX\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU2\tNV18\tNV18\t X \tNV18\tNV18\tNV18\tNV18\tNV18\tNODE\tNODE\tPIX\tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU3\tNV18\tNV18\tNV18\t X \tNV18\tNV18\tNV18\tNV18\tNODE\tNODE\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t0-55,112-167\t0\t\tN/A\nGPU4\tNV18\tNV18\tNV18\tNV18\t X \tNV18\tNV18\tNV18\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tPIX\tNODE\tNODE\tNODE\t56-111,168-223\t1\t\tN/A\nGPU5\tNV18\tNV18\tNV18\tNV18\tNV18\t X \tNV18\tNV18\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tPIX\tNODE\tNODE\t56-111,168-223\t1\t\tN/A\nGPU6\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\t X \tNV18\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\tPIX\t56-111,168-223\t1\t\tN/A\nGPU7\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\tNV18\t X \tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tNODE\tNODE\t56-111,168-223\t1\t\tN/A\nNIC0\tPIX\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t X \tNODE\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC1\tNODE\tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\t X \tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC2\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\t X \tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC3\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\t X \tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC4\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tNODE\tNODE\t X \tPIX\tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC5\tNODE\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tNODE\tNODE\tPIX\t X \tSYS\tSYS\tSYS\tSYS\t\t\t\t\nNIC6\tSYS\tSYS\tSYS\tSYS\tPIX\tNODE\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\t X \tNODE\tNODE\tNODE\t\t\t\t\nNIC7\tSYS\tSYS\tSYS\tSYS\tNODE\tPIX\tNODE\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\t X \tNODE\tNODE\t\t\t\t\nNIC8\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\t X \tPIX\t\t\t\t\nNIC9\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\tNODE\tSYS\tSYS\tSYS\tSYS\tSYS\tSYS\tNODE\tNODE\tPIX\t X \t\t\t\t\n\nLegend:\n\n X = Self\n SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n PIX = Connection traversing at most a single PCIe bridge\n NV# = Connection traversing a bonded set of # NVLinks\n\nNIC Legend:\n\n NIC0: mlx5_0\n NIC1: mlx5_1\n NIC2: mlx5_2\n NIC3: mlx5_3\n NIC4: mlx5_4\n NIC5: mlx5_5\n NIC6: mlx5_6\n NIC7: mlx5_7\n NIC8: mlx5_8\n NIC9: mlx5_9\n\n", - "timestamp": "2026-05-22T15:26:36.627805", - "detected_gpu_type": "h100", - "gpu_label": "H100 SXM5" - }, - "memory_bench": { - "memory": { - "source": "pytorch", - "h2d_bandwidth_gbps": 11.8, - "d2h_bandwidth_gbps": 10.1, - "d2d_bandwidth_gbps": 829.0, - "peak_bandwidth_gbps": 3400, - "efficiency_pct": 24.4, - "test_sizes_mb": [ - 1, - 4, - 16, - 64, - 256, - 1024, - 4096 - ], - "bandwidth_by_size": { - "1": { - "h2d_gbps": 3.6, - "d2h_gbps": 1.4, - "d2d_gbps": 40.3 - }, - "4": { - "h2d_gbps": 7.7, - "d2h_gbps": 10.1, - "d2d_gbps": 159.5 - }, - "16": { - "h2d_gbps": 10.9, - "d2h_gbps": 1.9, - "d2d_gbps": 439.5 - }, - "64": { - "h2d_gbps": 11.8, - "d2h_gbps": 1.4, - "d2d_gbps": 740.5 - }, - "256": { - "h2d_gbps": 9.0, - "d2h_gbps": 1.4, - "d2d_gbps": 792.1 - }, - "1024": { - "h2d_gbps": 8.4, - "d2h_gbps": 1.4, - "d2d_gbps": 818.9 - }, - "4096": { - "h2d_gbps": 6.1, - "d2h_gbps": 1.4, - "d2d_gbps": 829.0 - } - }, - "per_gpu": [] - } - }, - "compute_bench": { - "compute": { - "per_dtype_tflops": { - "fp32": 51.9, - "tf32": 357.8, - "fp16": 667.2, - "bf16": 699.1, - "fp8": 1146.2 - }, - "peak_tflops": { - "fp32": 67, - "tf32": 495, - "fp16": 990, - "bf16": 990, - "fp8": 1979 - }, - "efficiency_pct": { - "fp32": 77.5, - "tf32": 72.3, - "fp16": 67.4, - "bf16": 70.6, - "fp8": 57.9 - }, - "pass_thresholds_tflops": { - "fp32": 54, - "tf32": 444, - "fp16": 734, - "bf16": 745, - "fp8": 1400 - }, - "per_gpu": [ - { - "index": 0, - "fp32": 51.9, - "tf32": 357.8, - "fp16": 667.2, - "bf16": 699.1, - "fp8": 1146.2 - } - ], - "matrix_size": 8192, - "warmup": 50, - "iterations": 500 - } - } -} \ No newline at end of file diff --git a/reports_single_gpu_aikubeworker0016.md b/reports_single_gpu_aikubeworker0016.md deleted file mode 100644 index 49f9f45..0000000 --- a/reports_single_gpu_aikubeworker0016.md +++ /dev/null @@ -1,54 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22 15:27:53 -- **Host:** aikubeworker0016 -- **GPU:** NVIDIA H100 80GB HBM3 x8 -- **Driver:** 580.159.03 | **CUDA:** 13.0 - -## Summary - -| Test | Result | -|------|--------| -| GPU Info | PASS (8 GPUs detected) | -| Memory Bandwidth | WARN (829 GB/s via PyTorch fallback) | -| Compute Throughput | FAIL (worst TF32 358 vs >= 444) | - -## GPU Information - -| GPU | Model | VRAM | Temp | Power | SM Clock | -|-----|-------|------|------|-------|----------| -| 0 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 70/700W | 345 MHz | -| 1 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 67/700W | 345 MHz | -| 2 | NVIDIA H100 80GB HBM3 | 81559 MB | 21C | 67/700W | 345 MHz | -| 3 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 67/700W | 345 MHz | -| 4 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 67/700W | 345 MHz | -| 5 | NVIDIA H100 80GB HBM3 | 81559 MB | 22C | 69/700W | 345 MHz | -| 6 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 68/700W | 345 MHz | -| 7 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 66/700W | 345 MHz | - -## Memory Bandwidth - -Source: pytorch - -| Metric | Value | Peak | Efficiency | -|--------|-------|------|------------| -| H2D (PCIe) | 11.8 GB/s | 0 GB/s | 0.0% | -| D2H (PCIe) | 10.1 GB/s | 0 GB/s | 0.0% | -| D2D (NVLink) | 829.0 GB/s | 3400 GB/s | 24.4% | - -**Verdict: WARN** (D2D 829.0 GB/s via PyTorch fallback; nvbandwidth unavailable — figure is indicative only, not a true HBM peak) - -## Compute Throughput - -| DType | Achieved (TFLOPS) | Peak | Threshold | Status | -|-------|-------------------|------|------------|--------| -| FP32 | 51.9 | 67 | >= 54 | WARN | -| TF32 | 357.8 | 495 | >= 444 | FAIL | -| FP16 | 667.2 | 990 | >= 734 | WARN | -| BF16 | 699.1 | 990 | >= 745 | WARN | -| FP8 | 1146.2 | 1979 | >= 1400 | FAIL | - -**Verdict: FAIL** (absolute TFLOPS thresholds; worst efficiency 57.9%) - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_stress_smoke_reasons_aikubeworker0012.json b/reports_stress_smoke_reasons_aikubeworker0012.json deleted file mode 100644 index 2722c96..0000000 --- a/reports_stress_smoke_reasons_aikubeworker0012.json +++ /dev/null @@ -1,165 +0,0 @@ -{ - "stress": { - "source": "pytorch", - "passed": false, - "duration_sec": 45, - "elapsed_sec": 45.4, - "gpu_status": { - "0": "PASS", - "1": "PASS", - "2": "PASS", - "3": "PASS", - "4": "PASS", - "5": "PASS", - "6": "PASS", - "7": "PASS" - }, - "telemetry": { - "passed": false, - "samples": 39, - "steady_samples": 31, - "warmup_sec": 9.0, - "max_temp_c": { - "0": 59.0, - "1": 58.0, - "2": 65.0, - "3": 54.0, - "4": 59.0, - "5": 66.0, - "6": 62.0, - "7": 55.0 - }, - "avg_power_w": { - "0": 697.0, - "1": 697.4, - "2": 697.9, - "3": 698.0, - "4": 697.8, - "5": 697.6, - "6": 697.9, - "7": 698.2 - }, - "temp_delta_c": 12.0, - "throttle_events": [ - { - "gpu": 0, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 1, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 2, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 3, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 4, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 5, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 6, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 7, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 0, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 1, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 2, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 3, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 4, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 5, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 6, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 7, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 0, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 1, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 2, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 3, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - } - ], - "throttle_event_count": 248, - "xid_events": [], - "tflops_jitter_pct": 4.07, - "steady_tflops_samples": 781, - "failures": [ - "GPU temperature delta 12.0C exceeds 5.0C", - "non-idle throttle reasons observed in 248 samples (first: GPU 0 0x4)" - ], - "thresholds": { - "max_temp_c": 80.0, - "max_temp_delta_c": 5.0, - "min_power_w": 630.0, - "max_tflops_jitter_pct": 5.0, - "warmup_sec": 10.0, - "min_steady_samples": 10 - } - }, - "timestamp": "2026-05-22T17:52:09.074859" - }, - "timestamp": "2026-05-22T17:52:09.082873" -} \ No newline at end of file diff --git a/reports_stress_smoke_reasons_aikubeworker0012.md b/reports_stress_smoke_reasons_aikubeworker0012.md deleted file mode 100644 index cea30e2..0000000 --- a/reports_stress_smoke_reasons_aikubeworker0012.md +++ /dev/null @@ -1,29 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T17:52:09.082873 -- **Host:** aikubeworker0012 - -## Summary - -| Test | Result | -|------|--------| -| Stress Test | FAIL | - -## Stress Test - -- **Source:** pytorch -- **Duration:** 45s (requested 45s) -- **Telemetry samples:** 39 -- **Max temp:** {'0': 59.0, '1': 58.0, '2': 65.0, '3': 54.0, '4': 59.0, '5': 66.0, '6': 62.0, '7': 55.0} -- **Avg power:** {'0': 697.0, '1': 697.4, '2': 697.9, '3': 698.0, '4': 697.8, '5': 697.6, '6': 697.9, '7': 698.2} -- **Temp delta:** 12.0 C -- **TFLOPS jitter:** 4.07% -- **Throttle events:** 248 -- **XID events:** 0 -- **Failure reasons:** - - GPU temperature delta 12.0C exceeds 5.0C - - non-idle throttle reasons observed in 248 samples (first: GPU 0 0x4) -- **Result: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_stress_smoke_reasons_aikubeworker0016.json b/reports_stress_smoke_reasons_aikubeworker0016.json deleted file mode 100644 index 8d39f58..0000000 --- a/reports_stress_smoke_reasons_aikubeworker0016.json +++ /dev/null @@ -1,165 +0,0 @@ -{ - "stress": { - "source": "pytorch", - "passed": false, - "duration_sec": 45, - "elapsed_sec": 45.4, - "gpu_status": { - "0": "PASS", - "1": "PASS", - "2": "PASS", - "3": "PASS", - "4": "PASS", - "5": "PASS", - "6": "PASS", - "7": "PASS" - }, - "telemetry": { - "passed": false, - "samples": 39, - "steady_samples": 31, - "warmup_sec": 9.0, - "max_temp_c": { - "0": 50.0, - "1": 56.0, - "2": 57.0, - "3": 52.0, - "4": 51.0, - "5": 58.0, - "6": 53.0, - "7": 51.0 - }, - "avg_power_w": { - "0": 698.3, - "1": 698.5, - "2": 697.6, - "3": 697.9, - "4": 697.8, - "5": 698.0, - "6": 697.5, - "7": 698.0 - }, - "temp_delta_c": 8.0, - "throttle_events": [ - { - "gpu": 0, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 1, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 2, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 3, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 4, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 5, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 6, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 7, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 0, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 1, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 2, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 3, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 4, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 5, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 6, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 7, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 0, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 1, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 2, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - }, - { - "gpu": 3, - "throttle": "0x0000000000000004", - "real_throttle": "0x4" - } - ], - "throttle_event_count": 248, - "xid_events": [], - "tflops_jitter_pct": 3.77, - "steady_tflops_samples": 787, - "failures": [ - "GPU temperature delta 8.0C exceeds 5.0C", - "non-idle throttle reasons observed in 248 samples (first: GPU 0 0x4)" - ], - "thresholds": { - "max_temp_c": 80.0, - "max_temp_delta_c": 5.0, - "min_power_w": 630.0, - "max_tflops_jitter_pct": 5.0, - "warmup_sec": 10.0, - "min_steady_samples": 10 - } - }, - "timestamp": "2026-05-22T17:53:02.058687" - }, - "timestamp": "2026-05-22T17:53:02.066792" -} \ No newline at end of file diff --git a/reports_stress_smoke_reasons_aikubeworker0016.md b/reports_stress_smoke_reasons_aikubeworker0016.md deleted file mode 100644 index 9f9c3ab..0000000 --- a/reports_stress_smoke_reasons_aikubeworker0016.md +++ /dev/null @@ -1,29 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T17:53:02.066792 -- **Host:** aikubeworker0016 - -## Summary - -| Test | Result | -|------|--------| -| Stress Test | FAIL | - -## Stress Test - -- **Source:** pytorch -- **Duration:** 45s (requested 45s) -- **Telemetry samples:** 39 -- **Max temp:** {'0': 50.0, '1': 56.0, '2': 57.0, '3': 52.0, '4': 51.0, '5': 58.0, '6': 53.0, '7': 51.0} -- **Avg power:** {'0': 698.3, '1': 698.5, '2': 697.6, '3': 697.9, '4': 697.8, '5': 698.0, '6': 697.5, '7': 698.0} -- **Temp delta:** 8.0 C -- **TFLOPS jitter:** 3.77% -- **Throttle events:** 248 -- **XID events:** 0 -- **Failure reasons:** - - GPU temperature delta 8.0C exceeds 5.0C - - non-idle throttle reasons observed in 248 samples (first: GPU 0 0x4) -- **Result: FAIL** - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_test_all_latest_aikubeworker0012_20260522_203246.md b/reports_test_all_latest_aikubeworker0012_20260522_203246.md deleted file mode 100644 index 8853d18..0000000 --- a/reports_test_all_latest_aikubeworker0012_20260522_203246.md +++ /dev/null @@ -1,322 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T20:32:51.687830 -- **Host:** aikubeworker0012 -- **GPU:** NVIDIA H100 80GB HBM3 x8 -- **Driver:** 580.159.03 | **CUDA:** 13.0 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Failed or unverified items: -- Compute Throughput: FAIL (FP16 spread 3.04% > 3%) -- NCCL: FAIL -- Stress Test: FAIL -- RDMA: FAIL - -## Summary - -| Test | Result | -|------|--------| -| GPU Info | PASS (8 GPUs detected) | -| Health Check | PASS | -| Memory Bandwidth | PASS (108.1%) | -| Compute Throughput | FAIL (FP16 spread 3.04% > 3%) | -| NVLink/NVSwitch | PASS | -| DCGM | PASS | -| NCCL | FAIL | -| Stress Test | FAIL | -| RDMA | FAIL | -| Training | PASS (216498 tokens/sec) | - -## GPU Information - -| GPU | Model | VRAM | Temp | Power | SM Clock | -|-----|-------|------|------|-------|----------| -| 0 | NVIDIA H100 80GB HBM3 | 81559 MB | 25C | 69/700W | 345 MHz | -| 1 | NVIDIA H100 80GB HBM3 | 81559 MB | 25C | 73/700W | 345 MHz | -| 2 | NVIDIA H100 80GB HBM3 | 81559 MB | 26C | 69/700W | 345 MHz | -| 3 | NVIDIA H100 80GB HBM3 | 81559 MB | 24C | 69/700W | 345 MHz | -| 4 | NVIDIA H100 80GB HBM3 | 81559 MB | 24C | 69/700W | 345 MHz | -| 5 | NVIDIA H100 80GB HBM3 | 81559 MB | 27C | 70/700W | 345 MHz | -| 6 | NVIDIA H100 80GB HBM3 | 81559 MB | 25C | 70/700W | 345 MHz | -| 7 | NVIDIA H100 80GB HBM3 | 81559 MB | 24C | 71/700W | 345 MHz | - -## Health Check - -**Overall: PASS** - -| GPU | Temp | Power | ECC | PCIe | Throttle | Status | -|-----|------|-------|-----|------|----------|--------| -| 0 | 25C PASS | 69W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 1 | 25C PASS | 73W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 2 | 26C PASS | 69W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 3 | 24C PASS | 70W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 4 | 24C PASS | 69W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 5 | 27C PASS | 70W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 6 | 25C PASS | 70W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 7 | 24C PASS | 71W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | - -## Memory Bandwidth - -Source: nvbandwidth - -| Metric | Value | Peak | Efficiency | -|--------|-------|------|------------| -| H2D (PCIe) | 55.4 GB/s | 64 GB/s | 86.6% | -| D2H (PCIe) | 54.0 GB/s | 64 GB/s | 84.4% | -| D2D (NVLink) | 486.5 GB/s | 450 GB/s | 108.1% | - -**Verdict: PASS** (D2D efficiency 108.1%) - -## Compute Throughput - -| DType | Achieved (TFLOPS) | Peak | Threshold | Status | -|-------|-------------------|------|------------|--------| -| FP32 | 51.9 | 67 | >= 54 | FAIL | -| TF32 | 364.9 | 495 | >= 444 | FAIL | -| FP16 | 680.0 | 990 | >= 734 | FAIL | -| BF16 | 713.2 | 990 | >= 745 | FAIL | -| FP8 | 1170.4 | 1979 | >= 1400 | FAIL | -| FP64 | 46.9 | 67 | >= 63 | FAIL | -| INT8 | 100.4 | 1979 | >= 1536 | FAIL | - -**Verdict: FAIL** (absolute TFLOPS thresholds; worst efficiency 5.1%) - -### Compute Consistency - -| DType | Min | Mean | Max | Spread | Limit | Status | -|-------|-----|------|-----|--------|-------|--------| -| FP32 | 51.9 | 52.0 | 52.1 | 0.38% | <= 3% | PASS | -| TF32 | 361.0 | 364.9 | 369.0 | 2.19% | <= 3% | PASS | -| FP16 | 667.3 | 680.0 | 688.0 | 3.04% | <= 3% | FAIL | -| BF16 | 703.0 | 713.3 | 735.7 | 4.58% | <= 3% | FAIL | -| FP8 | 1156.9 | 1170.5 | 1186.1 | 2.49% | <= 3% | PASS | -| FP64 | 45.9 | 46.9 | 47.5 | 3.41% | <= 3% | FAIL | -| INT8 | 100.4 | 100.4 | 100.4 | 0.00% | <= 3% | PASS | - -### Compute Per-GPU TFLOPS - -| GPU | FP32 | TF32 | FP16 | BF16 | FP8 | FP64 | INT8 | -|---|---|---|---|---|---|---|---| -| 0 | 52.0 | 369.0 | 688.0 | 735.7 | 1186.1 | 47.5 | 100.4 | -| 1 | 51.9 | 365.6 | 675.3 | 711.6 | 1171.0 | 47.0 | 100.4 | -| 2 | 51.9 | 364.9 | 685.7 | 715.3 | 1175.3 | 47.1 | 100.4 | -| 3 | 51.9 | 364.0 | 679.9 | 704.0 | 1167.6 | 47.4 | 100.4 | -| 4 | 51.9 | 367.7 | 681.2 | 719.0 | 1178.0 | 46.6 | 100.4 | -| 5 | 52.0 | 364.3 | 680.8 | 712.3 | 1165.5 | 46.8 | 100.4 | -| 6 | 52.1 | 362.9 | 681.8 | 703.0 | 1156.9 | 46.9 | 100.4 | -| 7 | 51.9 | 361.0 | 667.3 | 705.3 | 1163.2 | 45.9 | 100.4 | - -## NVLink/NVSwitch - -**Overall: PASS** - -| GPU | Active Links | Issues | -|-----|--------------|--------| -| 0 | 18/18 | OK | -| 1 | 18/18 | OK | -| 2 | 18/18 | OK | -| 3 | 18/18 | OK | -| 4 | 18/18 | OK | -| 5 | 18/18 | OK | -| 6 | 18/18 | OK | -| 7 | 18/18 | OK | - -## DCGM Diagnostic - -**Overall: PASS** - -| Subtest | Status | -|---------|--------| -| Deployment/software/GPU0 | PASS | -| Deployment/software/GPU1 | PASS | -| Deployment/software/GPU2 | PASS | -| Deployment/software/GPU3 | PASS | -| Deployment/software/GPU4 | PASS | -| Deployment/software/GPU5 | PASS | -| Deployment/software/GPU6 | PASS | -| Deployment/software/GPU7 | PASS | -| Deployment/software/summary | PASS | -| Hardware/memory/GPU0 | PASS | -| Hardware/memory/GPU1 | PASS | -| Hardware/memory/GPU2 | PASS | -| Hardware/memory/GPU3 | PASS | -| Hardware/memory/GPU4 | PASS | -| Hardware/memory/GPU5 | PASS | -| Hardware/memory/GPU6 | PASS | -| Hardware/memory/GPU7 | PASS | -| Hardware/memory/summary | PASS | -| Hardware/diagnostic/GPU0 | PASS | -| Hardware/diagnostic/GPU1 | PASS | -| Hardware/diagnostic/GPU2 | PASS | -| Hardware/diagnostic/GPU3 | PASS | -| Hardware/diagnostic/GPU4 | PASS | -| Hardware/diagnostic/GPU5 | PASS | -| Hardware/diagnostic/GPU6 | PASS | -| Hardware/diagnostic/GPU7 | PASS | -| Hardware/diagnostic/summary | PASS | -| Hardware/nvbandwidth/GPU0 | PASS | -| Hardware/nvbandwidth/GPU1 | PASS | -| Hardware/nvbandwidth/GPU2 | PASS | -| Hardware/nvbandwidth/GPU3 | PASS | -| Hardware/nvbandwidth/GPU4 | PASS | -| Hardware/nvbandwidth/GPU5 | PASS | -| Hardware/nvbandwidth/GPU6 | PASS | -| Hardware/nvbandwidth/GPU7 | PASS | -| Hardware/nvbandwidth/summary | PASS | -| Integration/pcie/GPU0 | PASS | -| Integration/pcie/GPU1 | PASS | -| Integration/pcie/GPU2 | PASS | -| Integration/pcie/GPU3 | PASS | -| Integration/pcie/GPU4 | PASS | -| Integration/pcie/GPU5 | PASS | -| Integration/pcie/GPU6 | PASS | -| Integration/pcie/GPU7 | PASS | -| Integration/pcie/summary | PASS | -| Stress/targeted_stress/GPU0 | PASS | -| Stress/targeted_stress/GPU1 | PASS | -| Stress/targeted_stress/GPU2 | PASS | -| Stress/targeted_stress/GPU3 | PASS | -| Stress/targeted_stress/GPU4 | PASS | -| Stress/targeted_stress/GPU5 | PASS | -| Stress/targeted_stress/GPU6 | PASS | -| Stress/targeted_stress/GPU7 | PASS | -| Stress/targeted_stress/summary | PASS | -| Stress/targeted_power/GPU0 | PASS | -| Stress/targeted_power/GPU1 | PASS | -| Stress/targeted_power/GPU2 | PASS | -| Stress/targeted_power/GPU3 | PASS | -| Stress/targeted_power/GPU4 | PASS | -| Stress/targeted_power/GPU5 | PASS | -| Stress/targeted_power/GPU6 | PASS | -| Stress/targeted_power/GPU7 | PASS | -| Stress/targeted_power/summary | PASS | - -## NCCL Multi-GPU - -Source: nccl-tests | GPUs: 8 - -| Operation | Bus BW (GB/s) | Threshold | Status | -|-----------|---------------|-----------|--------| -| allreduce | 472.3 | >= 405 | FAIL | -| alltoall | 343.3 | >= 315 | FAIL | -| broadcast | 364.1 | >= 360 | FAIL | -| reducescatter | 352.8 | >= 405 | FAIL | -| allgather | 366.4 | >= 405 | FAIL | -| sendrecv | 369.0 | >= 360 | FAIL | - -### NCCL allreduce by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 24.9, 25.0, 24.7 | 24.7 | 24.9 | 0.50% | >= 405 | FAIL | -| 256M | 421.6, 421.8, 421.6 | 421.6 | 421.7 | 0.02% | >= 405 | PASS | -| 2G | 472.8, 472.7, 471.5 | 471.5 | 472.3 | 0.13% | >= 405 | PASS | - -### NCCL alltoall by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 8.1, 8.0, 8.0 | 8.0 | 8.0 | 0.59% | >= 315 | FAIL | -| 256M | 305.3, 314.9, 313.1 | 305.3 | 311.1 | 1.34% | >= 315 | FAIL | -| 2G | 342.1, 342.5, 345.4 | 342.1 | 343.3 | 0.43% | >= 315 | PASS | - -### NCCL broadcast by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.5, 14.6, 14.2 | 14.2 | 14.4 | 1.18% | >= 360 | FAIL | -| 256M | 344.2, 345.9, 344.6 | 344.2 | 344.9 | 0.21% | >= 360 | FAIL | -| 2G | 364.2, 364.0, 364.1 | 364.0 | 364.1 | 0.02% | >= 360 | PASS | - -### NCCL reducescatter by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.1, 13.8, 14.2 | 13.8 | 14.0 | 1.21% | >= 405 | FAIL | -| 256M | 328.6, 328.3, 328.2 | 328.2 | 328.4 | 0.05% | >= 405 | FAIL | -| 2G | 352.6, 352.4, 353.3 | 352.4 | 352.8 | 0.11% | >= 405 | FAIL | - -### NCCL allgather by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.6, 14.3, 14.4 | 14.3 | 14.4 | 0.86% | >= 405 | FAIL | -| 256M | 350.5, 350.4, 349.9 | 349.9 | 350.3 | 0.07% | >= 405 | FAIL | -| 2G | 366.3, 366.6, 366.2 | 366.2 | 366.4 | 0.05% | >= 405 | FAIL | - -### NCCL sendrecv by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 18.4, 18.4, 18.4 | 18.4 | 18.4 | 0.00% | >= 360 | FAIL | -| 256M | 350.9, 351.6, 351.4 | 350.9 | 351.3 | 0.08% | >= 360 | FAIL | -| 2G | 368.9, 369.1, 368.9 | 368.9 | 369.0 | 0.03% | >= 360 | PASS | - -**Overall: FAIL** - -## Stress Test - -- **Source:** pytorch -- **Duration:** 1800s (requested 1800s) -- **Telemetry samples:** 1266 -- **Max temp:** {0: 60.0, 1: 60.0, 2: 68.0, 3: 56.0, 4: 60.0, 5: 68.0, 6: 64.0, 7: 56.0} -- **Avg power:** {0: 697.7, 1: 697.5, 2: 697.1, 3: 697.8, 4: 697.8, 5: 697.9, 6: 697.7, 7: 698.3} -- **Temp delta:** 12.0 C -- **TFLOPS jitter:** 4.37% -- **Steady TFLOPS samples:** 37672 -- **Throttle events:** 9712 -- **XID events:** 0 -- **Failure reasons:** - - GPU temperature delta 12.0C exceeds 5.0C - - non-idle throttle reasons observed in 9712 samples (first: GPU 0 0x4) -- **Result: FAIL** - -## RDMA/InfiniBand - -### RDMA Port Checks - -| Device | Port | State | Rate | Required | Status | -|--------|------|-------|------|----------|--------| -| mlx5_0 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_1 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_4 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_5 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_6 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_7 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | - -| Test | Value | Threshold | Status | -|------|-------|-----------|--------| -| ib_write_bw | 49.5 GB/s | >= 47 GB/s | PASS | -| ib_read_bw | 39.1 GB/s | >= 47 GB/s | FAIL | -| ib_write_lat | 1.25 us | <= 2 us | PASS | -| ib_read_lat | 2.60 us | <= 3.5 us | PASS | -| ibping | local_loopback target=0x58 count=5 | 0% packet loss | PASS | - -- **PFC/ECN/CNP/congestion counters checked:** 146 -- **PFC/ECN/CNP/congestion non-zero:** no -- **Failure reasons:** - - mlx5_4 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - mlx5_5 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - ib_read_bw bandwidth 39.12GB/s < 47GB/s -**Overall: FAIL** - -## Training Simulation - -| Metric | Value | -|--------|-------| -| Model | synthetic_transformer_1.5b | -| Params | 1470.5M | -| Throughput | 216498 tokens/sec | -| Avg Step Time | 75.7 ms | -| Warmup Steps | 5 | -| Peak Memory | 18.1 GB | -| Final Loss | 0.0039 | -| Step Jitter | 1.89% | -| Distributed Mode | ddp | -| Verdict | PASS (216498 tokens/sec) | - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_test_all_latest_aikubeworker0016_20260522_203447.md b/reports_test_all_latest_aikubeworker0016_20260522_203447.md deleted file mode 100644 index 3a4077f..0000000 --- a/reports_test_all_latest_aikubeworker0016_20260522_203447.md +++ /dev/null @@ -1,322 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T20:34:52.129246 -- **Host:** aikubeworker0016 -- **GPU:** NVIDIA H100 80GB HBM3 x8 -- **Driver:** 580.159.03 | **CUDA:** 13.0 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Failed or unverified items: -- Compute Throughput: FAIL (BF16 spread 3.44% > 3%) -- NCCL: FAIL -- Stress Test: FAIL -- RDMA: FAIL - -## Summary - -| Test | Result | -|------|--------| -| GPU Info | PASS (8 GPUs detected) | -| Health Check | PASS | -| Memory Bandwidth | PASS (108.1%) | -| Compute Throughput | FAIL (BF16 spread 3.44% > 3%) | -| NVLink/NVSwitch | PASS | -| DCGM | PASS | -| NCCL | FAIL | -| Stress Test | FAIL | -| RDMA | FAIL | -| Training | PASS (216683 tokens/sec) | - -## GPU Information - -| GPU | Model | VRAM | Temp | Power | SM Clock | -|-----|-------|------|------|-------|----------| -| 0 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 70/700W | 345 MHz | -| 1 | NVIDIA H100 80GB HBM3 | 81559 MB | 21C | 68/700W | 345 MHz | -| 2 | NVIDIA H100 80GB HBM3 | 81559 MB | 21C | 67/700W | 345 MHz | -| 3 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 67/700W | 345 MHz | -| 4 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 68/700W | 345 MHz | -| 5 | NVIDIA H100 80GB HBM3 | 81559 MB | 22C | 69/700W | 345 MHz | -| 6 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 68/700W | 345 MHz | -| 7 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 66/700W | 345 MHz | - -## Health Check - -**Overall: PASS** - -| GPU | Temp | Power | ECC | PCIe | Throttle | Status | -|-----|------|-------|-----|------|----------|--------| -| 0 | 20C PASS | 70W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 1 | 21C PASS | 68W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 2 | 21C PASS | 67W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 3 | 20C PASS | 67W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 4 | 20C PASS | 68W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 5 | 22C PASS | 69W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 6 | 20C PASS | 68W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 7 | 20C PASS | 66W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | - -## Memory Bandwidth - -Source: nvbandwidth - -| Metric | Value | Peak | Efficiency | -|--------|-------|------|------------| -| H2D (PCIe) | 55.4 GB/s | 64 GB/s | 86.6% | -| D2H (PCIe) | 54.4 GB/s | 64 GB/s | 85.0% | -| D2D (NVLink) | 486.6 GB/s | 450 GB/s | 108.1% | - -**Verdict: PASS** (D2D efficiency 108.1%) - -## Compute Throughput - -| DType | Achieved (TFLOPS) | Peak | Threshold | Status | -|-------|-------------------|------|------------|--------| -| FP32 | 52.1 | 67 | >= 54 | FAIL | -| TF32 | 366.7 | 495 | >= 444 | FAIL | -| FP16 | 682.7 | 990 | >= 734 | FAIL | -| BF16 | 717.3 | 990 | >= 745 | FAIL | -| FP8 | 1173.5 | 1979 | >= 1400 | FAIL | -| FP64 | 47.4 | 67 | >= 63 | FAIL | -| INT8 | 100.4 | 1979 | >= 1536 | FAIL | - -**Verdict: FAIL** (absolute TFLOPS thresholds; worst efficiency 5.1%) - -### Compute Consistency - -| DType | Min | Mean | Max | Spread | Limit | Status | -|-------|-----|------|-----|--------|-------|--------| -| FP32 | 51.9 | 52.1 | 52.2 | 0.58% | <= 3% | PASS | -| TF32 | 362.3 | 366.7 | 369.2 | 1.88% | <= 3% | PASS | -| FP16 | 674.4 | 682.7 | 693.1 | 2.74% | <= 3% | PASS | -| BF16 | 705.3 | 717.2 | 730.0 | 3.44% | <= 3% | FAIL | -| FP8 | 1155.2 | 1173.5 | 1186.2 | 2.64% | <= 3% | PASS | -| FP64 | 46.3 | 47.4 | 48.5 | 4.64% | <= 3% | FAIL | -| INT8 | 100.4 | 100.4 | 100.4 | 0.00% | <= 3% | PASS | - -### Compute Per-GPU TFLOPS - -| GPU | FP32 | TF32 | FP16 | BF16 | FP8 | FP64 | INT8 | -|---|---|---|---|---|---|---|---| -| 0 | 52.2 | 362.3 | 674.4 | 714.3 | 1159.0 | 46.3 | 100.4 | -| 1 | 51.9 | 366.5 | 674.7 | 721.4 | 1185.4 | 47.7 | 100.4 | -| 2 | 52.2 | 367.4 | 693.1 | 730.0 | 1185.7 | 48.5 | 100.4 | -| 3 | 52.2 | 367.8 | 682.2 | 708.2 | 1163.4 | 47.4 | 100.4 | -| 4 | 52.0 | 366.4 | 686.9 | 714.1 | 1186.2 | 47.3 | 100.4 | -| 5 | 52.0 | 369.2 | 679.9 | 721.1 | 1155.2 | 47.3 | 100.4 | -| 6 | 51.9 | 365.1 | 677.7 | 705.3 | 1169.0 | 47.0 | 100.4 | -| 7 | 52.2 | 369.0 | 692.8 | 723.5 | 1184.3 | 47.6 | 100.4 | - -## NVLink/NVSwitch - -**Overall: PASS** - -| GPU | Active Links | Issues | -|-----|--------------|--------| -| 0 | 18/18 | OK | -| 1 | 18/18 | OK | -| 2 | 18/18 | OK | -| 3 | 18/18 | OK | -| 4 | 18/18 | OK | -| 5 | 18/18 | OK | -| 6 | 18/18 | OK | -| 7 | 18/18 | OK | - -## DCGM Diagnostic - -**Overall: PASS** - -| Subtest | Status | -|---------|--------| -| Deployment/software/GPU0 | PASS | -| Deployment/software/GPU1 | PASS | -| Deployment/software/GPU2 | PASS | -| Deployment/software/GPU3 | PASS | -| Deployment/software/GPU4 | PASS | -| Deployment/software/GPU5 | PASS | -| Deployment/software/GPU6 | PASS | -| Deployment/software/GPU7 | PASS | -| Deployment/software/summary | PASS | -| Hardware/memory/GPU0 | PASS | -| Hardware/memory/GPU1 | PASS | -| Hardware/memory/GPU2 | PASS | -| Hardware/memory/GPU3 | PASS | -| Hardware/memory/GPU4 | PASS | -| Hardware/memory/GPU5 | PASS | -| Hardware/memory/GPU6 | PASS | -| Hardware/memory/GPU7 | PASS | -| Hardware/memory/summary | PASS | -| Hardware/diagnostic/GPU0 | PASS | -| Hardware/diagnostic/GPU1 | PASS | -| Hardware/diagnostic/GPU2 | PASS | -| Hardware/diagnostic/GPU3 | PASS | -| Hardware/diagnostic/GPU4 | PASS | -| Hardware/diagnostic/GPU5 | PASS | -| Hardware/diagnostic/GPU6 | PASS | -| Hardware/diagnostic/GPU7 | PASS | -| Hardware/diagnostic/summary | PASS | -| Hardware/nvbandwidth/GPU0 | PASS | -| Hardware/nvbandwidth/GPU1 | PASS | -| Hardware/nvbandwidth/GPU2 | PASS | -| Hardware/nvbandwidth/GPU3 | PASS | -| Hardware/nvbandwidth/GPU4 | PASS | -| Hardware/nvbandwidth/GPU5 | PASS | -| Hardware/nvbandwidth/GPU6 | PASS | -| Hardware/nvbandwidth/GPU7 | PASS | -| Hardware/nvbandwidth/summary | PASS | -| Integration/pcie/GPU0 | PASS | -| Integration/pcie/GPU1 | PASS | -| Integration/pcie/GPU2 | PASS | -| Integration/pcie/GPU3 | PASS | -| Integration/pcie/GPU4 | PASS | -| Integration/pcie/GPU5 | PASS | -| Integration/pcie/GPU6 | PASS | -| Integration/pcie/GPU7 | PASS | -| Integration/pcie/summary | PASS | -| Stress/targeted_stress/GPU0 | PASS | -| Stress/targeted_stress/GPU1 | PASS | -| Stress/targeted_stress/GPU2 | PASS | -| Stress/targeted_stress/GPU3 | PASS | -| Stress/targeted_stress/GPU4 | PASS | -| Stress/targeted_stress/GPU5 | PASS | -| Stress/targeted_stress/GPU6 | PASS | -| Stress/targeted_stress/GPU7 | PASS | -| Stress/targeted_stress/summary | PASS | -| Stress/targeted_power/GPU0 | PASS | -| Stress/targeted_power/GPU1 | PASS | -| Stress/targeted_power/GPU2 | PASS | -| Stress/targeted_power/GPU3 | PASS | -| Stress/targeted_power/GPU4 | PASS | -| Stress/targeted_power/GPU5 | PASS | -| Stress/targeted_power/GPU6 | PASS | -| Stress/targeted_power/GPU7 | PASS | -| Stress/targeted_power/summary | PASS | - -## NCCL Multi-GPU - -Source: nccl-tests | GPUs: 8 - -| Operation | Bus BW (GB/s) | Threshold | Status | -|-----------|---------------|-----------|--------| -| allreduce | 472.4 | >= 405 | FAIL | -| alltoall | 344.3 | >= 315 | FAIL | -| broadcast | 363.6 | >= 360 | FAIL | -| reducescatter | 353.1 | >= 405 | FAIL | -| allgather | 366.4 | >= 405 | FAIL | -| sendrecv | 368.9 | >= 360 | FAIL | - -### NCCL allreduce by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 24.9, 24.4, 24.9 | 24.4 | 24.7 | 0.95% | >= 405 | FAIL | -| 256M | 421.9, 421.1, 421.9 | 421.1 | 421.6 | 0.09% | >= 405 | PASS | -| 2G | 472.6, 472.0, 472.5 | 472.0 | 472.4 | 0.06% | >= 405 | PASS | - -### NCCL alltoall by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 7.9, 7.8, 8.1 | 7.8 | 7.9 | 1.57% | >= 315 | FAIL | -| 256M | 298.7, 312.7, 303.2 | 298.7 | 304.9 | 1.91% | >= 315 | FAIL | -| 2G | 342.2, 345.4, 345.2 | 342.2 | 344.3 | 0.43% | >= 315 | PASS | - -### NCCL broadcast by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.5, 14.3, 14.4 | 14.3 | 14.4 | 0.57% | >= 360 | FAIL | -| 256M | 344.1, 344.3, 344.8 | 344.1 | 344.4 | 0.09% | >= 360 | FAIL | -| 2G | 364.0, 363.6, 363.3 | 363.3 | 363.6 | 0.08% | >= 360 | PASS | - -### NCCL reducescatter by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.0, 14.2, 14.3 | 14.0 | 14.2 | 0.88% | >= 405 | FAIL | -| 256M | 328.8, 328.7, 328.4 | 328.4 | 328.6 | 0.05% | >= 405 | FAIL | -| 2G | 351.9, 353.8, 353.6 | 351.9 | 353.1 | 0.24% | >= 405 | FAIL | - -### NCCL allgather by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.4, 13.9, 14.0 | 13.9 | 14.1 | 1.53% | >= 405 | FAIL | -| 256M | 350.2, 350.4, 350.7 | 350.2 | 350.4 | 0.06% | >= 405 | FAIL | -| 2G | 366.9, 366.4, 366.0 | 366.0 | 366.4 | 0.10% | >= 405 | FAIL | - -### NCCL sendrecv by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 18.4, 18.3, 18.5 | 18.3 | 18.4 | 0.44% | >= 360 | FAIL | -| 256M | 351.1, 351.4, 351.3 | 351.1 | 351.3 | 0.04% | >= 360 | FAIL | -| 2G | 368.9, 368.8, 368.9 | 368.8 | 368.9 | 0.01% | >= 360 | PASS | - -**Overall: FAIL** - -## Stress Test - -- **Source:** pytorch -- **Duration:** 1800s (requested 1800s) -- **Telemetry samples:** 1295 -- **Max temp:** {0: 51.0, 1: 59.0, 2: 61.0, 3: 53.0, 4: 53.0, 5: 62.0, 6: 56.0, 7: 52.0} -- **Avg power:** {0: 698.8, 1: 697.8, 2: 698.1, 3: 697.9, 4: 697.9, 5: 698.2, 6: 698.0, 7: 697.8} -- **Temp delta:** 11.0 C -- **TFLOPS jitter:** 3.4% -- **Steady TFLOPS samples:** 37874 -- **Throttle events:** 9944 -- **XID events:** 0 -- **Failure reasons:** - - GPU temperature delta 11.0C exceeds 5.0C - - non-idle throttle reasons observed in 9944 samples (first: GPU 0 0x4) -- **Result: FAIL** - -## RDMA/InfiniBand - -### RDMA Port Checks - -| Device | Port | State | Rate | Required | Status | -|--------|------|-------|------|----------|--------| -| mlx5_0 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_1 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_4 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_5 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_6 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_7 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | - -| Test | Value | Threshold | Status | -|------|-------|-----------|--------| -| ib_write_bw | 48.6 GB/s | >= 47 GB/s | PASS | -| ib_read_bw | 40.3 GB/s | >= 47 GB/s | FAIL | -| ib_write_lat | 1.29 us | <= 2 us | PASS | -| ib_read_lat | 2.59 us | <= 3.5 us | PASS | -| ibping | local_loopback target=0x4b count=5 | 0% packet loss | PASS | - -- **PFC/ECN/CNP/congestion counters checked:** 146 -- **PFC/ECN/CNP/congestion non-zero:** no -- **Failure reasons:** - - mlx5_4 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - mlx5_5 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - ib_read_bw bandwidth 40.29GB/s < 47GB/s -**Overall: FAIL** - -## Training Simulation - -| Metric | Value | -|--------|-------| -| Model | synthetic_transformer_1.5b | -| Params | 1470.5M | -| Throughput | 216683 tokens/sec | -| Avg Step Time | 75.6 ms | -| Warmup Steps | 5 | -| Peak Memory | 18.1 GB | -| Final Loss | 0.0039 | -| Step Jitter | 1.2% | -| Distributed Mode | ddp | -| Verdict | PASS (216683 tokens/sec) | - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_test_all_latest_summary_cn_20260523.md b/reports_test_all_latest_summary_cn_20260523.md deleted file mode 100644 index 87f4eab..0000000 --- a/reports_test_all_latest_summary_cn_20260523.md +++ /dev/null @@ -1,101 +0,0 @@ -# H100 单节点 test all 中文汇总 - -生成时间：2026-05-23 -测试范围：`aikubeworker0012`、`aikubeworker0016` 单节点 `python gpu_tester.py --test all --report --format md` - -原始报告： - -- `reports_test_all_latest_aikubeworker0012_20260522_203246.md` -- `reports_test_all_latest_aikubeworker0016_20260522_203447.md` - -## 总结论 - -| 机器 | Suite | PDF 验收结论 | 主要失败项 | -|---|---:|---|---| -| aikubeworker0012 | 6/10 PASS | FAIL | Compute、NCCL、Stress、RDMA | -| aikubeworker0016 | 6/10 PASS | FAIL | Compute、NCCL、Stress、RDMA | - -按 PDF 口径，任一必测子项 FAIL，则整机 FAIL。因此两台机器当前都不通过生产验收。 - -## 通过项 - -| 项目 | aikubeworker0012 | aikubeworker0016 | 说明 | -|---|---|---|---| -| GPU Info | PASS | PASS | 8 张 H100 | -| Health | PASS | PASS | 温度、空闲功耗、ECC、PCIe、空闲 throttle 正常 | -| Memory Bandwidth | PASS | PASS | D2D 效率均约 108.1% | -| NVLink/NVSwitch | PASS | PASS | 8 卡均 18/18 links | -| DCGM diag -r 3 | PASS | PASS | software、memory、diagnostic、nvbandwidth、pcie、targeted stress/power 全 PASS | -| Training Simulation | PASS | PASS | 8 卡 DDP synthetic 1.5B，loss finite | - -Training 结果： - -| 机器 | Throughput | Step jitter | Peak memory | Verdict | -|---|---:|---:|---:|---| -| aikubeworker0012 | 216498 tokens/s | 1.89% | 18.08 GB | PASS | -| aikubeworker0016 | 216683 tokens/s | 1.20% | 18.08 GB | PASS | - -## 失败项 - -### Compute - -两台机器都未达到当前 H100 绝对 TFLOPS 阈值，且部分 dtype 的跨 GPU spread 超过 3%。 - -| 机器 | 代表性失败 | -|---|---| -| aikubeworker0012 | FP16 spread 3.04%，BF16 spread 4.58%，FP64 spread 3.41%；FP32/TF32/FP16/BF16/FP8/FP64/INT8 绝对阈值均 FAIL | -| aikubeworker0016 | BF16 spread 3.44%，FP64 spread 4.64%；FP32/TF32/FP16/BF16/FP8/FP64/INT8 绝对阈值均 FAIL | - -### NCCL - -NCCL 已经使用真实 `nccl-tests` bus BW，不是 torchrun fallback。失败主要来自小 size 以及部分 256M/2G op 未达阈值。 - -| 机器 | allreduce best | alltoall best | broadcast best | reducescatter best | allgather best | sendrecv best | Verdict | -|---|---:|---:|---:|---:|---:|---:|---| -| aikubeworker0012 | 472.3 | 343.3 | 364.1 | 352.8 | 366.4 | 369.0 | FAIL | -| aikubeworker0016 | 472.4 | 344.3 | 363.6 | 353.1 | 366.4 | 368.9 | FAIL | - -关键原因： - -- `1M` size 在所有 op 上都明显低于阈值。 -- `reducescatter`、`allgather` 的 2G 也低于 405 GB/s 阈值。 -- `broadcast/sendrecv` 的 256M 低于 360 GB/s 阈值。 - -### Stress - -两台机器的 1800 秒 PyTorch BF16 GEMM 压力测试均跑满，但 telemetry 判定 FAIL。 - -| 机器 | 平均稳态功耗 | 最高温度范围 | 温差 | TFLOPS jitter | throttle events | XID | Verdict | -|---|---|---|---:|---:|---:|---:|---| -| aikubeworker0012 | 约 697-698W/GPU | 56-68C | 12C | 4.37% | 9712 | 0 | FAIL | -| aikubeworker0016 | 约 698W/GPU | 51-62C | 11C | 3.40% | 9944 | 0 | FAIL | - -失败原因： - -- GPU 间温差超过 5C 阈值。 -- 观测到大量非 idle throttle，首个原因是 `0x4`，即 `sw_power_cap`。 - -### RDMA/InfiniBand - -本轮 `test all` 是单节点 RDMA 路径，`ibping` 显示为 `local_loopback`。这份结果不能替代跨节点 RDMA 验收，但仍反映单节点 perftest read bandwidth 未达标。 - -| 机器 | ib_write_bw | ib_read_bw | ib_write_lat | ib_read_lat | Verdict | -|---|---:|---:|---:|---:|---| -| aikubeworker0012 | 49.5 GB/s PASS | 39.1 GB/s FAIL | 1.25 us PASS | 2.60 us PASS | FAIL | -| aikubeworker0016 | 48.6 GB/s PASS | 40.3 GB/s FAIL | 1.29 us PASS | 2.59 us PASS | FAIL | - -另外，两台机器都有 `mlx5_4`、`mlx5_5` 处于 ACTIVE 但速率为 100 Gb/sec，低于当前 400G 端口阈值，因此 RDMA port check 也有 FAIL。 - -## 当前阻塞 - -1. Compute 阈值口径较严，当前实测绝对 TFLOPS 全 dtype 未达配置阈值，尤其 INT8 路径仅约 100 TFLOPS。 -2. NCCL 真实 bus BW 已可测，但多 op/size 未达 PDF 阈值。 -3. Stress 负载可跑满 30 分钟，但温差和 `sw_power_cap` throttle 导致 FAIL。 -4. 单节点 RDMA read bandwidth 未达 47 GB/s，且部分 IB 端口速率低于 400G。 -5. 跨节点 RDMA 需要继续使用单独 server/client 报告；不能把本轮 `local_loopback` 当作跨节点验收。 - -## 状态判断 - -脚本能力已经基本补齐到 PDF 验收口径：真实 nccl-tests、30 分钟 stress telemetry、NVLink、DCGM r3、RDMA perftest/ibping/counter、逐 GPU compute、8 卡 DDP training、最终任一 FAIL 即整机 FAIL 都已经跑通。 - -当前剩余问题主要不是脚本缺项，而是两台机器的实际验收数据有多项未达标。 diff --git a/reports_test_all_pdf_aikubeworker0012_20260522_182656.md b/reports_test_all_pdf_aikubeworker0012_20260522_182656.md deleted file mode 100644 index 283d875..0000000 --- a/reports_test_all_pdf_aikubeworker0012_20260522_182656.md +++ /dev/null @@ -1,259 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T18:27:01.103760 -- **Host:** aikubeworker0012 -- **GPU:** NVIDIA H100 80GB HBM3 x8 -- **Driver:** 580.159.03 | **CUDA:** 13.0 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Failed or unverified items: -- Compute Throughput: FAIL (worst FP32 52 vs >= 54) -- DCGM: ERROR: dcgmi diag -r 3 timeout after 1200s -- NCCL: FAIL -- Stress Test: FAIL -- RDMA: FAIL -- Training: FAIL (188741 tokens/sec) - -## Summary - -| Test | Result | -|------|--------| -| GPU Info | PASS (8 GPUs detected) | -| Health Check | PASS | -| Memory Bandwidth | PASS (108.1%) | -| Compute Throughput | FAIL (worst FP32 52 vs >= 54) | -| NVLink/NVSwitch | PASS | -| DCGM | ERROR: dcgmi diag -r 3 timeout after 1200s | -| NCCL | FAIL | -| Stress Test | FAIL | -| RDMA | FAIL | -| Training | FAIL (188741 tokens/sec) | - -## GPU Information - -| GPU | Model | VRAM | Temp | Power | SM Clock | -|-----|-------|------|------|-------|----------| -| 0 | NVIDIA H100 80GB HBM3 | 81559 MB | 25C | 70/700W | 345 MHz | -| 1 | NVIDIA H100 80GB HBM3 | 81559 MB | 25C | 73/700W | 345 MHz | -| 2 | NVIDIA H100 80GB HBM3 | 81559 MB | 26C | 69/700W | 345 MHz | -| 3 | NVIDIA H100 80GB HBM3 | 81559 MB | 24C | 70/700W | 345 MHz | -| 4 | NVIDIA H100 80GB HBM3 | 81559 MB | 24C | 69/700W | 345 MHz | -| 5 | NVIDIA H100 80GB HBM3 | 81559 MB | 27C | 70/700W | 345 MHz | -| 6 | NVIDIA H100 80GB HBM3 | 81559 MB | 25C | 71/700W | 345 MHz | -| 7 | NVIDIA H100 80GB HBM3 | 81559 MB | 24C | 72/700W | 345 MHz | - -## Health Check - -**Overall: PASS** - -| GPU | Temp | Power | ECC | PCIe | Throttle | Status | -|-----|------|-------|-----|------|----------|--------| -| 0 | 25C PASS | 70W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 1 | 25C PASS | 73W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 2 | 26C PASS | 69W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 3 | 24C PASS | 70W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 4 | 24C PASS | 69W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 5 | 27C PASS | 70W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 6 | 25C PASS | 71W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 7 | 24C PASS | 72W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | - -## Memory Bandwidth - -Source: nvbandwidth - -| Metric | Value | Peak | Efficiency | -|--------|-------|------|------------| -| H2D (PCIe) | 55.5 GB/s | 64 GB/s | 86.7% | -| D2H (PCIe) | 54.3 GB/s | 64 GB/s | 84.8% | -| D2D (NVLink) | 486.6 GB/s | 450 GB/s | 108.1% | - -**Verdict: PASS** (D2D efficiency 108.1%) - -## Compute Throughput - -| DType | Achieved (TFLOPS) | Peak | Threshold | Status | -|-------|-------------------|------|------------|--------| -| FP32 | 52.0 | 67 | >= 54 | FAIL | -| TF32 | 364.8 | 495 | >= 444 | FAIL | -| FP16 | 685.0 | 990 | >= 734 | FAIL | -| BF16 | 715.9 | 990 | >= 745 | FAIL | -| FP8 | 1166.6 | 1979 | >= 1400 | FAIL | -| FP64 | 46.9 | 0 | >= 63 | FAIL | -| INT8 | 100.4 | 0 | >= 1536 | FAIL | - -**Verdict: FAIL** (absolute TFLOPS thresholds; worst efficiency 58.9%) - -### Compute Consistency - -| DType | Min | Mean | Max | Spread | Limit | Status | -|-------|-----|------|-----|--------|-------|--------| -| FP32 | 51.9 | 52.0 | 52.2 | 0.58% | <= 3% | PASS | -| TF32 | 360.9 | 364.9 | 368.2 | 2.00% | <= 3% | PASS | -| FP16 | 676.0 | 685.0 | 689.9 | 2.03% | <= 3% | PASS | -| BF16 | 697.3 | 715.9 | 730.2 | 4.60% | <= 3% | FAIL | -| FP8 | 1141.8 | 1166.6 | 1180.3 | 3.30% | <= 3% | FAIL | -| FP64 | 45.8 | 46.9 | 47.7 | 4.05% | <= 3% | FAIL | -| INT8 | 100.4 | 100.4 | 100.4 | 0.00% | <= 3% | PASS | - -### Compute Per-GPU TFLOPS - -| GPU | FP32 | TF32 | FP16 | BF16 | FP8 | FP64 | INT8 | -|---|---|---|---|---|---|---|---| -| 0 | 51.9 | 368.2 | 689.5 | 730.2 | 1180.3 | 47.1 | 100.4 | -| 1 | 51.9 | 366.8 | 688.7 | 721.6 | 1170.1 | 47.7 | 100.4 | -| 2 | 51.9 | 366.3 | 689.9 | 711.3 | 1167.8 | 47.2 | 100.4 | -| 3 | 51.9 | 363.0 | 677.6 | 699.2 | 1176.3 | 46.6 | 100.4 | -| 4 | 52.2 | 365.3 | 685.0 | 725.4 | 1163.0 | 46.8 | 100.4 | -| 5 | 52.1 | 363.9 | 684.2 | 725.0 | 1172.1 | 46.9 | 100.4 | -| 6 | 51.9 | 364.4 | 688.8 | 717.3 | 1161.2 | 46.9 | 100.4 | -| 7 | 51.9 | 360.9 | 676.0 | 697.3 | 1141.8 | 45.8 | 100.4 | - -## NVLink/NVSwitch - -**Overall: PASS** - -| GPU | Active Links | Issues | -|-----|--------------|--------| -| 0 | 18/18 | OK | -| 1 | 18/18 | OK | -| 2 | 18/18 | OK | -| 3 | 18/18 | OK | -| 4 | 18/18 | OK | -| 5 | 18/18 | OK | -| 6 | 18/18 | OK | -| 7 | 18/18 | OK | - -## DCGM Diagnostic - -**Overall: FAIL** (dcgmi diag -r 3 timeout after 1200s) - -## NCCL Multi-GPU - -Source: nccl-tests | GPUs: 8 - -| Operation | Bus BW (GB/s) | Threshold | Status | -|-----------|---------------|-----------|--------| -| allreduce | 472.4 | >= 405 | FAIL | -| alltoall | 344.4 | >= 315 | FAIL | -| broadcast | 363.8 | >= 360 | FAIL | -| reducescatter | 353.0 | >= 405 | FAIL | -| allgather | 366.4 | >= 405 | FAIL | -| sendrecv | 368.9 | >= 360 | FAIL | - -### NCCL allreduce by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 24.0, 24.9, 24.7 | 24.0 | 24.5 | 1.57% | >= 405 | FAIL | -| 256M | 421.4, 421.7, 421.4 | 421.4 | 421.5 | 0.03% | >= 405 | PASS | -| 2G | 471.8, 473.0, 472.3 | 471.8 | 472.4 | 0.10% | >= 405 | PASS | - -### NCCL alltoall by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 8.1, 8.0, 8.0 | 8.0 | 8.0 | 0.59% | >= 315 | FAIL | -| 256M | 312.3, 310.9, 319.2 | 310.9 | 314.1 | 1.15% | >= 315 | FAIL | -| 2G | 343.1, 346.2, 344.0 | 343.1 | 344.4 | 0.38% | >= 315 | PASS | - -### NCCL broadcast by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.6, 13.6, 14.5 | 13.6 | 14.2 | 3.16% | >= 360 | FAIL | -| 256M | 343.8, 344.2, 344.5 | 343.8 | 344.2 | 0.08% | >= 360 | FAIL | -| 2G | 363.5, 363.3, 364.7 | 363.3 | 363.8 | 0.17% | >= 360 | PASS | - -### NCCL reducescatter by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.1, 14.3, 14.3 | 14.1 | 14.2 | 0.66% | >= 405 | FAIL | -| 256M | 328.1, 328.3, 328.3 | 328.1 | 328.2 | 0.03% | >= 405 | FAIL | -| 2G | 354.0, 352.6, 352.3 | 352.3 | 353.0 | 0.21% | >= 405 | FAIL | - -### NCCL allgather by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.5, 14.5, 14.3 | 14.3 | 14.4 | 0.65% | >= 405 | FAIL | -| 256M | 350.7, 350.7, 350.5 | 350.5 | 350.6 | 0.03% | >= 405 | FAIL | -| 2G | 366.6, 366.3, 366.3 | 366.3 | 366.4 | 0.04% | >= 405 | FAIL | - -### NCCL sendrecv by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 18.5, 18.4, 18.1 | 18.1 | 18.3 | 0.93% | >= 360 | FAIL | -| 256M | 352.3, 350.6, 350.5 | 350.5 | 351.1 | 0.24% | >= 360 | FAIL | -| 2G | 368.8, 369.0, 368.8 | 368.8 | 368.9 | 0.03% | >= 360 | PASS | - -**Overall: FAIL** - -## Stress Test - -- **Source:** pytorch -- **Duration:** 1800s (requested 1800s) -- **Telemetry samples:** 1541 -- **Max temp:** {0: 60.0, 1: 60.0, 2: 68.0, 3: 56.0, 4: 60.0, 5: 68.0, 6: 65.0, 7: 56.0} -- **Avg power:** {0: 697.7, 1: 697.4, 2: 697.2, 3: 697.7, 4: 697.5, 5: 698.0, 6: 697.8, 7: 698.4} -- **Temp delta:** 12.0 C -- **TFLOPS jitter:** 3.16% -- **Steady TFLOPS samples:** 37676 -- **Throttle events:** 11912 -- **XID events:** 0 -- **Failure reasons:** - - GPU temperature delta 12.0C exceeds 5.0C - - non-idle throttle reasons observed in 11912 samples (first: GPU 0 0x4) -- **Result: FAIL** - -## RDMA/InfiniBand - -### RDMA Port Checks - -| Device | Port | State | Rate | Required | Status | -|--------|------|-------|------|----------|--------| -| mlx5_0 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_1 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_4 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_5 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_6 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_7 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | - -| Test | Value | Threshold | Status | -|------|-------|-----------|--------| -| ib_write_bw | 49.2 GB/s | >= 47 GB/s | PASS | -| ib_read_bw | 39.1 GB/s | >= 47 GB/s | FAIL | -| ib_write_lat | 5.68 us | <= 2 us | FAIL | -| ib_read_lat | 16.00 us | <= 3.5 us | FAIL | -| ibping | target=0x58 count=5 | 0% packet loss | PASS | - -- **PFC/ECN/CNP/congestion counters checked:** 0 -- **PFC/ECN/CNP/congestion non-zero:** no -- **Failure reasons:** - - mlx5_4 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - mlx5_5 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - ib_read_bw bandwidth 39.11GB/s < 47GB/s - - ib_write_lat latency 5.68us > 2.0us - - ib_read_lat latency 16.0us > 3.5us -**Overall: FAIL** - -## Training Simulation - -| Metric | Value | -|--------|-------| -| Model | synthetic_transformer_1.5b | -| Params | 1470.5M | -| Throughput | 188741 tokens/sec | -| Avg Step Time | 86.8 ms | -| Peak Memory | 18.1 GB | -| Final Loss | 0.0041 | -| Step Jitter | 626.74% | -| Distributed Mode | ddp | -| Verdict | FAIL (188741 tokens/sec) | - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_test_all_pdf_aikubeworker0016_20260522_182856.md b/reports_test_all_pdf_aikubeworker0016_20260522_182856.md deleted file mode 100644 index dbee788..0000000 --- a/reports_test_all_pdf_aikubeworker0016_20260522_182856.md +++ /dev/null @@ -1,259 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T18:29:01.245683 -- **Host:** aikubeworker0016 -- **GPU:** NVIDIA H100 80GB HBM3 x8 -- **Driver:** 580.159.03 | **CUDA:** 13.0 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Failed or unverified items: -- Compute Throughput: FAIL (worst FP32 52 vs >= 54) -- DCGM: ERROR: dcgmi diag -r 3 timeout after 1200s -- NCCL: FAIL -- Stress Test: FAIL -- RDMA: FAIL -- Training: FAIL (193836 tokens/sec) - -## Summary - -| Test | Result | -|------|--------| -| GPU Info | PASS (8 GPUs detected) | -| Health Check | PASS | -| Memory Bandwidth | PASS (108.1%) | -| Compute Throughput | FAIL (worst FP32 52 vs >= 54) | -| NVLink/NVSwitch | PASS | -| DCGM | ERROR: dcgmi diag -r 3 timeout after 1200s | -| NCCL | FAIL | -| Stress Test | FAIL | -| RDMA | FAIL | -| Training | FAIL (193836 tokens/sec) | - -## GPU Information - -| GPU | Model | VRAM | Temp | Power | SM Clock | -|-----|-------|------|------|-------|----------| -| 0 | NVIDIA H100 80GB HBM3 | 81559 MB | 19C | 70/700W | 345 MHz | -| 1 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 67/700W | 345 MHz | -| 2 | NVIDIA H100 80GB HBM3 | 81559 MB | 20C | 67/700W | 345 MHz | -| 3 | NVIDIA H100 80GB HBM3 | 81559 MB | 19C | 67/700W | 345 MHz | -| 4 | NVIDIA H100 80GB HBM3 | 81559 MB | 19C | 67/700W | 345 MHz | -| 5 | NVIDIA H100 80GB HBM3 | 81559 MB | 21C | 69/700W | 345 MHz | -| 6 | NVIDIA H100 80GB HBM3 | 81559 MB | 19C | 68/700W | 345 MHz | -| 7 | NVIDIA H100 80GB HBM3 | 81559 MB | 19C | 66/700W | 345 MHz | - -## Health Check - -**Overall: PASS** - -| GPU | Temp | Power | ECC | PCIe | Throttle | Status | -|-----|------|-------|-----|------|----------|--------| -| 0 | 19C PASS | 70W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 1 | 20C PASS | 67W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 2 | 20C PASS | 67W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 3 | 19C PASS | 67W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 4 | 19C PASS | 67W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 5 | 21C PASS | 69W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 6 | 19C PASS | 68W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | -| 7 | 19C PASS | 66W PASS | S:0 D:0 | Gen5x16 | PASS | **PASS** | - -## Memory Bandwidth - -Source: nvbandwidth - -| Metric | Value | Peak | Efficiency | -|--------|-------|------|------------| -| H2D (PCIe) | 55.5 GB/s | 64 GB/s | 86.7% | -| D2H (PCIe) | 54.7 GB/s | 64 GB/s | 85.5% | -| D2D (NVLink) | 486.6 GB/s | 450 GB/s | 108.1% | - -**Verdict: PASS** (D2D efficiency 108.1%) - -## Compute Throughput - -| DType | Achieved (TFLOPS) | Peak | Threshold | Status | -|-------|-------------------|------|------------|--------| -| FP32 | 52.0 | 67 | >= 54 | FAIL | -| TF32 | 366.2 | 495 | >= 444 | FAIL | -| FP16 | 684.8 | 990 | >= 734 | FAIL | -| BF16 | 720.7 | 990 | >= 745 | FAIL | -| FP8 | 1180.3 | 1979 | >= 1400 | FAIL | -| FP64 | 47.3 | 0 | >= 63 | FAIL | -| INT8 | 100.5 | 0 | >= 1536 | FAIL | - -**Verdict: FAIL** (absolute TFLOPS thresholds; worst efficiency 59.6%) - -### Compute Consistency - -| DType | Min | Mean | Max | Spread | Limit | Status | -|-------|-----|------|-----|--------|-------|--------| -| FP32 | 51.9 | 52.0 | 52.2 | 0.58% | <= 3% | PASS | -| TF32 | 361.1 | 366.2 | 368.9 | 2.13% | <= 3% | PASS | -| FP16 | 672.6 | 684.8 | 695.0 | 3.27% | <= 3% | FAIL | -| BF16 | 703.6 | 720.7 | 734.2 | 4.25% | <= 3% | FAIL | -| FP8 | 1158.6 | 1180.3 | 1241.8 | 7.05% | <= 3% | FAIL | -| FP64 | 46.7 | 47.3 | 48.0 | 2.75% | <= 3% | PASS | -| INT8 | 100.4 | 100.5 | 101.1 | 0.70% | <= 3% | PASS | - -### Compute Per-GPU TFLOPS - -| GPU | FP32 | TF32 | FP16 | BF16 | FP8 | FP64 | INT8 | -|---|---|---|---|---|---|---|---| -| 0 | 51.9 | 361.1 | 673.3 | 703.6 | 1158.6 | 46.7 | 100.4 | -| 1 | 52.0 | 367.0 | 684.0 | 725.7 | 1184.3 | 47.3 | 100.4 | -| 2 | 52.2 | 368.7 | 695.0 | 734.2 | 1197.7 | 48.0 | 100.4 | -| 3 | 51.9 | 367.8 | 688.0 | 708.1 | 1174.8 | 47.3 | 100.4 | -| 4 | 52.0 | 365.2 | 688.4 | 718.2 | 1160.5 | 47.0 | 101.1 | -| 5 | 52.1 | 368.9 | 684.2 | 733.7 | 1160.5 | 47.3 | 100.4 | -| 6 | 51.9 | 364.0 | 672.6 | 715.6 | 1164.4 | 47.1 | 100.4 | -| 7 | 51.9 | 367.0 | 692.5 | 726.5 | 1241.8 | 47.6 | 100.4 | - -## NVLink/NVSwitch - -**Overall: PASS** - -| GPU | Active Links | Issues | -|-----|--------------|--------| -| 0 | 18/18 | OK | -| 1 | 18/18 | OK | -| 2 | 18/18 | OK | -| 3 | 18/18 | OK | -| 4 | 18/18 | OK | -| 5 | 18/18 | OK | -| 6 | 18/18 | OK | -| 7 | 18/18 | OK | - -## DCGM Diagnostic - -**Overall: FAIL** (dcgmi diag -r 3 timeout after 1200s) - -## NCCL Multi-GPU - -Source: nccl-tests | GPUs: 8 - -| Operation | Bus BW (GB/s) | Threshold | Status | -|-----------|---------------|-----------|--------| -| allreduce | 472.5 | >= 405 | FAIL | -| alltoall | 344.2 | >= 315 | FAIL | -| broadcast | 363.8 | >= 360 | FAIL | -| reducescatter | 352.5 | >= 405 | FAIL | -| allgather | 366.8 | >= 405 | FAIL | -| sendrecv | 369.0 | >= 360 | FAIL | - -### NCCL allreduce by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 24.7, 24.1, 24.5 | 24.1 | 24.4 | 1.02% | >= 405 | FAIL | -| 256M | 421.8, 422.1, 421.4 | 421.4 | 421.8 | 0.07% | >= 405 | PASS | -| 2G | 472.8, 472.2, 472.6 | 472.2 | 472.5 | 0.05% | >= 405 | PASS | - -### NCCL alltoall by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 8.0, 8.0, 7.9 | 7.9 | 8.0 | 0.59% | >= 315 | FAIL | -| 256M | 326.8, 315.4, 315.8 | 315.4 | 319.3 | 1.65% | >= 315 | PASS | -| 2G | 344.2, 343.8, 344.6 | 343.8 | 344.2 | 0.09% | >= 315 | PASS | - -### NCCL broadcast by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.4, 14.2, 14.1 | 14.1 | 14.2 | 0.88% | >= 360 | FAIL | -| 256M | 345.3, 344.9, 344.4 | 344.4 | 344.9 | 0.11% | >= 360 | FAIL | -| 2G | 363.6, 363.9, 363.8 | 363.6 | 363.8 | 0.03% | >= 360 | PASS | - -### NCCL reducescatter by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.3, 14.1, 14.1 | 14.1 | 14.2 | 0.67% | >= 405 | FAIL | -| 256M | 328.2, 328.3, 328.4 | 328.2 | 328.3 | 0.02% | >= 405 | FAIL | -| 2G | 352.2, 352.7, 352.6 | 352.2 | 352.5 | 0.06% | >= 405 | FAIL | - -### NCCL allgather by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 14.2, 14.5, 14.3 | 14.2 | 14.3 | 0.87% | >= 405 | FAIL | -| 256M | 350.6, 350.6, 350.5 | 350.5 | 350.6 | 0.01% | >= 405 | FAIL | -| 2G | 367.0, 366.8, 366.5 | 366.5 | 366.8 | 0.06% | >= 405 | FAIL | - -### NCCL sendrecv by size - -| Size | Runs Bus BW (GB/s) | Worst | Mean | StdDev | Threshold | Status | -|------|---------------------|-------|------|--------|-----------|--------| -| 1M | 18.4, 18.2, 18.6 | 18.2 | 18.4 | 0.89% | >= 360 | FAIL | -| 256M | 350.7, 350.8, 351.1 | 350.7 | 350.9 | 0.05% | >= 360 | FAIL | -| 2G | 369.0, 369.0, 368.9 | 368.9 | 369.0 | 0.01% | >= 360 | PASS | - -**Overall: FAIL** - -## Stress Test - -- **Source:** pytorch -- **Duration:** 1800s (requested 1800s) -- **Telemetry samples:** 1541 -- **Max temp:** {0: 51.0, 1: 59.0, 2: 62.0, 3: 53.0, 4: 53.0, 5: 62.0, 6: 57.0, 7: 53.0} -- **Avg power:** {0: 698.7, 1: 698.0, 2: 698.1, 3: 697.9, 4: 697.7, 5: 698.2, 6: 698.0, 7: 697.7} -- **Temp delta:** 11.0 C -- **TFLOPS jitter:** 3.05% -- **Steady TFLOPS samples:** 37841 -- **Throttle events:** 11912 -- **XID events:** 0 -- **Failure reasons:** - - GPU temperature delta 11.0C exceeds 5.0C - - non-idle throttle reasons observed in 11912 samples (first: GPU 0 0x4) -- **Result: FAIL** - -## RDMA/InfiniBand - -### RDMA Port Checks - -| Device | Port | State | Rate | Required | Status | -|--------|------|-------|------|----------|--------| -| mlx5_0 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_1 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_4 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_5 | 1 | 4: ACTIVE | 100 Gb/sec (2X HDR) | >= 400Gbps ACTIVE | FAIL | -| mlx5_6 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | -| mlx5_7 | 1 | 4: ACTIVE | 400 Gb/sec (4X NDR) | >= 400Gbps ACTIVE | PASS | - -| Test | Value | Threshold | Status | -|------|-------|-----------|--------| -| ib_write_bw | 48.4 GB/s | >= 47 GB/s | PASS | -| ib_read_bw | 40.3 GB/s | >= 47 GB/s | FAIL | -| ib_write_lat | 2.44 us | <= 2 us | FAIL | -| ib_read_lat | 16.00 us | <= 3.5 us | FAIL | -| ibping | target=0x4b count=5 | 0% packet loss | PASS | - -- **PFC/ECN/CNP/congestion counters checked:** 0 -- **PFC/ECN/CNP/congestion non-zero:** no -- **Failure reasons:** - - mlx5_4 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - mlx5_5 port 1 state/rate failed (4: ACTIVE, 100 Gb/sec (2X HDR); required >= 400.0Gbps ACTIVE) - - ib_read_bw bandwidth 40.29GB/s < 47GB/s - - ib_write_lat latency 2.44us > 2.0us - - ib_read_lat latency 16.0us > 3.5us -**Overall: FAIL** - -## Training Simulation - -| Metric | Value | -|--------|-------| -| Model | synthetic_transformer_1.5b | -| Params | 1470.5M | -| Throughput | 193836 tokens/sec | -| Avg Step Time | 84.5 ms | -| Peak Memory | 18.1 GB | -| Final Loss | 0.004 | -| Step Jitter | 521.24% | -| Distributed Mode | ddp | -| Verdict | FAIL (193836 tokens/sec) | - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_training_warmup_aikubeworker0012_20260522_194528.md b/reports_training_warmup_aikubeworker0012_20260522_194528.md deleted file mode 100644 index 948e866..0000000 --- a/reports_training_warmup_aikubeworker0012_20260522_194528.md +++ /dev/null @@ -1,43 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T19:46:07.450315 -- **Host:** aikubeworker0012 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- DCGM - -## Summary - -| Test | Result | -|------|--------| -| Training | PASS (216654 tokens/sec) | - -## Training Simulation - -| Metric | Value | -|--------|-------| -| Model | synthetic_transformer_1.5b | -| Params | 1470.5M | -| Throughput | 216654 tokens/sec | -| Avg Step Time | 75.6 ms | -| Warmup Steps | 5 | -| Peak Memory | 18.1 GB | -| Final Loss | 0.0039 | -| Step Jitter | 0.87% | -| Distributed Mode | ddp | -| Verdict | PASS (216654 tokens/sec) | - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file diff --git a/reports_training_warmup_aikubeworker0016_20260522_194609.md b/reports_training_warmup_aikubeworker0016_20260522_194609.md deleted file mode 100644 index 61570ca..0000000 --- a/reports_training_warmup_aikubeworker0016_20260522_194609.md +++ /dev/null @@ -1,43 +0,0 @@ -# GPU Test Report - -- **Date:** 2026-05-22T19:46:48.023650 -- **Host:** aikubeworker0016 - -## Overall Acceptance Verdict - -**Result: FAIL** - -Missing required evidence: -- GPU Info -- Health Check -- Memory Bandwidth -- Compute Throughput -- NVLink/NVSwitch -- NCCL -- Stress Test -- RDMA -- DCGM - -## Summary - -| Test | Result | -|------|--------| -| Training | PASS (217236 tokens/sec) | - -## Training Simulation - -| Metric | Value | -|--------|-------| -| Model | synthetic_transformer_1.5b | -| Params | 1470.5M | -| Throughput | 217236 tokens/sec | -| Avg Step Time | 75.4 ms | -| Warmup Steps | 5 | -| Peak Memory | 18.1 GB | -| Final Loss | 0.0039 | -| Step Jitter | 1.23% | -| Distributed Mode | ddp | -| Verdict | PASS (217236 tokens/sec) | - ---- -*Generated by GPU Test Suite v0.2.0* \ No newline at end of file