han.zhao/test_gpu_scripts
1
0
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity
test_gpu_scripts/reports_multinode_nccl_diagnostic_2x8_sshfix.md

2.2 KiB
Raw Blame History

GPU Test Report

  • Date: 2026-05-23T07:46:11.464439
  • Host: aikubeworker0012

Overall Acceptance Verdict

Result: FAIL

Missing required evidence:

  • GPU Info
  • Health Check
  • Memory Bandwidth
  • Compute Throughput
  • NVLink/NVSwitch
  • NCCL
  • Stress Test
  • RDMA
  • DCGM
  • Training

Summary

Test Result
Multi-node NCCL FAIL

Multi-node NCCL / Cross Leaf

Source: nccl-tests-mpirun | Mode: diagnostic

  • Hosts: nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16)
  • Preflight: PASS

Multi-node NCCL allreduce

Topology Peak Bus BW Peak Size Avg Bus BW Threshold Status
2 nodes x 8 GPUs diagnostic 67.42 GB/s 256M 67.50 GB/s >= 480 GB/s FAIL
Topology NCCL Network GPU Direct RDMA GDR Disabled HCAs
2 nodes x 8 GPUs diagnostic IB DISABLED mlx5_0, mlx5_1, mlx5_6, mlx5_7
Topology Return Code Error / Output Tail
2 nodes x 8 GPUs diagnostic 0 orker0016:986293:986293 [1] NCCL INFO comm 0x563abe94c350 rank 9 nranks 16 cudaDev 1 busId 2a000 - Destroy COMPLETE aikubeworker0016:986292:986292 [0] NCCL INFO comm 0x560ffac51160 rank 8 nranks 16 cudaDev 0 busId 18000 - Destroy COMPLETE

Multi-node NCCL alltoall

Topology Peak Bus BW Peak Size Avg Bus BW Threshold Status
2 nodes x 8 GPUs diagnostic 9.56 GB/s 256M 9.55 GB/s >= 75 GB/s FAIL
Topology NCCL Network GPU Direct RDMA GDR Disabled HCAs
2 nodes x 8 GPUs diagnostic IB DISABLED mlx5_0, mlx5_1, mlx5_6, mlx5_7
Topology Return Code Error / Output Tail
2 nodes x 8 GPUs diagnostic 0 TE aikubeworker0012:2141982:2141982 [4] NCCL INFO comm 0x55d0bf9c6a00 rank 4 nranks 16 cudaDev 4 busId 9a000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 9.55234 # # Collective test concluded: alltoall_perf #

Overall: FAIL

Generated by GPU Test Suite v0.2.0