GPU Test Report
- Date: 2026-05-23T07:54:48.990378
- Host: aikubeworker0012
Overall Acceptance Verdict
Result: FAIL
Missing required evidence:
- GPU Info
- Health Check
- Memory Bandwidth
- Compute Throughput
- NVLink/NVSwitch
- NCCL
- Stress Test
- RDMA
- DCGM
- Training
Summary
| Test |
Result |
| Multi-node NCCL |
FAIL |
Multi-node NCCL / Cross Leaf
Source: nccl-tests-mpirun | Mode: sweep-nccl-2.27.7
- Hosts: nccl-gpu-1(172.72.8.12), nccl-gpu-2(172.72.8.16)
- Preflight: PASS
Multi-node NCCL allreduce
| Topology |
Peak Bus BW |
Peak Size |
Avg Bus BW |
Threshold |
Status |
| 2 nodes x 8 GPUs NCCL 2.27.7 sweep |
237.26 GB/s |
4G |
150.62 GB/s |
>= 480 GB/s |
FAIL |
| Topology |
NCCL Network |
GPU Direct RDMA |
GDR Enabled HCAs |
GDR Disabled HCAs |
| 2 nodes x 8 GPUs NCCL 2.27.7 sweep |
IB |
ENABLED |
mlx5_0, mlx5_1, mlx5_6, mlx5_7 |
- |
| Topology |
Return Code |
Error / Output Tail |
| 2 nodes x 8 GPUs NCCL 2.27.7 sweep |
0 |
aikubeworker0012:2145024:2145189 [0] NCCL INFO comm 0x561f7dc1f780 rank 0 nranks 16 cudaDev 0 busId 18000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 150.624 # # Collective test concluded: all_reduce_perf # |
Multi-node NCCL alltoall
| Topology |
Peak Bus BW |
Peak Size |
Avg Bus BW |
Threshold |
Status |
| 2 nodes x 8 GPUs NCCL 2.27.7 sweep |
28.78 GB/s |
1G |
23.57 GB/s |
>= 75 GB/s |
FAIL |
| Topology |
NCCL Network |
GPU Direct RDMA |
GDR Enabled HCAs |
GDR Disabled HCAs |
| 2 nodes x 8 GPUs NCCL 2.27.7 sweep |
IB |
ENABLED |
mlx5_0, mlx5_1, mlx5_6, mlx5_7 |
- |
| Topology |
Return Code |
Error / Output Tail |
| 2 nodes x 8 GPUs NCCL 2.27.7 sweep |
0 |
r0012:2145213:2145384 [7] NCCL INFO comm 0x558d54228110 rank 7 nranks 16 cudaDev 7 busId db000 - Destroy COMPLETE aikubeworker0016:1014703:1015544 [0] NCCL INFO comm 0x55ed6d99d8e0 rank 8 nranks 16 cudaDev 0 busId 18000 - Destroy COMPLETE |
Overall: FAIL
Generated by GPU Test Suite v0.2.0
