Some checks failed
CI / Benchmarks (yolo26n, macos-26, 3.13) (push) Has been cancelled
CI / Benchmarks (yolo26n, ubuntu-24.04-arm, 3.13) (push) Has been cancelled
CI / Benchmarks (yolo26n, ubuntu-latest, 3.13) (push) Has been cancelled
CI / Tests (macos-26, 3.13, latest) (push) Has been cancelled
CI / Tests (ubuntu-24.04-arm, 3.13, latest) (push) Has been cancelled
CI / Tests (ubuntu-latest, 3.13, latest) (push) Has been cancelled
CI / Tests (ubuntu-latest, 3.8, 1.8.0, 0.9.0) (push) Has been cancelled
CI / Tests (windows-latest, 3.13, latest) (push) Has been cancelled
CI / SlowTests (export-legacy-torch, ubuntu-latest, 3.10, 1.11.0, 0.12.0) (push) Has been cancelled
CI / SlowTests (export-legacy-torch, ubuntu-latest, 3.10, 1.12.0, 0.13.0) (push) Has been cancelled
CI / SlowTests (export-legacy-torch, ubuntu-latest, 3.10, 1.13.0, 0.14.0) (push) Has been cancelled
CI / SlowTests (export-legacy-torch, ubuntu-latest, 3.11, 2.0.0, 0.15.0) (push) Has been cancelled
CI / SlowTests (export-legacy-torch, ubuntu-latest, 3.11, 2.1.0, 0.16.0) (push) Has been cancelled
CI / SlowTests (export-legacy-torch, ubuntu-latest, 3.12, 2.2.0, 0.17.0) (push) Has been cancelled
CI / SlowTests (export-legacy-torch, ubuntu-latest, 3.9, 1.10.0, 0.11.0) (push) Has been cancelled
CI / SlowTests (export-legacy-torch, ubuntu-latest, 3.9, 1.9.0, 0.10.0) (push) Has been cancelled
CI / SlowTests (macos-26, 3.13, latest) (push) Has been cancelled
CI / SlowTests (ubuntu-24.04-arm, 3.13, latest) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.10.0, 0.25.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.3.0, 0.18.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.4.0, 0.19.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.5.0, 0.20.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.6.0, 0.21.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.7.0, 0.22.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.8.0, 0.23.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.9.0, 0.24.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.13, 2.11.0, 0.26.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.13, latest) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.8, 1.8.0, 0.9.0) (push) Has been cancelled
CI / SlowTests (windows-latest, 3.13, latest) (push) Has been cancelled
CI / IsolatedExports (push) Has been cancelled
CI / GPU (push) Has been cancelled
CI / RaspberryPi (push) Has been cancelled
CI / NVIDIA_Jetson (JetPack5.1.2, 1.23.5, https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.16.3-cp38-cp38-linux_aarch64.whl, 3.8, jetson-jp512, https://github.com/ultralytics/assets/releases/download/v0.0.0/torch-2.2.0-cp38-c… (push) Has been cancelled
CI / NVIDIA_Jetson (JetPack6.2, 1.26.4, https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.20.0-cp310-cp310-linux_aarch64.whl, 3.10, jetson-jp62, https://github.com/ultralytics/assets/releases/download/v0.0.0/torch-2.5.0a0+872d… (push) Has been cancelled
CI / Conda (ubuntu-latest, 3.12) (push) Has been cancelled
Publish Docker Images / Build (, Dockerfile-arm64, linux/arm64, ubuntu-24.04-arm, latest-arm64) (push) Has been cancelled
Publish Docker Images / Build (, Dockerfile-jetson-jetpack4, linux/arm64, ubuntu-24.04-arm, latest-jetson-jetpack4) (push) Has been cancelled
Publish Docker Images / Build (, Dockerfile-jetson-jetpack5, linux/arm64, ubuntu-24.04-arm, latest-jetson-jetpack5) (push) Has been cancelled
Publish Docker Images / Build (, Dockerfile-jetson-jetpack6, linux/arm64, ubuntu-24.04-arm, latest-jetson-jetpack6) (push) Has been cancelled
Publish Docker Images / Build (, Dockerfile-nvidia-arm64, linux/arm64, ubuntu-24.04-arm, latest-nvidia-arm64) (push) Has been cancelled
Publish Docker Images / Build (Dockerfile-jupyter,Dockerfile-cpu,Dockerfile-runner-cpu,Dockerfile-python-export, Dockerfile-python, linux/amd64, ubuntu-latest, latest-python) (push) Has been cancelled
Publish Docker Images / Build (Dockerfile-runner,Dockerfile-export, Dockerfile, linux/amd64, ubuntu-latest, latest) (push) Has been cancelled
Publish Docs / Docs (push) Has been cancelled
Publish to PyPI / check (push) Has been cancelled
CI / Summary (push) Has been cancelled
Publish Docker Images / trigger-actions (push) Has been cancelled
Publish Docker Images / notify (push) Has been cancelled
Publish to PyPI / build (push) Has been cancelled
Publish to PyPI / publish (push) Has been cancelled
Publish to PyPI / sbom (push) Has been cancelled
Publish to PyPI / notify (push) Has been cancelled
## InfiniBand / RDMA support (Dockerfile + entrypoint)
- Base image switched to registry.d-robotics.cc/.../yolo_distributed:v20260611_fix
- Install IB user-space libs: libibverbs1, librdmacm1, ibverbs-providers,
rdma-core, infiniband-diags, perftest
- Set NCCL IB env vars at build time (NCCL_IB_DISABLE=0, NCCL_IB_GID_INDEX=3,
NCCL_IB_HCA=mlx5, NCCL_IB_TC=106, NCCL_IB_SL=5, NCCL_NET_GDR_LEVEL=2)
- entrypoint.sh: print IB device info (ibv_devices) on startup for diagnostics
## Multi-node DDP bug fixes (ultralytics/engine/trainer.py)
Three bugs that only surface with NNODES > 1:
1. Wrong device index in _setup_ddp():
- Before: torch.cuda.set_device(RANK) and torch.device("cuda", RANK)
- After: torch.cuda.set_device(LOCAL_RANK) and torch.device("cuda", LOCAL_RANK)
- RANK is the global rank (0-15 across 2 nodes); LOCAL_RANK is the per-node
GPU index (0-7). Using global RANK as a CUDA device index fails on node 1
where RANK 8-15 don't correspond to real GPU indices.
2. Wrong device_ids in DistributedDataParallel constructor:
- Before: device_ids=[RANK]
- After: device_ids=[LOCAL_RANK], find_unused_parameters=False,
gradient_as_bucket_view=True
- Same RANK vs LOCAL_RANK confusion; also adds gradient_as_bucket_view for
lower memory overhead and disables find_unused_parameters (no unused params
in this model, the flag was causing a spurious warning).
3. Pod hangs after training completes (final_eval deadlock):
- Root cause: final_eval() called self.validator(model=best.pt) on ALL ranks
without a RANK guard. The standalone validator path uses
torch.device("cuda", RANK) — valid for ranks 0-7 on the master node but
invalid for ranks 8-15 on the worker node, causing immediate crash. Those
crashed ranks never reached the torch_distributed_zero_first barrier inside
the validator, so rank 0 waited at that barrier forever → pod hung for 20+
minutes until manual deletion.
- Fix: call dist.destroy_process_group() on all ranks before the validator,
then guard the validator call with if RANK in {-1, 0}. This lets worker
ranks exit cleanly and runs final validation only on rank 0 (which already
has the full best.pt checkpoint).
## DetectionValidator dist guard (docker/distributed/detect_val.py)
- After destroy_process_group(), dist.get_world_size() raises ValueError because
no default process group exists. DetectionValidator.gather_stats() and
_gather_image_metrics() call dist.get_world_size() unconditionally.
- Fix: add `if not dist.is_initialized(): return` at the top of both methods.
When rank 0 runs the standalone validator after destroying the group, the
stats are already fully accumulated from rank 0's inference on all images —
there is nothing to gather.
## Build script (build_docker.sh)
- Added REPO_ROOT so the build context is the repo root (needed because COPY
paths like ultralytics/engine/trainer.py are relative to repo root)
- Added DOCKER_PLATFORM variable (default linux/amd64) for cross-platform builds
- Default tag suffix changed from _fix to _ib
## VCJob manifests
- vcjob-ib.yaml: reference manifest with IB env vars, H20-3e nodeSelector,
PVC dataset mount, rdma/hca resource, batch=128
- vcjob-ib3.yaml: same manifest pointing to v20260611_ib3 (fully fixed image)
Tested on 2-node × 8-GPU H20-3e cluster. Both master and worker pods exit
with status Completed after 100 epochs; final validation metrics are printed
and results saved to /workspace/runs/detect/train/.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
85 lines
2.8 KiB
Bash
Executable File
85 lines
2.8 KiB
Bash
Executable File
#!/bin/bash
|
|
# Build YOLO distributed training Docker image and push to registry
|
|
# CUDA 12.8 + PyTorch 2.8.0, supports RTX 5090 (sm_120 / compute capability 12.0)
|
|
set -e
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
|
REGISTRY="registry.d-robotics.cc/fat-auwomo-fat-auwomo/yolo_distributed"
|
|
IMAGE_TAG="${IMAGE_TAG:-v$(date +%Y%m%d)_ib}"
|
|
LOCAL_TAG="yolo-train:${IMAGE_TAG}"
|
|
DOCKER_PLATFORM="${DOCKER_PLATFORM:-linux/amd64}"
|
|
|
|
echo "============================================"
|
|
echo " Building YOLO Distributed Training Image"
|
|
echo "============================================"
|
|
echo " Registry: ${REGISTRY}"
|
|
echo " Tag: ${IMAGE_TAG}"
|
|
echo " Platform: ${DOCKER_PLATFORM}"
|
|
echo " CUDA: 12.8 (supports RTX 5090 sm_120)"
|
|
echo " PyTorch: 2.8.0+cu128"
|
|
echo "============================================"
|
|
|
|
# Pass proxy env vars if set
|
|
PROXY_ARGS=()
|
|
[ -n "$http_proxy" ] && PROXY_ARGS+=(--build-arg "http_proxy=$http_proxy")
|
|
[ -n "$https_proxy" ] && PROXY_ARGS+=(--build-arg "https_proxy=$https_proxy")
|
|
[ -n "$no_proxy" ] && PROXY_ARGS+=(--build-arg "no_proxy=$no_proxy")
|
|
|
|
# Build
|
|
docker build \
|
|
--platform "${DOCKER_PLATFORM}" \
|
|
"${PROXY_ARGS[@]}" \
|
|
-t "${LOCAL_TAG}" \
|
|
-f "${SCRIPT_DIR}/Dockerfile" \
|
|
"${REPO_ROOT}"
|
|
|
|
echo ""
|
|
echo "Build complete: ${LOCAL_TAG}"
|
|
|
|
# Tag for registry
|
|
REMOTE_TAG="${REGISTRY}:${IMAGE_TAG}"
|
|
echo ""
|
|
echo "Tagging for registry: ${REMOTE_TAG}"
|
|
docker tag "${LOCAL_TAG}" "${REMOTE_TAG}"
|
|
|
|
# Push
|
|
echo ""
|
|
echo "Pushing to registry..."
|
|
docker push "${REMOTE_TAG}"
|
|
|
|
echo ""
|
|
echo "============================================"
|
|
echo " Push complete: ${REMOTE_TAG}"
|
|
echo "============================================"
|
|
echo ""
|
|
echo "Usage examples:"
|
|
echo ""
|
|
echo " # Pull image"
|
|
echo " docker pull ${REMOTE_TAG}"
|
|
echo ""
|
|
echo " # Single-node training (all GPUs)"
|
|
echo " docker run --gpus all --ipc=host \\"
|
|
echo " -v \$(pwd)/datasets:/workspace/datasets \\"
|
|
echo " ${REMOTE_TAG} bash /entrypoint.sh train data=kitti.yaml model=yolo26n.pt epochs=100"
|
|
echo ""
|
|
echo " # Distributed multi-node (node 0 - master)"
|
|
echo " docker run --gpus all --ipc=host --network=host \\"
|
|
echo " -v /shared/datasets:/workspace/datasets \\"
|
|
echo " -e MASTER_ADDR=10.0.0.1 \\"
|
|
echo " -e MASTER_PORT=29500 \\"
|
|
echo " -e NNODES=2 \\"
|
|
echo " -e NPROC_PER_NODE=4 \\"
|
|
echo " -e RANK=0 \\"
|
|
echo " ${REMOTE_TAG} bash /entrypoint.sh train data=kitti.yaml model=yolo26n.pt epochs=100"
|
|
echo ""
|
|
echo " # Distributed multi-node (node 1 - worker)"
|
|
echo " docker run --gpus all --ipc=host --network=host \\"
|
|
echo " -v /shared/datasets:/workspace/datasets \\"
|
|
echo " -e MASTER_ADDR=10.0.0.1 \\"
|
|
echo " -e MASTER_PORT=29500 \\"
|
|
echo " -e NNODES=2 \\"
|
|
echo " -e NPROC_PER_NODE=4 \\"
|
|
echo " -e RANK=1 \\"
|
|
echo " ${REMOTE_TAG} bash /entrypoint.sh train data=kitti.yaml model=yolo26n.pt epochs=100"
|