198 Commits

Author SHA1 Message Date
5a9bce1274 feat: add InfiniBand support and fix multi-node DDP for distributed training
Some checks failed
CI / SlowTests (ubuntu-latest, 3.12, 2.6.0, 0.21.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.7.0, 0.22.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.8.0, 0.23.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.12, 2.9.0, 0.24.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.13, 2.11.0, 0.26.0) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.13, latest) (push) Has been cancelled
CI / SlowTests (ubuntu-latest, 3.8, 1.8.0, 0.9.0) (push) Has been cancelled
CI / SlowTests (windows-latest, 3.13, latest) (push) Has been cancelled
CI / IsolatedExports (push) Has been cancelled
CI / GPU (push) Has been cancelled
CI / RaspberryPi (push) Has been cancelled
CI / NVIDIA_Jetson (JetPack5.1.2, 1.23.5, https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.16.3-cp38-cp38-linux_aarch64.whl, 3.8, jetson-jp512, https://github.com/ultralytics/assets/releases/download/v0.0.0/torch-2.2.0-cp38-c… (push) Has been cancelled
CI / NVIDIA_Jetson (JetPack6.2, 1.26.4, https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.20.0-cp310-cp310-linux_aarch64.whl, 3.10, jetson-jp62, https://github.com/ultralytics/assets/releases/download/v0.0.0/torch-2.5.0a0+872d… (push) Has been cancelled
CI / Conda (ubuntu-latest, 3.12) (push) Has been cancelled
Publish Docker Images / Build (, Dockerfile-arm64, linux/arm64, ubuntu-24.04-arm, latest-arm64) (push) Has been cancelled
Publish Docker Images / Build (, Dockerfile-jetson-jetpack4, linux/arm64, ubuntu-24.04-arm, latest-jetson-jetpack4) (push) Has been cancelled
Publish Docker Images / Build (, Dockerfile-jetson-jetpack5, linux/arm64, ubuntu-24.04-arm, latest-jetson-jetpack5) (push) Has been cancelled
Publish Docker Images / Build (, Dockerfile-jetson-jetpack6, linux/arm64, ubuntu-24.04-arm, latest-jetson-jetpack6) (push) Has been cancelled
Publish Docker Images / Build (, Dockerfile-nvidia-arm64, linux/arm64, ubuntu-24.04-arm, latest-nvidia-arm64) (push) Has been cancelled
Publish Docker Images / Build (Dockerfile-jupyter,Dockerfile-cpu,Dockerfile-runner-cpu,Dockerfile-python-export, Dockerfile-python, linux/amd64, ubuntu-latest, latest-python) (push) Has been cancelled
Publish Docker Images / Build (Dockerfile-runner,Dockerfile-export, Dockerfile, linux/amd64, ubuntu-latest, latest) (push) Has been cancelled
Publish Docs / Docs (push) Has been cancelled
Publish to PyPI / check (push) Has been cancelled
CI / Summary (push) Has been cancelled
Publish Docker Images / trigger-actions (push) Has been cancelled
Publish Docker Images / notify (push) Has been cancelled
Publish to PyPI / build (push) Has been cancelled
Publish to PyPI / publish (push) Has been cancelled
Publish to PyPI / sbom (push) Has been cancelled
Publish to PyPI / notify (push) Has been cancelled
## InfiniBand / RDMA support (Dockerfile + entrypoint)
- Base image switched to registry.d-robotics.cc/.../yolo_distributed:v20260611_fix
- Install IB user-space libs: libibverbs1, librdmacm1, ibverbs-providers,
  rdma-core, infiniband-diags, perftest
- Set NCCL IB env vars at build time (NCCL_IB_DISABLE=0, NCCL_IB_GID_INDEX=3,
  NCCL_IB_HCA=mlx5, NCCL_IB_TC=106, NCCL_IB_SL=5, NCCL_NET_GDR_LEVEL=2)
- entrypoint.sh: print IB device info (ibv_devices) on startup for diagnostics

## Multi-node DDP bug fixes (ultralytics/engine/trainer.py)
Three bugs that only surface with NNODES > 1:

1. Wrong device index in _setup_ddp():
   - Before: torch.cuda.set_device(RANK) and torch.device("cuda", RANK)
   - After:  torch.cuda.set_device(LOCAL_RANK) and torch.device("cuda", LOCAL_RANK)
   - RANK is the global rank (0-15 across 2 nodes); LOCAL_RANK is the per-node
     GPU index (0-7). Using global RANK as a CUDA device index fails on node 1
     where RANK 8-15 don't correspond to real GPU indices.

2. Wrong device_ids in DistributedDataParallel constructor:
   - Before: device_ids=[RANK]
   - After:  device_ids=[LOCAL_RANK], find_unused_parameters=False,
             gradient_as_bucket_view=True
   - Same RANK vs LOCAL_RANK confusion; also adds gradient_as_bucket_view for
     lower memory overhead and disables find_unused_parameters (no unused params
     in this model, the flag was causing a spurious warning).

3. Pod hangs after training completes (final_eval deadlock):
   - Root cause: final_eval() called self.validator(model=best.pt) on ALL ranks
     without a RANK guard. The standalone validator path uses
     torch.device("cuda", RANK) — valid for ranks 0-7 on the master node but
     invalid for ranks 8-15 on the worker node, causing immediate crash. Those
     crashed ranks never reached the torch_distributed_zero_first barrier inside
     the validator, so rank 0 waited at that barrier forever → pod hung for 20+
     minutes until manual deletion.
   - Fix: call dist.destroy_process_group() on all ranks before the validator,
     then guard the validator call with if RANK in {-1, 0}. This lets worker
     ranks exit cleanly and runs final validation only on rank 0 (which already
     has the full best.pt checkpoint).

## DetectionValidator dist guard (docker/distributed/detect_val.py)
- After destroy_process_group(), dist.get_world_size() raises ValueError because
  no default process group exists. DetectionValidator.gather_stats() and
  _gather_image_metrics() call dist.get_world_size() unconditionally.
- Fix: add `if not dist.is_initialized(): return` at the top of both methods.
  When rank 0 runs the standalone validator after destroying the group, the
  stats are already fully accumulated from rank 0's inference on all images —
  there is nothing to gather.

## Build script (build_docker.sh)
- Added REPO_ROOT so the build context is the repo root (needed because COPY
  paths like ultralytics/engine/trainer.py are relative to repo root)
- Added DOCKER_PLATFORM variable (default linux/amd64) for cross-platform builds
- Default tag suffix changed from _fix to _ib

## VCJob manifests
- vcjob-ib.yaml: reference manifest with IB env vars, H20-3e nodeSelector,
  PVC dataset mount, rdma/hca resource, batch=128
- vcjob-ib3.yaml: same manifest pointing to v20260611_ib3 (fully fixed image)

Tested on 2-node × 8-GPU H20-3e cluster. Both master and worker pods exit
with status Completed after 100 epochs; final validation metrics are printed
and results saved to /workspace/runs/detect/train/.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-11 16:57:57 +08:00
b7ec03d2fa feat: add YOLO distributed training Docker deployment
Some checks are pending
CI / SlowTests (ubuntu-latest, 3.12, 2.6.0, 0.21.0) (push) Waiting to run
CI / SlowTests (ubuntu-latest, 3.12, 2.7.0, 0.22.0) (push) Waiting to run
CI / SlowTests (ubuntu-latest, 3.12, 2.8.0, 0.23.0) (push) Waiting to run
CI / SlowTests (ubuntu-latest, 3.12, 2.9.0, 0.24.0) (push) Waiting to run
CI / SlowTests (ubuntu-latest, 3.13, 2.11.0, 0.26.0) (push) Waiting to run
CI / SlowTests (ubuntu-latest, 3.13, latest) (push) Waiting to run
CI / SlowTests (ubuntu-latest, 3.8, 1.8.0, 0.9.0) (push) Waiting to run
CI / SlowTests (windows-latest, 3.13, latest) (push) Waiting to run
CI / IsolatedExports (push) Waiting to run
CI / GPU (push) Waiting to run
CI / RaspberryPi (push) Waiting to run
CI / NVIDIA_Jetson (JetPack5.1.2, 1.23.5, https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.16.3-cp38-cp38-linux_aarch64.whl, 3.8, jetson-jp512, https://github.com/ultralytics/assets/releases/download/v0.0.0/torch-2.2.0-cp38-c… (push) Waiting to run
CI / NVIDIA_Jetson (JetPack6.2, 1.26.4, https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.20.0-cp310-cp310-linux_aarch64.whl, 3.10, jetson-jp62, https://github.com/ultralytics/assets/releases/download/v0.0.0/torch-2.5.0a0+872d… (push) Waiting to run
CI / Conda (ubuntu-latest, 3.12) (push) Waiting to run
CI / Summary (push) Blocked by required conditions
Publish Docker Images / Build (, Dockerfile-arm64, linux/arm64, ubuntu-24.04-arm, latest-arm64) (push) Waiting to run
Publish Docker Images / Build (, Dockerfile-jetson-jetpack4, linux/arm64, ubuntu-24.04-arm, latest-jetson-jetpack4) (push) Waiting to run
Publish Docker Images / Build (, Dockerfile-jetson-jetpack5, linux/arm64, ubuntu-24.04-arm, latest-jetson-jetpack5) (push) Waiting to run
Publish Docker Images / Build (, Dockerfile-jetson-jetpack6, linux/arm64, ubuntu-24.04-arm, latest-jetson-jetpack6) (push) Waiting to run
Publish Docker Images / Build (, Dockerfile-nvidia-arm64, linux/arm64, ubuntu-24.04-arm, latest-nvidia-arm64) (push) Waiting to run
Publish Docker Images / Build (Dockerfile-jupyter,Dockerfile-cpu,Dockerfile-runner-cpu,Dockerfile-python-export, Dockerfile-python, linux/amd64, ubuntu-latest, latest-python) (push) Waiting to run
Publish Docker Images / Build (Dockerfile-runner,Dockerfile-export, Dockerfile, linux/amd64, ubuntu-latest, latest) (push) Waiting to run
Publish Docker Images / trigger-actions (push) Blocked by required conditions
Publish Docker Images / notify (push) Blocked by required conditions
Publish Docs / Docs (push) Waiting to run
Publish to PyPI / check (push) Waiting to run
Publish to PyPI / build (push) Blocked by required conditions
Publish to PyPI / publish (push) Blocked by required conditions
Publish to PyPI / sbom (push) Blocked by required conditions
Publish to PyPI / notify (push) Blocked by required conditions
2026-06-11 11:45:35 +08:00
Glenn Jocher
31f7fc2563
Harden ARM CI against runner flakes (#24665)
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2026-06-02 17:29:36 +08:00
Glenn Jocher
158ced9c17
ultralytics 8.4.58 Isolated CI export environments (#24649)
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2026-05-31 19:23:43 +08:00
Glenn Jocher
c219cb2d5a
Update runner images and cache TensorRT (#24436)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2026-05-09 21:07:36 +08:00
Lakshantha Dissanayake
ed56d39f76
Auto-update tensorrt from 10.3 to 10.7 on JetPack 6 systems to fix YOLO26 int8 build issues (#24368)
Co-authored-by: Laughing-q <1185102784@qq.com>
2026-04-29 17:36:05 +08:00
Lakshantha Dissanayake
c7e1d6044a
Fix torch and torchvision versions on JetPack 5 Docker (#24199) 2026-04-11 09:37:33 +08:00
Glenn Jocher
e1996a35b5
FROM pytorch/pytorch:2.11.0-cuda12.8-cudnn9-runtime (#23991)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: Onuralp SEZER <onuralp@ultralytics.com>
2026-03-25 13:53:31 +01:00
Glenn Jocher
f514503165
Update Dockerfile-runner to v2.333.0 (#23974)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2026-03-22 20:25:40 +01:00
Jing Qiu
fbef54d9a3
Add gpg package for Dockerfile-runner-cpu (#23878) 2026-03-12 16:00:45 +08:00
Glenn Jocher
14e12fcf46
Add nodejs and npm to runner images (#23832) 2026-03-08 11:57:05 +01:00
Glenn Jocher
9a45fd3800
Update Dockerfile-runner-cpu with sudo (#23830)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2026-03-08 04:10:50 +01:00
Glenn Jocher
cc021913f8
Remove Runner SHA checks (#23829) 2026-03-08 02:51:29 +01:00
Glenn Jocher
1e1f188bda
Add Dockerfile-runner-cpu (#23827) 2026-03-08 02:38:48 +01:00
Glenn Jocher
c67d6b5b94
Update RUNNER_VERSION to 2.332.0 (#23826)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2026-03-07 23:49:52 +01:00
Glenn Jocher
1ce3b0c4e6
Add UV_BREAK_SYSTEM_PACKAGES environment variable (#23809)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 23:10:15 +01:00
Glenn Jocher
83989148fb
FROM pytorch/pytorch:2.10.0-cuda12.8-cudnn9-runtime (#23807)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2026-03-05 22:47:30 +01:00
Lakshantha Dissanayake
b880b608b7
Update JetPack 6 Docker image with torch 2.10 (#23788) 2026-03-04 11:51:38 +08:00
Glenn Jocher
38dca710a5
Round-robin GPU autoselect (#23544)
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2026-02-03 18:20:34 +00:00
Glenn Jocher
f2d3aed634
ultralytics 8.4.0 YOLO26 Models Release (#23176)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: Laughing-q <1185102784@qq.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2026-01-14 03:44:38 +00:00
Lakshantha Dissanayake
779b4319fa
ultralytics 8.3.249 Jetson AGX Thor and DGX Spark Docker (#23111)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: Onuralp SEZER <onuralp@ultralytics.com>
2026-01-07 12:54:20 +01:00
Glenn Jocher
3dd9ffed54
ultralytics 8.3.248 Target correct Python env for auto-install (#23118)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2026-01-04 18:20:21 +01:00
Glenn Jocher
171c26c6c9
Update Dockerfile examples (#23103) 2026-01-02 03:29:19 +01:00
Glenn Jocher
d5c1734554
YAML fixes (#23092)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-12-31 11:31:27 +01:00
Glenn Jocher
abd4c52dd9
Dockerfile improvements (#23090)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-12-31 11:23:49 +01:00
Lakshantha Dissanayake
6d7942385e
Add missing --runtime=nvidia usage description to Dockerfiles (#23080)
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-12-29 22:26:17 +01:00
Lakshantha Dissanayake
8c873a693b
Remove numpy pin for Sony IMX export and JetPack 6 systems (#23030)
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-12-24 18:47:11 +01:00
Lakshantha Dissanayake
9482d3a063
Pin numpy for JetPack 5 Docker builds (#23032) 2025-12-24 18:45:38 +01:00
Onuralp SEZER
c3b3500f7a
feat: 📦 update onnx version to 1.20.0 (#22867)
Signed-off-by: Onuralp SEZER <onuralp@ultralytics.com>
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-12-15 19:21:24 +00:00
Glenn Jocher
2ee0b4d870
Minor fixes (#22943)
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2025-12-14 22:46:21 +00:00
Lakshantha Dissanayake
4b884f4a44
ultralytics 8.3.235 Remove torch 2.8.0 pin for Sony IMX export (#22874)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: Jing Qiu <61612323+Laughing-q@users.noreply.github.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-12-04 10:35:19 +01:00
Jing Qiu
bcb11c1e4e
Update Dockerfile-jetson-jetpack6 to pin onnx<1.20.0 (#22864) 2025-12-02 20:20:26 +08:00
Onuralp SEZER
f31a4bf6cf
fix: 🐞 update Dockerfile to use Ubuntu 24.04 LTS and streamline package installation commands (#22763)
Signed-off-by: Onuralp SEZER <onuralp@ultralytics.com>
Co-authored-by: Lakshantha Dissanayake <lakshantha@ultralytics.com>
2025-11-24 14:48:00 +08:00
Glenn Jocher
d457669f2a
Remove pip install uv from Dockerfile (#22709) 2025-11-18 11:10:30 +01:00
Glenn Jocher
29448b21a9
Update Dockerfile-runner to v2.329.0 (#22707)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-11-18 01:14:46 +01:00
Glenn Jocher
74eff21d60
Update Dockerfile to 2.9.1-cuda12.8-cudnn9-runtime (#22708)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-11-18 01:13:56 +01:00
Glenn Jocher
4e85987d6f
Fix 429 forum links in Dockerfile-jetson-jetpack4 (#22576)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-11-04 10:07:40 +09:00
Jing Qiu
3f1a3e2e28
Pin torch<=2.8.0 in Dockerfile-python-export (#22392) 2025-10-15 19:47:56 +08:00
Glenn Jocher
e86384fd9d
Export in Dockerfile /tmp dir (#22107)
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2025-09-16 21:59:58 +01:00
Mohammed Yasin
cba61ac457
Update Dockerfile-export to use AutoInstall for TensorRT package (#22084)
Signed-off-by: Mohammed Yasin <32206511+Y-T-G@users.noreply.github.com>
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-09-16 16:37:10 +01:00
Glenn Jocher
3a327b2853
Parameterize runner version in Dockerfile-runner (#22049)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-09-11 13:40:04 +02:00
Glenn Jocher
9c56d4a15a
FROM pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime (#21675)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2025-08-13 13:33:09 +02:00
Glenn Jocher
0cc062807f
Use apt-get clean in Dockerfiles (#21665)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2025-08-13 00:42:42 +02:00
Glenn Jocher
c2569acc06
ultralytics 8.3.178 new lighter Dockerfile-python images (#21661)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2025-08-12 20:01:07 +02:00
Glenn Jocher
bc507dac58
Refactor derivative Dockerfile images (#21656)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2025-08-12 14:57:13 +02:00
Glenn Jocher
81a614cfe5
Combine Dockerfile RUN commands (#21649)
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2025-08-12 12:00:55 +02:00
Glenn Jocher
11e43791f4
Use opencv-python-headless in Dockerfiles (#21647)
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2025-08-12 00:04:48 +02:00
Михаил
dcc65e168e
ultralytics 8.3.157 Add faster-coco-eval package for COCO/LVIS evaluation (#17020)
Signed-off-by: Михаил <mike.milos@yandex.ru>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
Co-authored-by: Laughing <61612323+Laughing-q@users.noreply.github.com>
Co-authored-by: Ultralytics Assistant <135830346+UltralyticsAssistant@users.noreply.github.com>
Co-authored-by: Laughing-q <1185102784@qq.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-06-20 11:59:37 +01:00
Glenn Jocher
d0f344ba97
Remove g++ and libusb-1.0-0 from Docker linux installs (#20384)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2025-04-27 21:58:39 +02:00
Glenn Jocher
180dfa1e06
Dockerfile cleanup (#20382)
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
2025-04-27 20:14:43 +02:00