Model Benchmarking with Ultralytics YOLO#
Benchmark Visualization#
Introduction#
Once your model is trained and validated, the next logical step is to evaluate its performance in various real-world scenarios. Benchmark mode in Ultralytics YOLO26 serves this purpose by providing a robust framework for assessing the speed and accuracy of your model across a range of export formats.
See the unreleased YOLO27 preview for preliminary speed benchmarks.
Watch: Benchmark Ultralytics YOLO26 Models | How to Compare Model Performance on Different Hardware?
Why Is Benchmarking Crucial?#
- Informed Decisions: Gain insights into the trade-offs between speed and accuracy.
- Resource Allocation: Understand how different export formats perform on different hardware.
- Optimization: Learn which export format offers the best performance for your specific use case.
- Cost Efficiency: Make more efficient use of hardware resources based on benchmark results.
Key Metrics in Benchmark Mode#
- mAP50-95: For object detection, segmentation, and pose estimation.
- accuracy_top1: For image classification.
- Inference Time: Time taken for each image in milliseconds.
Supported Export Formats#
- ONNX: For optimal CPU performance
- TensorRT: For maximal GPU efficiency
- OpenVINO: For Intel hardware optimization
- CoreML, TensorFlow SavedModel, and More: For diverse deployment needs.
- Export to ONNX or OpenVINO for up to 3x CPU speedup.
- Export to TensorRT for up to 5x GPU speedup.
Usage Examples#
Install Ultralytics with export dependencies before benchmarking to avoid missing packages.
pip install ultralytics[export]Run YOLO26n benchmarks across all supported export formats (ONNX, TensorRT, etc.). See the Arguments section below for a full list of export options.
from ultralytics.utils.benchmarks import benchmark
# Benchmark on GPU
benchmark(model="yolo26n.pt", data="coco8.yaml", imgsz=640, device=0)
# Benchmark specific export format
benchmark(model="yolo26n.pt", data="coco8.yaml", imgsz=640, format="onnx")Arguments#
Arguments such as model, data, imgsz, quantize, device, verbose and format provide users with the flexibility to fine-tune the benchmarks to their specific needs and compare the performance of different export formats with ease.
| Key | Default Value | Description |
|---|---|---|
model | None | Specifies the path to the model file. Accepts both .pt and .yaml formats, e.g., "yolo26n.pt" for pretrained models or configuration files. |
data | None | Path to the dataset YAML for benchmarking, typically including paths and settings for validation data. Example: "coco8.yaml". Classification instead takes a dataset directory or a built-in dataset name (e.g., imagenet10). |
imgsz | 640 | The input image size for the model. Must be a single integer for square images (e.g., 640); benchmark() only supports square image sizes. |
quantize | None | Requested precision: 16 (FP16) or 8 (INT8; PTQ schemes need calibration data/fraction, weight-only ones do not); 32/unset is FP32 where the format supports it — a format that cannot export FP32 rejects an explicit 32 or falls back to the precision it requires. The native PyTorch row is not exported: only 16 affects it, selecting FP16 inference, while 8, 32 and unset run FP32. Each format then runs inference at its own runtime precision. Replaces the deprecated half/int8 flags. |
device | 'cpu' | Defines the computation device(s) for benchmarking, such as "cpu" or "cuda:0". |
verbose | False | Controls the level of detail in logging output. Set verbose=True for detailed logs. |
eps | 0.001 | Small epsilon (milliseconds) added to the per-image inference time before converting it to FPS, preventing division by zero. Rarely changed. |
format | '' | Benchmarks only the specified export format (e.g., format=onnx). Leave it blank to test every supported format automatically. |
The standalone benchmark() function (from ultralytics.utils.benchmarks import benchmark) uses its own signature defaults instead of the table values above, notably model="yolo26n.pt" and imgsz=160; pass imgsz explicitly to match the yolo benchmark CLI.
Export Formats#
Benchmarks will attempt to run automatically on all possible export formats listed below. Alternatively, you can run benchmarks for a specific format by using the format argument, which accepts any of the formats mentioned below.
| Format | format Argument | Model | Metadata | Arguments |
|---|---|---|---|---|
| PyTorch | - | yolo26n.pt | ✅ | - |
| TorchScript | torchscript | yolo26n.torchscript | ✅ | imgsz, quantize, dynamic, nms, batch, device |
| ONNX | onnx | yolo26n.onnx | ✅ | imgsz, quantize, dynamic, simplify, opset, nms, batch, data, fraction, device |
| OpenVINO | openvino | yolo26n_openvino_model/ | ✅ | imgsz, quantize, dynamic, nms, batch, data, fraction, device |
| TensorRT | engine | yolo26n.engine | ✅ | imgsz, quantize, dynamic, simplify, opset, workspace, nms, batch, data, fraction, device |
| CoreML | coreml | yolo26n.mlpackage | ✅ | imgsz, dynamic, quantize, nms, batch, device |
| TF SavedModel | saved_model | yolo26n_saved_model/ | ✅ | imgsz, |