Baidu's RT-DETR: A Vision Transformer-Based Real-Time Object Detector#

Overview#

Real-Time Detection Transformer (RT-DETR), developed by Baidu, is a cutting-edge end-to-end object detector that provides real-time performance while maintaining high accuracy. It is based on the idea of DETR (the NMS-free framework), meanwhile introducing conv-based backbone and an efficient hybrid encoder to gain real-time speed. RT-DETR efficiently processes multiscale features by decoupling intra-scale interaction and cross-scale fusion. The model is highly adaptable, supporting flexible adjustment of inference speed using different decoder layers without retraining. RT-DETR excels on accelerated backends like CUDA with TensorRT, outperforming many other real-time object detectors.

Overview of Baidu's RT-DETR. The RT-DETR model architecture diagram shows the last three stages of the backbone {S3, S4, S5} as the input to the encoder. The efficient hybrid encoder transforms multiscale features into a sequence of image features through intrascale feature interaction (AIFI) and cross-scale feature-fusion module (CCFM). The IoU-aware query selection is employed to select a fixed number of image features to serve as initial object queries for the decoder. Finally, the decoder with auxiliary prediction heads iteratively optimizes object queries to generate boxes and confidence scores (source).

Key Features#

• Efficient Hybrid Encoder: Baidu's RT-DETR uses an efficient hybrid encoder that processes multiscale features by decoupling intra-scale interaction and cross-scale fusion. This unique Vision Transformers-based design reduces computational costs and allows for real-time object detection.
• IoU-aware Query Selection: Baidu's RT-DETR improves object query initialization by utilizing IoU-aware query selection. This allows the model to focus on the most relevant objects in the scene, enhancing the detection accuracy.
• Adaptable Inference Speed: Baidu's RT-DETR supports flexible adjustments of inference speed by using different decoder layers without the need for retraining. This adaptability facilitates practical application in various real-time object detection scenarios.
• NMS-Free Framework: Based on DETR, RT-DETR eliminates the need for non-maximum suppression post-processing, simplifying the detection pipeline and potentially improving efficiency.
• Anchor-Free Detection: As an anchor-free detector, RT-DETR simplifies the detection process and may improve generalization across different datasets.

Pretrained Models#

The Ultralytics Python API provides pretrained PaddlePaddle RT-DETR models with different scales:

• RT-DETR-L: 53.0% AP on COCO val2017, 114 FPS on T4 GPU
• RT-DETR-X: 54.8% AP on COCO val2017, 74 FPS on T4 GPU

Additionally, Baidu has released RTDETRv2 in July 2024, which further improves upon the original architecture with enhanced performance metrics.

Usage Examples#

This example provides simple RT-DETR training and inference examples. For full documentation on these and other modes see the Predict, Train, Val and Export docs pages. Models can also be trained on cloud GPUs through Ultralytics Platform.

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR("rtdetr-l.pt")

# Display model information (optional)
model.info()

# Train the model on the COCO8 example dataset for 100 epochs
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)

# Run inference with the RT-DETR-l model on the 'bus.jpg' image
results = model("path/to/bus.jpg")

from ultralytics import RTDETR

rtdetr = RTDETR("rtdetr-l.pt")
head = rtdetr.model.model[-1]

# Choose one or both settings after validating the speed/accuracy trade-off.
head.decoder.eval_idx = 3  # Use 4 of 6 decoder layers.
head.num_queries = 100  # Use fewer object queries.

results = rtdetr("path/to/image.jpg")

# Export uses the same decoder and query settings, including TensorRT exports.
rtdetr.export(format="engine", device=0, quantize=16)

Supported Tasks and Modes#

This table presents the model types, the specific pretrained weights, the tasks supported by each model, and the various modes (Train, Val, Predict, Export) that are supported, indicated by ✅ emojis.

Ideal Use Cases#

RT-DETR is particularly well-suited for applications requiring both high accuracy and real-time performance:

• Autonomous Driving: For reliable environmental perception in self-driving systems where both speed and accuracy are critical. Learn more about AI in self-driving cars.