Detect and track objects with ML Kit on Android

  • ML Kit provides object detection and tracking in images and videos, identifying up to five objects per image and assigning unique IDs for tracking in video streams.

  • Setting up ML Kit requires Android API level 21 or higher and including the object-detection dependency from Google's Maven repository.

  • Developers can configure detection mode, multiple object detection, and classification using ObjectDetectorOptions and process images from various sources like files, byte buffers, or bitmaps.

  • Best practices include optimizing for real-time performance by limiting detection frequency and disabling unnecessary features like classification if not needed.

  • User experience can be enhanced by guiding users on capturing suitable images for object detection and implementing handling for unknown objects when using classification.

You can use ML Kit to detect and track objects in successive video frames.

When you pass an image to ML Kit, it detects up to five objects in the image along with the position of each object in the image. When detecting objects in video streams, each object has a unique ID that you can use to track the object from frame to frame. You can also optionally enable coarse object classification, which labels objects with broad category descriptions.

Try it out

Before you begin

  1. In your project-level build.gradle file, make sure to include Google's Maven repository in both your buildscript and allprojects sections.
  2. Add the dependencies for the ML Kit Android libraries to your module's app-level gradle file, which is usually app/build.gradle: 
    dependencies {
  // ...

  implementation 'com.google.mlkit:object-detection:17.0.2'

}

1. Configure the object detector

To detect and track objects, first create an instance of ObjectDetector and optionally specify any detector settings that you want to change from the default.

  1. Configure the object detector for your use case with an ObjectDetectorOptions object. You can change the following settings:

    Object Detector Settings
    Detection mode STREAM_MODE (default) | SINGLE_IMAGE_MODE

    In STREAM_MODE (default), the object detector runs with low latency, but might produce incomplete results (such as unspecified bounding boxes or category labels) on the first few invocations of the detector. Also, in STREAM_MODE, the detector assigns tracking IDs to objects, which you can use to track objects across frames. Use this mode when you want to track objects, or when low latency is important, such as when processing video streams in real time.

    In SINGLE_IMAGE_MODE, the object detector returns the result after the object's bounding box is determined. If you also enable classification it returns the result after the bounding box and category label are both available. As a consequence, detection latency is potentially higher. Also, in SINGLE_IMAGE_MODE, tracking IDs are not assigned. Use this mode if latency isn't critical and you don't want to deal with partial results.
    Detect and track multiple objects false (default) | true

    Whether to detect and track up to five objects or only the most prominent object (default).
    Classify objects false (default) | true

    Whether or not to classify detected objects into coarse categories. When enabled, the object detector classifies objects into the following categories: fashion goods, food, home goods, places, and plants.

    The object detection and tracking API is optimized for these two core use cases:

    • Live detection and tracking of the most prominent object in the camera viewfinder.
    • The detection of multiple objects from a static image.

    To configure the API for these use cases:

    Kotlin

    // Live detection and tracking
val options = ObjectDetectorOptions.Builder()
        .setDetectorMode(ObjectDetectorOptions.STREAM_MODE)
        .enableClassification()  // Optional
        .build()

// Multiple object detection in static images
val options = ObjectDetectorOptions.Builder()
        .setDetectorMode(ObjectDetectorOptions.SINGLE_IMAGE_MODE)
        .enableMultipleObjects()
        .enableClassification()  // Optional
        .build()

    Java

    // Live detection and tracking
ObjectDetectorOptions options =
        new ObjectDetectorOptions.Builder()
                .setDetectorMode(ObjectDetectorOptions.STREAM_MODE)
                .enableClassification()  // Optional
                .build();

// Multiple object detection in static images
ObjectDetectorOptions options =
        new ObjectDetectorOptions.Builder()
                .setDetectorMode(ObjectDetectorOptions.SINGLE_IMAGE_MODE)
                .enableMultipleObjects()
                .enableClassification()  // Optional
                .build();

  2. Get an instance of ObjectDetector:

    Kotlin

    val objectDetector = ObjectDetection.getClient(options)

    Java

    ObjectDetector objectDetector = ObjectDetection.getClient(options);

2. Prepare the input image

To detect and track objects, pass images to the ObjectDetector instance's process() method.

The object detector runs directly from a Bitmap, NV21 ByteBuffer or a YUV_420_888 media.Image. Constructing an InputImage from those sources are recommended if you have direct access to one of them. If you construct an InputImage from other sources, we will handle the conversion internally for you and it might be less efficient.

For each frame of video or image in a sequence, do the following:

You can create an InputImage object from different sources, each is explained below.

Using a media.Image

To create an InputImage object from a media.Image object, such as when you capture an image from a device's camera, pass the media.Image object and the image's rotation to InputImage.fromMediaImage().

If you use the