Multi-camera API

Note: This page refers to the Camera2 package. Unless your app requires specific, low-level features from Camera2, we recommend using CameraX. Both CameraX and Camera2 support Android 5.0 (API level 21) and higher.

Multi-camera was introduced with Android 9 (API level 28). Since its release, devices have come to market that support the API. Many multi-camera use cases are tightly coupled with a specific hardware configuration. In other words, not all use cases are compatible with every device,  which makes multi-camera features a good candidate for Play Feature Delivery.

Some typical use cases include:

  • Zoom: switching between cameras depending on crop region or desired focal length.
  • Depth: using multiple cameras to build a depth map.
  • Bokeh: using inferred depth information to simulate a DSLR-like narrow focus range.

The difference between logical and physical cameras

Understanding the multi-camera API requires understanding the difference between logical and physical cameras. For reference, consider a device with three back-facing cameras. In this example, each of the three back cameras is considered a physical camera. A logical camera is then a grouping of two or more of those physical cameras. The output of the logical camera can be a stream that comes from one of the underlying physical cameras, or a fused stream coming from more than one underlying physical camera simultaneously. Either way, the stream is handled by the camera Hardware Abstraction Layer (HAL).

Many phone manufacturers develop first-party camera applications, which usually come pre-installed on their devices. To use all of the hardware's capabilities, they may use private or hidden APIs or receive special treatment from the driver implementation that other applications do not have access to. Some devices implement the concept of logical cameras by providing a fused stream of frames from the different physical cameras, but only to certain privileged applications. Often, only one of the physical cameras is exposed to the framework. The situation for third-party developers prior to Android 9 is illustrated in the following diagram:

Figure 1. Camera capabilities typically only available to privileged applications

Beginning with Android 9, private APIs are no longer allowed in Android apps. With the inclusion of multi-camera support in the framework, Android best practices strongly recommend that phone manufacturers expose a logical camera for all physical cameras facing the same direction. The following is what third-party developers should expect to see on devices running Android 9 and higher:

Figure 2. Full developer access to all camera devices starting in Android 9

What the logical camera provides is entirely dependent on the OEM implementation of the Camera HAL. For example, a device like Pixel 3 implements its logical camera in such a way that it chooses one of its physical cameras based on the requested focal length and crop region.

The multi-camera API

The new API adds the following new constants, classes, and methods:

Due to changes to the Android Compatibility Definition Document (CDD), the multi-camera API also comes with certain expectations from developers. Devices with dual cameras existed prior to Android 9, but opening more than one camera simultaneously involved trial and error. On Android 9 and higher, multi-camera gives a set of rules to specify when it is possible to open a pair of physical cameras that are part of the same logical camera.

In most cases, devices running Android 9 and higher expose all physical cameras (except possibly for less-common sensor types like infrared) along with an easier-to-use logical camera. For every combination of streams that are guaranteed to work, one stream belonging to a logical camera can be replaced by two streams from the underlying physical cameras.

Multiple streams simultaneously

Using multiple camera streams simultaneously covers the rules for using multiple streams simultaneously in a single camera. With one notable addition, the same rules apply for multiple cameras. CameraMetadata.REQUEST_AVAILABLE_CAPABILITIES_LOGICAL_MULTI_CAMERA explains how to replace a logical YUV_420_888 or raw stream with two physical streams. That is, each stream of type YUV or RAW can be replaced with two streams of identical type and size. You can start with a camera stream of the following guaranteed configuration for single-camera devices:

  • Stream 1: YUV type, MAXIMUM size from logical camera id = 0

Then, a device with multi-camera support allows you to create a session replacing that logical YUV stream with two physical streams:

  • Stream 1: YUV type, MAXIMUM size from physical camera id = 1
  • Stream 2: YUV type, MAXIMUM size from physical camera id = 2

You can replace a YUV or RAW stream with two equivalent streams if and only if those two cameras are part of a logical camera grouping,  which is listed under CameraCharacteristics.getPhysicalCameraIds().

The guarantees provided by the framework are just the bare minimum required to get frames from more than one physical camera simultaneously. Additional streams are supported in most devices, sometimes even allowing opening multiple physical camera devices independently. Since it's not a hard guarantee from the framework, doing that requires performing per-device testing and tuning using trial and error.

Creating a session with multiple physical cameras

When using physical cameras on a multi-camera enabled device, open a single CameraDevice (the logical camera) and interact with it within a single session. Create the single session using the API CameraDevice.createCaptureSession(SessionConfiguration config), which was added in API level 28. The session configuration has a number of output configurations, each of which has a set of output targets and, optionally, a desired physical camera ID.

Figure 3. SessionConfiguration and OutputConfiguration model

Capture requests have an output target associated with them. The framework determines which physical (or logical) camera the requests are sent to based on what output target is attached. If the output target corresponds to one of the output targets that was sent as an output configuration along with a physical camera ID, then that physical camera receives and processes the request.

Using a pair of physical cameras

Another addition to the camera APIs for multi-camera is the ability to identify logical cameras and find the physical cameras behind them. You can define a function to help identify potential pairs of physical cameras that you can use to replace one of the logical camera streams:

Kotlin

/**
     * Helper class used to encapsulate a logical camera and two underlying
     * physical cameras
     */
    data class DualCamera(val logicalId: String, val physicalId1: String, val physicalId2: String)

    fun findDualCameras(manager: CameraManager, facing: Int? = null): List {
        val dualCameras = MutableList()

        // Iterate over all the available camera characteristics
        manager.cameraIdList.map {
            Pair(manager.getCameraCharacteristics(it), it)
        }.filter {
            // Filter by cameras facing the requested direction
            facing == null || it.first.get(CameraCharacteristics.LENS_FACING) == facing
        }.filter {
            // Filter by logical cameras
            // CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES_LOGICAL_MULTI_CAMERA requires API >= 28
            it.first.get(CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES)!!.contains(
                CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES_LOGICAL_MULTI_CAMERA)
        }.forEach {
            // All possible pairs from the list of physical cameras are valid results
            // NOTE: There could be N physical cameras as part of a logical camera grouping
            // getPhysicalCameraIds() requires API >= 28
            val physicalCameras = it.first.physicalCameraIds.toTypedArray()
            for (idx1 in 0 until physicalCameras.size) {
                for (idx2 in