WebCodecs

1. Definitions

Codec

Refers generically to an instance of AudioDecoder, AudioEncoder, VideoDecoder, or VideoEncoder.

Key Chunk

An encoded chunk that does not depend on any other frames for decoding. Also commonly referred to as a "key frame".

Internal Pending Output

Codec outputs such as VideoFrames that currently reside in the internal pipeline of the underlying codec implementation. The underlying codec implementation MAY emit new outputs only when new inputs are provided. The underlying codec implementation MUST emit all outputs in response to a flush.

Codec System Resources

Resources including CPU memory, GPU memory, and exclusive handles to specific decoding/encoding hardware that MAY be allocated by the User Agent as part of codec configuration or generation of AudioData and VideoFrame objects. Such resources MAY be quickly exhausted and SHOULD be released immediately when no longer in use.

Temporal Layer

A grouping of EncodedVideoChunks whose timestamp cadence produces a particular framerate. See scalabilityMode.

Progressive Image

An image that supports decoding to multiple levels of detail, with lower levels becoming available while the encoded data is not yet fully buffered.

Progressive Image Frame Generation

A generational identifier for a given Progressive Image decoded output. Each successive generation adds additional detail to the decoded output. The mechanism for computing a frame’s generation is implementer defined.

Primary Image Track

An image track that is marked by the given image file as being the default track. The mechanism for indicating a primary track is format defined.

RGB Format

A VideoPixelFormat containing red, green, and blue color channels in any order or layout (interleaved or planar), and irrespective of whether an alpha channel is present.

sRGB Color Space

A VideoColorSpace object, initialized as follows:

[[primaries]] is set to bt709,
[[transfer]] is set to iec61966-2-1,
[[matrix]] is set to rgb,
[[full range]] is set to true

Display P3 Color Space

A VideoColorSpace object, initialized as follows:

[[primaries]] is set to smpte432,
[[transfer]] is set to iec61966-2-1,
[[matrix]] is set to rgb,
[[full range]] is set to true

REC709 Color Space

A VideoColorSpace object, initialized as follows:

[[primaries]] is set to bt709,
[[transfer]] is set to bt709,
[[matrix]] is set to bt709,
[[full range]] is set to false

Codec Saturation

The state of an underlying codec implementation where the number of active decoding or encoding requests has reached an implementation specific maximum such that it is temporarily unable to accept more work. The maximum may be any value greater than 1, including infinity (no maximum). While saturated, additional calls to decode() or encode() will be buffered in the control message queue, and will increment the respective decodeQueueSize and encodeQueueSize attributes. The codec implementation will become unsaturated after making sufficient progress on the current workload.

2. Codec Processing Model

2.1. Background

This section is non-normative.

The codec interfaces defined by the specification are designed such that new codec tasks can be scheduled while previous tasks are still pending. For example, web authors can call decode() without waiting for a previous decode() to complete. This is achieved by offloading underlying codec tasks to a separate parallel queue for parallel execution.

This section describes threading behaviors as they are visible from the perspective of web authors. Implementers can choose to use more threads, as long as the externally visible behaviors of blocking and sequencing are maintained as follows.

2.2. Control Messages

A control message defines a sequence of steps corresponding to a method invocation on a codec instance (e.g. encode()).

A control message queue is a queue of control messages. Each codec instance has a control message queue stored in an internal slot named [[control message queue]].

Queuing a control message means enqueuing the message to a codec’s [[control message queue]]. Invoking codec methods will generally queue a control message to schedule work.

Running a control message means performing a sequence of steps specified by the method that enqueued the message.

The steps of a given control message can block processing later messages in the control message queue. Each codec instance has a boolean internal slot named [[message queue blocked]] that is set to true when this occurs. A blocking message will conclude by setting [[message queue blocked]] to false and rerunning the Process the control message queue steps.

All control messages will return either "processed" or "not processed". Returning "processed" indicates the message steps are being (or have been) executed and the message may be removed from the control message queue. "not processed" indicates the message must not be processed at this time and should remain in the control message queue to be retried later.

To Process the control message queue, run these steps:

While [[message queue blocked]] is false and [[control message queue]] is not empty:
1. Let front message be the first message in [[control message queue]].
2. Let outcome be the result of running the control message steps described by front message.
3. If outcome equals "not processed", break.
4. Otherwise, dequeue front message from the [[control message queue]].

2.3. Codec Work Parallel Queue

Each codec instance has an internal slot named [[codec work queue]] that is a parallel queue.

Each codec instance has an internal slot named [[codec implementation]] that refers to the underlying platform encoder or decoder. Except for the initial assignment, any steps that reference [[codec implementation]] will be enqueued to the [[codec work queue]].

Each codec instance has a unique codec task source. Tasks queued from the [[codec work queue]] to the event loop will use the codec task source.

3. AudioDecoder Interface

[Exposed=(Window,DedicatedWorker), SecureContext]
interface AudioDecoder : EventTarget {
  constructor(AudioDecoderInit init);

  readonly attribute CodecState state;
  readonly attribute unsigned long decodeQueueSize;
  attribute EventHandler ondequeue;

  undefined configure(AudioDecoderConfig config);
  undefined decode(EncodedAudioChunk chunk);
  Promise<undefined> flush();
  undefined reset();
  undefined close();

  static Promise<AudioDecoderSupport> isConfigSupported(AudioDecoderConfig config);
};

dictionary AudioDecoderInit {
  required AudioDataOutputCallback output;
  required WebCodecsErrorCallback error;
};

callback AudioDataOutputCallback = undefined(AudioData output);

3.1. Internal Slots

[[control message queue]]: A queue of control messages to be performed upon this codec instance. See [[control message queue]].
[[message queue blocked]]: A boolean indicating when processing the [[control message queue]] is blocked by a pending control message. See [[message queue blocked]].
[[codec implementation]]: Underlying decoder implementation provided by the User Agent. See [[codec implementation]].
[[codec work queue]]: A parallel queue used for running parallel steps that reference the [[codec implementation]]. See [[codec work queue]].
[[codec saturated]]: A boolean indicating when the [[codec implementation]] is unable to accept additional decoding work.
[[output callback]]: Callback given at construction for decoded outputs.
[[error callback]]: Callback given at construction for decode errors.
[[key chunk required]]: A boolean indicating that the next chunk passed to decode() MUST describe a key chunk as indicated by [[type]].
[[state]]: The current CodecState of this AudioDecoder.
[[decodeQueueSize]]: The number of pending decode requests. This number will decrease as the underlying codec is ready to accept new input.
[[pending flush promises]]: A list of unresolved promises returned by calls to flush().
[[dequeue event scheduled]]: A boolean indicating whether a dequeue event is already scheduled to fire. Used to avoid event spam.

3.2. Constructors


  AudioDecoder(init)

Let d be a new AudioDecoder object.
Assign a new queue to [[control message queue]].
Assign false to [[message queue blocked]].
Assign null to [[codec implementation]].
Assign the result of starting a new parallel queue to [[codec work queue]].
Assign false to [[codec saturated]].
Assign init.output to [[output callback]].
Assign init.error to [[error callback]].
Assign true to [[key chunk required]].
Assign "unconfigured" to [[state]]
Assign 0 to [[decodeQueueSize]].
Assign a new list to [[pending flush promises]].
Assign false to [[dequeue event scheduled]].
Return d.

3.3. Attributes

state, of type CodecState, readonly: Returns the value of [[state]].
decodeQueueSize, of type unsigned long, readonly: Returns the value of [[decodeQueueSize]].
ondequeue, of type EventHandler: An event handler IDL attribute whose event handler event type is dequeue.

3.4. Event Summary

dequeue: Fired at the AudioDecoder when the decodeQueueSize has decreased.

3.5. Methods

configure(config)

Enqueues a control message to configure the audio decoder for decoding chunks as described by config.

NOTE: This method will trigger a NotSupportedError if the User Agent does not support config. Authors are encouraged to first check support by calling isConfigSupported() with config. User Agents don’t have to support any particular codec type or configuration.

When invoked, run these steps:

If config is not a valid AudioDecoderConfig, throw a TypeError.
If [[state]] is “closed”, throw an InvalidStateError.
Set [[state]] to "configured".
Set [[key chunk required]] to true.
Queue a control message to configure the decoder with config.
Process the control message queue.

Running a control message to configure the decoder means running these steps:

Assign true to [[message queue blocked]].
Enqueue the following steps to [[codec work queue]]:
1. Let supported be the result of running the Check Configuration Support algorithm with config.
2. If supported is false, queue a task to run the Close AudioDecoder algorithm with NotSupportedError and abort these steps.
3. If needed, assign [[codec implementation]] with an implementation supporting config.
4. Configure [[codec implementation]] with config.
5. queue a task to run the following steps:
  1. Assign false to [[message queue blocked]].
  2. Queue a task to Process the control message queue.
Return "processed".

decode(chunk)

Enqueues a control message to decode the given chunk.

When invoked, run these steps:

If [[state]] is not "configured", throw an InvalidStateError.
If [[key chunk required]] is true:
1. If chunk.[[type]] is not key, throw a DataError.
2. Implementers SHOULD inspect the chunk’s [[internal data]] to verify that it is truly a key chunk. If a mismatch is detected, throw a DataError.
3. Otherwise, assign false to [[key chunk required]].
Increment [[decodeQueueSize]].
Queue a control message to decode the chunk.
Process the control message queue.

Running a control message to decode the chunk means performing these steps:

If [[codec saturated]] equals true, return "not processed".
If decoding chunk will cause the [[codec implementation]] to become saturated, assign true to [[codec saturated]].
Decrement [[decodeQueueSize]] and run the Schedule Dequeue Event algorithm.
Enqueue the following steps to the [[codec work queue]]:
1. Attempt to use [[codec implementation]] to decode the chunk.
2. If decoding results in an error, queue a task to run the Close AudioDecoder algorithm with EncodingError and return.
3. If [[codec saturated]] equals true and [[codec implementation]] is no longer saturated, queue a task to perform the following steps:
  1. Assign false to [[codec saturated]].
  2. Process the control message queue.
4. Let decoded outputs be a list of decoded audio data outputs emitted by [[codec implementation]].
5. If decoded outputs is not empty, queue a task to run the Output AudioData algorithm with decoded outputs.
Return "processed".

flush()

Completes all control messages in the control message queue and emits all outputs.

When invoked, run these steps:

If [[state]] is not "configured", return a promise rejected with InvalidStateError DOMException.
Set [[key chunk required]] to true.
Let promise be a new Promise.
Append promise to [[pending flush promises]].
Queue a control message to flush the codec with promise.
Process the control message queue.
Return promise.

Running a control message to flush the codec means performing these steps with promise.

Enqueue the following steps to the [[codec work queue]]:
1. Signal [[codec implementation]] to emit all internal pending outputs.
2. Let decoded outputs be a list of decoded audio data outputs emitted by [[codec implementation]].
3. Queue a task to perform these steps:
  1. If decoded outputs is not empty, run the Output AudioData algorithm with decoded outputs.
  2. Remove promise from [[pending flush promises]].
  3. Resolve promise.
Return "processed".

reset()

Immediately resets all state including configuration, control messages in the control message queue, and all pending callbacks.

When invoked, run the Reset AudioDecoder algorithm with an AbortError DOMException.

close()

Immediately aborts all pending work and releases system resources. Close is final.

When invoked, run the Close AudioDecoder algorithm with an AbortError DOMException.

isConfigSupported(config)

Returns a promise indicating whether the provided config is supported by the User Agent.

NOTE: The returned AudioDecoderSupport config will contain only the dictionary members that User Agent recognized. Unrecognized dictionary members will be ignored. Authors can detect unrecognized dictionary members by comparing config to their provided config.

When invoked, run these steps:

If config is not a valid AudioDecoderConfig, return a promise rejected with TypeError.
Let p be a new Promise.
Let checkSupportQueue be the result of starting a new parallel queue.
Enqueue the following steps to checkSupportQueue:
1. Let supported be the result of running the Check Configuration Support algorithm with config.
2. Queue a task to run the following steps:
  1. Let decoderSupport be a newly constructed AudioDecoderSupport, initialized as follows:
    1. Set config to the result of running the Clone Configuration algorithm with config.
    2. Set supported to supported.
  2. Resolve p with decoderSupport.
Return p.

3.6. Algorithms

Schedule Dequeue Event

If [[dequeue event scheduled]] equals true, return.
Assign true to [[dequeue event scheduled]].
Queue a task to run the following steps:
1. Fire a simple event named dequeue at this.
2. Assign false to [[dequeue event scheduled]].

Output AudioData (with outputs)

Run these steps:

For each output in outputs:
1. Let data be an AudioData, initialized as follows:
  1. Assign false to [[Detached]].
  2. Let resource be the media resource described by output.
  3. Let resourceReference be a reference to resource.
  4. Assign resourceReference to [[resource reference]].
  5. Let timestamp be the [[timestamp]] of the EncodedAudioChunk associated with output.
  6. Assign timestamp to [[timestamp]].
  7. If output uses a recognized AudioSampleFormat, assign that format to [[format]]. Otherwise, assign null to [[format]].
  8. Assign values to [[sample rate]], [[number of frames]], and [[number of channels]] as determined by output.
2. Invoke [[output callback]] with data.

Reset AudioDecoder (with exception)

Run these steps:

If [[state]] is "closed", throw an InvalidStateError.
Set [[state]] to "unconfigured".
Signal [[codec implementation]] to cease producing output for the previous configuration.
Remove all control messages from the [[control message queue]].
If [[decodeQueueSize]] is greater than zero:
1. Set [[decodeQueueSize]] to zero.
2. Run the Schedule Dequeue Event algorithm.
For each promise in [[pending flush promises]]:
1. Reject promise with exception.
2. Remove promise from [[pending flush promises]].

Close AudioDecoder (with exception)

Run these steps:

Run the Reset AudioDecoder algorithm with exception.
Set [[state]] to "closed".
Clear [[codec implementation]] and release associated system resources.
If exception is not an AbortError DOMException, invoke the [[error callback]] with exception.

4. VideoDecoder Interface

[Exposed=(Window,DedicatedWorker), SecureContext]
interface VideoDecoder : EventTarget {
  constructor(VideoDecoderInit init);

  readonly attribute CodecState state;
  readonly attribute unsigned long decodeQueueSize;
  attribute EventHandler ondequeue;

  undefined configure(VideoDecoderConfig config);
  undefined decode(EncodedVideoChunk chunk);
  Promise<undefined> flush();
  undefined reset();
  undefined close();

  static Promise<VideoDecoderSupport> isConfigSupported(VideoDecoderConfig config);
};

dictionary VideoDecoderInit {
  required VideoFrameOutputCallback output;
  required WebCodecsErrorCallback error;
};

callback VideoFrameOutputCallback = undefined(VideoFrame output);

4.1. Internal Slots

[[control message queue]]: A queue of control messages to be performed upon this codec instance. See [[control message queue]].
[[message queue blocked]]: A boolean indicating when processing the [[control message queue]] is blocked by a pending control message. See [[message queue blocked]].
[[codec implementation]]: Underlying decoder implementation provided by the User Agent. See [[codec implementation]].
[[codec work queue]]: A parallel queue used for running parallel steps that reference the [[codec implementation]]. See [[codec work queue]].
[[codec saturated]]: A boolean indicating when the [[codec implementation]] is unable to accept additional decoding work.
[[output callback]]: Callback given at construction for decoded outputs.
[[error callback]]: Callback given at construction for decode errors.
[[active decoder config]]: The VideoDecoderConfig that is actively applied.
[[key chunk required]]: A boolean indicating that the next chunk passed to decode() MUST describe a key chunk as indicated by type.
[[state]]: The current CodecState of this VideoDecoder.
[[decodeQueueSize]]: The number of pending decode requests. This number will decrease as the underlying codec is ready to accept new input.
[[pending flush promises]]: A list of unresolved promises returned by calls to flush().
[[dequeue event scheduled]]: A boolean indicating whether a dequeue event is already scheduled to fire. Used to avoid event spam.

4.2. Constructors


  VideoDecoder(init)

Let d be a new VideoDecoder object.
Assign a new queue to [[control message queue]].
Assign false to [[message queue blocked]].
Assign null to [[codec implementation]].
Assign the result of starting a new parallel queue to [[codec work queue]].
Assign false to [[codec saturated]].
Assign init.output to [[output callback]].
Assign init.error to [[error callback]].
Assign null to [[active decoder config]].
Assign true to [[key chunk required]].
Assign "unconfigured" to [[state]]
Assign 0 to [[decodeQueueSize]].
Assign a new list to [[pending flush promises]].
Assign false to [[dequeue event scheduled]].
Return d.

4.3. Attributes

state, of type CodecState, readonly: Returns the value of [[state]].
decodeQueueSize, of type unsigned long, readonly: Returns the value of [[decodeQueueSize]].
ondequeue, of type EventHandler: An event handler IDL attribute whose event handler event type is dequeue.

4.4. Event Summary

dequeue: Fired at the VideoDecoder when the decodeQueueSize has decreased.

4.5. Methods

configure(config)

Enqueues a control message to configure the video decoder for decoding chunks as described by config.

When invoked, run these steps:

If config is not a valid VideoDecoderConfig, throw a TypeError.
If [[state]] is “closed”, throw an InvalidStateError.
Set [[state]] to "configured".
Set [[key chunk required]] to true.
Queue a control message to configure the decoder with config.
Process the control message queue.

Running a control message to configure the decoder means running these steps:

Assign true to [[message queue blocked]].
Enqueue the following steps to [[codec work queue]]:
1. Let supported be the result of running the Check Configuration Support algorithm with config.
2. If supported is false, queue a task to run the Close VideoDecoder algorithm with NotSupportedError and abort these steps.
3. If needed, assign [[codec implementation]] with an implementation supporting config.
4. Configure [[codec implementation]] with config.
5. Assign config to [[active decoder config]].
6. queue a task to run the following steps:
  1. Assign false to [[message queue blocked]].
  2. Queue a task to Process the control message queue.
Return "processed".

decode(chunk)

Enqueues a control message to decode the given chunk.

NOTE: Authors are encouraged to call close() on output VideoFrames immediately when frames are no longer needed. The underlying media resources are owned by the VideoDecoder and failing to release them (or waiting for garbage collection) can cause decoding to stall.

NOTE: VideoDecoder requires that frames are output in the order they expect to be presented, commonly known as presentation order. When using some [[codec implementation]]s the User Agent will have to reorder outputs into presentation order.

When invoked, run these steps:

If [[state]] is not "configured", throw an InvalidStateError.
If [[key chunk required]] is true:
1. If chunk.type is not key, throw a DataError.
2. Implementers SHOULD inspect the chunk’s [[internal data]] to verify that it is truly a key chunk. If a mismatch is detected, throw a DataError.
3. Otherwise, assign false to [[key chunk required]].
Increment [[decodeQueueSize]].
Queue a control message to decode the chunk.
Process the control message queue.

Running a control message to decode the chunk means performing these steps:

If [[codec saturated]] equals true, return "not processed".
If decoding chunk will cause the [[codec implementation]] to become saturated, assign true to [[codec saturated]].
Decrement [[decodeQueueSize]] and run the Schedule Dequeue Event algorithm.
Enqueue the following steps to the [[codec work queue]]:
1. Attempt to use [[codec implementation]] to decode the chunk.
2. If decoding results in an error, queue a task to run the Close VideoDecoder algorithm with EncodingError and return.
3. If [[codec saturated]] equals true and [[codec implementation]] is no longer saturated, queue a task to perform the following steps:
  1. Assign false to [[codec saturated]].
  2. Process the control message queue.
4. Let decoded outputs be a list of decoded video data outputs emitted by [[codec implementation]] in presentation order.
5. If decoded outputs is not empty, queue a task to run the Output VideoFrame algorithm with decoded outputs.
Return "processed".

flush()

Completes all control messages in the control message queue and emits all outputs.

When invoked, run these steps:

If [[state]] is not "configured", return a promise rejected with InvalidStateError DOMException.
Set [[key chunk required]] to true.
Let promise be a new Promise.
Append promise to [[pending flush promises]].
Queue a control message to flush the codec with promise.
Process the control message queue.
Return promise.

Running a control message to flush the codec means performing these steps with promise.

Enqueue the following steps to the [[codec work queue]]:
1. Signal [[codec implementation]] to emit all internal pending outputs.
2. Let decoded outputs be a list of decoded video data outputs emitted by [[codec implementation]].
3. Queue a task to perform these steps:
  1. If decoded outputs is not empty, run the Output VideoFrame algorithm with decoded outputs.
  2. Remove promise from [[pending flush promises]].
  3. Resolve promise.
Return "processed".

reset()

Immediately resets all state including configuration, control messages in the control message queue, and all pending callbacks.

When invoked, run the Reset VideoDecoder algorithm with an AbortError DOMException.

close()

Immediately aborts all pending work and releases system resources. Close is final.

When invoked, run the Close VideoDecoder algorithm with an AbortError DOMException.

isConfigSupported(config)

Returns a promise indicating whether the provided config is supported by the User Agent.

NOTE: The returned VideoDecoderSupport config will contain only the dictionary members that User Agent recognized. Unrecognized dictionary members will be ignored. Authors can detect unrecognized dictionary members by comparing config to their provided config.

When invoked, run these steps:

If config is not a valid VideoDecoderConfig, return a promise rejected with TypeError.
Let p be a new Promise.
Let checkSupportQueue be the result of starting a new parallel queue.
Enqueue the following steps to checkSupportQueue:
1. Let supported be the result of running the Check Configuration Support algorithm with config.
2. Queue a task to run the following steps:
  1. Let decoderSupport be a newly constructed VideoDecoderSupport, initialized as follows:
    1. Set config to the result of running the Clone Configuration algorithm with config.
    2. Set supported to supported.
  2. Resolve p with decoderSupport.
Return p.

4.6. Algorithms

Schedule Dequeue Event

If [[dequeue event scheduled]] equals true, return.
Assign true to [[dequeue event scheduled]].
Queue a task to run the following steps:
1. Fire a simple event named dequeue at this.
2. Assign false to [[dequeue event scheduled]].

Output VideoFrames (with outputs)

Run these steps:

For each output in outputs:
1. Let timestamp and duration be the timestamp and duration from the EncodedVideoChunk associated with output.
2. Let displayAspectWidth and displayAspectHeight be undefined.
3. If displayAspectWidth and displayAspectHeight exist in the [[active decoder config]], assign their values to displayAspectWidth and displayAspectHeight respectively.
4. Let colorSpace be the VideoColorSpace for output as detected by the codec implementation. If no VideoColorSpace is detected, let colorSpace be undefined.
  
  NOTE: The codec implementation can detect a VideoColorSpace by analyzing the bitstream. Detection is made on a best-effort basis. The exact method of detection is implementer defined and codec-specific. Authors can override the detected VideoColorSpace by providing a colorSpace in the VideoDecoderConfig.
5. If colorSpace exists in the [[active decoder config]], assign its value to colorSpace. In that case, User Agents MAY replace null members of colorSpace with the corresponding values detected by the codec implementation. FIXME: Properly specify the case of null members.
6. Assign the values of rotation and flip to rotation and flip respectively.
7. Let frame be the result of running the Create a VideoFrame algorithm with output, timestamp, duration, displayAspectWidth, displayAspectHeight, colorSpace, rotation, and flip.
8. Invoke [[output callback]] with frame.

Reset VideoDecoder (with exception)

Run these steps:

If state is "closed", throw an InvalidStateError.
Set state to "unconfigured".
Signal [[codec implementation]] to cease producing output for the previous configuration.
Remove all control messages from the [[control message queue]].
If [[decodeQueueSize]] is greater than zero:
1. Set [[decodeQueueSize]] to zero.
2. Run the Schedule Dequeue Event algorithm.
For each promise in [[pending flush promises]]:
1. Reject promise with exception.
2. Remove promise from [[pending flush promises]].

Close VideoDecoder (with exception)

Run these steps:

Run the Reset VideoDecoder algorithm with exception.
Set state to "closed".
Clear [[codec implementation]] and release associated system resources.
If exception is not an AbortError DOMException, invoke the [[error callback]] with exception.

5. AudioEncoder Interface

[Exposed=(Window,DedicatedWorker), SecureContext]
interface AudioEncoder : EventTarget {
  constructor(AudioEncoderInit init);

  readonly attribute CodecState state;
  readonly attribute unsigned long encodeQueueSize;
  attribute EventHandler ondequeue;

  undefined configure(AudioEncoderConfig config);
  undefined encode(AudioData data);
  Promise<undefined> flush();
  undefined reset();
  undefined close();

  static Promise<AudioEncoderSupport> isConfigSupported(AudioEncoderConfig config);
};

dictionary AudioEncoderInit {
  required EncodedAudioChunkOutputCallback output;
  required WebCodecsErrorCallback error;
};

callback EncodedAudioChunkOutputCallback =
    undefined (EncodedAudioChunk output,
               optional EncodedAudioChunkMetadata metadata = {});

5.1. Internal Slots

[[control message queue]]: A queue of control messages to be performed upon this codec instance. See [[control message queue]].
[[message queue blocked]]: A boolean indicating when processing the [[control message queue]] is blocked by a pending control message. See [[message queue blocked]].
[[codec implementation]]: Underlying encoder implementation provided by the User Agent. See [[codec implementation]].
[[codec work queue]]: A parallel queue used for running parallel steps that reference the [[codec implementation]]. See [[codec work queue]].
[[codec saturated]]: A boolean indicating when the [[codec implementation]] is unable to accept additional encoding work.
[[output callback]]: Callback given at construction for encoded outputs.
[[error callback]]: Callback given at construction for encode errors.
[[active encoder config]]: The AudioEncoderConfig that is actively applied.
[[active output config]]: The AudioDecoderConfig that describes how to decode the most recently emitted EncodedAudioChunk.
[[state]]: The current CodecState of this AudioEncoder.
[[encodeQueueSize]]: The number of pending encode requests. This number will decrease as the underlying codec is ready to accept new input.
[[pending flush promises]]: A list of unresolved promises returned by calls to flush().
[[dequeue event scheduled]]: A boolean indicating whether a dequeue event is already scheduled to fire. Used to avoid event spam.

5.2. Constructors


  AudioEncoder(init)

Let e be a new AudioEncoder object.
Assign a new queue to [[control message queue]].
Assign false to [[message queue blocked]].
Assign null to [[codec implementation]].
Assign the result of starting a new parallel queue to [[codec work queue]].
Assign false to [[codec saturated]].
Assign init.output to [[output callback]].
Assign init.error to [[error callback]].
Assign null to [[active encoder config]].
Assign null to [[active output config]].
Assign "unconfigured" to [[state]]
Assign 0 to [[encodeQueueSize]].
Assign a new list to [[pending flush promises]].
Assign false to [[dequeue event scheduled]].
Return e.

5.3. Attributes

state, of type CodecState, readonly: Returns the value of [[state]].
encodeQueueSize, of type unsigned long, readonly: Returns the value of [[encodeQueueSize]].
ondequeue, of type EventHandler: An event handler IDL attribute whose event handler event type is dequeue.

5.4. Event Summary

dequeue: Fired at the AudioEncoder when the encodeQueueSize has decreased.

5.5. Methods

configure(config)

Enqueues a control message to configure the audio encoder for encoding audio data as described by config.

When invoked, run these steps:

If config is not a valid AudioEncoderConfig, throw a TypeError.
If [[state]] is "closed", throw an InvalidStateError.
Set [[state]] to "configured".
Queue a control message to configure the encoder using config.
Process the control message queue.

Running a control message to configure the encoder means performing these steps:

Assign true to [[message queue blocked]].
Enqueue the following steps to [[codec work queue]]:
1. Let supported be the result of running the Check Configuration Support algorithm with config.
2. If supported is false, queue a task to run the Close AudioEncoder algorithm with NotSupportedError and abort these steps.
3. If needed, assign [[codec implementation]] with an implementation supporting config.
4. Configure [[codec implementation]] with config.
5. Assign config to [[active encoder config]].
6. queue a task to run the following steps:
  1. Assign false to [[message queue blocked]].
  2. Queue a task to Process the control message queue.
Return "processed".

encode(data)

Enqueues a control message to encode the given data.

When invoked, run these steps:

If the value of data’s [[Detached]] internal slot is true, throw a TypeError.
If [[state]] is not "configured", throw an InvalidStateError.
Let dataClone hold the result of running the Clone AudioData algorithm with data.
Increment [[encodeQueueSize]].
Queue a control message to encode dataClone.
Process the control message queue.

Running a control message to encode the data means performing these steps:

If [[codec saturated]] equals true, return "not processed".
If encoding data will cause the [[codec implementation]] to become saturated, assign true to [[codec saturated]].
Decrement [[encodeQueueSize]] and run the Schedule Dequeue Event algorithm.
Enqueue the following steps to the [[codec work queue]]:
1. Attempt to use [[codec implementation]] to encode the media resource described by dataClone.
2. If encoding results in an error, queue a task to run the Close AudioEncoder algorithm with EncodingError and return.
3. If [[codec saturated]] equals true and [[codec implementation]] is no longer saturated, queue a task to perform the following steps:
  1. Assign false to [[codec saturated]].
  2. Process the control message queue.
4. Let encoded outputs be a list of encoded audio data outputs emitted by [[codec implementation]].
5. If encoded outputs is not empty, queue a task to run the Output EncodedAudioChunks algorithm with encoded outputs.
Return "processed".

flush()

Completes all control messages in the control message queue and emits all outputs.

When invoked, run these steps:

If [[state]] is not "configured", return a promise rejected with InvalidStateError DOMException.
Let promise be a new Promise.
Append promise to [[pending flush promises]].
Queue a control message to flush the codec with promise.
Process the control message queue.
Return promise.

Running a control message to flush the codec means performing these steps with promise.

Enqueue the following steps to the [[codec work queue]]:
1. Signal [[codec implementation]] to emit all internal pending outputs.
2. Let encoded outputs be a list of encoded audio data outputs emitted by [[codec implementation]].
3. Queue a task to perform these steps:
  1. If encoded outputs is not empty, run the Output EncodedAudioChunks algorithm with encoded outputs.
  2. Remove promise from [[pending flush promises]].
  3. Resolve promise.
Return "processed".

reset()

Immediately resets all state including configuration, control messages in the control message queue, and all pending callbacks.

When invoked, run the Reset AudioEncoder algorithm with an AbortError DOMException.

close()

Immediately aborts all pending work and releases system resources. Close is final.

When invoked, run the Close AudioEncoder algorithm with an AbortError DOMException.

isConfigSupported(config)

Returns a promise indicating whether the provided config is supported by the User Agent.

NOTE: The returned AudioEncoderSupport config will contain only the dictionary members that User Agent recognized. Unrecognized dictionary members will be ignored. Authors can detect unrecognized dictionary members by comparing config to their provided config.

When invoked, run these steps:

If config is not a valid AudioEncoderConfig, return a promise rejected with TypeError.
Let p be a new Promise.
Let checkSupportQueue be the result of starting a new parallel queue.
Enqueue the following steps to checkSupportQueue:
1. Let supported be the result of running the Check Configuration Support algorithm with config.
2. Queue a task to run the following steps:
  1. Let encoderSupport be a newly constructed AudioEncoderSupport, initialized as follows:
    1. Set config to the result of running the Clone Configuration algorithm with config.
    2. Set supported to supported.
  2. Resolve p with encoderSupport.
Return p.

5.6. Algorithms

Schedule Dequeue Event

If [[dequeue event scheduled]] equals true, return.
Assign true to [[dequeue event scheduled]].
Queue a task to run the following steps:
1. Fire a simple event named dequeue at this.
2. Assign false to [[dequeue event scheduled]].

Output EncodedAudioChunks (with outputs)

Run these steps:

For each output in outputs:
1. Let chunkInit be an EncodedAudioChunkInit with the following keys:
  1. Let data contain the encoded audio data from output.
  2. Let type be the EncodedAudioChunkType of output.
  3. Let timestamp be the timestamp from the AudioData associated with output.
  4. Let duration be the duration from the AudioData associated with output.
2. Let chunk be a new EncodedAudioChunk constructed with chunkInit.
3. Let chunkMetadata be a new EncodedAudioChunkMetadata.
4. Let encoderConfig be the [[active encoder config]].
5. Let outputConfig be a new AudioDecoderConfig that describes output. Initialize outputConfig as follows:
  1. Assign encoderConfig.codec to outputConfig.codec.
  2. Assign encoderConfig.sampleRate to outputConfig.sampleRate.
  3. Assign to encoderConfig.numberOfChannels to outputConfig.numberOfChannels.
  4. Assign outputConfig.description with a sequence of codec specific bytes as determined by the [[codec implementation]]. The User Agent MUST ensure that the provided description could be used to correctly decode output.
    
    NOTE: The codec specific requirements for populating the description are described in the [WEBCODECS-CODEC-REGISTRY].
6. If outputConfig and [[active output config]] are not equal dictionaries:
  1. Assign outputConfig to chunkMetadata.decoderConfig.
  2. Assign outputConfig to [[active output config]].
7. Invoke [[output callback]] with chunk and chunkMetadata.

Reset AudioEncoder (with exception)

Run these steps:

If [[state]] is "closed", throw an InvalidStateError.
Set [[state]] to "unconfigured".
Set [[active encoder config]] to null.
Set [[active output config]] to null.
Signal [[codec implementation]] to cease producing output for the previous configuration.
Remove all control messages from the [[control message queue]].
If [[encodeQueueSize]] is greater than zero:
1. Set [[encodeQueueSize]] to zero.
2. Run the Schedule Dequeue Event algorithm.
For each promise in [[pending flush promises]]:
1. Reject promise with exception.
2. Remove promise from [[pending flush promises]].

Close AudioEncoder (with exception)

Run these steps:

Run the Reset AudioEncoder algorithm with exception.
Set [[state]] to "closed".
Clear [[codec implementation]] and release associated system resources.
If exception is not an AbortError DOMException, invoke the [[error callback]] with exception.

5.7. EncodedAudioChunkMetadata

The following metadata dictionary is emitted by the EncodedAudioChunkOutputCallback alongside an associated EncodedAudioChunk.

dictionary EncodedAudioChunkMetadata {
  AudioDecoderConfig decoderConfig;
};

decoderConfig, of type AudioDecoderConfig: A AudioDecoderConfig that authors MAY use to decode the associated EncodedAudioChunk.

6. VideoEncoder Interface

[Exposed=(Window,DedicatedWorker), SecureContext]
interface VideoEncoder : EventTarget {
  constructor(VideoEncoderInit init);

  readonly attribute CodecState state;
  readonly attribute unsigned long encodeQueueSize;
  attribute EventHandler ondequeue;

  undefined configure(VideoEncoderConfig config);
  undefined encode(VideoFrame frame, optional VideoEncoderEncodeOptions options = {});
  Promise<undefined> flush();
  undefined reset();
  undefined close();

  static Promise<VideoEncoderSupport> isConfigSupported(VideoEncoderConfig config);
};

dictionary VideoEncoderInit {
  required EncodedVideoChunkOutputCallback output;
  required WebCodecsErrorCallback error;
};

callback EncodedVideoChunkOutputCallback =
    undefined (EncodedVideoChunk chunk,
               optional EncodedVideoChunkMetadata metadata = {});

6.1. Internal Slots

[[control message queue]]: A queue of control messages to be performed upon this codec instance. See [[control message queue]].
[[message queue blocked]]: A boolean indicating when processing the [[control message queue]] is blocked by a pending control message. See [[message queue blocked]].
[[codec implementation]]: Underlying encoder implementation provided by the User Agent. See [[codec implementation]].
[[codec work queue]]: A parallel queue used for running parallel steps that reference the [[codec implementation]]. See [[codec work queue]].
[[codec saturated]]: A boolean indicating when the [[codec implementation]] is unable to accept additional encoding work.
[[output callback]]: Callback given at construction for encoded outputs.
[[error callback]]: Callback given at construction for encode errors.
[[active encoder config]]: The VideoEncoderConfig that is actively applied.
[[active output config]]: The VideoDecoderConfig that describes how to decode the most recently emitted EncodedVideoChunk.
[[state]]: The current CodecState of this VideoEncoder.
[[encodeQueueSize]]: The number of pending encode requests. This number will decrease as the underlying codec is ready to accept new input.
[[pending flush promises]]: A list of unresolved promises returned by calls to flush().
[[dequeue event scheduled]]: A boolean indicating whether a dequeue event is already scheduled to fire. Used to avoid event spam.
[[active orientation]]: An integer and boolean pair indicating the [[flip]] and [[rotation]] of the first VideoFrame given to encode() after configure().

6.2. Constructors


  VideoEncoder(init)

Let e be a new VideoEncoder object.
Assign a new queue to [[control message queue]].
Assign false to [[message queue blocked]].
Assign null to [[codec implementation]].
Assign the result of starting a new parallel queue to [[codec work queue]].
Assign false to [[codec saturated]].
Assign init.output to [[output callback]].
Assign init.error to [[error callback]].
Assign null to [[active encoder config]].
Assign null to [[active output config]].
Assign "unconfigured" to [[state]]
Assign 0 to [[encodeQueueSize]].
Assign a new list to [[pending flush promises]].
Assign false to [[dequeue event scheduled]].
Return e.

6.3. Attributes

state, of type CodecState, readonly: Returns the value of [[state]].
encodeQueueSize, of type unsigned long, readonly: Returns the value of [[encodeQueueSize]].
ondequeue, of type EventHandler: An event handler IDL attribute whose event handler event type is dequeue.

6.4. Event Summary

dequeue: Fired at the VideoEncoder when the encodeQueueSize has decreased.

6.5. Methods

configure(config)

Enqueues a control message to configure the video encoder for encoding video frames as described by config.

When invoked, run these steps:

If config is not a valid VideoEncoderConfig, throw a TypeError.
If [[state]] is "closed", throw an InvalidStateError.
Set [[state]] to "configured".
Set [[active orientation]] to null.
Queue a control message to configure the encoder using config.
Process the control message queue.

Running a control message to configure the encoder means performing these steps:

Assign true to [[message queue blocked]].
Enqueue the following steps to [[codec work queue]]:
1. Let supported be the result of running the Check Configuration Support algorithm with config.
2. If supported is false, queue a task to run the Close VideoEncoder algorithm with NotSupportedError and abort these steps.
3. If needed, assign [[codec implementation]] with an implementation supporting config.
4. Configure [[codec implementation]] with config.
5. Assign config to [[active encoder config]].
6. queue a task to run the following steps:
  1. Assign false to [[message queue blocked]].
  2. Queue a task to Process the control message queue.
Return "processed".

encode(frame, options)

Enqueues a control message to encode the given frame.

When invoked, run these steps:

If the value of frame’s [[Detached]] internal slot is true, throw a TypeError.
If [[state]] is not "configured", throw an InvalidStateError.
If [[active orientation]] is not null and does not match frame’s [[rotation]] and [[flip]] throw a DataError.
If [[active orientation]] is null, set it to frame’s [[rotation]] and [[flip]].
Let frameClone hold the result of running the Clone VideoFrame algorithm with frame.
Increment [[encodeQueueSize]].
Queue a control message to encode frameClone.
Process the control message queue.

Running a control message to encode the frame means performing these steps:

If [[codec saturated]] equals true, return "not processed".
If encoding frame will cause the [[codec implementation]] to become saturated, assign true to [[codec saturated]].
Decrement [[encodeQueueSize]] and run the Schedule Dequeue Event algorithm.
Enqueue the following steps to the [[codec work queue]]:
1. Attempt to use [[codec implementation]] to encode the frameClone according to options.
2. If encoding results in an error, queue a task to run the Close VideoEncoder algorithm with EncodingError and return.
3. If [[codec saturated]] equals true and [[codec implementation]] is no longer saturated, queue a task to perform the following steps:
  1. Assign false to [[codec saturated]].
  2. Process the control message queue.
4. Let encoded outputs be a list of encoded video data outputs emitted by [[codec implementation]].
5. If encoded outputs is not empty, queue a task to run the Output EncodedVideoChunks algorithm with encoded outputs.
Return "processed".

flush()

Completes all control messages in the control message queue and emits all outputs.

When invoked, run these steps:

If [[state]] is not "configured", return a promise rejected with InvalidStateError DOMException.
Let promise be a new Promise.
Append promise to [[pending flush promises]].
Queue a control message to flush the codec with promise.
Process the control message queue.
Return promise.

Running a control message to flush the codec means performing these steps with promise:

Enqueue the following steps to the [[codec work queue]]:
1. Signal [[codec implementation]] to emit all internal pending outputs.
2. Let encoded outputs be a list of encoded video data outputs emitted by [[codec implementation]].
3. Queue a task to perform these steps:
  1. If encoded outputs is not empty, run the Output EncodedVideoChunks algorithm with encoded outputs.
  2. Remove promise from [[pending flush promises]].
  3. Resolve promise.
Return "processed".

reset()

Immediately resets all state including configuration, control messages in the control message queue, and all pending callbacks.

When invoked, run the Reset VideoEncoder algorithm with an AbortError DOMException.

close()

Immediately aborts all pending work and releases system resources. Close is final.

When invoked, run the Close VideoEncoder algorithm with an AbortError DOMException.

isConfigSupported(config)

Returns a promise indicating whether the provided config is supported by the User Agent.

NOTE: The returned VideoEncoderSupport config will contain only the dictionary members that User Agent recognized. Unrecognized dictionary members will be ignored. Authors can detect unrecognized dictionary members by comparing config to their provided config.

When invoked, run these steps:

If config is not a valid VideoEncoderConfig, return a promise rejected with TypeError.
Let p be a new Promise.
Let checkSupportQueue be the result of starting a new parallel queue.
Enqueue the following steps to checkSupportQueue:
1. Let supported be the result of running the Check Configuration Support algorithm with config.
2. Queue a task to run the following steps:
  1. Let encoderSupport be a newly constructed VideoEncoderSupport, initialized as follows:
    1. Set config to the result of running the Clone Configuration algorithm with config.
    2. Set supported to supported.
3. Resolve p with encoderSupport.
Return p.

6.6. Algorithms

Schedule Dequeue Event

If [[dequeue event scheduled]] equals true, return.
Assign true to [[dequeue event scheduled]].
Queue a task to run the following steps:
1. Fire a simple event named dequeue at this.
2. Assign false to [[dequeue event scheduled]].

Output EncodedVideoChunks (with outputs)

Run these steps:

For each output in outputs:
1. Let chunkInit be an EncodedVideoChunkInit with the following keys:
  1. Let data contain the encoded video data from output.
  2. Let type be the EncodedVideoChunkType of output.
  3. Let timestamp be the [[timestamp]] from the VideoFrame associated with output.
  4. Let duration be the [[duration]] from the VideoFrame associated with output.
2. Let chunk be a new EncodedVideoChunk constructed with chunkInit.
3. Let chunkMetadata be a new EncodedVideoChunkMetadata.
4. Let encoderConfig be the [[active encoder config]].
5. Let outputConfig be a VideoDecoderConfig that describes output. Initialize outputConfig as follows:
  1. Assign encoderConfig.codec to outputConfig.codec.
  2. Assign encoderConfig.width to outputConfig.codedWidth.
  3. Assign encoderConfig.height to outputConfig.codedHeight.
  4. Assign encoderConfig.displayWidth to outputConfig.displayAspectWidth.
  5. Assign encoderConfig.displayHeight to outputConfig.displayAspectHeight.
  6. Assign [[rotation]] from the VideoFrame associated with output to outputConfig.rotation.
  7. Assign [[flip]] from the VideoFrame associated with output to outputConfig.flip.
  8. Assign the remaining keys of outputConfig as determined by [[codec implementation]]. The User Agent MUST ensure that the configuration is completely described such that outputConfig could be used to correctly decode output.
    
    NOTE: The codec specific requirements for populating the description are described in the [WEBCODECS-CODEC-REGISTRY].
6. If outputConfig and [[active output config]] are not equal dictionaries:
  1. Assign outputConfig to chunkMetadata.decoderConfig.
  2. Assign outputConfig to [[active output config]].
7. If encoderConfig.scalabilityMode describes multiple temporal layers:
  1. Let svc be a new SvcOutputMetadata instance.
  2. Let temporal_layer_id be the zero-based index describing the temporal layer for output.
  3. Assign temporal_layer_id to svc.temporalLayerId.
  4. Assign svc to chunkMetadata.svc.
8. If encoderConfig.alpha is set to "keep":
  1. Let alphaSideData be the encoded alpha data in output.
  2. Assign alphaSideData to chunkMetadata.alphaSideData.
9. Invoke [[output callback]] with chunk and chunkMetadata.

Reset VideoEncoder (with exception)

Run these steps:

If [[state]] is "closed", throw an InvalidStateError.
Set [[state]] to "unconfigured".
Set [[active encoder config]] to null.
Set [[active output config]] to null.
Signal [[codec implementation]] to cease producing output for the previous configuration.
Remove all control messages from the [[control message queue]].
If [[encodeQueueSize]] is greater than zero:
1. Set [[encodeQueueSize]] to zero.
2. Run the Schedule Dequeue Event algorithm.
For each promise in [[pending flush promises]]:
1. Reject promise with exception.
2. Remove promise from [[pending flush promises]].

Close VideoEncoder (with exception)

Run these steps:

Run the Reset VideoEncoder algorithm with exception.
Set [[state]] to "closed".
Clear [[codec implementation]] and release associated system resources.
If exception is not an AbortError DOMException, invoke the [[error callback]] with exception.

6.7. EncodedVideoChunkMetadata

The following metadata dictionary is emitted by the EncodedVideoChunkOutputCallback alongside an associated EncodedVideoChunk.

dictionary EncodedVideoChunkMetadata {
  VideoDecoderConfig decoderConfig;
  SvcOutputMetadata svc;
  BufferSource alphaSideData;
};

dictionary SvcOutputMetadata {
  unsigned long temporalLayerId;
};

decoderConfig, of type VideoDecoderConfig: A VideoDecoderConfig that authors MAY use to decode the associated EncodedVideoChunk.
svc, of type SvcOutputMetadata: A collection of metadata describing this EncodedVideoChunk with respect to the configured scalabilityMode.
alphaSideData, of type BufferSource: A BufferSource that contains the EncodedVideoChunk’s extra alpha channel data.
temporalLayerId, of type unsigned long: A number that identifies the temporal layer for the associated EncodedVideoChunk.

7. Configurations

7.1. Check Configuration Support (with `config`)

Run these steps:

If the codec string in config.codec is not a valid codec string or is otherwise unrecognized by the User Agent, return false.
If config is an AudioDecoderConfig or VideoDecoderConfig and the User Agent can’t provide a codec that can decode the exact profile (where present), level (where present), and constraint bits (where present) indicated by the codec string in config.codec, return false.
If config is an AudioEncoderConfig or VideoEncoderConfig:
1. If the codec string in config.codec contains a profile and the User Agent can’t provide a codec that can encode the exact profile indicated by config.codec, return false.
2. If the codec string in config.codec contains a level and the User Agent can’t provide a codec that can encode to a level less than or equal to the level indicated by config.codec, return false.
3. If the codec string in config.codec contains constraint bits and the User Agent can’t provide a codec that can produce an encoded bitstream at least as constrained as indicated by config.codec, return false.
If the User Agent can provide a codec to support all entries of the config, including applicable default values for keys that are not included, return true.

NOTE: The types AudioDecoderConfig, VideoDecoderConfig, AudioEncoderConfig, and VideoEncoderConfig each define their respective configuration entries and defaults.

NOTE: Support for a given configuration can change dynamically if the hardware is altered (e.g. external GPU unplugged) or if essential hardware resources are exhausted. User Agents describe support on a best-effort basis given the resources that are available at the time of the query.
Otherwise, return false.

7.2. Clone Configuration (with `config`)

NOTE: This algorithm will copy only the dictionary members that the User Agent recognizes as part of the dictionary type.

Run these steps:

Let dictType be the type of dictionary config.
Let clone be a new empty instance of dictType.
For each dictionary member m defined on dictType:
1. If m does not exist in config, then continue.
2. If config[m] is a nested dictionary, set clone[m] to the result of recursively running the Clone Configuration algorithm with config[m].
3. Otherwise, assign a copy of config[m] to clone[m].

Note: This implements a "deep-copy". These configuration objects are frequently used as the input of asynchronous operations. Copying means that modifying the original object while the operation is in flight won’t change the operation’s outcome.

7.3. Signalling Configuration Support

7.3.1. AudioDecoderSupport

dictionary AudioDecoderSupport {
  boolean supported;
  AudioDecoderConfig config;
};

supported, of type boolean: A boolean indicating the whether the corresponding config is supported by the User Agent.
config, of type AudioDecoderConfig: An AudioDecoderConfig used by the User Agent in determining the value of supported.

7.3.2. VideoDecoderSupport

dictionary VideoDecoderSupport {
  boolean supported;
  VideoDecoderConfig config;
};

supported, of type boolean: A boolean indicating the whether the corresponding config is supported by the User Agent.
config, of type VideoDecoderConfig: A VideoDecoderConfig used by the User Agent in determining the value of supported.

7.3.3. AudioEncoderSupport

dictionary AudioEncoderSupport {
  boolean supported;
  AudioEncoderConfig config;
};

supported, of type boolean: A boolean indicating the whether the corresponding config is supported by the User Agent.
config, of type AudioEncoderConfig: An AudioEncoderConfig used by the User Agent in determining the value of supported.

7.3.4. VideoEncoderSupport

dictionary VideoEncoderSupport {
  boolean supported;
  VideoEncoderConfig config;
};

supported, of type boolean: A boolean indicating the whether the corresponding config is supported by the User Agent.
config, of type VideoEncoderConfig: A VideoEncoderConfig used by the User Agent in determining the value of supported.

7.4. Codec String

A codec string describes a given codec format to be used for encoding or decoding.

A valid codec string MUST meet the following conditions.

Is valid per the relevant codec specification (see examples below).
It describes a single codec.
It is unambiguous about codec profile, level, and constraint bits for codecs that define these concepts.

NOTE: In other media specifications, codec strings historically accompanied a MIME type as the "codecs=" parameter (isTypeSupported(), canPlayType()) [RFC6381]. In this specification, encoded media is not containerized; hence, only the value of the codecs parameter is accepted.

NOTE: Encoders for codecs that define level and constraint bits have flexibility around these parameters, but won’t produce bitstreams that have a higher level or are less constrained than requested.

The format and semantics for codec strings are defined by codec registrations listed in the [WEBCODECS-CODEC-REGISTRY]. A compliant implementation MAY support any combination of codec registrations or none at all.

7.5. AudioDecoderConfig

dictionary AudioDecoderConfig {
  required DOMString codec;
  [EnforceRange] required unsigned long sampleRate;
  [EnforceRange] required unsigned long numberOfChannels;
  AllowSharedBufferSource description;
};

To check if an AudioDecoderConfig is a valid AudioDecoderConfig, run these steps:

If codec is empty after stripping leading and trailing ASCII whitespace, return false.
If description is [detached], return false.
If sampleRate or numberOfChannels are equal to zero, return false.
Return true.

codec, of type DOMString: Contains a codec string in config.codec describing the codec.
sampleRate, of type unsigned long: The number of frame samples per second.
numberOfChannels, of type unsigned long: The number of audio channels.
description, of type AllowSharedBufferSource: A sequence of codec specific bytes, commonly known as extradata.
NOTE: The registrations in the [WEBCODECS-CODEC-REGISTRY] describe whether/how to populate this sequence, corresponding to the provided codec.

7.6. VideoDecoderConfig

dictionary VideoDecoderConfig {
  required DOMString codec;
  AllowSharedBufferSource description;
  [EnforceRange] unsigned long codedWidth;
  [EnforceRange] unsigned long codedHeight;
  [EnforceRange] unsigned long displayAspectWidth;
  [EnforceRange] unsigned long displayAspectHeight;
  VideoColorSpaceInit colorSpace;
  HardwareAcceleration hardwareAcceleration = "no-preference";
  boolean optimizeForLatency = false;
  double rotation = 0;
  boolean flip = false;
};

To check if a VideoDecoderConfig is a valid VideoDecoderConfig, run these steps:

If codec is empty after stripping leading and trailing ASCII whitespace, return false.
If one of codedWidth or codedHeight is provided but the other isn’t, return false.
If codedWidth = 0 or codedHeight = 0, return false.
If one of displayAspectWidth or displayAspectHeight is provided but the other isn’t, return false.
If displayAspectWidth = 0 or displayAspectHeight = 0, return false.
If description is [detached], return false.
Return true.

codec, of type DOMString: Contains a codec string describing the codec.
description, of type AllowSharedBufferSource: A sequence of codec specific bytes, commonly known as extradata.
NOTE: The registrations in the [WEBCODECS-CODEC-REGISTRY] describes whether/how to populate this sequence, corresponding to the provided codec.
codedWidth, of type unsigned long: Width of the VideoFrame in pixels, potentially including non-visible padding, and prior to considering potential ratio adjustments.
codedHeight, of type unsigned long: Height of the VideoFrame in pixels, potentially including non-visible padding, and prior to considering potential ratio adjustments.
NOTE: codedWidth and codedHeight are used when selecting a [[codec implementation]].
displayAspectWidth, of type unsigned long: Horizontal dimension of the VideoFrame’s aspect ratio when displayed.
displayAspectHeight, of type unsigned long: Vertical dimension of the VideoFrame’s aspect ratio when displayed.
NOTE: displayWidth and displayHeight can both be different from displayAspectWidth and displayAspectHeight, but have identical ratios, after scaling is applied when creating the video frame.
colorSpace, of type VideoColorSpaceInit: Configures the VideoFrame.colorSpace for VideoFrames associated with this VideoDecoderConfig. If colorSpace exists, the provided values will override any in-band values from the bitsream.
hardwareAcceleration, of type HardwareAcceleration, defaulting to "no-preference": Hint that configures hardware acceleration for this codec. See HardwareAcceleration.
optimizeForLatency, of type boolean, defaulting to false: Hint that the selected decoder SHOULD be configured to minimize the number of EncodedVideoChunks that have to be decoded before a VideoFrame is output.
NOTE: In addition to User Agent and hardware limitations, some codec bitstreams require a minimum number of inputs before any output can be produced.
rotation, of type double, defaulting to 0: Sets the rotation attribute on decoded frames.
flip, of type boolean, defaulting to false: Sets the flip attribute on decoded frames.

7.7. AudioEncoderConfig

dictionary AudioEncoderConfig {
  required DOMString codec;
  [EnforceRange] required unsigned long sampleRate;
  [EnforceRange] required unsigned long numberOfChannels;
  [EnforceRange] unsigned long long bitrate;
  BitrateMode bitrateMode = "variable";
};

NOTE: Codec-specific extensions to AudioEncoderConfig are described in their registrations in the [WEBCODECS-CODEC-REGISTRY].

To check if an AudioEncoderConfig is a valid AudioEncoderConfig, run these steps:

If codec is empty after stripping leading and trailing ASCII whitespace, return false.
If the AudioEncoderConfig has a codec-specific extension and the corresponding registration in the [WEBCODECS-CODEC-REGISTRY] defines steps to check whether the extension is a valid extension, return the result of running those steps.
If sampleRate or numberOfChannels are equal to zero, return false.
Return true.

codec, of type DOMString: Contains a codec string describing the codec.
sampleRate, of type unsigned long: The number of frame samples per second.
numberOfChannels, of type unsigned long: The number of audio channels.
bitrate, of type unsigned long long: The average bitrate of the encoded audio given in units of bits per second.
bitrateMode, of type BitrateMode, defaulting to "variable": Configures the encoder to use a constant or variable bitrate as defined by [MEDIASTREAM-RECORDING].
NOTE: Not all audio codecs support specific BitrateModes, Authors are encouraged to check by calling isConfigSupported() with config.

7.8. VideoEncoderConfig

dictionary VideoEncoderConfig {
  required DOMString codec;
  [EnforceRange] required unsigned long width;
  [EnforceRange] required unsigned long height;
  [EnforceRange] unsigned long displayWidth;
  [EnforceRange] unsigned long displayHeight;
  [EnforceRange] unsigned long long bitrate;
  double framerate;
  HardwareAcceleration hardwareAcceleration = "no-preference";
  AlphaOption alpha = "discard";
  DOMString scalabilityMode;
  VideoEncoderBitrateMode bitrateMode = "variable";
  LatencyMode latencyMode = "quality";
  DOMString contentHint;
};

NOTE: Codec-specific extensions to VideoEncoderConfig are described in their registrations in the [WEBCODECS-CODEC-REGISTRY].

To check if a VideoEncoderConfig is a valid VideoEncoderConfig, run these steps:

If codec is empty after stripping leading and trailing ASCII whitespace, return false.
If width = 0 or height = 0, return false.
If displayWidth = 0 or displayHeight = 0, return false.
Return true.

codec, of type DOMString: Contains a codec string in config.codec describing the codec.
width, of type unsigned long: The encoded width of output EncodedVideoChunks in pixels, prior to any display aspect ratio adjustments.
The encoder MUST scale any VideoFrame whose [[visible width]] differs from this value.
height, of type unsigned long: The encoded height of output EncodedVideoChunks in pixels, prior to any display aspect ratio adjustments.
The encoder MUST scale any VideoFrame whose [[visible height]] differs from this value.

displayWidth, of type unsigned long: The intended display width of output EncodedVideoChunks in pixels. Defaults to width if not present.
displayHeight, of type unsigned long: The intended display height of output EncodedVideoChunks in pixels. Defaults to width if not present.

NOTE: Providing a displayWidth or displayHeight that differs from width and height signals that chunks are to be scaled after decoding to arrive at the final display aspect ratio.

For many codecs this is merely pass-through information, but some codecs can sometimes include display sizing in the bitstream.

bitrate, of type unsigned long long

The average bitrate of the encoded video given in units of bits per second.

NOTE: Authors are encouraged to additionally provide a framerate to inform rate control.

framerate, of type double

The expected frame rate in frames per second, if known. This value, along with the frame timestamp, SHOULD be used by the video encoder to calculate the optimal byte length for each encoded frame. Additionally, the value SHOULD be considered a target deadline for outputting encoding chunks when latencyMode is set to realtime.

hardwareAcceleration, of type HardwareAcceleration, defaulting to "no-preference"

Hint that configures hardware acceleration for this codec. See HardwareAcceleration.

alpha, of type AlphaOption, defaulting to "discard"

Whether the alpha component of the VideoFrame inputs SHOULD be kept or discarded prior to encoding. If alpha is equal to discard, alpha data is always discarded, regardless of a VideoFrame’s [[format]].

scalabilityMode, of type DOMString

An encoding scalability mode identifier as defined by [WebRTC-SVC].

bitrateMode, of type VideoEncoderBitrateMode, defaulting to "variable"

Configures encoding to use one of the rate control modes specified by VideoEncoderBitrateMode.

NOTE: The precise degree of bitrate fluctuation in either mode is implementation defined.

latencyMode, of type LatencyMode, defaulting to "quality"

Configures latency related behaviors for this codec. See LatencyMode.

contentHint, of type DOMString

An encoding video content hint as defined by [mst-content-hint].

The User Agent MAY use this hint to set expectations about incoming VideoFrames and to improve encoding quality. If using this hint:

The User Agent MUST respect other explicitly set encoding options when configuring the encoder, whether they are codec-specific encoding options or not.
The User Agent SHOULD make a best-effort attempt to use additional configuration options to improve encoding quality, according to the goals defined by the corresponding video content hint.

NOTE: Some encoder options are implementation specific, and mappings between contentHint and those options cannot be prescribed.

The User Agent MUST NOT refuse the configuration if it doesn’t support this content hint. See isConfigSupported().

7.9. Hardware Acceleration

enum HardwareAcceleration {
  "no-preference",
  "prefer-hardware",
  "prefer-software",
};

When supported, hardware acceleration offloads encoding or decoding to specialized hardware. prefer-hardware and prefer-software are hints. While User Agents SHOULD respect these values when possible, User Agents may ignore these values in some or all circumstances for any reason.

To prevent fingerprinting, if a User Agent implements [media-capabilities], the User Agent MUST ensure rejection or acceptance of a given HardwareAcceleration preference reveals no additional information on top of what is inherent to the User Agent and revealed by [media-capabilities]. If a User Agent does not implement [media-capabilities] for reasons of fingerprinting, they SHOULD ignore the HardwareAcceleration preference.

NOTE: Good examples of when a User Agent can ignore prefer-hardware or prefer-software are for reasons of user privacy or circumstances where the User Agent determines an alternative setting would better serve the end user.

Most authors will be best served by using the default of no-preference. This gives the User Agent flexibility to optimize based on its knowledge of the system and configuration. A common strategy will be to prioritize hardware acceleration at higher resolutions with a fallback to software codecs if hardware acceleration fails.

Authors are encouraged to carefully weigh the tradeoffs when setting a hardware acceleration preference. The precise tradeoffs will be device-specific, but authors can generally expect the following:

Setting a value of prefer-hardware or prefer-software can significantly restrict what configurations are supported. It can occur that the user’s device does not offer acceleration for any codec, or only for the most common profiles of older codecs. It can also occur that a given User Agent lacks a software based codec implementation.
Hardware acceleration does not simply imply faster encoding / decoding. Hardware acceleration often has higher startup latency but more consistent throughput performance. Acceleration will generally reduce CPU load.
For decoding, hardware acceleration is often less robust to inputs that are mislabeled or violate the relevant codec specification.
Hardware acceleration will often be more power efficient than purely software based codecs.
For lower resolution content, the overhead added by hardware acceleration can yield decreased performance and power efficiency compared to purely software based codecs.

Given these tradeoffs, a good example of using "prefer-hardware" would be if an author intends to provide their own software based fallback via WebAssembly.

Alternatively, a good example of using "prefer-software" would be if an author is especially sensitive to the higher startup latency or decreased robustness generally associated with hardware acceleration.

no-preference: Indicates that the User Agent MAY use hardware acceleration if it is available and compatible with other aspects of the codec configuration.
prefer-software: Indicates that the User Agent SHOULD prefer a software codec implementation. User Agents may ignore this value for any reason.
NOTE: This can cause the configuration to be unsupported on platforms where an unaccelerated codec is unavailable or is incompatible with other aspects of the codec configuration.
prefer-hardware: Indicates that the User Agent SHOULD prefer hardware acceleration. User Agents may ignore this value for any reason.
NOTE: This can cause the configuration to be unsupported on platforms where an accelerated codec is unavailable or is incompatible with other aspects of the codec configuration.

7.10. Alpha Option

enum AlphaOption {
  "keep",
  "discard",
};

Describes how the user agent SHOULD behave when dealing with alpha channels, for a variety of different operations.

keep: Indicates that the user agent SHOULD preserve alpha channel data for VideoFrames, if it is present.
discard: Indicates that the user agent SHOULD ignore or remove VideoFrame’s alpha channel data.

7.11. Latency Mode

enum LatencyMode {
  "quality",
  "realtime"
};

quality

Indicates that the User Agent SHOULD optimize for encoding quality. In this mode:

User Agents MAY increase encoding latency to improve quality.
User Agents MUST not drop frames to achieve the target bitrate and/or framerate.
framerate SHOULD not be used as a target deadline for emitting encoded chunks.

realtime

Indicates that the User Agent SHOULD optimize for low latency. In this mode:

User Agents MAY sacrifice quality to improve latency.
User Agents MAY drop frames to achieve the target bitrate and/or framerate.
framerate SHOULD be used as a target deadline for emitting encoded chunks.

7.12. Configuration Equivalence

Two dictionaries are equal dictionaries if they contain the same keys and values. For nested dictionaries, apply this definition recursively.

7.13. VideoEncoderEncodeOptions

dictionary VideoEncoderEncodeOptions {
  boolean keyFrame = false;
};

NOTE: Codec-specific extensions to VideoEncoderEncodeOptions are described in their registrations in the [WEBCODECS-CODEC-REGISTRY].

keyFrame, of type boolean, defaulting to false: A value of true indicates that the given frame MUST be encoded as a key frame. A value of false indicates that the User Agent has flexibility to decide whether the frame will be encoded as a key frame.

7.14. VideoEncoderBitrateMode

enum VideoEncoderBitrateMode {
  "constant",
  "variable",
  "quantizer"
};

constant: Encode at a constant bitrate. See bitrate.
variable: Encode using a variable bitrate, allowing more space to be used for complex signals and less space for less complex signals. See bitrate.
quantizer: Encode using a quantizer, that is specified for each video frame in codec specific extensions of VideoEncoderEncodeOptions.

7.15. CodecState

enum CodecState {
  "unconfigured",
  "configured",
  "closed"
};

unconfigured: The codec is not configured for encoding or decoding.
configured: A valid configuration has been provided. The codec is ready for encoding or decoding.
closed: The codec is no longer usable and underlying system resources have been released.

7.16. WebCodecsErrorCallback

callback WebCodecsErrorCallback = undefined(DOMException error);

8. Encoded Media Interfaces (Chunks)

These interfaces represent chunks of encoded media.

8.1. EncodedAudioChunk Interface

[Exposed=(Window,DedicatedWorker), Serializable]
interface EncodedAudioChunk {
  constructor(EncodedAudioChunkInit init);
  readonly attribute EncodedAudioChunkType type;
  readonly attribute long long timestamp;          // microseconds
  readonly attribute unsigned long long? duration; // microseconds
  readonly attribute unsigned long byteLength;

  undefined copyTo(AllowSharedBufferSource destination);
};

dictionary EncodedAudioChunkInit {
  required EncodedAudioChunkType type;
  [EnforceRange] required long long timestamp;    // microseconds
  [EnforceRange] unsigned long long duration;     // microseconds
  required AllowSharedBufferSource data;
  sequence<ArrayBuffer> transfer = [];
};

enum EncodedAudioChunkType {
    "key",
    "delta",
};

8.1.1. Internal Slots

[[internal data]]: An array of bytes representing the encoded chunk data.
[[type]]: Describes whether the chunk is a key chunk.
[[timestamp]]: The presentation timestamp, given in microseconds.
[[duration]]: The presentation duration, given in microseconds.
[[byte length]]: The byte length of [[internal data]].

8.1.2. Constructors


  EncodedAudioChunk(init)

If init.transfer contains more than one reference to the same ArrayBuffer, then throw a DataCloneError DOMException.
For each transferable in init.transfer:
1. If [[Detached]] internal slot is true, then throw a DataCloneError DOMException.
Let chunk be a new EncodedAudioChunk object, initialized as follows
1. Assign init.type to [[type]].
2. Assign init.timestamp to [[timestamp]].
3. If init.duration exists, assign it to [[duration]], or assign null otherwise.
4. Assign init.data.byteLength to [[byte length]];
5. If init.transfer contains an ArrayBuffer referenced by init.data the User Agent MAY choose to:
  1. Let resource be a new media resource referencing sample data in init.data.
6. Otherwise:
  1. Assign a copy of init.data to [[internal data]].
For each transferable in init.transfer:
1. Perform DetachArrayBuffer on transferable
Return chunk.

8.1.3. Attributes

type, of type EncodedAudioChunkType, readonly: Returns the value of [[type]].
timestamp, of type long long, readonly: Returns the value of [[timestamp]].
duration, of type unsigned long long, readonly, nullable: Returns the value of [[duration]].
byteLength, of type unsigned long, readonly: Returns the value of [[byte length]].

8.1.4. Methods

copyTo(destination)

When invoked, run these steps:

If the [[byte length]] of this EncodedAudioChunk is greater than in destination, throw a TypeError.
Copy the [[internal data]] into destination.

8.1.5. Serialization

The EncodedAudioChunk serialization steps (with value, serialized, and forStorage) are:

If forStorage is true, throw a DataCloneError.
For each EncodedAudioChunk internal slot in value, assign the value of each internal slot to a field in serialized with the same name as the internal slot.

The EncodedAudioChunk deserialization steps (with serialized and value) are:

For all named fields in serialized, assign the value of each named field to the EncodedAudioChunk internal slot in value with the same name as the named field.

NOTE: Since EncodedAudioChunks are immutable, User Agents can choose to implement serialization using a reference counting model similar to § 9.2.6 Transfer and Serialization.

8.2. EncodedVideoChunk Interface

[Exposed=(Window,DedicatedWorker), Serializable]
interface EncodedVideoChunk {
  constructor(EncodedVideoChunkInit init);
  readonly attribute EncodedVideoChunkType type;
  readonly attribute long long timestamp;             // microseconds
  readonly attribute unsigned long long? duration;    // microseconds
  readonly attribute unsigned long byteLength;

  undefined copyTo(AllowSharedBufferSource destination);
};

dictionary EncodedVideoChunkInit {
  required EncodedVideoChunkType type;
  [EnforceRange] required long long timestamp;        // microseconds
  [EnforceRange] unsigned long long duration;         // microseconds
  required AllowSharedBufferSource data;
  sequence<ArrayBuffer> transfer = [];
};

enum EncodedVideoChunkType {
    "key",
    "delta",
};

8.2.1. Internal Slots

[[internal data]]: An array of bytes representing the encoded chunk data.
[[type]]: The EncodedVideoChunkType of this EncodedVideoChunk;
[[timestamp]]: The presentation timestamp, given in microseconds.
[[duration]]: The presentation duration, given in microseconds.
[[byte length]]: The byte length of [[internal data]].

8.2.2. Constructors


  EncodedVideoChunk(init)

If init.transfer contains more than one reference to the same ArrayBuffer, then throw a DataCloneError DOMException.
For each transferable in init.transfer:
1. If [[Detached]] internal slot is true, then throw a DataCloneError DOMException.
Let chunk be a new EncodedVideoChunk object, initialized as follows
1. Assign init.type to [[type]].
2. Assign init.timestamp to [[timestamp]].
3. If duration is present in init, assign init.duration to [[duration]]. Otherwise, assign null to [[duration]].
4. Assign init.data.byteLength to [[byte length]];
5. If init.transfer contains an ArrayBuffer referenced by init.data the User Agent MAY choose to:
  1. Let resource be a new media resource referencing sample data in init.data.
6. Otherwise:
  1. Assign a copy of init.data to [[internal data]].
For each transferable in init.transfer:
1. Perform DetachArrayBuffer on transferable
Return chunk.

8.2.3. Attributes

type, of type EncodedVideoChunkType, readonly: Returns the value of [[type]].
timestamp, of type long long, readonly: Returns the value of [[timestamp]].
duration, of type unsigned long long, readonly, nullable: Returns the value of [[duration]].
byteLength, of type unsigned long, readonly: Returns the value of [[byte length]].

8.2.4. Methods

copyTo(destination)

When invoked, run these steps:

If [[byte length]] is greater than the [[byte length]] of destination, throw a TypeError.
Copy the [[internal data]] into destination.

8.2.5. Serialization

The EncodedVideoChunk serialization steps (with value, serialized, and forStorage) are:

If forStorage is true, throw a DataCloneError.
For each EncodedVideoChunk internal slot in value, assign the value of each internal slot to a field in serialized with the same name as the internal slot.

The EncodedVideoChunk deserialization steps (with serialized and value) are:

For all named fields in serialized, assign the value of each named field to the EncodedVideoChunk internal slot in value with the same name as the named field.

NOTE: Since EncodedVideoChunks are immutable, User Agents can choose to implement serialization using a reference counting model similar to § 9.4.7 Transfer and Serialization.

9. Raw Media Interfaces

These interfaces represent unencoded (raw) media.

9.1. Memory Model

9.1.1. Background

This section is non-normative.

Decoded media data MAY occupy a large amount of system memory. To minimize the need for expensive copies, this specification defines a scheme for reference counting (clone() and close()).

NOTE: Authors are encouraged to call close() immediately when frames are no longer needed.

9.1.2. Reference Counting

A media resource is storage for the actual pixel data or the audio sample data described by a VideoFrame or AudioData.

The AudioData [[resource reference]] and VideoFrame [[resource reference]] internal slots hold a reference to a media resource.

VideoFrame.clone() and AudioData.clone() return new objects whose [[resource reference]] points to the same media resource as the original object.

VideoFrame.close() and AudioData.close() will clear their [[resource reference]] slot, releasing the reference their media resource.

A media resource MUST remain alive at least as long as it continues to be referenced by a [[resource reference]].

NOTE: When a media resource is no longer referenced by a [[resource reference]], the resource can be destroyed. User Agents are encouraged to destroy such resources quickly to reduce memory pressure and facilitate resource reuse.

9.1.3. Transfer and Serialization

This section is non-normative.

AudioData and VideoFrame are both transferable and serializable objects. Their transfer and serialization steps are defined in § 9.2.6 Transfer and Serialization and § 9.4.7 Transfer and Serialization respectively.

Transferring an AudioData or VideoFrame moves its [[resource reference]] to the destination object and closes (as in close()) the source object. Authors MAY use this facility to move an AudioData or VideoFrame between realms without copying the underlying media resource.

Serializing an AudioData or VideoFrame effectively clones (as in clone()) the source object, resulting in two objects that reference the same media resource. Authors MAY use this facility to clone an AudioData or VideoFrame to another realm without copying the underlying media resource.

9.2. AudioData Interface

[Exposed=(Window,DedicatedWorker), Serializable, Transferable]
interface AudioData {
  constructor(AudioDataInit init);

  readonly attribute AudioSampleFormat? format;
  readonly attribute float sampleRate;
  readonly attribute unsigned long numberOfFrames;
  readonly attribute unsigned long numberOfChannels;
  readonly attribute unsigned long long duration;  // microseconds
  readonly attribute long long timestamp;          // microseconds

  unsigned long allocationSize(AudioDataCopyToOptions options);
  undefined copyTo(AllowSharedBufferSource destination, AudioDataCopyToOptions options);
  AudioData clone();
  undefined close();
};

dictionary AudioDataInit {
  required AudioSampleFormat format;
  required float sampleRate;
  [EnforceRange] required unsigned long numberOfFrames;
  [EnforceRange] required unsigned long numberOfChannels;
  [EnforceRange] required long long timestamp;  // microseconds
  required AllowSharedBufferSource data;
  sequence<ArrayBuffer> transfer = [];
};

9.2.1. Internal Slots

[[resource reference]]: A reference to a media resource that stores the audio sample data for this AudioData.
[[format]]: The AudioSampleFormat used by this AudioData. Will be null whenever the underlying format does not map to an AudioSampleFormat or when [[Detached]] is true.
[[sample rate]]: The sample-rate, in Hz, for this AudioData.
[[number of frames]]: The number of frames for this AudioData.
[[number of channels]]: The number of audio channels for this AudioData.
[[timestamp]]: The presentation timestamp, in microseconds, for this AudioData.

9.2.2. Constructors


  AudioData(init)

If init is not a valid AudioDataInit, throw a TypeError.
If init.transfer contains more than one reference to the same ArrayBuffer, then throw a DataCloneError DOMException.
For each transferable in init.transfer:
1. If [[Detached]] internal slot is true, then throw a DataCloneError DOMException.
Let frame be a new AudioData object, initialized as follows:
1. Assign false to [[Detached]].
2. Assign init.format to [[format]].
3. Assign init.sampleRate to [[sample rate]].
4. Assign init.numberOfFrames to [[number of frames]].
5. Assign init.numberOfChannels to [[number of channels]].
6. Assign init.timestamp to [[timestamp]].
7. If init.transfer contains an ArrayBuffer referenced by init.data the User Agent MAY choose to:
  1. Let resource be a new media resource referencing sample data in data.
8. Otherwise:
  1. Let resource be a media resource containing a copy of init.data.
9. Let resourceReference be a reference to resource.
10. Assign resourceReference to [[resource reference]].
For each transferable in init.transfer:
1. Perform DetachArrayBuffer on transferable
Return frame.

9.2.3. Attributes

format, of type AudioSampleFormat, readonly, nullable

The AudioSampleFormat used by this AudioData. Will be null whenever the underlying format does not map to a AudioSampleFormat or when [[Detached]] is true.

The format getter steps are to return [[format]].

sampleRate, of type float, readonly

The sample-rate, in Hz, for this AudioData.

The sampleRate getter steps are to return [[sample rate]].

numberOfFrames, of type unsigned long, readonly

The number of frames for this AudioData.

The numberOfFrames getter steps are to return [[number of frames]].

numberOfChannels, of type unsigned long, readonly

The number of audio channels for this AudioData.

The numberOfChannels getter steps are to return [[number of channels]].

timestamp, of type long long, readonly

The presentation timestamp, in microseconds, for this AudioData.

The numberOfChannels getter steps are to return [[timestamp]].

duration, of type unsigned long long, readonly

The duration, in microseconds, for this AudioData.

The duration getter steps are to:

Let microsecondsPerSecond be 1,000,000.
Let durationInSeconds be the result of dividing [[number of frames]] by [[sample rate]].
Return the product of durationInSeconds and microsecondsPerSecond.

9.2.4. Methods

allocationSize(options)

Returns the number of bytes required to hold the samples as described by options.

When invoked, run these steps:

If [[Detached]] is true, throw an InvalidStateError DOMException.
Let copyElementCount be the result of running the Compute Copy Element Count algorithm with options.
Let destFormat be the value of [[format]].
If options.format exists, assign options.format to destFormat.
Let bytesPerSample be the number of bytes per sample, as defined by the destFormat.
Return the product of multiplying bytesPerSample by copyElementCount.

copyTo(destination, options)

Copies the samples from the specified plane of the AudioData to the destination buffer.

When invoked, run these steps:

If [[Detached]] is true, throw an InvalidStateError DOMException.
Let copyElementCount be the result of running the Compute Copy Element Count algorithm with options.
Let destFormat be the value of [[format]].
If options.format exists, assign options.format to destFormat.
Let bytesPerSample be the number of bytes per sample, as defined by the destFormat.
If the product of multiplying bytesPerSample by copyElementCount is greater than destination.byteLength, throw a RangeError.
Let resource be the media resource referenced by [[resource reference]].
Let planeFrames be the region of resource corresponding to options.planeIndex.
Copy elements of planeFrames into destination, starting with the frame positioned at options.frameOffset and stopping after copyElementCount samples have been copied. If destFormat does not equal [[format]], convert elements to the destFormat AudioSampleFormat while making the copy.

clone()

Creates a new AudioData with a reference to the same media resource.

When invoked, run these steps:

If [[Detached]] is true, throw an InvalidStateError DOMException.
Return the result of running the Clone AudioData algorithm with this.

close()

Clears all state and releases the reference to the media resource. Close is final.

When invoked, run the Close AudioData algorithm with this.

9.2.5. Algorithms

Compute Copy Element Count (with options)

Run these steps:

Let destFormat be the value of [[format]].
If options.format exists, assign options.format to destFormat.
If destFormat describes an interleaved AudioSampleFormat and options.planeIndex is greater than 0, throw a RangeError.
Otherwise, if destFormat describes a planar AudioSampleFormat and if options.planeIndex is greater or equal to [[number of channels]], throw a RangeError.
If [[format]] does not equal destFormat and the User Agent does not support the requested AudioSampleFormat conversion, throw a NotSupportedError DOMException. Conversion to f32-planar MUST always be supported.
Let frameCount be the number of frames in the plane identified by options.planeIndex.
If options.frameOffset is greater than or equal to frameCount, throw a RangeError.
Let copyFrameCount be the difference of subtracting options.frameOffset from frameCount.
If options.frameCount exists:
1. If options.frameCount is greater than copyFrameCount, throw a RangeError.
2. Otherwise, assign options.frameCount to copyFrameCount.
Let elementCount be copyFrameCount.
If destFormat describes an interleaved AudioSampleFormat, multiply elementCount by [[number of channels]]
return elementCount.

Clone AudioData (with data)

Run these steps:

Let clone be a new AudioData initialized as follows:
1. Let resource be the media resource referenced by data’s [[resource reference]].
2. Let reference be a new reference to resource.
3. Assign reference to [[resource reference]].
4. Assign the values of data’s [[Detached]], [[format]], [[sample rate]], [[number of frames]], [[number of channels]], and [[timestamp]] slots to the corresponding slots in clone.
Return clone.

Close AudioData (with data)

Run these steps:

Assign true to data’s [[Detached]] internal slot.
Assign null to data’s [[resource reference]].
Assign 0 to data’s [[sample rate]].
Assign 0 to data’s [[number of frames]].
Assign 0 to data’s [[number of channels]].
Assign null to data’s [[format]].

To check if a AudioDataInit is a valid AudioDataInit, run these steps:

If sampleRate less than or equal to 0, return false.
If numberOfFrames = 0, return false.
If numberOfChannels = 0, return false.
Verify data has enough data by running the following steps:
1. Let totalSamples be the product of multiplying numberOfFrames by numberOfChannels.
2. Let bytesPerSample be the number of bytes per sample, as defined by the format.
3. Let totalSize be the product of multiplying bytesPerSample with totalSamples.
4. Let dataSize be the size in bytes of data.
5. If dataSize is less than totalSize, return false.
Return true.

Note: It’s expected that AudioDataInit’s data’s memory layout matches the expectations of the planar or interleaved format. There is no real way to verify whether the samples conform to their AudioSampleFormat.

9.2.6. Transfer and Serialization

The AudioData transfer steps (with value and dataHolder) are:

If value’s [[Detached]] is true, throw a DataCloneError DOMException.
For all AudioData internal slots in value, assign the value of each internal slot to a field in dataHolder with the same name as the internal slot.
Run the Close AudioData algorithm with value.

The AudioData transfer-receiving steps (with dataHolder and value) are:

For all named fields in dataHolder, assign the value of each named field to the AudioData internal slot in value with the same name as the named field.

The AudioData serialization steps (with value, serialized, and forStorage) are:

If value’s [[Detached]] is true, throw a DataCloneError DOMException.
If forStorage is true, throw a DataCloneError.
Let resource be the media resource referenced by value’s [[resource reference]].
Let newReference be a new reference to resource.
Assign newReference to |serialized.resource reference|.
For all remaining AudioData internal slots (excluding [[resource reference]]) in value, assign the value of each internal slot to a field in serialized with the same name as the internal slot.

The AudioData deserialization steps (with serialized and value) are:

For all named fields in serialized, assign the value of each named field to the AudioData internal slot in value with the same name as the named field.

9.2.7. AudioDataCopyToOptions

dictionary AudioDataCopyToOptions {
  [EnforceRange] required unsigned long planeIndex;
  [EnforceRange] unsigned long frameOffset = 0;
  [EnforceRange] unsigned long frameCount;
  AudioSampleFormat format;
};

planeIndex, of type unsigned long

The index identifying the plane to copy from.

frameOffset, of type unsigned long, defaulting to 0

An offset into the source plane data indicating which frame to begin copying from. Defaults to 0.

frameCount, of type unsigned long

The number of frames to copy. If not provided, the copy will include all frames in the plane beginning with frameOffset.

format, of type AudioSampleFormat

The output AudioSampleFormat for the destination data. If not provided, the resulting copy will use this AudioData’s [[format]]. Invoking copyTo() will throw a NotSupportedError if conversion to the requested format is not supported. Conversion from any AudioSampleFormat to f32-planar MUST always be supported.

NOTE: Authors seeking to integrate with [WEBAUDIO] can request f32-planar and use the resulting copy to create and AudioBuffer or render via AudioWorklet.

9.3. Audio Sample Format

An audio sample format describes the numeric type used to represent a single sample (e.g. 32-bit floating point) and the arrangement of samples from different channels as either interleaved or planar. The audio sample type refers solely to the numeric type and interval used to store the data, this is u8, s16, s32, or f32 for respectively unsigned 8-bits, signed 16-bits, signed 32-bits, and 32-bits floating point number. The audio buffer arrangement refers solely to the way the samples are laid out in memory (planar or interleaved).

A sample refers to a single value that is the magnitude of a signal at a particular point in time in a particular channel.

A frame or (sample-frame) refers to a set of values of all channels of a multi-channel signal, that happen at the exact same time.

NOTE: Consequently, if an audio signal is mono (has only one channel), a frame and a sample refer to the same thing.

All audio samples in this specification are using linear pulse-code modulation (Linear PCM): quantization levels are uniform between values.

NOTE: The Web Audio API, that is expected to be used with this specification, also uses Linear PCM.

enum AudioSampleFormat {
  "u8",
  "s16",
  "s32",
  "f32",
  "u8-planar",
  "s16-planar",
  "s32-planar",
  "f32-planar",
};

u8: 8-bit unsigned integer samples with interleaved channel arrangement.
s16: 16-bit signed integer samples with interleaved channel arrangement.
s32: 32-bit signed integer samples with interleaved channel arrangement.
f32: 32-bit float samples with interleaved channel arrangement.
u8-planar: 8-bit unsigned integer samples with planar channel arrangement.
s16-planar: 16-bit signed integer samples with planar channel arrangement.
s32-planar: 32-bit signed integer samples with planar channel arrangement.
f32-planar: 32-bit float samples with planar channel arrangement.

9.3.1. Arrangement of audio buffer

When an AudioData has an AudioSampleFormat that is interleaved, the audio samples from different channels are laid out consecutively in the same buffer, in the order described in the section § 9.3.3 Audio channel ordering. The AudioData has a single plane, that contains a number of elements therefore equal to [[number of frames]] * [[number of channels]].

When an AudioData has an AudioSampleFormat that is planar, the audio samples from different channels are laid out in different buffers, themselves arranged in an order described in the section § 9.3.3 Audio channel ordering. The AudioData has a number of planes equal to the AudioData’s [[number of channels]]. Each plane contains [[number of frames]] elements.

NOTE: The Web Audio API currently uses f32-planar exclusively.

NOTE: The following diagram exemplifies the memory layout of planar versus interleaved AudioSampleFormats

Graphical representation the memory layout of interleaved and planar
formats

9.3.2. Magnitude of the audio samples

The minimum value and maximum value of an audio sample, for a particular audio sample type, are the values below which (respectively above which) audio clipping might occur. They are otherwise regular types, that can hold values outside this interval during intermediate processing.

The bias value for an audio sample type is the value that often corresponds to the middle of the range (but often the range is not symmetrical). An audio buffer comprised only of values equal to the bias value is silent.

Sample type	IDL type	Minimum value	Bias value	Maximum value
`u8`	octet	0	128	+255
`s16`	short	-32768	0	+32767
`s32`	long	-2147483648	0	+2147483647
`f32`	float	-1.0	0.0	+1.0

NOTE: There is no data type that can hold 24 bits of information conveniently, but audio content using 24-bit samples is common, so 32-bits integers are commonly used to hold 24-bit content.

AudioData containing 24-bit samples SHOULD store those samples in s32 or f32. When samples are stored in s32, each sample MUST be left-shifted by 8 bits. By virtue of this process, samples outside of the valid 24-bit range ([-8388608, +8388607]) will be clipped. To avoid clipping and ensure lossless transport, samples MAY be converted to f32.

NOTE: While clipping is unavoidable in u8, s16, and s32 samples due to their storage types, implementations SHOULD take care not to clip internally when handling f32 samples.

9.3.3. Audio channel ordering

When decoding, the ordering of the audio channels in the resulting AudioData MUST be the same as what is present in the EncodedAudioChunk.

When encoding, the ordering of the audio channels in the resulting EncodedAudioChunk MUST be the same as what is preset in the given AudioData.

In other terms, no channel reordering is performed when encoding and decoding.

NOTE: The container either implies or specifies the channel mapping: the channel attributed to a particular channel index.

9.4. VideoFrame Interface

NOTE: VideoFrame is a CanvasImageSource. A VideoFrame can be passed to any method accepting a CanvasImageSource, including CanvasDrawImage’s drawImage().

[Exposed=(Window,DedicatedWorker), Serializable, Transferable]
interface VideoFrame {
  constructor(CanvasImageSource image, optional VideoFrameInit init = {});
  constructor(AllowSharedBufferSource data, VideoFrameBufferInit init);

  readonly attribute VideoPixelFormat? format;
  readonly attribute unsigned long codedWidth;
  readonly attribute unsigned long codedHeight;
  readonly attribute DOMRectReadOnly? codedRect;
  readonly attribute DOMRectReadOnly? visibleRect;
  readonly attribute double rotation;
  readonly attribute boolean flip;
  readonly attribute unsigned long displayWidth;
  readonly attribute unsigned long displayHeight;
  readonly attribute unsigned long long? duration;  // microseconds
  readonly attribute long long timestamp;           // microseconds
  readonly attribute VideoColorSpace colorSpace;

  VideoFrameMetadata metadata();

  unsigned long allocationSize(
      optional VideoFrameCopyToOptions options = {});
  Promise<sequence<PlaneLayout>> copyTo(
      AllowSharedBufferSource destination,
      optional VideoFrameCopyToOptions options = {});
  VideoFrame clone();
  undefined close();
};

dictionary VideoFrameInit {
  unsigned long long duration;  // microseconds
  long long timestamp;          // microseconds
  AlphaOption alpha = "keep";

  // Default matches image. May be used to efficiently crop. Will trigger
  // new computation of displayWidth and displayHeight using image's pixel
  // aspect ratio unless an explicit displayWidth and displayHeight are given.
  DOMRectInit visibleRect;

  double rotation = 0;
  boolean flip = false;

  // Default matches image unless visibleRect is provided.
  [EnforceRange] unsigned long displayWidth;
  [EnforceRange] unsigned long displayHeight;

  VideoFrameMetadata metadata;
};

dictionary VideoFrameBufferInit {
  required VideoPixelFormat format;
  required [EnforceRange] unsigned long codedWidth;
  required [EnforceRange] unsigned long codedHeight;
  required [EnforceRange] long long timestamp;  // microseconds
  [EnforceRange] unsigned long long duration;  // microseconds

  // Default layout is tightly-packed.
  sequence<PlaneLayout> layout;

  // Default visible rect is coded size positioned at (0,0)
  DOMRectInit visibleRect;

  double rotation = 0;
  boolean flip = false;

  // Default display dimensions match visibleRect.
  [

WebCodecs

Abstract

Status of this document

1. Definitions

2. Codec Processing Model

2.1. Background

2.2. Control Messages

2.3. Codec Work Parallel Queue

3. AudioDecoder Interface

3.1. Internal Slots

3.2. Constructors

3.3. Attributes

3.4. Event Summary

3.5. Methods

3.6. Algorithms

4. VideoDecoder Interface

4.1. Internal Slots

4.2. Constructors

4.3. Attributes

4.4. Event Summary

4.5. Methods

4.6. Algorithms

5. AudioEncoder Interface

5.1. Internal Slots

5.2. Constructors

5.3. Attributes

5.4. Event Summary

5.5. Methods

5.6. Algorithms

5.7. EncodedAudioChunkMetadata

6. VideoEncoder Interface

6.1. Internal Slots

6.2. Constructors

6.3. Attributes

6.4. Event Summary

6.5. Methods

6.6. Algorithms

6.7. EncodedVideoChunkMetadata

7. Configurations

7.1. Check Configuration Support (with config)

7.2. Clone Configuration (with config)

7.3. Signalling Configuration Support

7.3.1. AudioDecoderSupport

7.3.2. VideoDecoderSupport

7.3.3. AudioEncoderSupport

7.3.4. VideoEncoderSupport

7.4. Codec String

7.5. AudioDecoderConfig

7.6. VideoDecoderConfig

7.7. AudioEncoderConfig

7.8. VideoEncoderConfig

7.9. Hardware Acceleration

7.10. Alpha Option

7.11. Latency Mode

7.12. Configuration Equivalence

7.13. VideoEncoderEncodeOptions

7.14. VideoEncoderBitrateMode

7.15. CodecState

7.16. WebCodecsErrorCallback

8. Encoded Media Interfaces (Chunks)

8.1. EncodedAudioChunk Interface

8.1.1. Internal Slots

8.1.2. Constructors

8.1.3. Attributes

8.1.4. Methods

8.1.5. Serialization

8.2. EncodedVideoChunk Interface

8.2.1. Internal Slots

8.2.2. Constructors

8.2.3. Attributes

8.2.4. Methods

8.2.5. Serialization

9. Raw Media Interfaces

9.1. Memory Model

9.1.1. Background

9.1.2. Reference Counting

9.1.3. Transfer and Serialization

9.2. AudioData Interface

9.2.1. Internal Slots

9.2.2. Constructors

7.1. Check Configuration Support (with `config`)

7.2. Clone Configuration (with `config`)