RFC 6716: Definition of the Opus Audio Codec
- JM. Valin,
- K. Vos,
- T. Terriberry
Proposed Standard
1 subscribed
This RFC was updated, see
Internet Engineering Task Force (IETF) JM. Valin
Request for Comments: 6716 Mozilla Corporation
Category: Standards Track K. Vos
ISSN: 2070-1721 Skype Technologies S.A.
T. Terriberry
Mozilla Corporation
September 2012
Definition of the Opus Audio Codec
Abstract
This document defines the Opus interactive speech and audio codec.
Opus is designed to handle a wide range of interactive audio
applications, including Voice over IP, videoconferencing, in-game
chat, and even live, distributed music performances. It scales from
low bitrate narrowband speech at 6 kbit/s to very high quality stereo
music at 510 kbit/s. Opus uses both Linear Prediction (LP) and the
Modified Discrete Cosine Transform (MDCT) to achieve good compression
of both speech and music.
Status of This Memo
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Further information on
Internet Standards is available in Section 2 of RFC 5741.
Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc6716.
Valin, et al. Standards Track [Page 1]
RFC 6716 Interactive Audio Codec September 2012 Copyright Notice Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. The licenses granted by the IETF Trust to this RFC under Section 3.c of the Trust Legal Provisions shall also include the right to extract text from Sections 1 through 8 and Appendix A and Appendix B of this RFC and create derivative works from these extracts, and to copy, publish, display and distribute such derivative works in any medium and for any purpose, provided that no such derivative work shall be presented, displayed or published in a manner that states or implies that it is part of this RFC or any other IETF Document. Table of Contents 1. Introduction ....................................................5 1.1. Notation and Conventions ...................................6 2. Opus Codec Overview .............................................8 2.1. Control Parameters ........................................10 2.1.1. Bitrate ............................................10 2.1.2. Number of Channels (Mono/Stereo) ...................11 2.1.3. Audio Bandwidth ....................................11 2.1.4. Frame Duration .....................................11 2.1.5. Complexity .........................................11 2.1.6. Packet Loss Resilience .............................12 2.1.7. Forward Error Correction (FEC) .....................12 2.1.8. Constant/Variable Bitrate ..........................12 2.1.9. Discontinuous Transmission (DTX) ...................13 3. Internal Framing ...............................................13 3.1. The TOC Byte ..............................................13 3.2. Frame Packing .............................................16 3.2.1. Frame Length Coding ................................16 3.2.2. Code 0: One Frame in the Packet ....................16 3.2.3. Code 1: Two Frames in the Packet, Each with Equal Compressed Size ..............................17 3.2.4. Code 2: Two Frames in the Packet, with Different Compressed Sizes .........................17 Valin, et al. Standards Track [Page 2]
RFC 6716 Interactive Audio Codec September 2012 3.2.5. Code 3: A Signaled Number of Frames in the Packet ..18 3.3. Examples ..................................................21 3.4. Receiving Malformed Packets ...............................22 4. Opus Decoder ...................................................23 4.1. Range Decoder .............................................23 4.1.1. Range Decoder Initialization .......................25 4.1.2. Decoding Symbols ...................................25 4.1.3. Alternate Decoding Methods .........................27 4.1.4. Decoding Raw Bits ..................................29 4.1.5. Decoding Uniformly Distributed Integers ............29 4.1.6. Current Bit Usage ..................................30 4.2. SILK Decoder ..............................................32 4.2.1. SILK Decoder Modules ...............................32 4.2.2. LP Layer Organization ..............................33 4.2.3. Header Bits ........................................35 4.2.4. Per-Frame LBRR Flags ...............................36 4.2.5. LBRR Frames ........................................36 4.2.6. Regular SILK Frames ................................37 4.2.7. SILK Frame Contents ................................37 4.2.7.1. Stereo Prediction Weights .................40 4.2.7.2. Mid-Only Flag .............................42 4.2.7.3. Frame Type ................................43 4.2.7.4. Subframe Gains ............................44 4.2.7.5. Normalized Line Spectral Frequency (LSF) and Linear Predictive Coding (LPC) Coeffieients ..............................46 4.2.7.6. Long-Term Prediction (LTP) Parameters .....74 4.2.7.7. Linear Congruential Generator (LCG) Seed ..86 4.2.7.8. Excitation ................................86 4.2.7.9. SILK Frame Reconstruction .................98 4.2.8. Stereo Unmixing ...................................102 4.2.9. Resampling ........................................103 4.3. CELT Decoder .............................................104 4.3.1. Transient Decoding ................................108 4.3.2. Energy Envelope Decoding ..........................108 4.3.3. Bit Allocation ....................................110 4.3.4. Shape Decoding ....................................116 4.3.5. Anti-collapse Processing ..........................120 4.3.6. Denormalization ...................................121 4.3.7. Inverse MDCT ......................................121 4.4. Packet Loss Concealment (PLC) ............................122 4.4.1. Clock Drift Compensation ..........................122 4.5. Configuration Switching ..................................123 4.5.1. Transition Side Information (Redundancy) ..........124 4.5.2. State Reset .......................................127 4.5.3. Summary of Transitions ............................128 5. Opus Encoder ..................................................131 5.1. Range Encoder ............................................132 Valin, et al. Standards Track [Page 3]
RFC 6716 Interactive Audio Codec September 2012 5.1.1. Encoding Symbols ..................................133 5.1.2. Alternate Encoding Methods ........................134 5.1.3. Encoding Raw Bits .................................135 5.1.4. Encoding Uniformly Distributed Integers ...........135 5.1.5. Finalizing the Stream .............................135 5.1.6. Current Bit Usage .................................136 5.2. SILK Encoder .............................................136 5.2.1. Sample Rate Conversion ............................137 5.2.2. Stereo Mixing .....................................137 5.2.3. SILK Core Encoder .................................138 5.3. CELT Encoder .............................................150 5.3.1. Pitch Pre-filter ..................................150 5.3.2. Bands and Normalization ...........................151 5.3.3. Energy Envelope Quantization ......................151 5.3.4. Bit Allocation ....................................151 5.3.5. Stereo Decisions ..................................152 5.3.6. Time-Frequency Decision ...........................153 5.3.7. Spreading Values Decision .........................153 5.3.8. Spherical Vector Quantization .....................154 6. Conformance ...................................................155 6.1. Testing ..................................................155 6.2. Opus Custom ..............................................156 7. Security Considerations .......................................157 8. Acknowledgements ..............................................158 9. References ....................................................159 9.1. Normative References .....................................159 9.2. Informative References ...................................159 Appendix A. Reference Implementation .............................163 A.1. Extracting the Source ....................................164 A.2. Up-to-Date Implementation ................................164 A.3. Base64-Encoded Source Code ...............................164 A.4. Test Vectors .............................................321 Appendix B. Self-Delimiting Framing ..............................321 Valin, et al. Standards Track [Page 4]
RFC 6716 Interactive Audio Codec September 2012 1. Introduction The Opus codec is a real-time interactive audio codec designed to meet the requirements described in [REQUIREMENTS]. It is composed of a layer based on Linear Prediction (LP) [LPC] and a layer based on the Modified Discrete Cosine Transform (MDCT) [MDCT]. The main idea behind using two layers is as follows: in speech, linear prediction techniques (such as Code-Excited Linear Prediction, or CELP) code low frequencies more efficiently than transform (e.g., MDCT) domain techniques, while the situation is reversed for music and higher speech frequencies. Thus, a codec with both layers available can operate over a wider range than either one alone and can achieve better quality by combining them than by using either one individually. The primary normative part of this specification is provided by the source code in Appendix A. Only the decoder portion of this software is normative, though a significant amount of code is shared by both the encoder and decoder. Section 6 provides a decoder conformance test. The decoder contains a great deal of integer and fixed-point arithmetic that needs to be performed exactly, including all rounding considerations, so any useful specification requires domain-specific symbolic language to adequately define these operations. Additionally, any conflict between the symbolic representation and the included reference implementation must be resolved. For the practical reasons of compatibility and testability, it would be advantageous to give the reference implementation priority in any disagreement. The C language is also one of the most widely understood, human-readable symbolic representations for machine behavior. For these reasons, this RFC uses the reference implementation as the sole symbolic representation of the codec. While the symbolic representation is unambiguous and complete, it is not always the easiest way to understand the codec's operation. For this reason, this document also describes significant parts of the codec in prose and takes the opportunity to explain the rationale behind many of the more surprising elements of the design. These descriptions are intended to be accurate and informative, but the limitations of common English sometimes result in ambiguity, so it is expected that the reader will always read them alongside the symbolic representation. Numerous references to the implementation are provided for this purpose. The descriptions sometimes differ from the reference in ordering or through mathematical simplification wherever such deviation makes an explanation easier to understand. For example, the right shift and left shift operations in the reference implementation are often described using division and Valin, et al. Standards Track [Page 5]
RFC 6716 Interactive Audio Codec September 2012 multiplication in the text. In general, the text is focused on the "what" and "why" while the symbolic representation most clearly provides the "how". 1.1. Notation and Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Various operations in the codec require bit-exact fixed-point behavior, even when writing a floating point implementation. The notation "Q<n>", where n is an integer, denotes the number of binary digits to the right of the decimal point in a fixed-point number. For example, a signed Q14 value in a 16-bit word can represent values from -2.0 to 1.99993896484375, inclusive. This notation is for informational purposes only. Arithmetic, when described, always operates on the underlying integer. For example, the text will explicitly indicate any shifts required after a multiplication. Expressions, where included in the text, follow C operator rules and precedence, with the exception that the syntax "x**y" indicates x raised to the power y. The text also makes use of the following functions. 1.1.1. min(x,y) The smallest of two values x and y. 1.1.2. max(x,y) The largest of two values x and y. 1.1.3. clamp(lo,x,hi) clamp(lo,x,hi) = max(lo,min(x,hi)) With this definition, if lo > hi, then lo is returned. 1.1.4. sign(x) The sign of x, i.e., ( -1, x < 0 sign(x) = < 0, x == 0 ( 1, x > 0 Valin, et al. Standards Track [Page 6]
RFC 6716 Interactive Audio Codec September 2012 1.1.5. abs(x) The absolute value of x, i.e., abs(x) = sign(x)*x 1.1.6. floor(f) The largest integer z such that z <= f. 1.1.7. ceil(f) The smallest integer z such that z >= f. 1.1.8. round(f) The integer z nearest to f, with ties rounded towards negative infinity, i.e., round(f) = ceil(f - 0.5) 1.1.9. log2(f) The base-two logarithm of f. 1.1.10. ilog(n) The minimum number of bits required to store a positive integer n in binary, or 0 for a non-positive integer n. ( 0, n <= 0 ilog(n) = < ( floor(log2(n))+1, n > 0 Examples: o ilog(-1) = 0 o ilog(0) = 0 o ilog(1) = 1 o ilog(2) = 2 o ilog(3) = 2 o ilog(4) = 3 Valin, et al. Standards Track [Page 7]
RFC 6716 Interactive Audio Codec September 2012 o ilog(7) = 3 2. Opus Codec Overview The Opus codec scales from 6 kbit/s narrowband mono speech to 510 kbit/s fullband stereo music, with algorithmic delays ranging from 5 ms to 65.2 ms. At any given time, either the LP layer, the MDCT layer, or both, may be active. It can seamlessly switch between all of its various operating modes, giving it a great deal of flexibility to adapt to varying content and network conditions without renegotiating the current session. The codec allows input and output of various audio bandwidths, defined as follows: +----------------------+-----------------+-------------------------+ | Abbreviation | Audio Bandwidth | Sample Rate (Effective) | +----------------------+-----------------+-------------------------+ | NB (narrowband) | 4 kHz | 8 kHz | | | | | | MB (medium-band) | 6 kHz | 12 kHz | | | | | | WB (wideband) | 8 kHz | 16 kHz | | | | | | SWB (super-wideband) | 12 kHz | 24 kHz | | | | | | FB (fullband) | 20 kHz (*) | 48 kHz | +----------------------+-----------------+-------------------------+ Table 1 (*) Although the sampling theorem allows a bandwidth as large as half the sampling rate, Opus never codes audio above 20 kHz, as that is the generally accepted upper limit of human hearing. Opus defines super-wideband (SWB) with an effective sample rate of 24 kHz, unlike some other audio coding standards that use 32 kHz. This was chosen for a number of reasons. The band layout in the MDCT layer naturally allows skipping coefficients for frequencies over 12 kHz, but does not allow cleanly dropping just those frequencies over 16 kHz. A sample rate of 24 kHz also makes resampling in the MDCT layer easier, as 24 evenly divides 48, and when 24 kHz is sufficient, it can save computation in other processing, such as Acoustic Echo Cancellation (AEC). Experimental changes to the band layout to allow a 16 kHz cutoff (32 kHz effective sample rate) showed potential quality degradations at other sample rates, and, at typical bitrates, the number of bits saved by using such a cutoff instead of coding in fullband (FB) mode is very small. Therefore, if an application wishes to process a signal sampled at 32 kHz, it should just use FB. Valin, et al. Standards Track [Page 8]
RFC 6716 Interactive Audio Codec September 2012 The LP layer is based on the SILK codec [SILK]. It supports NB, MB, or WB audio and frame sizes from 10 ms to 60 ms, and requires an additional 5 ms look-ahead for noise shaping estimation. A small additional delay (up to 1.5 ms) may be required for sampling rate conversion. Like Vorbis [VORBIS-WEBSITE] and many other modern codecs, SILK is inherently designed for variable bitrate (VBR) coding, though the encoder can also produce constant bitrate (CBR) streams. The version of SILK used in Opus is substantially modified from, and not compatible with, the stand-alone SILK codec previously deployed by Skype. This document does not serve to define that format, but those interested in the original SILK codec should see [SILK] instead. The MDCT layer is based on the Constrained-Energy Lapped Transform (CELT) codec [CELT]. It supports NB, WB, SWB, or FB audio and frame sizes from 2.5 ms to 20 ms, and requires an additional 2.5 ms look- ahead due to the overlapping MDCT windows. The CELT codec is inherently designed for CBR coding, but unlike many CBR codecs, it is not limited to a set of predetermined rates. It internally allocates bits to exactly fill any given target budget, and an encoder can produce a VBR stream by varying the target on a per-frame basis. The MDCT layer is not used for speech when the audio bandwidth is WB or less, as it is not useful there. On the other hand, non-speech signals are not always adequately coded using linear prediction. Therefore, the MDCT layer should be used for music signals. A "Hybrid" mode allows the use of both layers simultaneously with a frame size of 10 or 20 ms and an SWB or FB audio bandwidth. The LP layer codes the low frequencies by resampling the signal down to WB. The MDCT layer follows, coding the high frequency portion of the signal. The cutoff between the two lies at 8 kHz, the maximum WB audio bandwidth. In the MDCT layer, all bands below 8 kHz are discarded, so there is no coding redundancy between the two layers. The sample rate (in contrast to the actual audio bandwidth) can be chosen independently on the encoder and decoder side, e.g., a fullband signal can be decoded as wideband, or vice versa. This approach ensures a sender and receiver can always interoperate, regardless of the capabilities of their actual audio hardware. Internally, the LP layer always operates at a sample rate of twice the audio bandwidth, up to a maximum of 16 kHz, which it continues to use for SWB and FB. The decoder simply resamples its output to support different sample rates. The MDCT layer always operates internally at a sample rate of 48 kHz. Since all the supported sample rates evenly divide this rate, and since the decoder may easily zero out the high frequency portion of the spectrum in the frequency domain, it can simply decimate the MDCT layer output to achieve the other supported sample rates very cheaply. Valin, et al. Standards Track [Page 9]
RFC 6716 Interactive Audio Codec September 2012 After conversion to the common, desired output sample rate, the decoder simply adds the output from the two layers together. To compensate for the different look-ahead required by each layer, the CELT encoder input is delayed by an additional 2.7 ms. This ensures that low frequencies and high frequencies arrive at the same time. This extra delay may be reduced by an encoder by using less look- ahead for noise shaping or using a simpler resampler in the LP layer, but this will reduce quality. However, the base 2.5 ms look-ahead in the CELT layer cannot be reduced in the encoder because it is needed for the MDCT overlap, whose size is fixed by the decoder. Both layers use the same entropy coder, avoiding any waste from "padding bits" between them. The hybrid approach makes it easy to support both CBR and VBR coding. Although the LP layer is VBR, the bit allocation of the MDCT layer can produce a final stream that is CBR by using all the bits left unused by the LP layer. 2.1. Control Parameters The Opus codec includes a number of control parameters that can be changed dynamically during regular operation of the codec, without interrupting the audio stream from the encoder to the decoder. These parameters only affect the encoder since any impact they have on the bitstream is signaled in-band such that a decoder can decode any Opus stream without any out-of-band signaling. Any Opus implementation can add or modify these control parameters without affecting interoperability. The most important encoder control parameters in the reference encoder are listed below. 2.1.1. Bitrate Opus supports all bitrates from 6 kbit/s to 510 kbit/s. All other parameters being equal, higher bitrate results in higher quality. For a frame size of 20 ms, these are the bitrate "sweet spots" for Opus in various configurations: o 8-12 kbit/s for NB speech, o 16-20 kbit/s for WB speech, o 28-40 kbit/s for FB speech, o 48-64 kbit/s for FB mono music, and o 64-128 kbit/s for FB stereo music. Valin, et al. Standards Track [Page 10]
RFC 6716 Interactive Audio Codec September 2012 2.1.2. Number of Channels (Mono/Stereo) Opus can transmit either mono or stereo frames within a single stream. When decoding a mono frame in a stereo decoder, the left and right channels are identical, and when decoding a stereo frame in a mono decoder, the mono output is the average of the left and right channels. In some cases, it is desirable to encode a stereo input stream in mono (e.g., because the bitrate is too low to encode stereo with sufficient quality). The number of channels encoded can be selected in real-time, but by default the reference encoder attempts to make the best decision possible given the current bitrate. 2.1.3. Audio Bandwidth The audio bandwidths supported by Opus are listed in Table 1. Just like for the number of channels, any decoder can decode audio that is encoded at any bandwidth. For example, any Opus decoder operating at 8 kHz can decode an FB Opus frame, and any Opus decoder operating at 48 kHz can decode an NB frame. Similarly, the reference encoder can take a 48 kHz input signal and encode it as NB. The higher the audio bandwidth, the higher the required bitrate to achieve acceptable quality. The audio bandwidth can be explicitly specified in real- time, but, by default, the reference encoder attempts to make the best bandwidth decision possible given the current bitrate. 2.1.4. Frame Duration Opus can encode frames of 2.5, 5, 10, 20, 40, or 60 ms. It can also combine multiple frames into packets of up to 120 ms. For real-time applications, sending fewer packets per second reduces the bitrate, since it reduces the overhead from IP, UDP, and RTP headers. However, it increases latency and sensitivity to packet losses, as losing one packet constitutes a loss of a bigger chunk of audio. Increasing the frame duration also slightly improves coding efficiency, but the gain becomes small for frame sizes above 20 ms. For this reason, 20 ms frames are a good choice for most applications. 2.1.5. Complexity There are various aspects of the Opus encoding process where trade- offs can be made between CPU complexity and quality/bitrate. In the reference encoder, the complexity is selected using an integer from 0 to 10, where 0 is the lowest complexity and 10 is the highest. Examples of computations for which such trade-offs may occur are: o The order of the pitch analysis whitening filter [WHITENING], Valin, et al. Standards Track [Page 11]
RFC 6716 Interactive Audio Codec September 2012 o The order of the short-term noise shaping filter, o The number of states in delayed decision quantization of the residual signal, and o The use of certain bitstream features such as variable time- frequency resolution and the pitch post-filter. 2.1.6. Packet Loss Resilience Audio codecs often exploit inter-frame correlations to reduce the bitrate at a cost in error propagation: after losing one packet, several packets need to be received before the decoder is able to accurately reconstruct the speech signal. The extent to which Opus exploits inter-frame dependencies can be adjusted on the fly to choose a trade-off between bitrate and amount of error propagation. 2.1.7. Forward Error Correction (FEC) Another mechanism providing robustness against packet loss is the in- band Forward Error Correction (FEC). Packets that are determined to contain perceptually important speech information, such as onsets or transients, are encoded again at a lower bitrate and this re-encoded information is added to a subsequent packet. 2.1.8. Constant/Variable Bitrate Opus is more efficient when operating with variable bitrate (VBR), which is the default. When low-latency transmission is required over a relatively slow connection, then constrained VBR can also be used. This uses VBR in a way that simulates a "bit reservoir" and is equivalent to what MP3 (MPEG 1, Layer 3) and AAC (Advanced Audio Coding) call CBR (i.e., not true CBR due to the bit reservoir). In some (rare) applications, constant bitrate (CBR) is required. There are two main reasons to operate in CBR mode: o When the transport only supports a fixed size for each compressed frame, or o When encryption is used for an audio stream that is either highly constrained (e.g., yes/no, recorded prompts) or highly sensitive [SRTP-VBR]. Bitrate may still be allowed to vary, even with sensitive data, as long as the variation is not driven by the input signal (for example, to match changing network conditions). To achieve this, an application should still run Opus in CBR mode, but change the target rate before each packet. Valin, et al. Standards Track [Page 12]
RFC 6716 Interactive Audio Codec September 2012 2.1.9. Discontinuous Transmission (DTX) Discontinuous Transmission (DTX) reduces the bitrate during silence or background noise. When DTX is enabled, only one frame is encoded every 400 milliseconds. 3. Internal Framing The Opus encoder produces "packets", which are each a contiguous set of bytes meant to be transmitted as a single unit. The packets described here do not include such things as IP, UDP, or RTP headers, which are normally found in a transport-layer packet. A single packet may contain multiple audio frames, so long as they share a common set of parameters, including the operating mode, audio bandwidth, frame size, and channel count (mono vs. stereo). This section describes the possible combinations of these parameters and the internal framing used to pack multiple frames into a single packet. This framing is not self-delimiting. Instead, it assumes that a lower layer (such as UDP or RTP [RFC3550] or Ogg [RFC3533] or Matroska [MATROSKA-WEBSITE]) will communicate the length, in bytes, of the packet, and it uses this information to reduce the framing overhead in the packet itself. A decoder implementation MUST support the framing described in this section. An alternative, self- delimiting variant of the framing is described in Appendix B. Support for that variant is OPTIONAL. All bit diagrams in this document number the bits so that bit 0 is the most significant bit of the first byte, and bit 7 is the least significant. Bit 8 is thus the most significant bit of the second byte, etc. Well-formed Opus packets obey certain requirements, marked [R1] through [R7] below. These are summarized in Section 3.4 along with appropriate means of handling malformed packets. 3.1. The TOC Byte A well-formed Opus packet MUST contain at least one byte [R1]. This byte forms a table-of-contents (TOC) header that signals which of the various modes and configurations a given packet uses. It is composed of a configuration number, "config", a stereo flag, "s", and a frame count code, "c", arranged as illustrated in Figure 1. A description of each of these fields follows. Valin, et al. Standards Track [Page 13]
RFC 6716 Interactive Audio Codec September 2012 0 0 1 2 3 4 5 6 7 +-+-+-+-+-+-+-+-+ | config |s| c | +-+-+-+-+-+-+-+-+ Figure 1: The TOC Byte The top five bits of the TOC byte, labeled "config", encode one of 32 possible configurations of operating mode, audio bandwidth, and frame size. As described, the LP (SILK) layer and MDCT (CELT) layer can be combined in three possible operating modes: 1. A SILK-only mode for use in low bitrate connections with an audio bandwidth of WB or less, 2. A Hybrid (SILK+CELT) mode for SWB or FB speech at medium bitrates, and 3. A CELT-only mode for very low delay speech transmission as well as music transmission (NB to FB). The 32 possible configurations each identify which one of these operating modes the packet uses, as well as the audio bandwidth and the frame size. Table 2 lists the parameters for each configuration. Valin, et al. Standards Track [Page 14]
RFC 6716 Interactive Audio Codec September 2012 +-----------------------+-----------+-----------+-------------------+ | Configuration | Mode | Bandwidth | Frame Sizes | | Number(s) | | | | +-----------------------+-----------+-----------+-------------------+ | 0...3 | SILK-only | NB | 10, 20, 40, 60 ms | | | | | | | 4...7 | SILK-only | MB | 10, 20, 40, 60 ms | | | | | | | 8...11 | SILK-only | WB | 10, 20, 40, 60 ms | | | | | | | 12...13 | Hybrid | SWB | 10, 20 ms | | | | | | | 14...15 | Hybrid | FB | 10, 20 ms | | | | | | | 16...19 | CELT-only | NB | 2.5, 5, 10, 20 ms | | | | | | | 20...23 | CELT-only | WB | 2.5, 5, 10, 20 ms | | | | | | | 24...27 | CELT-only | SWB | 2.5, 5, 10, 20 ms | | | | | | | 28...31 | CELT-only | FB | 2.5, 5, 10, 20 ms | +-----------------------+-----------+-----------+-------------------+ Table 2: TOC Byte Configuration Parameters The configuration numbers in each range (e.g., 0...3 for NB SILK- only) correspond to the various choices of frame size, in the same order. For example, configuration 0 has a 10 ms frame size and configuration 3 has a 60 ms frame size. One additional bit, labeled "s", signals mono vs. stereo, with 0 indicating mono and 1 indicating stereo. The remaining two bits of the TOC byte, labeled "c", code the number of frames per packet (codes 0 to 3) as follows: o 0: 1 frame in the packet o 1: 2 frames in the packet, each with equal compressed size o 2: 2 frames in the packet, with different compressed sizes o 3: an arbitrary number of frames in the packet This document refers to a packet as a code 0 packet, code 1 packet, etc., based on the value of "c". Valin, et al. Standards Track [Page 15]
RFC 6716 Interactive Audio Codec September 2012 3.2. Frame Packing This section describes how frames are packed according to each possible value of "c" in the TOC byte. 3.2.1. Frame Length Coding When a packet contains multiple VBR frames (i.e., code 2 or 3), the compressed length of one or more of these frames is indicated with a one- or two-byte sequence, with the meaning of the first byte as follows: o 0: No frame (Discontinuous Transmission (DTX) or lost packet) o 1...251: Length of the frame in bytes o 252...255: A second byte is needed. The total length is (second_byte*4)+first_byte The special length 0 indicates that no frame is available, either because it was dropped during transmission by some intermediary or because the encoder chose not to transmit it. Any Opus frame in any mode MAY have a length of 0. The maximum representable length is 255*4+255=1275 bytes. For 20 ms frames, this represents a bitrate of 510 kbit/s, which is approximately the highest useful rate for lossily compressed fullband stereo music. Beyond this point, lossless codecs are more appropriate. It is also roughly the maximum useful rate of the MDCT layer as, shortly thereafter, quality no longer improves with additional bits due to limitations on the codebook sizes. No length is transmitted for the last frame in a VBR packet, or for any of the frames in a CBR packet, as it can be inferred from the total size of the packet and the size of all other data in the packet. However, the length of any individual frame MUST NOT exceed 1275 bytes [R2] to allow for repacketization by gateways, conference bridges, or other software. 3.2.2. Code 0: One Frame in the Packet For code 0 packets, the TOC byte is immediately followed by N-1 bytes of compressed data for a single frame (where N is the size of the packet), as illustrated in Figure 2. Valin, et al. Standards Track [Page 16]
RFC 6716 Interactive Audio Codec September 2012 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | config |s|0|0| | +-+-+-+-+-+-+-+-+ | | Compressed frame 1 (N-1 bytes)... : : | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 2: A Code 0 Packet 3.2.3. Code 1: Two Frames in the Packet, Each with Equal Compressed Size For code 1 packets, the TOC byte is immediately followed by the (N-1)/2 bytes of compressed data for the first frame, followed by (N-1)/2 bytes of compressed data for the second frame, as illustrated in Figure 3. The number of payload bytes available for compressed data, N-1, MUST be even for all code 1 packets [R3]. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | config |s|0|1| | +-+-+-+-+-+-+-+-+ : | Compressed frame 1 ((N-1)/2 bytes)... | : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : | Compressed frame 2 ((N-1)/2 bytes)... | : +-+-+-+-+-+-+-+-+ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 3: A Code 1 Packet 3.2.4. Code 2: Two Frames in the Packet, with Different Compressed Sizes For code 2 packets, the TOC byte is followed by a one- or two-byte sequence indicating the length of the first frame (marked N1 in Figure 4), followed by N1 bytes of compressed data for the first frame. The remaining N-N1-2 or N-N1-3 bytes are the compressed data for the second frame. This is illustrated in Figure 4. A code 2 packet MUST contain enough bytes to represent a valid length. For example, a 1-byte code 2 packet is always invalid, and a 2-byte code 2 packet whose second byte is in the range 252...255 is also invalid. Valin, et al. Standards Track [Page 17]
RFC 6716 Interactive Audio Codec September 2012 The length of the first frame, N1, MUST also be no larger than the size of the payload remaining after decoding that length for all code 2 packets [R4]. This makes, for example, a 2-byte code 2 packet with a second byte in the range 1...251 invalid as well (the only valid 2-byte code 2 packet is one where the length of both frames is zero). 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | config |s|1|0| N1 (1-2 bytes): | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : | Compressed frame 1 (N1 bytes)... | : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | Compressed frame 2... : : | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4: A Code 2 Packet 3.2.5. Code 3: A Signaled Number of Frames in the Packet Code 3 packets signal the number of frames, as well as additional padding, called "Opus padding" to indicate that this padding is added at the Opus layer rather than at the transport layer. Code 3 packets MUST have at least 2 bytes [R6,R7]. The TOC byte is followed by a byte encoding the number of frames in the packet in bits 2 to 7 (marked "M" in Figure 5), with bit 1 indicating whether or not Opus padding is inserted (marked "p" in Figure 5), and bit 0 indicating VBR (marked "v" in Figure 5). M MUST NOT be zero, and the audio duration contained within a packet MUST NOT exceed 120 ms [R5]. This limits the maximum frame count for any frame size to 48 (for 2.5 ms frames), with lower limits for longer frame sizes. Figure 5 illustrates the layout of the frame count byte. 0 0 1 2 3 4 5 6 7 +-+-+-+-+-+-+-+-+ |v|p| M | +-+-+-+-+-+-+-+-+ Figure 5: The frame count byte When Opus padding is used, the number of bytes of padding is encoded in the bytes following the frame count byte. Values from 0...254 indicate that 0...254 bytes of padding are included, in addition to Valin, et al. Standards Track [Page 18]
RFC 6716 Interactive Audio Codec September 2012 the byte(s) used to indicate the size of the padding. If the value is 255, then the size of the additional padding is 254 bytes, plus the padding value encoded in the next byte. There MUST be at least one more byte in the packet in this case [R6,R7]. The additional padding bytes appear at the end of the packet and MUST be set to zero by the encoder to avoid creating a covert channel. The decoder MUST accept any value for the padding bytes, however. Although this encoding provides multiple ways to indicate a given number of padding bytes, each uses a different number of bytes to indicate the padding size and thus will increase the total packet size by a different amount. For example, to add 255 bytes to a packet, set the padding bit, p, to 1, insert a single byte after the frame count byte with a value of 254, and append 254 padding bytes with the value zero to the end of the packet. To add 256 bytes to a packet, set the padding bit to 1, insert two bytes after the frame count byte with the values 255 and 0, respectively, and append 254 padding bytes with the value zero to the end of the packet. By using the value 255 multiple times, it is possible to create a packet of any specific, desired size. Let P be the number of header bytes used to indicate the padding size plus the number of padding bytes themselves (i.e., P is the total number of bytes added to the packet). Then, P MUST be no more than N-2 [R6,R7]. In the CBR case, let R=N-2-P be the number of bytes remaining in the packet after subtracting the (optional) padding. Then, the compressed length of each frame in bytes is equal to R/M. The value R MUST be a non-negative integer multiple of M [R6]. The compressed data for all M frames follows, each of size R/M bytes, as illustrated in Figure 6. Valin, et al. Standards Track [Page 19]
RFC 6716 Interactive Audio Codec September 2012 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | config |s|1|1|0|p| M | Padding length (Optional) : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | : Compressed frame 1 (R/M bytes)... : | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | : Compressed frame 2 (R/M bytes)... : | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | : ... : | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | : Compressed frame M (R/M bytes)... : | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : Opus Padding (Optional)... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 6: A CBR Code 3 Packet In the VBR case, the (optional) padding length is followed by M-1 frame lengths (indicated by "N1" to "N[M-1]" in Figure 7), each encoded in a one- or two-byte sequence as described above. The packet MUST contain enough data for the M-1 lengths after removing the (optional) padding, and the sum of these lengths MUST be no larger than the number of bytes remaining in the packet after decoding them [R7]. The compressed data for all M frames follows, each frame consisting of the indicated number of bytes, with the final frame consuming any remaining bytes before the final padding, as illustrated in Figure 6. The number of header bytes (TOC byte, frame count byte, padding length bytes, and frame length bytes), plus the signaled length of the first M-1 frames themselves, plus the signaled length of the padding MUST be no larger than N, the total size of the packet. Valin, et al. Standards Track [Page 20]
RFC 6716 Interactive Audio Codec September 2012 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | config |s|1|1|1|p| M | Padding length (Optional) : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : N1 (1-2 bytes): N2 (1-2 bytes): ... : N[M-1] | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | : Compressed frame 1 (N1 bytes)... : | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | : Compressed frame 2 (N2 bytes)... : | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | : ... : | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | : Compressed frame M... : | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : Opus Padding (Optional)... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 7: A VBR Code 3 Packet 3.3. Examples Simplest case, one NB mono 20 ms SILK frame: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1 |0|0|0| compressed data... : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 8 Valin, et al. Standards Track [Page 21]
RFC 6716 Interactive Audio Codec September 2012 Two FB mono 5 ms CELT frames of the same compressed size: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 29 |0|0|1| compressed data... : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 9 Two FB mono 20 ms Hybrid frames of different compressed size: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 15 |0|1|1|1|0| 2 | N1 | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | compressed data... : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 10 Four FB stereo 20 ms CELT frames of the same compressed size: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 31 |1|1|1|0|0| 4 | compressed data... : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 11 3.4. Receiving Malformed Packets A receiver MUST NOT process packets that violate any of the rules above as normal Opus packets. They are reserved for future applications, such as in-band headers (containing metadata, etc.). Packets that violate these constraints may cause implementations of _this_ specification to treat them as malformed and discard them. These constraints are summarized here for reference: [R1] Packets are at least one byte. [R2] No implicit frame length is larger than 1275 bytes. [R3] Code 1 packets have an odd total length, N, so that (N-1)/2 is an integer. Valin, et al. Standards Track [Page 22]
RFC 6716 Interactive Audio Codec September 2012 [R4] Code 2 packets have enough bytes after the TOC for a valid frame length, and that length is no larger than the number of bytes remaining in the packet. [R5] Code 3 packets contain at least one frame, but no more than 120 ms of audio total. [R6] The length of a CBR code 3 packet, N, is at least two bytes, the number of bytes added to indicate the padding size plus the trailing padding bytes themselves, P, is no more than N-2, and the frame count, M, satisfies the constraint that (N-2-P) is a non-negative integer multiple of M. [R7] VBR code 3 packets are large enough to contain all the header bytes (TOC byte, frame count byte, any padding length bytes, and any frame length bytes), plus the length of the first M-1 frames, plus any trailing padding bytes. 4. Opus Decoder The Opus decoder consists of two main blocks: the SILK decoder and the CELT decoder. At any given time, one or both of the SILK and CELT decoders may be active. The output of the Opus decode is the sum of the outputs from the SILK and CELT decoders with proper sample rate conversion and delay compensation on the SILK side, and optional decimation (when decoding to sample rates less than 48 kHz) on the CELT side, as illustrated in the block diagram below. +---------+ +------------+ | SILK | | Sample | +->| Decoder |--->| Rate |----+ Bit- +---------+ | | | | Conversion | v stream | Range |---+ +---------+ +------------+ /---\ Audio ------->| Decoder | | + |------> | |---+ +---------+ +------------+ \---/ +---------+ | | CELT | | Decimation | ^ +->| Decoder |--->| (Optional) |----+ | | | | +---------+ +------------+ 4.1. Range Decoder Opus uses an entropy coder based on range coding [RANGE-CODING] [MARTIN79], which is itself a rediscovery of the FIFO arithmetic code introduced by [CODING-THESIS]. It is very similar to arithmetic encoding, except that encoding is done with digits in any base Valin, et al. Standards Track [Page 23]
RFC 6716 Interactive Audio Codec September 2012 instead of with bits, so it is faster when using larger bases (i.e., a byte). All of the calculations in the range coder must use bit- exact integer arithmetic. Symbols may also be coded as "raw bits" packed directly into the bitstream, bypassing the range coder. These are packed backwards starting at the end of the frame, as illustrated in Figure 12. This reduces complexity and makes the stream more resilient to bit errors, as corruption in the raw bits will not desynchronize the decoding process, unlike corruption in the input to the range decoder. Raw bits are only used in the CELT layer. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Range coder data (packed MSB to LSB) -> : + + : : + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : | <- Boundary occurs at an arbitrary bit position : +-+-+-+ + : <- Raw bits data (packed LSB to MSB) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Legend: LSB = Least Significant Bit MSB = Most Significant Bit Figure 12: Illustrative Example of Packing Range Coder and Raw Bits Data Each symbol coded by the range coder is drawn from a finite alphabet and coded in a separate "context", which describes the size of the alphabet and the relative frequency of each symbol in that alphabet. Suppose there is a context with n symbols, identified with an index that ranges from 0 to n-1. The parameters needed to encode or decode symbol k in this context are represented by a three-tuple (fl[k], fh[k], ft), all 16-bit unsigned integers, with 0 <= fl[k] < fh[k] <= ft <= 65535. The values of this tuple are derived from the probability model for the symbol, represented by traditional "frequency counts". Because Opus uses static contexts, those are not updated as symbols are decoded. Let f[i] be the frequency of symbol i. Then, the three-tuple corresponding to symbol k is given by the following: Valin, et al. Standards Track [Page 24]
RFC 6716 Interactive Audio Codec September 2012 k-1 n-1 __ __ fl[k] = \ f[i], fh[k] = fl[k] + f[k], ft = \ f[i] /_ /_ i=0 i=0 The range decoder extracts the symbols and integers encoded using the range encoder in Section 5.1. The range decoder maintains an internal state vector composed of the two-tuple (val, rng), where val represents the difference between the high end of the current range and the actual coded value, minus one, and rng represents the size of the current range. Both val and rng are 32-bit unsigned integer values. 4.1.1. Range Decoder Initialization Let b0 be an 8-bit unsigned integer containing first input byte (or containing zero if there are no bytes in this Opus frame). The decoder initializes rng to 128 and initializes val to (127 - (b0>>1)), where (b0>>1) is the top 7 bits of the first input byte. It saves the remaining bit, (b0&1), for use in the renormalization procedure described in Section 4.1.2.1, which the decoder invokes immediately after initialization to read additional bits and establish the invariant that rng > 2**23. 4.1.2. Decoding Symbols Decoding a symbol is a two-step process. The first step determines a 16-bit unsigned value fs, which lies within the range of some symbol in the current context. The second step updates the range decoder state with the three-tuple (fl[k], fh[k], ft) corresponding to that symbol. The first step is implemented by ec_decode() (entdec.c), which computes val fs = ft - min(------ + 1, ft) rng/ft The divisions here are integer division. The decoder then identifies the symbol in the current context corresponding to fs; i.e., the value of k whose three-tuple (fl[k], fh[k], ft) satisfies fl[k] <= fs < fh[k]. It uses this tuple to update val according to Valin, et al. Standards Track [Page 25]
RFC 6716 Interactive Audio Codec September 2012 rng val = val - --- * (ft - fh[k]) ft If fl[k] is greater than zero, then the decoder updates rng using rng rng = --- * (fh[k] - fl[k]) ft Otherwise, it updates rng using rng rng = rng - --- * (ft - fh[k]) ft Using a special case for the first symbol (rather than the last symbol, as is commonly done in other arithmetic coders) ensures that all the truncation error from the finite precision arithmetic accumulates in symbol 0. This makes the cost of coding a 0 slightly smaller, on average, than its estimated probability indicates and makes the cost of coding any other symbol slightly larger. When contexts are designed so that 0 is the most probable symbol, which is often the case, this strategy minimizes the inefficiency introduced by the finite precision. It also makes some of the special-case decoding routines in Section 4.1.3 particularly simple. After the updates, implemented by ec_dec_update() (entdec.c), the decoder normalizes the range using the procedure in the next section, and returns the index k. 4.1.2.1. Renormalization To normalize the range, the decoder repeats the following process, implemented by ec_dec_normalize() (entdec.c), until rng > 2**23. If rng is already greater than 2**23, the entire process is skipped. First, it sets rng to (rng<<8). Then, it reads the next byte of the Opus frame and forms an 8-bit value sym, using the leftover bit buffered from the previous byte as the high bit and the top 7 bits of the byte just read as the other 7 bits of sym. The remaining bit in the byte just read is buffered for use in the next iteration. If no more input bytes remain, it uses zero bits instead. See Section 4.1.1 for the initialization used to process the first byte. Then, it sets val = ((val<<8) + (255-sym)) & 0x7FFFFFFF Valin, et al. Standards Track [Page 26]
RFC 6716 Interactive Audio Codec September 2012 It is normal and expected that the range decoder will read several bytes into the data of the raw bits (if any) at the end of the frame by the time the frame is completely decoded, as illustrated in Figure 13. This same data MUST also be returned as raw bits when requested. The encoder is expected to terminate the stream in such a way that the range decoder will decode the intended values regardless of the data contained in the raw bits. Section 5.1.5 describes a procedure for doing this. If the range decoder consumes all of the bytes belonging to the current frame, it MUST continue to use zero when any further input bytes are required, even if there is additional data in the current packet from padding or other frames. n n+1 n+2 n+3 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : | <----------- Overlap region ------------> | : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ^ ^ | End of data buffered by the range coder | ...-----------------------------------------------+ | | End of data consumed by raw bits +-------------------------------------------------------... Figure 13: Illustrative Example of Raw Bits Overlapping Range Coder Data 4.1.3. Alternate Decoding Methods The reference implementation uses three additional decoding methods that are exactly equivalent to the above but make assumptions and simplifications that allow for a more efficient implementation. 4.1.3.1. ec_decode_bin() The first is ec_decode_bin() (entdec.c), defined using the parameter ftb instead of ft. It is mathematically equivalent to calling ec_decode() with ft = (1<<ftb), but it avoids one of the divisions. 4.1.3.2. ec_dec_bit_logp() The next is ec_dec_bit_logp() (entdec.c), which decodes a single binary symbol, replacing both the ec_decode() and ec_dec_update() steps. The context is described by a single parameter, logp, which is the absolute value of the base-2 logarithm of the probability of a "1". It is mathematically equivalent to calling ec_decode() with ft = (1<<logp), followed by ec_dec_update() with the 3-tuple (fl[k] = 0, fh[k] = (1<<logp) - 1, ft = (1<<logp)) if the returned Valin, et al. Standards Track [Page 27]
RFC 6716 Interactive Audio Codec September 2012 value of fs is less than (1<<logp) - 1 (a "0" was decoded), and with (fl[k] = (1<<logp) - 1, fh[k] = ft = (1<<logp)) otherwise (a "1" was decoded). The implementation requires no multiplications or divisions. 4.1.3.3. ec_dec_icdf() The last is ec_dec_icdf() (entdec.c), which decodes a single symbol with a table-based context of up to 8 bits, also replacing both the ec_decode() and ec_dec_update() steps, as well as the search for the decoded symbol in between. The context is described by two parameters, an icdf ("inverse" cumulative distribution function) table and ftb. As with ec_decode_bin(), (1<<ftb) is equivalent to ft. idcf[k], on the other hand, stores (1<<ftb)-fh[k], which is equal to (1<<ftb) - fl[k+1]. fl[0] is assumed to be 0, and the table is terminated by a value of 0 (where fh[k] == ft). The function is mathematically equivalent to calling ec_decode() with ft = (1<<ftb), using the returned value fs to search the table for the first entry where fs < (1<<ftb)-icdf[k], and calling ec_dec_update() with fl[k] = (1<<ftb) - icdf[k-1] (or 0 if k == 0), fh[k] = (1<<ftb) - idcf[k], and ft = (1<<ftb). Combining the search with the update allows the division to be replaced by a series of multiplications (which are usually much cheaper), and using an inverse CDF allows the use of an ftb as large as 8 in an 8-bit table without any special cases. This is the primary interface with the range decoder in the SILK layer, though it is used in a few places in the CELT layer as well. Although icdf[k] is more convenient for the code, the frequency counts, f[k], are a more natural representation of the probability distribution function (PDF) for a given symbol. Therefore, this document lists the latter, not the former, when describing the context in which a symbol is coded as a list, e.g., {4, 4, 4, 4}/16 for a uniform context with four possible values and ft = 16. The value of ft after the slash is always the sum of the entries in the PDF, but is included for convenience. Contexts with identical probabilities, f[k]/ft, but different values of ft (or equivalently, ftb) are not the same, and cannot, in general, be used in place of one another. An icdf table is also not capable of representing a PDF where the first symbol has 0 probability. In such contexts, ec_dec_icdf() can decode the symbol by using a table that drops the entries for any initial zero-probability values and by adding the constant offset of the first value with a non-zero probability to its return value. Valin, et al. Standards Track [Page 28]