§ 5.2  Module 5 — The Codec Wars and the Science of Compression

Opus: The Peace Treaty

5.2.1Two codecs in one bitstream

Opus is not a compromise between its parents; it is both of them, running as layers of a single codec. The SILK layer is a linear-predictive (LPC) speech coder: it fits a short filter to the vocal tract, a long-term predictor to the pitch, and transmits the filter coefficients plus a cheaply coded excitation signal. That model is devastatingly efficient for the one signal it assumes — a human voice — and it operates at three internal bandwidths: narrowband (4 kHz audio bandwidth, 8 kHz sampling), mediumband (6 kHz), and wideband (8 kHz). The CELT layer — Constrained-Energy Lapped Transform — is an MDCT transform coder like AAC or Vorbis, but engineered around one constraint those codecs never faced: the transform blocks must be short enough for conversation. CELT preserves the energy envelope of each frequency band exactly (that is the “constrained energy”), which keeps quality graceful as bitrate falls, and it runs on a 48 kHz clock with frames as short as 2.5 ms.

Between the pure modes sits the invention that ended the war: hybrid mode. For superwideband (12 kHz) and fullband (20 kHz) speech, SILK codes everything below 8 kHz and CELT codes everything above it, both layers sharing one range coder and one packet. Voice keeps the LPC efficiency where the vocal energy lives; sibilance and room tone keep the transform sparkle above. The encoder chooses the mode — SILK-only for low-rate speech, hybrid for high-quality speech, CELT-only for music and for latency-critical uses — and may switch seamlessly mid-stream as the signal or the target bitrate changes. A music-on-hold segment can flip the stream from hybrid to CELT and back without renegotiating anything: the decision is announced per packet, not per session.

The bandwidth ladder tops out at fullband: 20 kHz of audio bandwidth carried at 48 kHz sampling. Opus always presents itself to RTP as a 48 kHz codec — the RTP timestamp advances at 48,000 ticks per second no matter which internal bandwidth the encoder chose for a given packet.

5.2.2Frame sizes and the latency budget

An audio codec does not emit a continuous stream; it chops the signal into frames and codes each one. Opus frames come in 2.5, 5, 10, 20, 40, and 60 ms — though not every mode offers every size. SILK’s predictive model needs at least 10 ms of signal to fit; CELT’s transform can shrink to 2.5 ms; hybrid mode allows only 10 and 20 ms. The frame size is the first term of the latency bill: the encoder cannot code a frame until the last sample of that frame has been captured. Longer frames amortize packet overhead — one IP/UDP/RTP/SRTP header per 60 ms instead of per 10 ms — and give the coder more context, hence better quality per bit. Shorter frames cut delay. Interactive calls settle almost universally on 20 ms: at the default configuration Opus adds 26.5 ms of algorithmic delay (the 20 ms frame plus 6.5 ms of lookahead), which fits comfortably inside a conversational budget. CELT’s restricted low-delay mode drops the total to roughly 5 ms, which is why Opus turns up inside wireless studio gear, not just calls.

The packet rate follows directly: 20 ms frames mean 50 packets per second in each direction. Hold that number — you will measure it, and watch it collapse, in Problem 2.

5.2.3The TOC byte: a codec that answers for itself

Every Opus packet begins with a single table-of-contents (TOC) byte that announces, with no session context at all, how to read the rest. RFC 6716 §3.1 splits it, most-significant bits first, into three fields: five bits of config, one s bit for channel count, and two c bits for the frame-packing code. The 32 config values enumerate every legal combination of mode, bandwidth, and frame size:

Note the asymmetries — they encode the engineering. SILK stops at wideband because LPC gains little above 8 kHz; CELT skips mediumband because a transform coder has no use for a 12 kHz sampling rate; hybrid exists only where both layers earn their keep. The s bit distinguishes mono from stereo. The c code says how the packet packs frames: 0 means one frame, 1 means two frames of equal size, 2 means two frames of different sizes, and 3 means an arbitrary number announced by a following count byte. Decoding the TOC is Problem 1, and it is worth doing by hand once: it is the entire session-independence story of Opus in eight bits.

Check the mechanics before you code them.

5.2.4VBR, CBR, and the silence machine

Left to itself, Opus is variable-bitrate: a fricative costs more bits than a vowel, silence costs almost nothing, and the encoder spends where the signal demands. VBR buys real quality per average bit, and it is the default everywhere in WebRTC. But packet sizes are visible to anyone on the path even when the payload is encrypted — DTLS-SRTP encrypts contents, not lengths — and in 2008 Charles Wright, Lucas Ballard, Scott Coull, Fabian Monrose, and Gerald Masson showed the consequence: from the packet-length sequence of an encrypted VBR speech call alone, they could spot known phrases with alarming accuracy. That result is why RFC 6562 (Perkins and Valin, 2012) advises constant bitrate for calls whose content confidentiality matters against a traffic-observing adversary, and why Opus keeps a true CBR mode: every packet the same size, the side channel welded shut, a few kb/s of efficiency paid as the toll.

The other bandwidth lever is more cheerful. In a two-person call, each of you is silent roughly half the time; a codec that ships 50 packets a second of coded room tone is spending money on nothing. DTX — discontinuous transmission — lets the encoder stop. A voice-activity detector (a close cousin of the one in the voice pipeline) classifies each frame; once the signal goes quiet, the encoder sends a frame only every 400 ms or so, carrying just enough information for the receiver to synthesize comfort noise — a statistical sketch of the background, regenerated locally. Comfort noise matters psychologically: a truly silent line sounds dead, and people say “hello? hello?” into it. The packet rate during DTX silence drops from 50 per second to roughly 2.5 — a collapse you can observe from getStats, and will, in Problem 2.

5.2.5Losing gracefully: PLC and in-band FEC

Audio has a superpower video envies: it can genuinely repair loss. The first line of defense costs nothing on the wire. Packet-loss concealment (PLC) runs in the decoder: when a frame never arrives, the decoder extrapolates — repeating and decaying the last pitch period in SILK mode, extending the recent spectrum in CELT mode — and for a single lost 20 ms frame the fabrication is usually below the threshold of notice. Concealment degrades fast as gaps grow; two or three consecutive losses start to warble. But single scattered losses, the common case on a decent path, effectively vanish.

The second line is spent proactively. In-band FEC, called LBRR (low-bit-rate redundancy) in the SILK layer, has the encoder embed a coarse re-encoding of frame n inside packet n+1. If packet n dies in transit, the receiver waits one more packet, finds the stowaway copy, and decodes that instead of concealing: real signal, slightly blurred, in place of an educated guess. The costs are explicit. Redundancy competes with the primary encoding for bitrate, so the encoder adds LBRR only when the far side has declared it can use it (useinbandfec=1, § 5.2.6) and its own loss estimate says the insurance is worth buying; and recovery requires the jitter buffer to hold one extra frame of depth when loss is detected — you buy back words with about 20 ms of delay.

Now the asymmetry with video. A 20 ms Opus frame at conversational rates is on the order of 40–80 bytes; re-encoding it coarsely costs a fraction of that, so redundancy is cheap enough to carry always, just in case. A video frame is three orders of magnitude larger, and it does not stand alone: it is a link in a prediction chain of I- and P-frames, so a coarse duplicate of one frame neither fits in the budget nor stops the corruption propagating from a lost reference. Video therefore leans on reactive repair — NACK-driven retransmission and picture-loss recovery — which Module 6 dissects. Audio hides its losses; video has to ask for its packets back.

5.2.6The fmtp knobs — and whose preference they are

Everything above is encoder behavior, and in a browser you do not call the encoder. You reach it through the SDP that the offer/answer exchange carries. Opus appears in a Chrome offer like this:

m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1

Listing 5.2.1 — the Opus lines of a default Chrome audio offer. Payload type 111 is Chrome’s habit; Firefox uses 109. The number is arbitrary and scoped to the session.

The a=fmtp line carries the codec-specific parameters, and RFC 7587 gives them an easily misread semantics: they are declarations by the m-line’s owner about the stream it wants to receive, not announcements about what it will send. maxaveragebitrate=24000 means “cap what you send me at an average of 24 kb/s”; stereo=1 means “my decoder would like stereo”; useinbandfec=1 means “my decoder can exploit LBRR, so feel free to spend bits on it”; usedtx=1 means “I prefer that you go quiet when you have nothing to say.” (sprop-stereo is the odd one out — sprop- parameters describe what the sender is likely to transmit.) The practical consequence inverts many people’s intuition: to change what a peer’s encoder does, you edit the description that peer applies as its remote description. The fmtp your encoder obeys is the one in the far side’s SDP — or in whatever string you handed to setRemoteDescription.

Chrome’s default fmtp shows two more habits worth naming. minptime=10 is a libwebrtc house convention, not an RFC 7587 parameter — packetization time is properly signaled by the a=ptime and a=maxptime attributes, with Opus accepting 3–120 ms and defaulting to 20. And usedtx is absent, because DTX defaults to off. No W3C API flips it: setCodecPreferences only reorders codecs, and RTCRtpSender.setParameters adjusts bitrate but not fmtp. If you want DTX or a bitrate cap today, you edit the SDP string between creation and application — munging:

const offer = await pc.createOffer();
offer.sdp = offer.sdp.replace(
  'useinbandfec=1',
  'useinbandfec=1;usedtx=1');   // crude — Problem 2 does it properly
await pc.setLocalDescription(offer);

Listing 5.2.2 — the classic DTX munge. Because fmtp states receiver preferences, munging your offer configures what the answerer sends you; the answerer’s peer munges the description it applies to shape your encoder.

Munging is a scalpel, not a supported API — but for Opus fmtp parameters it remains the only tool the platform offers, which is exactly why this course makes you do it once, verify the negotiation, and watch the encoder obey. When the packet rate falls off a cliff the moment your oscillator goes quiet, the abstraction stack — fmtp string, negotiated parameters, encoder VAD, wire behavior — stops being folklore.

Two checks on the negotiation model before the problems.

Problems for § 5.2Check your understanding

First read the codec’s own handwriting — the TOC byte — then reach through the SDP to reconfigure a live encoder and watch DTX empty the wire.

¶Field quirks

What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.