§ 5.1  Module 5 — The Codec Wars and the Science of Compression

Why Raw Media Is Impossible: Perceptual Coding

Compression is the art of discarding what the brain would have discarded anyway — and for twenty years, that art was fenced with patents.— Module 5 epigraph

5.1.1The arithmetic that forbids raw media

Begin with the numbers, because they close every argument. Uncompressed audio at the rates WebRTC actually uses — 48,000 samples per second, 16 bits per sample, two channels — costs 48,000 × 16 × 2 = 1,536,000 bits per second: about 1.5 Mbps for a stereo microphone signal. That alone would saturate a mediocre uplink before any video is sent.

Video is worse by three orders of magnitude. A 1080p frame is 1920 × 1080 = 2,073,600 pixels. Captured as 8-bit RGB (or, equivalently, 4:4:4 YUV), each pixel costs 24 bits, so thirty frames per second cost 2,073,600 × 24 × 30 ≈ 1.49 Gbps. The first and crudest perceptual trick — 4:2:0 chroma subsampling, which keeps full-resolution brightness but quarter-resolution color because the retina resolves luminance far more finely than hue — cuts the average to 12 bits per pixel, or 2,073,600 × 12 × 30 = 746,496,000 ≈ 746 Mbps. A WebRTC encoder is typically asked to deliver that picture in about 2.5 Mbps. The required ratio is 746 ÷ 2.5 ≈ 300:1.

// Listing: the arithmetic, spelled out
const video = 1920 * 1080   // pixels per frame
            * 12            // bits per pixel, 8-bit 4:2:0
            * 30;           // frames per second
// => 746,496,000 b/s  (~746 Mbps)

const audio = 48000 * 16 * 2;   // => 1,536,000 b/s (~1.5 Mbps)

const ratio = video / 2.5e6;    // => ~299 : 1

Listing 5.1 — Raw bitrate arithmetic for 1080p30 video and 48 kHz stereo audio.

Could lossless coding close that gap? No — not within a factor of a hundred. Lossless compressors (FLAC for audio, or the intra-frame lossless modes of modern video coders) are bounded by the Shannon entropy of the source, and natural camera video simply contains too much genuine information: in practice lossless video lands near 2:1–3:1. The 300:1 gap can only be crossed by discarding information, and the entire discipline of video compression — and of the codecs whose licensing wars this module chronicles, H.264, VP8/VP9, and AV1 — is the study of which information can go missing without a human noticing. Lossy coding is not an engineering compromise bolted onto a communications problem; it is applied perceptual science.

5.1.2Masking, critical bands, and the just-noticeable difference

The ear is not a microphone; it is a filter bank. The basilar membrane inside the cochlea performs a mechanical frequency analysis, with each region of the membrane responding to a band of frequencies. Harvey Fletcher at Bell Labs — who had already mapped the equal-loudness contours with Wilden Munson in 1933 — showed in his 1940 paper "Auditory Patterns" that this analysis has a characteristic resolution: the critical band, the frequency neighborhood within which sounds compete for the same neural hardware. Eberhard Zwicker's 1961 measurements divided the audible range into 24 such bands (the Bark scale): roughly 100 Hz wide below 500 Hz, and widening to about 20 percent of center frequency above.

Critical bands are where masking lives. Simultaneous masking is the phenomenon Brandenburg's coders exploited: a loud tone raises the threshold of audibility for everything spectrally near it. Play a 1 kHz tone at 70 dB SPL and a 40 dB component at 1.1 kHz — clearly audible on its own — vanishes. The masking "skirt" is asymmetric: it spreads much farther upward in frequency than downward, because of how excitation propagates along the basilar membrane, so a masker hides its higher-frequency neighbors more effectively than its lower ones. Temporal masking extends the effect in time: a loud event masks quieter sounds for roughly 100–200 ms after it ends (forward or post-masking) and — counterintuitively, because the auditory system integrates before it reports — for up to about 20 ms before it begins (backward or pre-masking). Pre-masking is why a codec can afford a brief burst of quantization noise just before a drum hit.

The third pillar is the just-noticeable difference (JND): the smallest change in level a listener can detect, about 1 dB at conversational loudness. Together the three form the perceptual coder's contract. Transform a 20 ms slice of audio to the frequency domain; group the coefficients into critical bands; compute, from the loud components, a masking threshold across all bands; then allocate quantization bits so that the quantization noise in every band stays just below that threshold. Components entirely under the threshold get zero bits — they are simply not sent. That per-band signal-to-mask ratio computation is, almost verbatim, the psychoacoustic model of MP3, of AAC (1997), and of the CELT half of Opus. You will implement its core decision — which bands are masked and discardable — in Problem 1.

5.1.3The LPC–CELP lineage: coding the speaker, not the sound

Masking tells a codec what to omit from a waveform. Speech coding took a more audacious path: omit the waveform entirely and transmit a model of the talker. The lineage begins with Homer Dudley's vocoder at Bell Labs (1939, demonstrated as the Voder at the New York World's Fair), which analyzed speech into a buzz source and a set of filter-bank gains. Its mathematical successor is linear predictive coding: Fumitada Itakura and Shuzo Saito formulated the statistical version in 1966 in Japan, and Bishnu Atal at Bell Labs developed it into a practical coder from the late 1960s onward. LPC's premise is anatomical: the vocal tract is a resonant tube, well approximated by an all-pole filter, so ten-odd filter coefficients per 20 ms frame — plus a pitch value and a voiced/unvoiced flag — describe the speech apparatus far more compactly than the pressure wave it emits.

The lineage runs straight through the history this course has already touched. Danny Cohen's Network Voice Protocol carried LPC-compressed speech across the ARPANET in 1973–74 at a few kilobits per second — packet voice proved feasible precisely because LPC made it small enough. The U.S. Department of Defense standardized LPC-10 as FS-1015 (1984) at 2.4 kbps: intelligible, but notoriously robotic, because its crude two-state excitation (buzz or hiss) threw away everything idiosyncratic about a voice. Fixing the excitation was the next two decades' work. Atal and Remde's multipulse LPC (1982) let the encoder place excitation pulses freely; then Manfred Schroeder and Atal's code-excited linear prediction — CELP, ICASSP 1985 — chose the excitation from a shared codebook by analysis-by-synthesis: the encoder runs the decoder for every candidate and keeps the one whose output sounds closest to the input.

Here the two threads of this lesson braid together, because "sounds closest" is a perceptual judgment. CELP's error is measured through a perceptual weighting filter derived from the LPC coefficients themselves: it de-emphasizes error near the formant peaks, where the speech spectrum is loud and the quantization noise will be masked, and penalizes error in the spectral valleys where the ear would hear it. CELP is simultaneous masking, applied frame by frame to a model of your throat. Its descendants carried telephony for thirty years: FS-1016 CELP at 4.8 kbps (1991), ITU-T G.729's algebraic CELP at 8 kbps (1996), the 3GPP AMR codec of 1999 that still carries much of the world's mobile voice, Jean-Marc Valin's open-source Speex (2002), and Skype's SILK (2009) — which, fused with Xiph's CELT transform coder, became the speech half of Opus. When Opus runs in its SILK mode on your WebRTC call, it is spending most of its bits on an all-pole vocal-tract model whose family tree begins at Bell Labs.

From the archive

5.1.4Rate-distortion: bits where perception lives

There is a formal frame around all of this. Claude Shannon's rate-distortion theory — sketched in 1948 and completed in his 1959 paper "Coding Theorems for a Discrete Source with a Fidelity Criterion" — establishes that for any source and any tolerated distortion D there is a minimum achievable rate R(D), a smooth curve trading fidelity against bits. Two consequences matter for everything in this module. First, the curve is convex and steep near zero distortion: the final increments of fidelity are the most expensive, which is why "almost lossless" costs nearly as much as lossless and why practical codecs operate far down the curve. Second — and this is the deep point — the curve depends entirely on how you measure distortion. Measure it as mean-squared error on the waveform and you get one curve; measure it as perceived difference and you get a dramatically cheaper one, because perceptual distortion is zero for every change the ear or eye cannot detect. Perceptual coding does not cheat Shannon; it changes the fidelity criterion to the one that actually matters.

"Bits where perception lives" is therefore the single design rule beneath every codec you will meet in this module. Chroma subsampling puts bits in luminance rather than hue. MP3 and CELT put bits in unmasked bands and none under the threshold. CELP puts bits in the spectral valleys between formants where noise would be heard, and lets it hide under the formants where it would not. Video coders put bits in edges and motion the eye tracks, and starve the textures it does not. And in WebRTC the operating point itself is not chosen by the codec at all: the congestion controller — Google congestion control, later in this course — hands the encoder a bitrate target, and the encoder's rate control slides along its rate-distortion curve to meet it. The next lessons dissect the machinery: the GIPS voice pipeline Google bought for $68 million in 2010, Opus itself, and the color, prediction, and codec-war politics of video.

Problems for § 5.1Check your understanding

One coding problem — the discard decision at the heart of every perceptual audio codec — and one set of bitrate drills.

Problem 1 supplies band energies in dB and a masking curve given as threshold drops: maskingCurve[k] is how many dB below a masker's energy the threshold sits at a distance of k + 1 bands. Every band acts as a potential masker of every other band; a band is discardable when its own energy is strictly below the highest threshold any other band imposes on it. Return { discard, bitsSaved }, where discard lists the masked band indices in ascending order and bitsSaved assumes a naive 16 bits per discarded band.

¶Field quirks

What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.