§ 5.6  Module 5 — The Codec Wars and the Science of Compression

The Voice Pipeline: GIPS's Crown Jewels

From the archive

5.6.1Echo taxonomy: the render→capture loop

Telephony knows two species of echo. Line echo is electrical: the analog local loop is two wires carrying both directions, the long-distance plant is four, and the hybrid transformer that converts between them reflects energy whenever its impedance match is imperfect — which, across millions of miles of varied copper, is always. The network solved this with cancellers standardized in ITU-T G.168, and a WebRTC application meets line echo only at PSTN gateways. Acoustic echo is the one this lesson is about, and it is mechanical: your device renders the far end's voice through a loudspeaker, the sound crosses the room — a direct path plus every reflection off desk, walls, and ceiling — and re-enters your microphone. The room itself is a filter: its impulse response runs from tens to hundreds of milliseconds depending on how hard the surfaces are, and every sample your speaker plays is convolved through it into your capture. This render→capture loop is the defining topology of the problem: the signal to be removed is not noise, not interference, but a delayed, filtered copy of something your own machine just played on purpose.

Why is a little feedback so destructive? Delay. Telephone handsets intentionally return a trace of your own voice as sidetone; at near-zero delay the brain fuses it with your speech. But a WebRTC call carries 100–300 ms of one-way latency, so the echo of your sentence arrives as a separate acoustic event — a heckler repeating you a beat behind. Worse, the loop is closed at both ends: if each side leaks enough render into capture, the system sings, the same Larsen feedback howl a PA microphone makes near its own speaker. Note the direction of blame, because it decides every echo bug you will ever triage: the echo you hear was created on the other person's device, where your rendered voice leaked into their microphone. The sufferer is never the culprit.

5.6.2Adaptive filtering, double-talk, and why AEC stays hard

The canceller's premise is that the echo path is approximately linear: the capture's echo component is the render signal convolved with some unknown impulse response h. So the canceller maintains its own FIR filter ĥ, predicts the echo ŷ(n) = ĥ · x(n), subtracts it from the capture d(n), and treats the residual e(n) = d(n) − ŷ(n) as its error signal. The Widrow–Hoff LMS rule nudges every tap in the direction that would have reduced that error; its workhorse variant NLMS — normalized LMS — divides the step size by the recent power of the far-end signal, so adaptation speed stops depending on how loudly the far end happens to be talking. A room tail of 50–300 ms would demand thousands of time-domain taps at 48 kHz, so practical cancellers adapt in subbands or in the frequency domain, but the intuition survives translation: guess the room, subtract the guess, learn from what is left over.

Three enemies keep this simple idea from being a solved problem. The first is delay. The filter can only model the room if the reference and the capture are aligned in time, and between the sample your JavaScript "hears" being rendered and the sample that actually leaves the loudspeaker sit OS mixers, driver ring buffers, USB and Bluetooth stacks — a Bluetooth link alone can add well over 100 ms — amounting to a bulk delay Δ that is unknown, differs per device pair, jumps when the user switches headsets, and drifts, because the capture and render sides run on separate crystal oscillators that disagree by a few parts per million. WebRTC's third-generation canceller, AEC3, which replaced the long-serving desktop AEC as Chrome's default in 2018, is built around exactly this reality: it is delay-agnostic by design, running a bank of matched filters at candidate alignments and re-estimating the render→capture delay continuously, because field data showed the old startup-time delay estimate was wrong precisely when users changed devices mid-call.

The second enemy is double-talk. Adaptation assumes the residual is leftover echo; the moment the near talker speaks, the residual contains their voice, and a gradient update will happily bend the filter toward cancelling the customer. Every deployed AEC therefore carries a double-talk detector — the classic is the Geigel detector, a Bell Labs heuristic that declares near-end speech whenever the capture's magnitude exceeds a threshold relative to the recent far-end maximum; modern designs use coherence statistics — and its verdict does one thing: freeze or drastically slow adaptation until the near talker yields. You will watch this failure happen numerically in Problem 1: a fixed subtractive canceller is fine until both parties speak, and then its measured ERLE collapses, not because the echo returned but because the canceller cannot tell the near voice from its own unfinished work.

The third enemy is nonlinearity. The linear-path premise is a polite fiction on consumer hardware: a laptop speaker the size of a coin driven near its excursion limit distorts, its plastic enclosure buzzes, amplifiers clip. No linear filter can predict those components, so a correlated residue survives subtraction, and every practical AEC finishes with a residual echo suppressor — a spectral post-filter that attenuates whatever the linear stage could not explain. That suppression is why an aggressive AEC sounds "half-duplex," ducking your voice while the far end speaks: it is a controlled regression to the 1925 echo suppressor, applied only where the mathematics ran out. The pipeline even confesses its performance: the media-source section of getStats carries echoReturnLoss and echoReturnLossEnhancement, both in decibels, whenever the canceller is active.

5.6.3Noise suppression, gain control, and the sound of silence

AEC removes the far end's rendered signal. Everything else the microphone picks up that is not the talker — fans, traffic, the refrigerator — is the province of noise suppression (NS), a different algorithm with a different lineage. Steven Boll's 1979 spectral subtraction (IEEE Transactions on Acoustics, Speech, and Signal Processing) estimated the noise spectrum during speech pauses and subtracted it per frequency band; its signature artifact was "musical noise," fluttering spectral islands left where the subtraction over- and under-shot. Yariv Ephraim and David Malah's 1984 MMSE estimator tamed the flutter with statistically optimal per-band gains, and WebRTC's suppressor is in that tradition: track the noise floor, decide per band how speech-like the energy is, attenuate accordingly. Push any of these hard and speech itself gets carved — the "underwater robot" voice of an over-aggressive suppressor is speech bands wrongly taxed as noise. The modern turn is learned masks: Jean-Marc Valin's RNNoise (2017–18) replaced hand-tuned band decisions with a small recurrent network and opened the road to today's DNN denoisers.

Automatic gain control (AGC) answers a humbler question — how loud? — and answers it continuously, steering analog microphone gain through the OS and applying digital gain with a limiter so that a whisperer and a boomer land at comparable levels for the Opus encoder. Its characteristic failure is pumping: gain creeping up during pauses and slamming down after loud passages, so the room tone audibly breathes. Behind both NS and AGC sits voice activity detection (VAD) — WebRTC's classic detector is a Gaussian-mixture classifier over subband energies — which tells NS when it is safe to update the noise estimate, tells AGC when level measurements actually describe speech, and gates discontinuous transmission. And discontinuous transmission explains the strangest inhabitant of the pipeline: comfort noise. When VAD reports silence, sending nothing would be cheapest — but absolute digital silence is indistinguishable from a dead connection, and callers hang up on it. So the sender ships an occasional low-rate description of the background noise (the RTP comfort-noise payload, RFC 3389, 2002; Opus instead has DTX built into the codec itself), and the receiver synthesizes plausible hiss. The pipeline manufactures noise on purpose, at the same moment another stage works to remove it — both in service of perception, not the waveform.

5.6.4Constraints: toggling the pipeline from JavaScript

The whole pipeline sits behind three booleans in the Media Capture and Streams specification, spelled exactly echoCancellation, noiseSuppression, and autoGainControl, and passed as audio constraints to getUserMedia. Spelling is not pedantry here: unknown constraint names in the basic constraint set are silently ignored, so autoGainControl misspelled as automaticGainControl throws no error, logs no warning, and simply leaves the pipeline in its default state — one of the quieter footguns in the API. (Chrome's ancient goog-prefixed constraints — googEchoCancellation and kin — are long gone.) The defaults are all-on, tuned for a voice call; verifying what you actually received, as opposed to what you asked for, is track.getSettings(), and live changes go through track.applyConstraints().

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,   // AEC — subtract the far-end render
    noiseSuppression: true,   // NS  — attenuate non-speech noise
    autoGainControl:  true    // AGC — normalize the capture level
  }
});

const [track] = stream.getAudioTracks();
console.log(track.getSettings());
// => { echoCancellation: true, noiseSuppression: true,
//      autoGainControl: true, sampleRate: 48000, ... }

// A live-music capture wants the pipeline OUT of the way:
await track.applyConstraints({
  echoCancellation: false,
  noiseSuppression: false,
  autoGainControl:  false
});

Listing 5.6 — Requesting, verifying, and disabling the processing pipeline. Ask, then check getSettings() — constraints are requests, not guarantees.

When would you turn the crown jewels off? Music. AEC's residual suppressor ducks sustained program material; NS cannot distinguish a held violin note from stationary noise and will slowly eat it; AGC flattens the dynamics a performer worked for. Every serious in-browser recording or performance tool disables all three. The opposite trick is more common: headphones "solve" echo, and it is worth being precise about why. Headphones do not improve the canceller — they remove its problem, physically breaking the render→capture coupling so the echo path the AEC must model drops to nearly nothing (at high volume, leakage from open-backed cans to a laptop microphone still gives it a little work). This is why "please use headphones" remains the most effective echo mitigation ever deployed, and why the hardest calls are open-air laptops in reverberant kitchens. It also reframes the war story from the top of this lesson: the native app that captured system audio was injecting the rendered far-end voices into the send path electronically, a path with no acoustic attenuation at all — and one the browser's AEC never saw, because loopback capture bypasses the processing pipeline. Every remote participant echoed; the machine at fault was the only one that heard nothing wrong. Diagnosis in this domain always starts the same way: whose render is reaching whose capture, and which stage was supposed to stand between them?

Problems for § 5.6Check your understanding

One coding problem — a naive single-tap echo canceller, and the double-talk case that breaks it — and one quiz on the three A's.

Problem 1 hands you 48 kHz signals as Float32Arrays, rendered through an OfflineAudioContext band-limiting chain: far is what the loudspeaker played, and the capture contains that signal delayed by an unknown number of samples and scaled by an unknown gain. Implement four small functions: estimateDelay(far, near, maxDelay) (cross-correlation argmax over lags 0…maxDelay), estimateGain(far, near, delay) (the least-squares one-tap fit), cancelEcho(far, near, delay, gain) (delayed, scaled subtraction), and erle(near, residual) (10·log₁₀ of the power ratio, in dB). The final test runs your pipeline on a double-talk capture — and your canceller will fail it, by design: watch what the near voice does to the residual.

¶Field quirks

What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.