§ 5.5 Module 5 — The Codec Wars and the Science of Compression
Compression is the art of discarding what the brain would have discarded anyway — and for twenty years, that art was fenced with patents.Course notes
By the end of this lesson
The failure that forced the design
H.264 was technically superb and legally radioactive for the open web: MPEG-LA licensing versus browsers that ship for free. Google open-sourced VP8; the IETF spent years deadlocked on which codec browsers must implement; Cisco paid the H.264 license for everyone via a downloadable binary; and in 2014 the answer was 'both' — codified as RFC 7742 (2016). AV1 (2018) is the industry's attempt to make sure that war never happens again.
H.264's patent economics versus VP8's open gambit, the IETF MTI deadlock ending in the 2014 'implement both' compromise (codified as RFC 7742, 2016), Cisco's OpenH264 gambit, AOMedia and AV1, and the hardware-acceleration reality that keeps H.264 alive — plus real payload parsing and real codec negotiation.
Cross-references: The Video Codec Wars: H.264, VP8/VP9, AV1.
The operational claim is precise: MPEG-LA economics; On2/VP8; OpenH264; RFC 7742 MTI outcome. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. MPEG-LA economics; On2/VP8; OpenH264; RFC 7742 MTI outcome. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
RTCRtpSender.getCapabilities('video') and RTCRtpReceiver.getCapabilities('video') return different lists, and both vary by machine: a Chromebook may decode VP9 without encoding it; an Android device lists H.264 only while a hardware encoder is available. setCodecPreferences() lives on the transceiver rather than the peer connection, must be called before the createOffer that carries that m-line, and throws InvalidModificationError if you pass an entry the browser does not have. In Chrome it constrains both directions on that m-line, which surprises people trying to change only one; Safari shipped it late (around 15.4).
Query both capability lists at runtime on the actual device and reorder the objects you were given rather than constructing entries by hand. For send-only selection on recent Chrome, setParameters() with encodings[].codec avoids the m-line entirely. Always confirm through getStats() — join outbound-rtp.codecId to the codec object — and record the negotiated codec per session so you learn your fleet's real distribution.
H.264 is still the deployment baseline because hardware encoders are ubiquitous, but profile-level-id and packetization-mode must intersect. RTP carries complete NAL units when they fit, aggregates small units with STAP-A, and fragments large units with FU-A; the five low bits of the first octet identify the NAL type.
Engineering consequence. H.264 profiles/levels and RFC 6184 packetization (NAL, STAP-A, FU-A, emulation prevention). Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Interactive 3D instrument
H.264, VP8, VP9, AV1 — bits, watts, and lawyers
A 3D instrument you drive yourself, one variable at a time. It needs JavaScript and WebGL, so it is not shown in this static copy of the page.
The operational claim is precise: VP8 payload descriptor (RFC 7741); VP9; AV1 and AOMedia; HEVC's licensing pathology. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. VP8 payload descriptor (RFC 7741); VP9; AV1 and AOMedia; HEVC's licensing pathology. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Android hardware H.264 encoders commonly enforce an internal bitrate floor of 100–200 kbit/s at low resolutions and simply overshoot your cap; several emit keyframes only on explicit request; some produce SPS/PPS that Safari's decoder rejects, giving you a one-way video failure between two specific device families. Most importantly many SoCs support a single simultaneous hardware encode session, so a three-layer simulcast request quietly returns two layers or falls back to software with a CPU cliff.
Count your outbound-rtp objects: if you asked for three encodings and got two, a layer was dropped. Toggle chrome://flags/#disable-accelerated-video-encode as an A/B to confirm the hardware path is at fault. For Android-heavy fleets, prefer VP8 with software simulcast or restrict to two layers.
The operational claim is precise: Hardware encode reality on mobile; getCapabilities and setCodecPreferences. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Hardware encode reality on mobile; getCapabilities and setCodecPreferences. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Start with the claim “MPEG-LA economics; On2/VP8; OpenH264; RFC 7742 MTI outcome” and turn it into a falsifiable trace. Write down the input condition, the expected state transition, the counter or packet field that should change, and the deadline by which it must change. This prevents a familiar mistake: observing a symptom after several mechanisms have already reacted and assigning it to whichever mechanism has the most recognizable name. The causal order matters. A queue grows before loss; an ICE path fails before a restart selects a new pair; a decoder loses reference state before frames stop rendering. Preserve that order in the evidence.
Now hold every unrelated variable still. Use canned timestamps, SDP, stats-shaped records, or a permission-free loopback source so the same input can be replayed. Record both media directions separately. For cumulative counters, retain two samples and divide the difference by their timestamp interval. For state machines, retain the ordered event sequence rather than the last state alone. For negotiated features, compare the offer, answer, and live stats result. These three views answer different questions: what was proposed, what was accepted, and what the browser actually used.
The first lab, Parse the Payloads, applies this discipline to a bounded implementation problem. Implement classifyNal(bytes) over a canned Annex-B/RTP corpus (identify SPS/PPS/IDR/non-IDR, STAP-A, FU-A) and parseVp8PayloadDescriptor(bytes) (S bit, PID, keyframe inference). Assertions are exact per packet; hints catch emulation-prevention bytes (0x000003) and the 5-bit NAL type mask. Before coding, identify the invariant hidden in that brief and the tempting shortcut the tests should reject. Then run the reference trace, perturb one boundary value, and explain why the result changes. If the code passes but the explanation cannot predict the perturbed result, the implementation is memorized rather than understood.
Negotiating AV1 in a browser today almost always means libaom in software. It is excellent for low-bitrate screenshare, and it will thermally throttle a laptop at 720p30 camera, showing up as qualityLimitationReason: 'cpu' and a fan. AV1 also does not do RTP simulcast: you get scalability through scalabilityMode on the first encoding ('L1T3', 'L3T3_KEY'), which is a Chrome-only, non-standard-ish knob that other browsers ignore.
Use AV1 selectively — screenshare and low-bitrate audio-conference-with-video — not as a default. If you enable it, monitor totalEncodeTime/framesEncoded (over roughly 15 ms per frame at 30 fps means you are losing) and be ready to renegotiate down to VP8 or H.264.
A browser loopback is deliberately cleaner than the internet. It removes signaling latency, hostile NAT behavior, relay geography, wireless contention, device heat, and competing encoders. That makes it ideal for proving protocol and API invariants, but a dangerous basis for capacity claims. Carry the invariant into a second test with a canned hostile trace: reordered packets, a delayed candidate, a missing remote stats object, a bitrate step, or a state transition that recovers before the debounce timer. The implementation should remain correct even when optional fields are absent and events arrive more than once.
Feature detection is also part of correctness. Browser support for a codec, scalability mode, encoded transform, or transport API does not imply that the negotiated peer supports it or that hardware acceleration is active. Prefer capability checks, preserve a standards-based fallback, and label any simulation honestly. The graded result should measure the learner's algorithm; an optional live coda may demonstrate the real API without making a camera, operating-system codec, or network service a hidden test dependency.
Finally, connect the mechanism to the last topic in the lesson: “Hardware encode reality on mobile; getCapabilities and setCodecPreferences”. Advanced WebRTC failures cross boundaries. Congestion control changes encoder output; layer selection changes keyframe demand; end-to-end encryption constrains what an SFU can inspect; a watchdog can collide with negotiation; observability can mislabel a path if its joins are wrong. At every boundary, define ownership and an idempotent action. A retry must not create duplicate work, a stale event must not overwrite newer state, and an escalation must have a terminal cleanup path.
Use the problem set as an executable notebook. Keep a short record beside each answer: the invariant, the decisive evidence, the rejected alternative, and the production caveat. That record is more valuable than a green checkmark because it can be reused when a real call presents the same mechanism through noisier evidence.
A common error
SDP is a protocol WebRTC sends over the network — SDP is an inert description; transporting it is entirely the application's job.
Implement the decision rules, run every assertion, then use the explanations to distinguish a wrong result from a wrong model.
Parse the Payloads. Implement classifyNal(bytes) over a canned Annex-B/RTP corpus (identify SPS/PPS/IDR/non-IDR, STAP-A, FU-A) and parseVp8PayloadDescriptor(bytes) (S bit, PID, keyframe inference). Assertions are exact per packet; hints catch emulation-prevention bytes (0x000003) and the 5-bit NAL type mask.
Lab 1
Parse the Payloads
Graded in the browser against 3 assertions; the editor and harness require JavaScript.
Force the Codec, Prove It. Query RTCRtpSender.getCapabilities('video') in the learner's actual browser, then write a codec-preference function using setCodecPreferences on a loopback pair. Harness asserts the negotiated codec changed (via a harness stats helper reading the getStats codec mimeType) and quizzes tradeoff scenarios (battery-constrained mobile → which codec and why).
Lab 2
Force the Codec, Prove It
Graded in the browser against 3 assertions; the editor and harness require JavaScript.
Check your understanding
Auto-graded check
4 auto-graded questions with an explanation for every wrong answer. Requires JavaScript. (m5-l5-quiz)
What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.
Video codecs occupy the dynamic 96–127 range and the assignments differ between browsers, between browser versions, and between m-lines in the same session. A modern video m-line commonly carries 12 or more payload types: several H.264 profile/packetization combinations, VP8, VP9, AV1, an RTX type for each, plus RED and ULPFEC. Anything that hard-codes "96 is VP8" works until it does not, and the failure is a stream that negotiates fine and decodes garbage.
Resolve payload types through a=rtpmap or, better, through getStats(): join inbound-rtp.codecId to the codec object and read mimeType and payloadType. Never write a payload-type constant into an SFU.
WebRTC: From First Principles · Module 5, Lesson 5