Module 5 — The Codec Wars and the Science of Compression

The Module 5 Examination

WebRTC Browsers MUST implement the VP8 video codec as described in RFC 6386 and H.264 Constrained Baseline.— RFC 7742, §5, March 2016

Plan on 90 minutes. Part A is code, graded by the harness with hints disabled and partial credit per assertion; it ends on live instruments — a real VideoEncoder and a real RTCPeerConnection loopback whose negotiated codec you must prove from stats. Part B is forensic scenario work on canned artifacts: SDP excerpts, capability dumps, frame-size timelines. Part C is concepts and history, and, as every exam from Module 2 onward does, it reaches back with two spaced-review questions from earlier modules. The pass mark is 80 percent; retakes draw variants of these problems, so understanding beats memorizing the answer key.

AThe signal bench

Two dry-bench problems first. Problem 1 rebuilds the pixel pipeline from § 5.3 as a single pass: BT.601 conversion and 4:2:0 planar packing with exact buffer arithmetic. The coefficients are given in the starter — the exam tests the pipeline, not your memory of constants — but the grader is strict about sizes: an I420 buffer is exactly w·h·1.5 bytes, Y plane first, then Cb, then Cr, and each chroma sample is the rounded average of its 2×2 block, the way libyuv does it, not the top-left pixel alone.

Problem 2 is bitstream forensics on an unseen corpus: the Opus TOC byte (RFC 6716 §3.1) and the H.264 RTP payload headers of RFC 6184 (2011). Both tables are reproduced below; the skill being graded is exact bit surgery, not recall.

Opus TOC configurations (RFC 6716, Table 2):
  config  mode    bandwidth  frame sizes (ms)
   0- 3   SILK    NB         10, 20, 40, 60
   4- 7   SILK    MB         10, 20, 40, 60
   8-11   SILK    WB         10, 20, 40, 60
  12-13   Hybrid  SWB        10, 20
  14-15   Hybrid  FB         10, 20
  16-19   CELT    NB         2.5, 5, 10, 20
  20-23   CELT    WB         2.5, 5, 10, 20
  24-27   CELT    SWB        2.5, 5, 10, 20
  28-31   CELT    FB         2.5, 5, 10, 20

TOC byte layout:  | config (5 bits) | s (1 bit) | c (2 bits) |
  config = byte >> 3     s = (byte >> 2) & 1  (1 = stereo)
  c      = byte & 3      (frame-count code)

H.264 NAL header (RFC 6184): first payload byte
  F (1 bit) | NRI (2 bits) | type (5 bits) = byte & 0x1F
  types: 1 non-IDR slice, 5 IDR slice, 6 SEI, 7 SPS, 8 PPS,
         24 STAP-A (aggregation), 28 FU-A (fragmentation)
FU-A only: the second byte is the FU header
  S (0x80, first fragment) | E (0x40, last) | type (byte & 0x1F)

Listing E5.1 — the two tables Problem 2 is graded against.

Part A — Coding, dry benchHints disabled · partial credit per assertion

Pack the pixels, then read the bitstreams.

A′The live instruments

Problem 3 uses WebCodecs as an instrument, the way § 5.4 did: your code decides where the keyframes go — one every two seconds at 30 fps — a real VideoEncoder in latencyMode: 'realtime' executes the plan, and your analyzer must read the GOP structure back out of the emitted chunks. On engines without WebCodecs the harness substitutes a canned chunk trace; your functions are graded either way.

Problem 4 is the negotiation you will one day perform in production: force H.264 by munging the offer's m=video payload-type order, cap the send bitrate by inserting a b=AS line into the answer, and prove both from the running connection — the negotiated codec is read from getStats(), following the outbound-rtp entry's codecId to a codec entry and its mimeType. Remember the direction of b=AS: a bandwidth line in the description you receive caps what you send toward that peer, so the cap goes into the answer before the caller applies it.

Part A — Coding, liveReal VideoEncoder · real loopback · feature-detected fallbacks

Command a real encoder's GOP cadence, then force a codec and prove it from stats.

BScenario diagnostics

Four incidents, each anchored to an artifact of the kind webrtc-internals and getStats put in front of you. Read the artifact before answering; every correct diagnosis cites a specific line. Two of them turn on RTCP feedback — PLI (RFC 4585, 2006) and FIR (RFC 5104, 2008) — which Module 6 will dissect in full; here you need only what § 5.4 taught: a keyframe is both salvation and a bandwidth bomb, three to ten times the size of the deltas around it.

Artifact B-1 — offer and answer excerpts, and the symptom that followed.

-- offer, m=video excerpt (desktop app, hardware encoder) --
m=video 9 UDP/TLS/RTP/SAVPF 102 103 96 97
a=rtpmap:102 H264/90000
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640032
a=rtpmap:103 rtx/90000
a=fmtp:103 apt=102
a=rtpmap:96 VP8/90000
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96

-- answer, m=video excerpt (mid-range Android phone, Chrome) --
m=video 9 UDP/TLS/RTP/SAVPF 96 97
a=rtpmap:96 VP8/90000
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96

-- symptom --
call connects; the phone runs hot, battery drains fast, its video
drops to 15 fps; webrtc-internals on the phone: encoder = libvpx
(software)

Artifact B-2 — an SFU room with twelve publishers. Audio stays clean; every video tile pixelates for a second or two whenever someone joins.

publisher stats (one representative sender), cumulative counters:
  t=118 s  join #9   pliCount 14 -> 17    firCount 2
                     keyFramesEncoded 31 -> 34
                     outbound bitrate 1.8 Mbps -> 4.1 Mbps (one interval)
  t=141 s  join #10  pliCount 17 -> 20
                     keyFramesEncoded 34 -> 37
                     outbound bitrate 1.9 Mbps -> 4.3 Mbps
downlink packetsLost: spikes in the seconds after each join

Artifact B-3 — a frame timeline with one casualty, and the negotiated feedback lines. (This scenario waited for you since Module 1; you now have the GOP knowledge to read it.)

RTP seq      bytes    frame type        arrived?
2118          1042    delta (P)         yes
2119           998    delta (P)         yes
2120-2129    14880    KEY (I, 10 pkts)  seq 2124 lost -> frame undecodable
2130          1011    delta (P)         yes
2131           987    delta (P)         yes
2132          1020    delta (P)         yes
...

negotiated feedback for PT 96:
a=rtcp-fb:96 nack pli        <- present
a=rtcp-fb:96 ccm fir         <- present
(plain "a=rtcp-fb:96 nack" — packet retransmission — absent)

Artifact B-4 — sender capabilities on a phone, and one line from its chipset datasheet.

RTCRtpSender.getCapabilities('video').codecs (excerpt):
  { mimeType: 'video/VP8',  clockRate: 90000 }
  { mimeType: 'video/H264', clockRate: 90000, sdpFmtpLine:
    'level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f' }
  { mimeType: 'video/VP9',  clockRate: 90000, sdpFmtpLine: 'profile-id=0' }
  { mimeType: 'video/AV1',  clockRate: 90000 }

chipset datasheet: hardware video encode/decode blocks: H.264 only

Part B — ForensicsFour scenarios, canned artifacts

Diagnose from the artifact; every answer is anchored to a line above.

CConcepts, history, and spaced review

Short answers on the ideas that make the machinery legible: why a codec may silently discard a tone you are certain you would hear, what Opus's in-band FEC actually transmits, what the 2014 compromise actually mandated, and why the B in B-frames is banned from every real-time encoder. Then the spaced review reaches back — to Module 1's bitrate arithmetic and to Module 4's perfect negotiation under glare.

Part C — ConceptsPooled bank

Four questions on this module's ideas.

Part C — Spaced reviewModules 1 and 4

Two questions from earlier modules, as promised.