§ 5.4  Module 5 — The Codec Wars and the Science of Compression

The Grammar of Video: Frames, GOPs, and Rate Control

5.4.1Most of a frame is the previous frame

Start with the arithmetic of the problem. A 1920 × 1080 frame in 8-bit Y′CbCr with 4:2:0 chroma subsampling — the format § 5.3 left us with — is 1.5 bytes per pixel: about 3.1 MB per frame, 746 Mbps at 30 frames per second. A good encoder ships that scene at 2.5 Mbps, a ratio near 300:1. Subsampling bought a factor of two; the remaining factor of 150 comes from a single observation: most of a frame is the previous frame, moved slightly. A talking head is a nearly static room with a few thousand pixels of lips and eyelids in motion. A camera pan is the same image shifted eight pixels left. Sending pixels again because 33 ms have elapsed is the waste; the hybrid coder exists to stop it.

The machine works block by block. The frame is cut into macroblocks — 16 × 16 luma samples in H.261's vernacular, which H.264 and VP8 kept. For each block, motion estimation searches a window of the previously reconstructed frame for the best matching 16 × 16 patch; the offset to that patch is the motion vector, typically at half- or quarter-pixel precision. Motion compensation then builds a predicted frame by pasting each best match at its block's position, and the encoder subtracts prediction from reality. What is left — the residual — is mostly noise-level texture the prediction missed.

The residual is where Ahmed's transform earns its keep. Each residual block passes through a DCT (8 × 8 in MPEG-1, 4 × 4 in VP8 and much of H.264), which turns spatial samples into frequency coefficients: a flat patch becomes one significant number and a field of near-zeros. Quantization then divides every coefficient by a step size and rounds — the sole irreversible act in the pipeline, and therefore the codec's only quality knob. A larger step manufactures more zeros; long zero runs cost almost nothing after entropy coding. Everything a rate controller does, it does by moving that quantizer step up and down. The bitstream that leaves is just motion vectors plus entropy-coded quantized residuals — a set of instructions for editing the previous frame into this one.

5.4.2I, P, B, and the grammar of the GOP

Three frame types fall out of the design. An I-frame (intra, a keyframe) is coded entirely from itself, like a JPEG: no motion vectors, no dependencies, decodable in isolation. A P-frame predicts from an earlier decoded frame and carries motion vectors plus residuals. A B-frame predicts bidirectionally — from an earlier anchor and a later one — and is the cheapest of the three, because interpolating between two references leaves the smallest residual. The repeating sentence built from these words is the GOP, the group of pictures: one I-frame followed by a pattern of P- and B-frames until the next I-frame. Broadcast MPEG-2 (1995) settled on patterns like I B B P B B P … with a keyframe every 12–15 frames; DVDs and digital television still speak roughly that sentence.

Now the latency analysis the opener promised. B-frames reorder time. In display order I₀ B₁ B₂ P₃, frame B₁ references P₃ — a frame that, at capture time, does not exist yet. The encoder must hold B₁ and B₂ until P₃ arrives, encode P₃ first, and transmit in decode order I₀ P₃ B₁ B₂. At 30 fps, B₁ waits two frame intervals — about 67 ms — before it can even be encoded; the receiver must then hold P₃ in a reorder buffer (a structural delay separate from, and added to, the network jitter buffer) before display. ITU-T G.114 advises keeping one-way mouth-to-ear delay under 150 ms for comfortable conversation, and that budget must also cover capture, encoding, the network, decoding, and rendering. A three-frame reorder spends nearly half of it before the first packet leaves the machine. Offline encoders happily buffer far more: libvpx's quality modes use a lag-in-frames lookahead of up to 25 frames — over 800 ms at 30 fps — to plan allocations.

Real-time systems therefore ban the B-frame outright, and the ban is written into the standards' choice of profiles. H.264's Constrained Baseline profile — the profile RFC 7742 makes mandatory for WebRTC — contains no B-slices at all. VP8 (RFC 6386, November 2011) goes further: the bitstream simply defines no bidirectional prediction. Its extra reference frames — the golden frame and altref — point only backward in decode order, and the one libvpx trick that builds an altref from lookahead is disabled the moment you configure the encoder for real time, which forces lag-in-frames to zero. A WebRTC GOP is the dullest sentence in the language: one I, then P, P, P — each frame predicting from the last, capture order equal to transmit order equal to display order, structural delay zero.

5.4.3The keyframe: salvation and bandwidth bomb

The all-P diet has a structural weakness: every frame depends on the one before it. The keyframe exists to break that chain, and it is needed in exactly two situations. First, entry: a decoder that joins mid-stream has no previous frame to edit, so nothing decodes until an I-frame arrives. Second, repair: lose a packet and the damaged frame is wrong, and so is every frame that predicts from it — the error propagates, smearing artifacts forward indefinitely, until a keyframe resets the reference chain.

Salvation has a price. An I-frame spends bits on every block with no help from prediction; for the same picture at the same quality it typically costs 3–10× the bytes of a delta frame — you will measure the ratio on a live encoder in Problem 1. That price dictates opposite policies in the two halves of the video world. Streaming systems buy keyframes on a timer: HLS and DASH cut the stream into segments a few seconds long, and each segment must begin with a keyframe so that any segment is independently decodable and viewers can join, seek, or switch renditions at every boundary. Their latency budget — seconds — absorbs the cost invisibly.

WebRTC refuses the timer. Its encoders run an effectively infinite GOP: one keyframe at the start of the stream and then deltas forever, with fresh keyframes produced on demand. The demand arrives over the RTCP feedback channel: a receiver whose reference chain broke sends a Picture Loss Indication (PLI), a conferencing server sends a Full Intra Request (FIR), and an SFU asks the sender for a keyframe whenever a new subscriber needs a point of entry into an existing stream. Nothing about the wall clock triggers any of this; a stable one-to-one call can run for an hour on its opening keyframe.

The refusal is not thrift alone — it is self-defense, because a keyframe is a bandwidth bomb. At 1 Mbps and 30 fps, the average frame budget is about 4,167 bytes; a 31 KB keyframe is seven and a half frames of budget detonating in one frame interval. It leaves the encoder as a burst of RTP packets, momentarily driving the send rate far above target; queues build at the bottleneck, delay spikes, the receiver's jitter buffer stretches, and — in the ugliest feedback loop in conferencing — the resulting loss makes other receivers send PLIs, demanding more keyframes: a keyframe storm. Production SFUs throttle PLIs per sender and fan one fresh keyframe out to many requesters precisely to dampen this loop. You can watch every actor in this drama in webrtc-internals; Problem 2 asks you to predict the spike's size with arithmetic first.

Predict before you peek — everything below follows from § 5.4.2 and § 5.4.3.

5.4.4WebCodecs: a microscope for the encoder

Until recently none of this was observable from a web page. Inside RTCPeerConnection the encoder is sealed machinery: a track goes in, RTP comes out, and the GOP structure between them is invisible. WebCodecs unbundled it. Developed in the W3C Media Working Group (editors Paul Adenot of Mozilla, Bernard Aboba of Microsoft, and originally Chris Cunningham of Google) and shipped in Chrome 94 in September 2021, it hands JavaScript direct handles on the browser's codec implementations: VideoEncoder, VideoDecoder, and their audio twins. This course gives WebCodecs its full treatment in § 8.4, where it anchors the unbundled future; here we use it instrumentally, as a microscope pointed at everything this lesson has claimed.

const encoder = new VideoEncoder({
  output: (chunk, metadata) => record(chunk.type, chunk.byteLength),
  error: (e) => console.error(e),
});
encoder.configure({
  codec: 'vp8',
  width: 320, height: 240,
  bitrate: 500_000,           // bits per second — the encoder's contract
  framerate: 30,
  latencyMode: 'realtime',    // no lookahead, no reordering
});
const frame = new VideoFrame(canvas, { timestamp: 0 });
encoder.encode(frame, { keyFrame: true });   // demand an I-frame
frame.close();                // frames wrap scarce memory — release promptly
await encoder.flush();        // drain: resolves when every chunk is out

Listing 5.4.1 — the whole instrument. Each submitted VideoFrame comes back through output as an EncodedVideoChunk.

The API is a callback pipeline, not a promise-per-frame: encode() queues work and returns immediately, and finished chunks arrive through the output callback in decode order. Two details matter for the problems. First, every EncodedVideoChunk declares its place in the grammar: chunk.type is either 'key' or 'delta', and chunk.byteLength is its cost — type and size are the entire GOP structure, observable at last. Second, the encoder is asynchronous and buffered, so a program that returns without await encoder.flush() inspects an empty array; the flush is the point at which all submitted frames have become chunks.

5.4.5Rate control: three contracts with the channel

An encoder does not choose how good the video looks; it obeys a bit budget, and rate control is the policy that spends it — almost entirely by steering the quantizer step from § 5.4.1. Three contracts cover practice, and WebCodecs names them in its bitrateMode member. CBR ('constant') holds the wire rate steady and lets quality float frame to frame; it is the oldest contract, born with H.261 on ISDN circuits whose p×64 kbps did not flex, and it is real-time's contract today because the target handed down by the congestion controller is a hard ceiling, not a suggestion. VBR ('variable') holds average rate over a longer horizon and spends unevenly — hoarding bits in easy scenes to splurge on hard ones; it is the file-and-VOD contract, at its best in two-pass encodes that read the whole movie before allocating. CQ ('quantizer') pins quality by fixing the quantizer per frame and lets bitrate land wherever content takes it — the archivist's and screen-recorder's contract, unusable on a link with a ceiling.

In WebRTC the CBR target is not even yours to set: the congestion controller measures the network continuously and re-issues the encoder's target, second by second. The belief that "the codec decides the bitrate" is exactly backwards — the network decides, congestion control translates, the encoder obeys. Under that contract the keyframe becomes a debt instrument: the per-frame budget is targetBps ÷ fps ÷ 8 bytes, a 31 KB keyframe borrows seven-plus frames of it at once, and a CBR controller repays by starving the deltas that follow — the familiar post-keyframe blur that sharpens over the next half second. Problem 2 makes you the rate controller.

The last knob is time itself. Encoding is a search — more candidate motion vectors, finer partitions, more lookahead yield fewer bits at equal quality — so every encoder ships a speed ladder, and real time must buy the cheap rungs. WebCodecs compresses the whole ladder into one declaration: latencyMode: 'realtime' tells the implementation to disable lookahead and frame reordering, prefer speed presets that finish within a frame interval, and optimize for a stream that must leave now — the entire argument of this lesson, folded into a configuration field.

Problems for § 5.4Check your understanding

Two problems: put a real encoder under the microscope, then predict its worst habit with arithmetic.

¶Field quirks

What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.