§ 6.1 Module 6 — Surviving the Network: Loss, Congestion, and Jitter
"The network is reliable." — Fallacy #1 of distributed computingCourse notes
By the end of this lesson
The failure that forced the design
TCP's answer to loss — retransmit everything — is wrong for media, but selective retransmission is right sometimes: if the round trip is short enough, a NACKed packet can still make the jitter-buffer deadline. RFC 4585's feedback messages turned RTCP into a repair channel, and the escalation ladder ends at the nuclear option: request a whole new keyframe, and pay its bandwidth bomb.
Retransmission's latency budget (when NACK is affordable), RTX's separate SSRC, RED/ULPFEC/FlexFEC as pay-forward insurance, and the escalation ladder PLI → FIR — why one lost packet can freeze video for a second while audio just blips.
Cross-references: RTCP Feedback: NACK, RTX, PLI/FIR, FEC, REMB/TWCC.
A retransmission is useful only when the NACK, repaired packet, and decode work all finish before playout. RTX gives retransmissions a separate SSRC and sequence space, so receivers can distinguish repair traffic without corrupting the original stream's loss accounting.
Engineering consequence. NACK affordability: RTT vs jitter-buffer depth arithmetic. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Retransmissions ride a separate SSRC bound by a=ssrc-group:FID with their own payload type declared as a=fmtp:<pt> apt=<media-pt>. Depending on browser version those bytes fold into the media outbound-rtp (as retransmittedBytesSent, already included in bytesSent) or appear as a separate object. Under 5 % loss retransmissions can add 10–20 % to your wire rate, so a naive bitrate graph shows the bitrate rising as the network degrades, which reads as a bug in your rate control.
Graph original media rate as bytesSent - retransmittedBytesSent and plot retransmission rate as its own series. A rising retransmission ratio is one of the earliest, cleanest indicators of a deteriorating path — earlier than loss, because NACK repairs it.
The operational claim is precise: RTX (RFC 4588): separate SSRC, re-numbered sequence space. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. RTX (RFC 4588): separate SSRC, re-numbered sequence space. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Interactive minigame
Four repairs, one deadline
A 3D instrument you drive yourself, one variable at a time. It needs JavaScript and WebGL, so it is not shown in this static copy of the page.
Forward error correction spends bandwidth before a loss occurs. That trade is rational on paths whose round trip is longer than the playout reserve: recovery can be local and immediate, but the parity overhead is paid even when the network is clean.
Engineering consequence. RED, ULPFEC, FlexFEC; Opus in-band FEC callback to M5. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Video negotiates a=rtcp-fb:<pt> nack and RTX and genuinely recovers packets. Audio, in the configurations browsers actually use, relies on Opus in-band FEC plus NetEq concealment — so lost audio is never recovered, only fabricated. This is why packetsLost for audio only ever goes up while video's effective loss after repair is much lower than the reported figure, and why the same 3 % loss sounds bad while video looks fine.
Judge audio by concealment, not loss. If audio quality on lossy links is business-critical, RED redundancy (payload type 63, red/48000/2) is the lever that actually helps, at roughly double the audio bitrate — cheap next to video.
PLI asks for a new picture without pretending to identify a particular missing packet; FIR is a stronger, sequenced request. Both are expensive because a keyframe is much larger than a delta frame, so implementations coalesce requests and rate-limit escalation.
Engineering consequence. PLI vs FIR semantics; keyframe storms as the anti-pattern. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
The operational claim is precise: REMB and transport-cc as feedback messages — bridge to L6.2. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. REMB and transport-cc as feedback messages — bridge to L6.2. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
flexfec-03 has sat behind a flag in Chrome for years and is not enabled by default, and ULPFEC/RED for video is largely vestigial. So "we turned on FEC" almost always means RED-wrapped Opus FEC for audio, and video has no forward error correction at all — its only repair mechanism is NACK/RTX, which costs a round trip and is useless when RTT exceeds the jitter buffer depth. On a 300 ms satellite or double-relay path, video loss is simply unrecoverable.
Check fecPacketsReceived: if it is zero on video, you have no FEC regardless of what the SDP suggests. On high-RTT paths, buy resilience with temporal SVC (losing a top temporal layer costs frame rate, not decodability) rather than expecting retransmission to work.
Start with the claim “NACK affordability: RTT vs jitter-buffer depth arithmetic” and turn it into a falsifiable trace. Write down the input condition, the expected state transition, the counter or packet field that should change, and the deadline by which it must change. This prevents a familiar mistake: observing a symptom after several mechanisms have already reacted and assigning it to whichever mechanism has the most recognizable name. The causal order matters. A queue grows before loss; an ICE path fails before a restart selects a new pair; a decoder loses reference state before frames stop rendering. Preserve that order in the evidence.
Now hold every unrelated variable still. Use canned timestamps, SDP, stats-shaped records, or a permission-free loopback source so the same input can be replayed. Record both media directions separately. For cumulative counters, retain two samples and divide the difference by their timestamp interval. For state machines, retain the ordered event sequence rather than the last state alone. For negotiated features, compare the offer, answer, and live stats result. These three views answer different questions: what was proposed, what was accepted, and what the browser actually used.
The first lab, Build the NackTracker, applies this discipline to a bounded implementation problem. Implement a receiver-side NackTracker: consume a lossy, reordered sequence stream; emit NACKs with reorder tolerance (~2 packets) and RTT-based retry timers; process arriving RTX; emit PLI when a gap outlives the jitter-buffer depth. Assertions: NACKs exactly the gap seqs, no NACK spam, no NACK for later-received packets, PLI only when unrecoverable — the PLI-per-loss keyframe-storm anti-pattern is a trapped hint. Before coding, identify the invariant hidden in that brief and the tempting shortcut the tests should reject. Then run the reference trace, perturb one boundary value, and explain why the result changes. If the code passes but the explanation cannot predict the perturbed result, the implementation is memorized rather than understood.
Keyframes are often 10x a P-frame. Fifty subscribers joining an SFU stream in the same second each want a keyframe; if the SFU forwards every PLI, the publisher's encoder emits a burst of IDRs that consumes its entire bandwidth estimate, which triggers loss, which triggers more PLIs. Browsers rate-limit their own keyframe generation, which usually saves you, but a badly behaved SFU or a native publisher without that limiter produces a visible quality collapse for everyone at exactly the moment the meeting starts.
Cache the last keyframe per layer at the SFU and serve new subscribers from cache. Coalesce upstream PLIs with a floor of several hundred milliseconds. Watch pliCount and keyFramesEncoded in the publisher's outbound-rtp — a keyframe rate above roughly one per second is pathological.
A browser loopback is deliberately cleaner than the internet. It removes signaling latency, hostile NAT behavior, relay geography, wireless contention, device heat, and competing encoders. That makes it ideal for proving protocol and API invariants, but a dangerous basis for capacity claims. Carry the invariant into a second test with a canned hostile trace: reordered packets, a delayed candidate, a missing remote stats object, a bitrate step, or a state transition that recovers before the debounce timer. The implementation should remain correct even when optional fields are absent and events arrive more than once.
Feature detection is also part of correctness. Browser support for a codec, scalability mode, encoded transform, or transport API does not imply that the negotiated peer supports it or that hardware acceleration is active. Prefer capability checks, preserve a standards-based fallback, and label any simulation honestly. The graded result should measure the learner's algorithm; an optional live coda may demonstrate the real API without making a camera, operating-system codec, or network service a hidden test dependency.
Finally, connect the mechanism to the last topic in the lesson: “REMB and transport-cc as feedback messages — bridge to L6.2”. Advanced WebRTC failures cross boundaries. Congestion control changes encoder output; layer selection changes keyframe demand; end-to-end encryption constrains what an SFU can inspect; a watchdog can collide with negotiation; observability can mislabel a path if its joins are wrong. At every boundary, define ownership and an idempotent action. A retry must not create duplicate work, a stale event must not overwrite newer state, and an escalation must have a terminal cleanup path.
Use the problem set as an executable notebook. Keep a short record beside each answer: the invariant, the decisive evidence, the rejected alternative, and the production caveat. That record is more valuable than a green checkmark because it can be reused when a real call presents the same mechanism through noisier evidence.
A common error
onicecandidate firing with null means failure — it signals end of gathering.
Implement the decision rules, run every assertion, then use the explanations to distinguish a wrong result from a wrong model.
Build the NackTracker. Implement a receiver-side NackTracker: consume a lossy, reordered sequence stream; emit NACKs with reorder tolerance (~2 packets) and RTT-based retry timers; process arriving RTX; emit PLI when a gap outlives the jitter-buffer depth. Assertions: NACKs exactly the gap seqs, no NACK spam, no NACK for later-received packets, PLI only when unrecoverable — the PLI-per-loss keyframe-storm anti-pattern is a trapped hint.
Lab 1
Build the NackTracker
Graded in the browser against 3 assertions; the editor and harness require JavaScript.
Verify the Repair Contract, Live. On a loopback pair, parse the negotiated rtcp-fb capabilities per payload type from the answer SDP into a structured map, then cross-validate that map against live getStats by joining stats entries to codecs via codecId — asserting nackCount/pliCount appear exactly where nack/pli were negotiated (real parsing plus the SDP↔stats correspondence, not a key-existence check). A quiz item asks which repair applies when RTT is 400 ms (FEC/concealment, not NACK).
Lab 2
Verify the Repair Contract, Live
Graded in the browser against 2 assertions; the editor and harness require JavaScript.
Check your understanding
Auto-graded check
4 auto-graded questions with an explanation for every wrong answer. Requires JavaScript. (m6-l1-quiz)
WebRTC: From First Principles · Module 6, Lesson 1