§ 6.5 Module 6 — Surviving the Network: Loss, Congestion, and Jitter
"The network is reliable." — Fallacy #1 of distributed computingCourse notes
By the end of this lesson
The failure that forced the design
War stories: the Wi-Fi→LTE cliff — RTT spike, availableOutgoingBitrate crater, 4-second freeze until a debounced ICE restart landed; the calls that died at exactly t=30s (a stateful firewall's UDP timeout — the giveaway was the periodicity); one-way audio from asymmetric NAT filtering. Production WebRTC is the discipline of recognizing these signatures fast.
WiFi roaming and ICE restarts, LTE handover, VPNs forcing TURN, carrier-grade NAT, bufferbloat's smoking gun, one-way audio — six anonymized stats timelines to diagnose, plus a production-grade connection watchdog built live.
The operational claim is precise: Network-change handling: disconnected vs failed; debounced restartIce; escalation to full reconnect. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Network-change handling: disconnected vs failed; debounced restartIce; escalation to full reconnect. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Interactive minigame
Name that pathology — diagnosis from the instrument wall
A 3D instrument you drive yourself, one variable at a time. It needs JavaScript and WebGL, so it is not shown in this static copy of the page.
When connectionState is connected and one direction has no media, it is almost always a DTLS role problem, an expired TURN permission, or a firewall that permits the flow it initiated and drops the reverse. The distinguishing observation is that transport.bytesReceived keeps rising (STUN consent checks are arriving) while inbound-rtp.packetsReceived is flat — the network path is alive and the media is not. If inbound-rtp does not exist at all, the m-line was never negotiated for receive.
Check in order: transport.dtlsState and srtpCipher present; the relayed case's 5-minute onset (TURN permission); then packet counters on both ends. Do not start with the network — start with whether RTP is being sent at all, per outbound-rtp.packetsSent on the far side.
The operational claim is precise: Bufferbloat signature: RTT under load; VPN/MTU pathologies preview. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Bufferbloat signature: RTT under load; VPN/MTU pathologies preview. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
The operational claim is precise: CGNAT, UDP throttling, one-way audio (asymmetric filtering). Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. CGNAT, UDP throttling, one-way audio (asymmetric filtering). Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Three distinct pathologies share the "frozen video" symptom and are separated by two counters. framesReceived rising with framesDecoded flat and keyFramesDecoded === 0 means no keyframe ever arrived. framesDecoded rising with a solid-colour render means the hardware decoder failed and did not fall back — common on Android and on some Intel iGPUs with VP9 at unusual resolutions. framesDropped rising with qualityLimitationReason: 'cpu' means the machine is out of headroom.
Log all four counters together; the triage is mechanical once you do. For suspected decoder faults, toggle chrome://flags/#disable-accelerated-video-decode to confirm, then blocklist the affected codec for that device class.
The operational claim is precise: webrtc-internals dump autopsy methodology. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. webrtc-internals dump autopsy methodology. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Start with the claim “Network-change handling: disconnected vs failed; debounced restartIce; escalation to full reconnect” and turn it into a falsifiable trace. Write down the input condition, the expected state transition, the counter or packet field that should change, and the deadline by which it must change. This prevents a familiar mistake: observing a symptom after several mechanisms have already reacted and assigning it to whichever mechanism has the most recognizable name. The causal order matters. A queue grows before loss; an ICE path fails before a restart selects a new pair; a decoder loses reference state before frames stop rendering. Preserve that order in the evidence.
Now hold every unrelated variable still. Use canned timestamps, SDP, stats-shaped records, or a permission-free loopback source so the same input can be replayed. Record both media directions separately. For cumulative counters, retain two samples and divide the difference by their timestamp interval. For state machines, retain the ordered event sequence rather than the last state alone. For negotiated features, compare the offer, answer, and live stats result. These three views answer different questions: what was proposed, what was accepted, and what the browser actually used.
The first lab, Diagnose Six Sick Calls, applies this discipline to a bounded implementation problem. Implement diagnose(timeline) over six anonymized stats-timeline JSON dumps, returning {cause, evidenceKey}: cause is graded exactly against the categorical answer, and evidenceKey must be exactly one selection from an enumerated list of known stat/log keys (e.g. 'qualityLimitationReason') — multi-key shotgun answers are penalized; free-text rationale is collected as ungraded feedback. Assertions per case with hints referencing the exact telltale metric. Before coding, identify the invariant hidden in that brief and the tempting shortcut the tests should reject. Then run the reference trace, perturb one boundary value, and explain why the result changes. If the code passes but the explanation cannot predict the perturbed result, the implementation is memorized rather than understood.
Deep packet inspection appliances that permit STUN and small UDP datagrams while dropping or rate-limiting large ones let audio (about 200-byte packets) through perfectly and destroy video, because keyframes are large and bursty. You never decode a keyframe, so video never starts at all, and the PLI loop runs forever. Because audio is flawless, users and support both conclude the problem is video-specific in your app.
Correlate the failure with network, not with device or browser: if every affected user shares an egress, it is the network. Force TURN/TLS on 443 for those users, which reframes the traffic as a single TLS stream the appliance will pass. Confirm by testing with iceTransportPolicy: 'relay' against a TLS-only TURN configuration.
A browser loopback is deliberately cleaner than the internet. It removes signaling latency, hostile NAT behavior, relay geography, wireless contention, device heat, and competing encoders. That makes it ideal for proving protocol and API invariants, but a dangerous basis for capacity claims. Carry the invariant into a second test with a canned hostile trace: reordered packets, a delayed candidate, a missing remote stats object, a bitrate step, or a state transition that recovers before the debounce timer. The implementation should remain correct even when optional fields are absent and events arrive more than once.
Feature detection is also part of correctness. Browser support for a codec, scalability mode, encoded transform, or transport API does not imply that the negotiated peer supports it or that hardware acceleration is active. Prefer capability checks, preserve a standards-based fallback, and label any simulation honestly. The graded result should measure the learner's algorithm; an optional live coda may demonstrate the real API without making a camera, operating-system codec, or network service a hidden test dependency.
Finally, connect the mechanism to the last topic in the lesson: “webrtc-internals dump autopsy methodology”. Advanced WebRTC failures cross boundaries. Congestion control changes encoder output; layer selection changes keyframe demand; end-to-end encryption constrains what an SFU can inspect; a watchdog can collide with negotiation; observability can mislabel a path if its joins are wrong. At every boundary, define ownership and an idempotent action. A retry must not create duplicate work, a stale event must not overwrite newer state, and an escalation must have a terminal cleanup path.
Use the problem set as an executable notebook. Keep a short record beside each answer: the invariant, the decisive evidence, the rejected alternative, and the production caveat. That record is more valuable than a green checkmark because it can be reused when a real call presents the same mechanism through noisier evidence.
A common error
onicecandidate firing with null means failure — it signals end of gathering.
Implement the decision rules, run every assertion, then use the explanations to distinguish a wrong result from a wrong model.
Diagnose Six Sick Calls. Implement diagnose(timeline) over six anonymized stats-timeline JSON dumps, returning {cause, evidenceKey}: cause is graded exactly against the categorical answer, and evidenceKey must be exactly one selection from an enumerated list of known stat/log keys (e.g. 'qualityLimitationReason') — multi-key shotgun answers are penalized; free-text rationale is collected as ungraded feedback. Assertions per case with hints referencing the exact telltale metric.
Lab 1
Diagnose Six Sick Calls
Graded in the browser against 3 assertions; the editor and harness require JavaScript.
The Connection Watchdog. Implement a production-grade watchdog: detect disconnected, debounce 2s, trigger restartIce(), escalate to full reconnect on timeout. Grading is split: debounce/escalation timing logic runs against a harness-provided synthetic state-event source (deterministic schedules including sub-2s blips — no restart on blips, escalation ordering asserted), then one real kill is performed by closing the far peer — consent freshness (RFC 7675) stops being answered and iceConnectionState reaches 'disconnected' on browser-specific timers — with a single generous state-based assertion (restart attempted within a ~30s browser-calibrated window; reached disconnected → debounced → restartIce → escalation, never wall-clock-tight). Hints catch restart-on-first-disconnected flap amplification.
Lab 2
The Connection Watchdog
Graded in the browser against 3 assertions; the editor and harness require JavaScript.
Check your understanding
Auto-graded check
4 auto-graded questions with an explanation for every wrong answer. Requires JavaScript. (m6-l5-quiz)
What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.
The "works for everyone except one customer" class has a short suspect list: a VPN or PPPoE link with a 1400-byte MTU that fragments your larger packets, or a local security product (several consumer antivirus suites) that intercepts TLS and breaks the TURN/TLS handshake in a way indistinguishable from a firewall block. Both produce partial, intermittent, load-dependent failure that no server-side log explains.
Ask for three things before debugging code: MTU (a ping with the don't-fragment bit and decreasing sizes), whether a VPN is active, and whether any security software does HTTPS inspection. Disabling the product for a single test is a definitive answer and takes the user a minute.
WebRTC: From First Principles · Module 6, Lesson 5