§ 8.2 Module 8 — Production: Forensics, Observability, and the Frontier
"In theory there is no difference between theory and practice. In practice there is."Course notes
By the end of this lesson
The failure that forced the design
The failure catalog is the senior engineer's real curriculum: calls that die at exactly t=30s (a stateful firewall's UDP timer — the periodicity is the giveaway); DTLS handshakes that complete on WiFi and hang on VPN (MTU blackhole fragmenting the handshake); a hotel router's ALG helpfully rewriting SDP it half-understands. None of these are in the spec. All of them are in production.
Real bugs reproduced live on loopback — fingerprint tamper killing DTLS, payload-type remap producing black video, ufrag corruption stalling ICE, glare races sticking signalingState — plus canned incident bundles for the pathologies no browser will reproduce on demand (MTU blackholes, 30-second UDP timeouts, ALG-mangled SDP).
The operational claim is precise: Live-reproduced bugs: fingerprint tamper → DTLS fail; PT remap → black video; ufrag corruption → ICE stall; glare race → stuck state; muted-track vs no-keyframe black video. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Live-reproduced bugs: fingerprint tamper → DTLS fail; PT remap → black video; ufrag corruption → ICE stall; glare race → stuck state; muted-track vs no-keyframe black video. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Interactive minigame
The failure catalog — inject it, read it, repair it
A 3D instrument you drive yourself, one variable at a time. It needs JavaScript and WebGL, so it is not shown in this static copy of the page.
tc qdisc add dev eth0 root netem delay 100ms 20ms adds jitter by giving each packet an independent delay, which means packets overtake each other. Reordering is a genuinely different pathology from jitter — it triggers different NACK behaviour and different jitter buffer responses — so your "100 ms jitter" test is actually a reordering test and your results are unreproducible. Adding rate or the delay ... distribution plus a rate limit restores ordering.
Use netem delay 100ms 20ms rate 2mbit, or an explicit reorder 0%, and verify with a capture that sequence numbers arrive in order. Also remember loopback netem affects both directions, so your configured 5 % loss is closer to 10 % end to end.
The operational claim is precise: Canned pathologies: MTU blackhole, 30s UDP timeout, ALG mangling, iOS backgrounding. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Canned pathologies: MTU blackhole, 30s UDP timeout, ALG mangling, iOS backgrounding. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
The operational claim is precise: Decisive-evidence discipline: one key per diagnosis. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Decisive-evidence discipline: one key per diagnosis. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Two blind spots account for most unreproducible field reports. First, iceTransportPolicy: 'relay' confirms your TURN server allocates and forwards, but it does not test the thing that breaks in production — whether ICE discovers that UDP is blocked, how long the fallback to TURN over TLS takes, and whether your certificate and proxy configuration survive it. Second, your home or office network has working multicast, so Chrome's .local candidates resolve and same-LAN peers connect directly; the customer's enterprise WLAN blocks client-to-client multicast and the identical code relays, reported to you as "it's slow and our bandwidth spiked".
Keep an environment with UDP blocked at the firewall (a container with an egress rule permitting only TCP 443 is enough) and run the full suite there — it is the highest-yield WebRTC test environment and almost nobody has one. Add a multicast-blocked case, use chrome://flags/#enable-webrtc-hide-local-ips-with-mdns to A/B whether obfuscation is implicated, and log candidateType for every production session so the next spike explains itself.
The operational claim is precise: webrtc-internals event log and about:webrtc reading. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. webrtc-internals event log and about:webrtc reading. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Start with the claim “Live-reproduced bugs: fingerprint tamper → DTLS fail; PT remap → black video; ufrag corruption → ICE stall; glare race → stuck state; muted-track vs no-keyframe black video” and turn it into a falsifiable trace. Write down the input condition, the expected state transition, the counter or packet field that should change, and the deadline by which it must change. This prevents a familiar mistake: observing a symptom after several mechanisms have already reacted and assigning it to whichever mechanism has the most recognizable name. The causal order matters. A queue grows before loss; an ICE path fails before a restart selects a new pair; a decoder loses reference state before frames stop rendering. Preserve that order in the evidence.
Now hold every unrelated variable still. Use canned timestamps, SDP, stats-shaped records, or a permission-free loopback source so the same input can be replayed. Record both media directions separately. For cumulative counters, retain two samples and divide the difference by their timestamp interval. For state machines, retain the ordered event sequence rather than the last state alone. For negotiated features, compare the offer, answer, and live stats result. These three views answer different questions: what was proposed, what was accepted, and what the browser actually used.
The first lab, Name That Bug (Live Edition), applies this discipline to a bounded implementation problem. The harness reproduces six bugs live on loopback pairs (fingerprint tamper, PT remap, ufrag corruption, glare race, muted track, missing keyframe); the learner writes identifyBug(evidence) examining SDP, states, and stats to classify each. Assertions per case; hints teach the discriminating observation (e.g., dtlsState 'failed' vs ICE 'checking' forever). Before coding, identify the invariant hidden in that brief and the tempting shortcut the tests should reject. Then run the reference trace, perturb one boundary value, and explain why the result changes. If the code passes but the explanation cannot predict the perturbed result, the implementation is memorized rather than understood.
A large fraction of device-specific video bugs — green frames, garbage rows, wrong resolutions, missing simulcast layers, keyframes only on request — are hardware encoder or decoder bugs, and they are not reproducible on any other device. chrome://flags/#disable-accelerated-video-encode and #disable-accelerated-video-decode switch to the software path in one restart and answer the question definitively before you read any of your own code.
Make this step one of your device-bug playbook. If software fixes it, the remediation is a per-device-class codec blocklist or a layer-count reduction, not an application change. Record device model and GPU alongside your telemetry so the blocklist can be data-driven.
A browser loopback is deliberately cleaner than the internet. It removes signaling latency, hostile NAT behavior, relay geography, wireless contention, device heat, and competing encoders. That makes it ideal for proving protocol and API invariants, but a dangerous basis for capacity claims. Carry the invariant into a second test with a canned hostile trace: reordered packets, a delayed candidate, a missing remote stats object, a bitrate step, or a state transition that recovers before the debounce timer. The implementation should remain correct even when optional fields are absent and events arrive more than once.
Feature detection is also part of correctness. Browser support for a codec, scalability mode, encoded transform, or transport API does not imply that the negotiated peer supports it or that hardware acceleration is active. Prefer capability checks, preserve a standards-based fallback, and label any simulation honestly. The graded result should measure the learner's algorithm; an optional live coda may demonstrate the real API without making a camera, operating-system codec, or network service a hidden test dependency.
Finally, connect the mechanism to the last topic in the lesson: “webrtc-internals event log and about:webrtc reading”. Advanced WebRTC failures cross boundaries. Congestion control changes encoder output; layer selection changes keyframe demand; end-to-end encryption constrains what an SFU can inspect; a watchdog can collide with negotiation; observability can mislabel a path if its joins are wrong. At every boundary, define ownership and an idempotent action. A retry must not create duplicate work, a stale event must not overwrite newer state, and an escalation must have a terminal cleanup path.
Use the problem set as an executable notebook. Keep a short record beside each answer: the invariant, the decisive evidence, the rejected alternative, and the production caveat. That record is more valuable than a green checkmark because it can be reused when a real call presents the same mechanism through noisier evidence.
A common error
You rarely need TURN — 10-20% of real-world sessions require relay; production without TURN (including turns: over TCP 443) is broken.
Implement the decision rules, run every assertion, then use the explanations to distinguish a wrong result from a wrong model.
Name That Bug (Live Edition). The harness reproduces six bugs live on loopback pairs (fingerprint tamper, PT remap, ufrag corruption, glare race, muted track, missing keyframe); the learner writes identifyBug(evidence) examining SDP, states, and stats to classify each. Assertions per case; hints teach the discriminating observation (e.g., dtlsState 'failed' vs ICE 'checking' forever).
Lab 1
Name That Bug (Live Edition)
Graded in the browser against 3 assertions; the editor and harness require JavaScript.
The Incident Bundles. Implement diagnoseIncident(bundle) over six canned bundles (ICE event log + stats timeline + environment notes), returning {cause, evidenceKey} — cause graded exactly, evidenceKey a single selection from an enumerated key list (multi-key answers penalized; free-text rationale ungraded). Assertions: 6/6 with correct evidence ('failed at t=30s every time' → stateful UDP timeout; 'DTLS stuck in connecting, fine without VPN' → MTU blackhole).
Lab 2
The Incident Bundles
Graded in the browser against 2 assertions; the editor and harness require JavaScript.
Check your understanding
Auto-graded check
4 auto-graded questions with an explanation for every wrong answer. Requires JavaScript. (m8-l2-quiz)
What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.
Encoder overload is recognisable without a profiler: qualityLimitationReason becomes cpu, totalEncodeTime/framesEncoded exceeds roughly 15 ms at 30 fps (you have 33 ms per frame and you need the rest for everything else), outbound-rtp.framesPerSecond falls below media-source.framesPerSecond, and qualityLimitationResolutionChanges starts climbing. On laptops it appears two to five minutes into a call as thermal throttling engages, which is why short tests never catch it.
Run at least one ten-minute soak test per release on your lowest-spec target device and graph those four values. Remediate by reducing layers, lowering maxFramerate, or switching off a software codec, in that order.
WebRTC: From First Principles · Module 8, Lesson 2