§ 8.5 Module 8 — Production: Forensics, Observability, and the Frontier
"In theory there is no difference between theory and practice. In practice there is."Course notes
By the end of this lesson
The failure that forced the design
Every component in this build has a module behind it and a historical failure underneath it. The capstone is the argument of the course made executable: if you understand why each layer exists, composing them is engineering; if you don't, it's incantation.
No new material — a guided build-through composing everything: a mock-media call with perfect negotiation, simulcast, an E2EE transform, a data channel, a live stats HUD, and a connection watchdog, graded by integration assertions. The final-exam rehearsal.
Interactive lab bench
The full stack in one page — nine layers, one call
A 3D instrument you drive yourself, one variable at a time. It needs JavaScript and WebGL, so it is not shown in this static copy of the page.
The operational claim is precise: Integration architecture: composing harness-built components. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Integration architecture: composing harness-built components. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Two peer connections in one tab connect over loopback or a local interface, where the bandwidth estimate saturates at hundreds of megabits, RTT is under a millisecond, and there is no loss. Every adaptive behaviour you want to demonstrate — degradation preference, simulcast layer switching, concealment, jitter buffer growth — never engages. Both connections also share one audio device and one echo canceller, so an unmuted loopback demo howls.
Keep every element muted except a deliberate monitor, and add impairment to make the lab honest: iceTransportPolicy: 'relay' through a real TURN server, plus netem or Network Link Conditioner. A loopback demo with no impairment teaches the wrong intuitions.
A production watchdog treats disconnected as a potentially transient condition and failed as an escalation signal. Debouncing avoids turning a short Wi-Fi roam into a negotiation storm; a bounded timer then moves from restartIce() to a full session rebuild.
Engineering consequence. Cross-component pitfalls: encoded transforms with simulcast, negotiationneeded storms, watchdog-vs-glare interactions. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
The operational claim is precise: Stats HUD with automated pathology flags. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Stats HUD with automated pathology flags. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
In a same-page lab both peer connections fire onnegotiationneeded, and if both handlers offer you get a textbook glare — InvalidStateError on setRemoteDescription, and a lab that works on reload roughly half the time. Candidate cross-wiring is also easy to invert: pc1.onicecandidate must feed pc2.addIceCandidate, and the mistake produces a connection that never leaves checking with no clue as to why.
Designate one side as the offerer and give the other side no negotiation handler at all, or implement full perfect negotiation with polite/impolite roles. Name your variables so the crossing is obvious (caller/callee, not pc1/pc2), and log the direction on every candidate.
The operational claim is precise: Final-exam format preview. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Final-exam format preview. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Start with the claim “Integration architecture: composing harness-built components” and turn it into a falsifiable trace. Write down the input condition, the expected state transition, the counter or packet field that should change, and the deadline by which it must change. This prevents a familiar mistake: observing a symptom after several mechanisms have already reacted and assigning it to whichever mechanism has the most recognizable name. The causal order matters. A queue grows before loss; an ICE path fails before a restart selects a new pair; a decoder loses reference state before frames stop rendering. Preserve that order in the evidence.
Now hold every unrelated variable still. Use canned timestamps, SDP, stats-shaped records, or a permission-free loopback source so the same input can be replayed. Record both media directions separately. For cumulative counters, retain two samples and divide the difference by their timestamp interval. For state machines, retain the ordered event sequence rather than the last state alone. For negotiated features, compare the offer, answer, and live stats result. These three views answer different questions: what was proposed, what was accepted, and what the browser actually used.
The first lab, The One-Page Stack, applies this discipline to a bounded implementation problem. Guided capstone: two-peer in-page call with mock media, perfect negotiation under one injected glare, 3-layer simulcast with a layer toggle, an E2EE transform with one key rotation, a backpressured 1 MB data-channel transfer, live stats HUD, and a watchdog surviving one injected transport kill. Harness notes: the m7-l3 simulcast answer-munging shim is provided for the 3-layer simulcast, and the transport kill is implemented as closing the far peer (consent expiry drives 'disconnected'), asserted state-based with generous browser-calibrated windows. Graded pass/fail by 12 integration assertions with partial-credit reporting. Before coding, identify the invariant hidden in that brief and the tempting shortcut the tests should reject. Then run the reference trace, perturb one boundary value, and explain why the result changes. If the code passes but the explanation cannot predict the perturbed result, the implementation is memorized rather than understood.
An AudioContext constructed at module load is suspended and produces nothing. A <video> without muted will not autoplay and play() rejects. Chrome also gates audio output for a tab with no user activation. In a lab page with several instruments this manifests as some panels working and others silently dead, which reads as a bug in the instruments.
One Start button that does await ctx.resume(), attaches all srcObjects, and calls play() on each element with a catch. Assert ctx.state === 'running' afterwards and surface a visible warning if it is not, rather than letting it fail quietly.
A browser loopback is deliberately cleaner than the internet. It removes signaling latency, hostile NAT behavior, relay geography, wireless contention, device heat, and competing encoders. That makes it ideal for proving protocol and API invariants, but a dangerous basis for capacity claims. Carry the invariant into a second test with a canned hostile trace: reordered packets, a delayed candidate, a missing remote stats object, a bitrate step, or a state transition that recovers before the debounce timer. The implementation should remain correct even when optional fields are absent and events arrive more than once.
Feature detection is also part of correctness. Browser support for a codec, scalability mode, encoded transform, or transport API does not imply that the negotiated peer supports it or that hardware acceleration is active. Prefer capability checks, preserve a standards-based fallback, and label any simulation honestly. The graded result should measure the learner's algorithm; an optional live coda may demonstrate the real API without making a camera, operating-system codec, or network service a hidden test dependency.
Finally, connect the mechanism to the last topic in the lesson: “Final-exam format preview”. Advanced WebRTC failures cross boundaries. Congestion control changes encoder output; layer selection changes keyframe demand; end-to-end encryption constrains what an SFU can inspect; a watchdog can collide with negotiation; observability can mislabel a path if its joins are wrong. At every boundary, define ownership and an idempotent action. A retry must not create duplicate work, a stale event must not overwrite newer state, and an escalation must have a terminal cleanup path.
Use the problem set as an executable notebook. Keep a short record beside each answer: the invariant, the decisive evidence, the rejected alternative, and the production caveat. That record is more valuable than a green checkmark because it can be reused when a real call presents the same mechanism through noisier evidence.
A common error
You rarely need TURN — 10-20% of real-world sessions require relay; production without TURN (including turns: over TCP 443) is broken.
Implement the decision rules, run every assertion, then use the explanations to distinguish a wrong result from a wrong model.
The One-Page Stack. Guided capstone: two-peer in-page call with mock media, perfect negotiation under one injected glare, 3-layer simulcast with a layer toggle, an E2EE transform with one key rotation, a backpressured 1 MB data-channel transfer, live stats HUD, and a watchdog surviving one injected transport kill. Harness notes: the m7-l3 simulcast answer-munging shim is provided for the 3-layer simulcast, and the transport kill is implemented as closing the far peer (consent expiry drives 'disconnected'), asserted state-based with generous browser-calibrated windows. Graded pass/fail by 12 integration assertions with partial-credit reporting.
Lab 1
The One-Page Stack
Graded in the browser against 1 assertion; the editor and harness require JavaScript.
Check your understanding
Auto-graded check
4 auto-graded questions with an explanation for every wrong answer. Requires JavaScript. (m8-l5-quiz)
What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.
Background tabs get setTimeout and requestAnimationFrame throttled to about 1 Hz, and Chrome may freeze the tab entirely after several minutes. Your stats polling stops and your graphs flatline while the peer connection keeps running perfectly — so returning to the tab shows what looks like a total media failure that recovers the moment you look at it. On iOS the opposite happens: the connection genuinely dies while your JavaScript is suspended.
Annotate your timeline with document.visibilityState transitions and mark hidden periods as "no data" rather than plotting zeros. On resume, check connectionState and call restartIce() if needed — that single handler covers both the desktop illusion and the mobile reality.
WebRTC: From First Principles · Module 8, Lesson 5