§ 8.1 Module 8 — Production: Forensics, Observability, and the Frontier
"In theory there is no difference between theory and practice. In practice there is."Course notes
By the end of this lesson
The failure that forced the design
Every production WebRTC team eventually builds the same tool: something that turns a getStats forest into one sentence — 'relayed over TCP 443, VP8 at 400 kbps, RTT 210 ms, receiver CPU-limited.' This lesson builds that tool, because the joins (by id reference, across eight dictionary types) are exactly where dashboards silently lie.
The complete getStats object graph joined by id references — outbound-rtp → media-source → codec → transport → candidate-pair → candidates — run live and then against canned production-incident dumps, including the TURN-443 fingerprint from Module 3's war story.
The operational claim is precise: The id-reference graph and its eight dictionary types. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. The id-reference graph and its eight dictionary types. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Interactive minigame
The full join — close four incidents on the evidence
A 3D instrument you drive yourself, one variable at a time. It needs JavaScript and WebGL, so it is not shown in this static copy of the page.
The path is inbound-rtp → codecId → codec, and inbound-rtp → transportId → transport → selectedCandidatePairId → candidate-pair → localCandidateId/remoteCandidateId. Because BUNDLE puts every m-line on one transport, there is exactly one transport and one candidate pair for the whole session — so there is no per-stream RTT, no per-stream bandwidth estimate, and no way to say "the video path is worse than the audio path". They are the same path.
Build the join once into a helper that returns a flat per-stream record with transport fields denormalised onto it. Resist the urge to report per-stream RTT; report per-transport RTT and per-stream loss and delay, which are the things that genuinely differ.
ICE candidates describe possible transport addresses, while connectivity checks prove which pairs actually work. The selected pair—not the most attractive-looking candidate—determines whether media is direct, server-reflexive, or relayed.
Engineering consequence. Digest construction: selected pair candidate types, codec mimeTypes, per-track bitrates. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
The operational claim is precise: Dump pathologies: early samples, missing remote stats, multiple transports. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Dump pathologies: early samples, missing remote stats, multiple transports. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
outbound-rtp.bytesSent counts RTP payload only; headerBytesSent is a separate counter. transport.bytesSent includes RTP, RTCP, STUN, and DTLS. So the sum of your streams' bytesSent is systematically 10–20 % below the transport figure, and the gap widens with small packets (audio) and with relay framing. Many hours have been spent hunting the "missing bytes", which are headers doing their job.
Use bytesSent + headerBytesSent when comparing against transport totals or against network measurements, and use bytesSent alone when comparing against codec bitrate. Label both in your dashboards so nobody has to rediscover the difference.
The operational claim is precise: Triage methodology: connectivity → media presence → quality → resources. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Triage methodology: connectivity → media presence → quality → resources. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Start with the claim “The id-reference graph and its eight dictionary types” and turn it into a falsifiable trace. Write down the input condition, the expected state transition, the counter or packet field that should change, and the deadline by which it must change. This prevents a familiar mistake: observing a symptom after several mechanisms have already reacted and assigning it to whichever mechanism has the most recognizable name. The causal order matters. A queue grows before loss; an ICE path fails before a restart selects a new pair; a decoder loses reference state before frames stop rendering. Preserve that order in the evidence.
Now hold every unrelated variable still. Use canned timestamps, SDP, stats-shaped records, or a permission-free loopback source so the same input can be replayed. Record both media directions separately. For cumulative counters, retain two samples and divide the difference by their timestamp interval. For state machines, retain the ordered event sequence rather than the last state alone. For negotiated features, compare the offer, answer, and live stats result. These three views answer different questions: what was proposed, what was accepted, and what the browser actually used.
The first lab, Build the StatsDigest, applies this discipline to a bounded implementation problem. Implement a StatsDigest walker joining outbound-rtp → media-source → codec → transport → candidate-pair → local/remote-candidate by id references, run live on loopback and then on three canned production dumps. Assertions: complete join on the live PC; dump #2 correctly reports the relay-TCP-443 path; graceful handling of an early-call dump with absent remote-inbound-rtp; a trapped hint for assuming exactly one transport (the unbundled legacy case). Before coding, identify the invariant hidden in that brief and the tempting shortcut the tests should reject. Then run the reference trace, perturb one boundary value, and explain why the result changes. If the code passes but the explanation cannot predict the perturbed result, the implementation is memorized rather than understood.
codec stats objects are per-payload-type per-direction and appear only while in use, so they materialise and vanish over a call. inbound-rtp.codecId can change if the sender switches codecs, and the object it pointed at may be gone by the time you dereference it in a later snapshot. It is also the only place you learn clockRate, which you need for any timestamp arithmetic — 90000 for video, 48000 for Opus, 8000 for G.711.
Resolve codecId within the same report you read it from, never across snapshots, and carry mimeType and clockRate forward into your flattened record. Log codec changes as timeline events; an unexpected switch to a different codec mid-call is a real finding.
A browser loopback is deliberately cleaner than the internet. It removes signaling latency, hostile NAT behavior, relay geography, wireless contention, device heat, and competing encoders. That makes it ideal for proving protocol and API invariants, but a dangerous basis for capacity claims. Carry the invariant into a second test with a canned hostile trace: reordered packets, a delayed candidate, a missing remote stats object, a bitrate step, or a state transition that recovers before the debounce timer. The implementation should remain correct even when optional fields are absent and events arrive more than once.
Feature detection is also part of correctness. Browser support for a codec, scalability mode, encoded transform, or transport API does not imply that the negotiated peer supports it or that hardware acceleration is active. Prefer capability checks, preserve a standards-based fallback, and label any simulation honestly. The graded result should measure the learner's algorithm; an optional live coda may demonstrate the real API without making a camera, operating-system codec, or network service a hidden test dependency.
Finally, connect the mechanism to the last topic in the lesson: “Triage methodology: connectivity → media presence → quality → resources”. Advanced WebRTC failures cross boundaries. Congestion control changes encoder output; layer selection changes keyframe demand; end-to-end encryption constrains what an SFU can inspect; a watchdog can collide with negotiation; observability can mislabel a path if its joins are wrong. At every boundary, define ownership and an idempotent action. A retry must not create duplicate work, a stale event must not overwrite newer state, and an escalation must have a terminal cleanup path.
Use the problem set as an executable notebook. Keep a short record beside each answer: the invariant, the decisive evidence, the rejected alternative, and the production caveat. That record is more valuable than a green checkmark because it can be reused when a real call presents the same mechanism through noisier evidence.
A common error
You rarely need TURN — 10-20% of real-world sessions require relay; production without TURN (including turns: over TCP 443) is broken.
Implement the decision rules, run every assertion, then use the explanations to distinguish a wrong result from a wrong model.
Build the StatsDigest. Implement a StatsDigest walker joining outbound-rtp → media-source → codec → transport → candidate-pair → local/remote-candidate by id references, run live on loopback and then on three canned production dumps. Assertions: complete join on the live PC; dump #2 correctly reports the relay-TCP-443 path; graceful handling of an early-call dump with absent remote-inbound-rtp; a trapped hint for assuming exactly one transport (the unbundled legacy case).
Lab 1
Build the StatsDigest
Graded in the browser against 1 assertion; the editor and harness require JavaScript.
Check: The Triage Tree. Ordering items: given a symptom, choose the correct next artifact to inspect (ICE states before inbound-rtp before qualityLimitationReason), including the connectionState-connected-but-black-video gap.
Check your understanding
Auto-graded check
4 auto-graded questions with an explanation for every wrong answer. Requires JavaScript. (m8-l1-ex2)
What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.
Counters reset when the SSRC changes (remove/add track, some ICE restarts, codec renegotiation) and when a new candidate pair is selected. The result is a negative difference between consecutive snapshots. Clamping to zero hides a real event; treating it as data produces impossible bitrates and negative loss rates that then poison every percentile you compute downstream.
Detect negative deltas explicitly, discard that window, restart the accumulator, and emit a marker so the discontinuity is visible on the timeline. Key all accumulators on the stat object's id plus ssrc so a new object cannot be mistaken for a continuation of the old one.
WebRTC: From First Principles · Module 8, Lesson 1