§ 8.4 Module 8 — Production: Forensics, Observability, and the Frontier
"In theory there is no difference between theory and practice. In practice there is."Course notes
By the end of this lesson
The failure that forced the design
ORTC lost the standards war but won the argument: WebRTC is a monolith — you get GCC, NetEQ, and a decade of opinions whether you want them or not. WebCodecs (2021) hands you the encoder; WebTransport (2022) hands you QUIC datagrams; Media over QUIC sketches the relay fabric. The frontier question is the course's first question wearing new clothes: what timing guarantees do you actually need, and what are you willing to pay for them?
The monolith critique — ORTC's revenge: WebCodecs gives raw encoder access, WebTransport gives QUIC streams and datagrams (client-server only), MoQ sketches relay-based media distribution. A working encode/transport/decode pipeline with loss recovery, and the judgment call that closes the course's thesis.
Cross-references: WebCodecs & WebTransport.
WebCodecs exposes encoder and decoder state directly; WebTransport supplies QUIC streams and datagrams to a server. Together they offer control but do not recreate WebRTC's ICE, congestion control, jitter buffering, A/V synchronization, or peer-to-peer security for free.
Engineering consequence. WebCodecs: VideoEncoder/VideoDecoder control, hardware access, decoder state discipline. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
WebTransport over HTTP/3 offers datagrams and reliable streams with genuine WritableStream backpressure via writer.ready — the thing data channels lack. The costs: it is client-to-server only, so there is no P2P, and for self-signed operation serverCertificateHashes restricts you to ECDSA certificates with a validity window of at most about two weeks, which means a certificate rotation job as part of your deployment. Safari support arrived late and remains less complete than Chrome's.
Use WebTransport for client-server media or bulk data where you control the server and want proper flow control, and keep WebRTC for anything peer-to-peer or anything needing echo cancellation. Feature-detect and keep a data channel fallback if your audience includes older Safari.
The operational claim is precise: WebTransport: QUIC streams vs datagrams, client-server only; MoQ relay/pub-sub sketch. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. WebTransport: QUIC streams vs datagrams, client-server only; MoQ relay/pub-sub sketch. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Interactive 3D instrument
The unbundling — control, and what it costs
A 3D instrument you drive yourself, one variable at a time. It needs JavaScript and WebGL, so it is not shown in this static copy of the page.
WebCodecs exposes encoder and decoder state directly; WebTransport supplies QUIC streams and datagrams to a server. Together they offer control but do not recreate WebRTC's ICE, congestion control, jitter buffering, A/V synchronization, or peer-to-peer security for free.
Engineering consequence. DIY stack (WebCodecs+WebTransport) vs RTCPeerConnection tradeoff table. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Rolling your own stack gets you codec control and no SDP, and it gets you none of the parts that took a decade: no jitter buffer, no packet loss concealment, no bandwidth estimation, no pacer, no echo cancellation, no automatic gain control. Video is the easy half — a naive buffer plus dropped frames is tolerable. Audio is where these projects fail, because acceptable concealment and clock-drift-tracking time stretching is genuinely hard and users notice every millisecond of it.
If you go this route, do it for video only and keep audio on WebRTC in a parallel peer connection, or accept that you are building NetEq. Budget for congestion control too: without it you will be an unfair, loss-inducing flow on shared links.
The operational claim is precise: The when-not-WebRTC framework; voice-AI realtime APIs as server-side peers. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. The when-not-WebRTC framework; voice-AI realtime APIs as server-side peers. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
The operational claim is precise: WebRTC-NV direction. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. WebRTC-NV direction. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
VideoEncoder defaults to quality mode, which reorders and buffers frames and destroys interactivity; you must pass latencyMode: 'realtime' in configure(). For H.264 you must also decide avc: { format: 'annexb' } versus 'avcc', which changes whether parameter sets are inline or in a description blob — get it wrong and the decoder never initialises. And the frame pool is small: every VideoFrame and EncodedVideoChunk must be close()d or capture stalls with no error.
Configure explicitly, use encoder.encodeQueueSize as your backpressure signal, and close every frame in a finally. Test with a long-running capture; leaks present as a stall after a few seconds, not as growing memory.
Start with the claim “WebCodecs: VideoEncoder/VideoDecoder control, hardware access, decoder state discipline” and turn it into a falsifiable trace. Write down the input condition, the expected state transition, the counter or packet field that should change, and the deadline by which it must change. This prevents a familiar mistake: observing a symptom after several mechanisms have already reacted and assigning it to whichever mechanism has the most recognizable name. The causal order matters. A queue grows before loss; an ICE path fails before a restart selects a new pair; a decoder loses reference state before frames stop rendering. Preserve that order in the evidence.
Now hold every unrelated variable still. Use canned timestamps, SDP, stats-shaped records, or a permission-free loopback source so the same input can be replayed. Record both media directions separately. For cumulative counters, retain two samples and divide the difference by their timestamp interval. For state machines, retain the ordered event sequence rather than the last state alone. For negotiated features, compare the offer, answer, and live stats result. These three views answer different questions: what was proposed, what was accepted, and what the browser actually used.
The first lab, Roll Your Own Pipeline, applies this discipline to a bounded implementation problem. Build canvas → VideoEncoder → a harness-provided jitter-and-loss-injecting async queue → VideoDecoder → canvas, implementing keyframe-on-corruption recovery (drop deltas after a loss until the next key, request keys via callback). Assertions: pipeline survives injected loss with ≤1s freeze; the decoder never throws (never fed deltas after loss — the assertion that teaches decoder state); GOP structure matches learner config. Feature-detected with graceful degradation. Before coding, identify the invariant hidden in that brief and the tempting shortcut the tests should reject. Then run the reference trace, perturb one boundary value, and explain why the result changes. If the code passes but the explanation cannot predict the perturbed result, the implementation is memorized rather than understood.
If latency tolerance is above roughly two seconds, LL-HLS or LL-DASH is cheaper, simpler, and scales through CDNs you already pay for. If you are moving files, HTTP is better in every dimension. What no other browser API offers is the capture-side audio stack — echo cancellation, noise suppression, automatic gain control — plus NetEq on the receive side. That is the moat, and it is the reason "we replaced WebRTC with WebSockets and WebCodecs" projects come back for the audio.
Choose by latency budget and by whether you need microphone capture in a room with a speaker. Sub-500 ms conversational audio: WebRTC. Two-second one-to-many video: HLS. Reliable bulk data: HTTP. Mixed requirements are legitimately mixed stacks.
A browser loopback is deliberately cleaner than the internet. It removes signaling latency, hostile NAT behavior, relay geography, wireless contention, device heat, and competing encoders. That makes it ideal for proving protocol and API invariants, but a dangerous basis for capacity claims. Carry the invariant into a second test with a canned hostile trace: reordered packets, a delayed candidate, a missing remote stats object, a bitrate step, or a state transition that recovers before the debounce timer. The implementation should remain correct even when optional fields are absent and events arrive more than once.
Feature detection is also part of correctness. Browser support for a codec, scalability mode, encoded transform, or transport API does not imply that the negotiated peer supports it or that hardware acceleration is active. Prefer capability checks, preserve a standards-based fallback, and label any simulation honestly. The graded result should measure the learner's algorithm; an optional live coda may demonstrate the real API without making a camera, operating-system codec, or network service a hidden test dependency.
Finally, connect the mechanism to the last topic in the lesson: “WebRTC-NV direction”. Advanced WebRTC failures cross boundaries. Congestion control changes encoder output; layer selection changes keyframe demand; end-to-end encryption constrains what an SFU can inspect; a watchdog can collide with negotiation; observability can mislabel a path if its joins are wrong. At every boundary, define ownership and an idempotent action. A retry must not create duplicate work, a stale event must not overwrite newer state, and an escalation must have a terminal cleanup path.
Use the problem set as an executable notebook. Keep a short record beside each answer: the invariant, the decisive evidence, the rejected alternative, and the production caveat. That record is more valuable than a green checkmark because it can be reused when a real call presents the same mechanism through noisier evidence.
A common error
You rarely need TURN — 10-20% of real-world sessions require relay; production without TURN (including turns: over TCP 443) is broken.
Implement the decision rules, run every assertion, then use the explanations to distinguish a wrong result from a wrong model.
Roll Your Own Pipeline. Build canvas → VideoEncoder → a harness-provided jitter-and-loss-injecting async queue → VideoDecoder → canvas, implementing keyframe-on-corruption recovery (drop deltas after a loss until the next key, request keys via callback). Assertions: pipeline survives injected loss with ≤1s freeze; the decoder never throws (never fed deltas after loss — the assertion that teaches decoder state); GOP structure matches learner config. Feature-detected with graceful degradation.
Lab 1
Roll Your Own Pipeline
Graded in the browser against 3 assertions; the editor and harness require JavaScript.
The Transport Judgment Call. Implement chooseTransport(useCase) over 10 canned product briefs (1:1 call, 100k-viewer broadcast, file sync, cloud game, IoT telemetry, voice-AI agent...) graded against a reasoned truth table; the transport choice is graded exactly and the decisive rationale is a single selection from an enumerated reason list (free-text rationale is ungraded feedback — keyword-checking is gameable). Wrong answers get the production reasoning ('WebRTC for 100k one-way viewers: WHEP/LL-HLS/MoQ economics win').
Lab 2
The Transport Judgment Call
Graded in the browser against 3 assertions; the editor and harness require JavaScript.
Check your understanding
Auto-graded check
4 auto-graded questions with an explanation for every wrong answer. Requires JavaScript. (m8-l4-quiz)
WebRTC: From First Principles · Module 8, Lesson 4