§ 7.4 Module 7 — From Two to Two Thousand: Data, Topologies, and Trust
March 2020: every mesh-topology product melted at four participants, and the architecture that won can read your media.Course notes
By the end of this lesson
The failure that forced the design
In 2020, Zoom's 'end-to-end encrypted' marketing collided with the fact that its servers held the keys — a scandal that made the industry define terms. A standard WebRTC SFU is a lawful-intercept point by architecture: it terminates DTLS-SRTP on every hop. Insertable Streams gave JavaScript a hook inside the pipeline — between encoder and packetizer — so applications can encrypt frames with keys the SFU never sees. SFrame (RFC 9605) standardized the envelope.
The SFU reopened the Snowden problem: it terminates DTLS-SRTP and sees every encoded frame. Insertable Streams (encoded transforms) as the escape hatch, SFrame's frame-level encryption, the key-distribution problem, and what E2EE costs — proven experientially with real transforms on a live connection.
Cross-references: E2EE: Insertable Streams & SFrame.
The operational claim is precise: Hop-by-hop vs end-to-end; what the SFU sees (encoded frames, timing, sizes). Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Hop-by-hop vs end-to-end; what the SFU sees (encoded frames, timing, sizes). Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Chrome exposes RTCRtpSender.createEncodedStreams() / RTCRtpReceiver.createEncodedStreams(), which requires { encodedInsertableStreams: true } in the RTCPeerConnection constructor and runs the transform wherever you read the streams. The specification and Firefox/Safari implement RTCRtpScriptTransform, which is constructed with a Worker and assigned to sender.transform. They are not shims of each other: the worker-based API forces your crypto off the main thread and has a different event surface. Any real E2EE deployment ships both paths.
Feature-detect ('transform' in RTCRtpSender.prototype versus 'createEncodedStreams' in RTCRtpSender.prototype) and factor your frame transform so the same code runs in both. Prefer the worker path where available; it is both the spec direction and better for jank.
The operational claim is precise: RTCRtpScriptTransform / createEncodedStreams; transform placement in the pipeline. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. RTCRtpScriptTransform / createEncodedStreams; transform placement in the pipeline. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Interactive 3D instrument
Trusting the middleman — what the SFU must read, and what it must not
A 3D instrument you drive yourself, one variable at a time. It needs JavaScript and WebGL, so it is not shown in this static copy of the page.
Scalable video coding places spatial and temporal layers in one dependency graph. An SFU may discard enhancement frames only when no retained frame depends on them; RTP dependency metadata makes that decision possible without parsing or decrypting the encoded frame body.
Engineering consequence. SFrame (RFC 9605): frame encryption, key ids, what stays cleartext and why — cross-reference to L7.3's AV1 Dependency Descriptor as the cleartext layer map that keeps E2EE SVC forwardable; Cryptex (RFC 9335) as a one-line sidebar. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
An SFU has to detect keyframes and, for SVC, read the dependency structure — so it needs the codec payload descriptor in the clear. Encrypt the entire frame and the SFU can no longer identify keyframes, cannot do layer selection, and cannot cache for new subscribers. The characteristic bug: E2EE works perfectly in a two-party test and video never starts through the SFU, because the SFU is waiting for a keyframe it can no longer recognise.
Leave the codec header unencrypted — the VP8 payload descriptor (1–10 bytes plus the 3-byte or 10-byte payload header on keyframes) or the H.264 NAL header — and encrypt from there. Append your IV and key generation as a trailer rather than a prefix so header offsets stay fixed. SFrame specifies this split; follow it rather than inventing your own.
The operational claim is precise: Key management: the MLS pointer; E2EE costs (no server-side recording/compositing). Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Key management: the MLS pointer; E2EE costs (no server-side recording/compositing). Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Start with the claim “Hop-by-hop vs end-to-end; what the SFU sees (encoded frames, timing, sizes)” and turn it into a falsifiable trace. Write down the input condition, the expected state transition, the counter or packet field that should change, and the deadline by which it must change. This prevents a familiar mistake: observing a symptom after several mechanisms have already reacted and assigning it to whichever mechanism has the most recognizable name. The causal order matters. A queue grows before loss; an ICE path fails before a restart selects a new pair; a decoder loses reference state before frames stop rendering. Preserve that order in the evidence.
Now hold every unrelated variable still. Use canned timestamps, SDP, stats-shaped records, or a permission-free loopback source so the same input can be replayed. Record both media directions separately. For cumulative counters, retain two samples and divide the difference by their timestamp interval. For state machines, retain the ordered event sequence rather than the last state alone. For negotiated features, compare the offer, answer, and live stats result. These three views answer different questions: what was proposed, what was accepted, and what the browser actually used.
The first lab, Encrypt Past the SFU, applies this discipline to a bounded implementation problem. Implement sender/receiver encoded transforms on a live loopback doing keystream encryption (WebCrypto AES-GCM) with a prepended key-id byte, preserving the leading unencrypted VP8 frame-header bytes (3 for delta frames, 10 for keyframes) that the depacketizer/decoder and the SFU need — the RTP payload descriptor does not exist yet at transform time; it is added later by the packetizer, reinforcing the pipeline-placement objective. Assertions: matched keys → remote pixel-probe passes; mismatched key → framesDecoded stalls (the SFU-can't-see-it property, proven); key rotation mid-stream causes ≤1 frame gap; encrypting the whole frame (breaking routing metadata) is a trapped, explained failure. Feature-detected with a simulation fallback. Before coding, identify the invariant hidden in that brief and the tempting shortcut the tests should reject. Then run the reference trace, perturb one boundary value, and explain why the result changes. If the code passes but the explanation cannot predict the perturbed result, the implementation is memorized rather than understood.
Rotate a key and there are still packets in flight encrypted with the old one, plus retransmissions that can arrive seconds later, plus a jitter buffer holding frames from before the switch. A receiver that discards the old key at rotation drops those frames, which reads as a burst of loss at every rotation — and if you rotate on participant join in a busy meeting, that is constant. There is no browser-provided ratchet; the whole scheme, including the generation identifier, is yours.
Carry an explicit key generation counter in the frame trailer and keep the previous one or two generations for a window of at least 2 s. Rotate on membership change, but debounce joins so a burst of arrivals produces one rotation.
A browser loopback is deliberately cleaner than the internet. It removes signaling latency, hostile NAT behavior, relay geography, wireless contention, device heat, and competing encoders. That makes it ideal for proving protocol and API invariants, but a dangerous basis for capacity claims. Carry the invariant into a second test with a canned hostile trace: reordered packets, a delayed candidate, a missing remote stats object, a bitrate step, or a state transition that recovers before the debounce timer. The implementation should remain correct even when optional fields are absent and events arrive more than once.
Feature detection is also part of correctness. Browser support for a codec, scalability mode, encoded transform, or transport API does not imply that the negotiated peer supports it or that hardware acceleration is active. Prefer capability checks, preserve a standards-based fallback, and label any simulation honestly. The graded result should measure the learner's algorithm; an optional live coda may demonstrate the real API without making a camera, operating-system codec, or network service a hidden test dependency.
Finally, connect the mechanism to the last topic in the lesson: “Key management: the MLS pointer; E2EE costs (no server-side recording/compositing)”. Advanced WebRTC failures cross boundaries. Congestion control changes encoder output; layer selection changes keyframe demand; end-to-end encryption constrains what an SFU can inspect; a watchdog can collide with negotiation; observability can mislabel a path if its joins are wrong. At every boundary, define ownership and an idempotent action. A retry must not create duplicate work, a stale event must not overwrite newer state, and an escalation must have a terminal cleanup path.
Use the problem set as an executable notebook. Keep a short record beside each answer: the invariant, the decisive evidence, the rejected alternative, and the production caveat. That record is more valuable than a green checkmark because it can be reused when a real call presents the same mechanism through noisier evidence.
A common error
Symmetric NAT always kills P2P — direct traversal fails roughly when both sides have address/port-dependent mapping; one-sided symmetric usually connects.
Implement the decision rules, run every assertion, then use the explanations to distinguish a wrong result from a wrong model.
Encrypt Past the SFU. Implement sender/receiver encoded transforms on a live loopback doing keystream encryption (WebCrypto AES-GCM) with a prepended key-id byte, preserving the leading unencrypted VP8 frame-header bytes (3 for delta frames, 10 for keyframes) that the depacketizer/decoder and the SFU need — the RTP payload descriptor does not exist yet at transform time; it is added later by the packetizer, reinforcing the pipeline-placement objective. Assertions: matched keys → remote pixel-probe passes; mismatched key → framesDecoded stalls (the SFU-can't-see-it property, proven); key rotation mid-stream causes ≤1 frame gap; encrypting the whole frame (breaking routing metadata) is a trapped, explained failure. Feature-detected with a simulation fallback.
Lab 1
Encrypt Past the SFU
Graded in the browser against 2 assertions; the editor and harness require JavaScript.
Check: What Does the Middleman Know?. Structured scenario: for a standard SFU and an SFrame-enabled SFU, mark what each can access (frame bytes, frame sizes/timing, routing metadata, decoded pixels) — auto-graded per cell; includes the 'DTLS-SRTP means my SFU call is E2EE' misconception.
Check your understanding
Auto-graded check
4 auto-graded questions with an explanation for every wrong answer. Requires JavaScript. (m7-l4-ex2)
What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.
If your encoded transform function throws, or fails to call controller.enqueue(), frames simply stop being delivered. There is no error event on the sender, receiver, or peer connection; connectionState stays connected and packet counters on the far side keep rising while nothing decodes. The same silence occurs on the receive side when the first frames arrive before your key has been distributed and your transform passes ciphertext through to the decoder, which then poisons its own state.
Wrap the entire transform body in try/catch, log via postMessage from the worker, and on failure enqueue the frame unchanged rather than dropping it (or drop deliberately and count it). Before your key arrives, drop frames explicitly instead of passing them through. Remember E2EE also disables server-side recording, transcription, and any PSTN leg.
WebRTC: From First Principles · Module 7, Lesson 4