§ 7.2 Module 7 — From Two to Two Thousand: Data, Topologies, and Trust
March 2020: every mesh-topology product melted at four participants, and the architecture that won can read your media.Course notes
By the end of this lesson
The failure that forced the design
Conference bridges were once hardware MCUs in telco closets: decode everyone, mix, re-encode — expensive, high-latency, and in full possession of your media. Mesh seemed like the P2P dream until the arithmetic hit: each peer uploads to every other peer, and at four participants a home uplink dies. The SFU (~2012, the Jitsi/Licode era) split the difference: route encoded packets, decode nothing — the insight that carried the pandemic.
The n(n−1) uplink arithmetic that kills mesh, MCU's CPU bill and latency tax, why the SFU's 'route, don't decode' model won the pandemic, and the TURN/bandwidth economics of each — computed, plotted, and asserted.
Cross-references: Media Topologies: Mesh, MCU, SFU.
Mesh requires every participant to upload and encode for every other participant: per-client uplink grows as (n−1)×bitrate. An SFU keeps one publisher uplink and forwards encoded packets, whereas an MCU pays decode, composition, and re-encode costs to produce a single mixed stream.
Engineering consequence. Mesh: n-1 uplinks AND n-1 encodes; the n≈4 wall. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Interactive minigame
Where does the n-fold multiplication go?
A 3D instrument you drive yourself, one variable at a time. It needs JavaScript and WebGL, so it is not shown in this static copy of the page.
Everyone computes the uplink: five peers at 1 Mbit/s is 5 Mbit/s up, above many residential and most mobile uplinks. The harder limit is CPU. Chrome shares an encoder across senders within one peer connection, but a mesh means five separate RTCPeerConnection objects and therefore five independent encoder instances producing five near-identical bitstreams, each with its own bandwidth estimate that competes with the other four on the same uplink. All five estimators then underestimate, so quality collapses faster than the arithmetic suggests.
Cap mesh at three or four participants and switch to an SFU above that. If you must mesh, drop to a single low resolution for everyone and consider audio-only mesh with a separate SFU for video.
The operational claim is precise: MCU: transcode cost, latency tax, composition; where MCUs still win (PSTN gateways, recording). Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. MCU: transcode cost, latency tax, composition; where MCUs still win (PSTN gateways, recording). Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Simulcast sends independent encodings identified by RID, commonly at several resolutions and bitrates. The SFU can switch among them without decoding, but uplink cost rises and switching logic must move down quickly, move up slowly, and avoid keyframe-request storms.
Engineering consequence. SFU: route-don't-decode; downlink still (n-1)×b without layering — the reason simulcast exists. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
When an SFU switches a subscriber from a high simulcast layer to a low one, the two layers have independent sequence number and timestamp spaces. Forward them unmodified and the receiver sees a gap or a jump, concludes packets were lost, and NACKs for packets that never existed — which the SFU cannot supply, so the receiver retries, and the jitter buffer stalls waiting for them. The symptom is a freeze on every layer switch, which correlates with network conditions and therefore looks like the network.
Rewrite SSRC, sequence numbers, timestamps, and picture ids continuously across layer switches, and switch only on a keyframe boundary of the target layer. If you are evaluating an SFU, ask specifically how it handles this — it separates the mature implementations from the demos.
The operational claim is precise: Economics: server bandwidth, TURN interaction, ecosystem map. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.
Engineering consequence. Economics: server bandwidth, TURN interaction, ecosystem map. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.
Start with the claim “Mesh: n-1 uplinks AND n-1 encodes; the n≈4 wall” and turn it into a falsifiable trace. Write down the input condition, the expected state transition, the counter or packet field that should change, and the deadline by which it must change. This prevents a familiar mistake: observing a symptom after several mechanisms have already reacted and assigning it to whichever mechanism has the most recognizable name. The causal order matters. A queue grows before loss; an ICE path fails before a restart selects a new pair; a decoder loses reference state before frames stop rendering. Preserve that order in the evidence.
Now hold every unrelated variable still. Use canned timestamps, SDP, stats-shaped records, or a permission-free loopback source so the same input can be replayed. Record both media directions separately. For cumulative counters, retain two samples and divide the difference by their timestamp interval. For state machines, retain the ordered event sequence rather than the last state alone. For negotiated features, compare the offer, answer, and live stats result. These three views answer different questions: what was proposed, what was accepted, and what the browser actually used.
The first lab, The Topology Cost Calculator, applies this discipline to a bounded implementation problem. Implement topologyCost(n, bitrate, topology) returning per-client up/down bandwidth and server bandwidth/CPU for mesh/MCU/SFU plus a simulcast-aware SFU variant; the harness plots it and asserts crossover points. Assertions include mesh uplink =(n-1)×b, SFU uplink = b (or simulcast sum), and the exact max viable mesh n under a 4 Mbps uplink; hints catch forgetting mesh's n-1 encodes (CPU, not just bandwidth). Before coding, identify the invariant hidden in that brief and the tempting shortcut the tests should reject. Then run the reference trace, perturb one boundary value, and explain why the result changes. If the code passes but the explanation cannot predict the perturbed result, the implementation is memorized rather than understood.
One peer connection per direction gives you a single BUNDLE transport with one congestion controller managing all streams, which is what you want. Splitting subscriptions across several peer connections gives each one its own estimator, and they cannot see each other: on a constrained uplink or downlink they collectively overshoot, induce loss, all back off, and oscillate. The pattern is attractive for code organisation and it is a performance mistake.
Use one peer connection per direction (or one total) and multiplex with transceivers. If you need isolation for reliability, accept the estimator cost knowingly and cap each connection's maxBitrate so their sum stays inside a plausible total.
A browser loopback is deliberately cleaner than the internet. It removes signaling latency, hostile NAT behavior, relay geography, wireless contention, device heat, and competing encoders. That makes it ideal for proving protocol and API invariants, but a dangerous basis for capacity claims. Carry the invariant into a second test with a canned hostile trace: reordered packets, a delayed candidate, a missing remote stats object, a bitrate step, or a state transition that recovers before the debounce timer. The implementation should remain correct even when optional fields are absent and events arrive more than once.
Feature detection is also part of correctness. Browser support for a codec, scalability mode, encoded transform, or transport API does not imply that the negotiated peer supports it or that hardware acceleration is active. Prefer capability checks, preserve a standards-based fallback, and label any simulation honestly. The graded result should measure the learner's algorithm; an optional live coda may demonstrate the real API without making a camera, operating-system codec, or network service a hidden test dependency.
Finally, connect the mechanism to the last topic in the lesson: “Economics: server bandwidth, TURN interaction, ecosystem map”. Advanced WebRTC failures cross boundaries. Congestion control changes encoder output; layer selection changes keyframe demand; end-to-end encryption constrains what an SFU can inspect; a watchdog can collide with negotiation; observability can mislabel a path if its joins are wrong. At every boundary, define ownership and an idempotent action. A retry must not create duplicate work, a stale event must not overwrite newer state, and an escalation must have a terminal cleanup path.
Use the problem set as an executable notebook. Keep a short record beside each answer: the invariant, the decisive evidence, the rejected alternative, and the production caveat. That record is more valuable than a green checkmark because it can be reused when a real call presents the same mechanism through noisier evidence.
A common error
Symmetric NAT always kills P2P — direct traversal fails roughly when both sides have address/port-dependent mapping; one-sided symmetric usually connects.
Implement the decision rules, run every assertion, then use the explanations to distinguish a wrong result from a wrong model.
The Topology Cost Calculator. Implement topologyCost(n, bitrate, topology) returning per-client up/down bandwidth and server bandwidth/CPU for mesh/MCU/SFU plus a simulcast-aware SFU variant; the harness plots it and asserts crossover points. Assertions include mesh uplink =(n-1)×b, SFU uplink = b (or simulcast sum), and the exact max viable mesh n under a 4 Mbps uplink; hints catch forgetting mesh's n-1 encodes (CPU, not just bandwidth).
Lab 1
The Topology Cost Calculator
Graded in the browser against 3 assertions; the editor and harness require JavaScript.
Check: Capacity Planning. Harness-posed scenarios (webinar with 200 viewers, 8-person standup, 2-person interview with recording) — choose topology and justify via numeric bandwidth fields graded exactly.
Check your understanding
Auto-graded check
4 auto-graded questions with an explanation for every wrong answer. Requires JavaScript. (m7-l2-ex2)
What the specifications say, and what the browsers actually do, are two different documents. These are the divergences that cost production teams their weekends.
ontrack fires for each inbound transceiver, and event.streams[0].id is a string the SFU picked — often a UUID that means nothing to your client, sometimes absent entirely. With fifty subscriptions, the events arrive in an order you do not control and before your signaling has necessarily told you who is who. Every "wrong video in the wrong tile" bug traces to inferring identity from media rather than from signaling.
Have the SFU signal an explicit MID-to-participant mapping and key your UI on event.transceiver.mid. Render a placeholder until the mapping arrives rather than guessing, and re-read the mapping after every renegotiation because MIDs can be recycled.
WebRTC: From First Principles · Module 7, Lesson 2