§ 7.3  Module 7 — From Two to Two Thousand: Data, Topologies, and Trust

Simulcast and SVC: One Stream, Many Screens

March 2020: every mesh-topology product melted at four participants, and the architecture that won can read your media.Course notes

The SFU's core dilemma — heterogeneous receivers — solved by sending multiple independent encodings (simulcast, RIDs) or one layered bitstream (SVC): spatial/temporal layers, scalabilityMode grammar, and SFU layer-switching logic with hysteresis, built against a war story.

Cross-references: Simulcast & SVC.

7.3.0Learning objectives

• Configure real 3-layer simulcast via addTransceiver sendEncodings and verify per-rid stats
• Walk SVC dependency graphs: compute which frames an SFU may drop for a target layer
• Read scalabilityMode grammar (L1T3, L3T3_KEY, S-modes) and the simulcast-vs-SVC decision matrix
• Implement SFU layer selection with hysteresis and keyframe-request rate limiting

7.3.1Simulcast

Simulcast sends independent encodings identified by RID, commonly at several resolutions and bitrates. The SFU can switch among them without decoding, but uplink cost rises and switching logic must move down quickly, move up slowly, and avoid keyframe-request storms.

Engineering consequence. Simulcast: rid/sendEncodings, scaleResolutionDownBy, per-layer bitrate/keyframes, a=simulcast SDP. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.

7.3.2SVC

H.264 is still the deployment baseline because hardware encoders are ubiquitous, but profile-level-id and packetization-mode must intersect. RTP carries complete NAL units when they fit, aggregates small units with STAP-A, and fragments large units with FU-A; the five low bits of the first octet identify the NAL type.

Engineering consequence. SVC: temporal/spatial/quality layers; H.264 Annex G's market failure; VP9/AV1 scalabilityMode. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.

7.3.3Dependency graphs and safe frame dropping; switch points

The operational claim is precise: Dependency graphs and safe frame dropping; switch points. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.

Engineering consequence. Dependency graphs and safe frame dropping; switch points. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.

7.3.4The AV1 Dependency Descriptor RTP header extension

Scalable video coding places spatial and temporal layers in one dependency graph. An SFU may discard enhancement frames only when no retained frame depends on them; RTP dependency metadata makes that decision possible without parsing or decrypting the encoded frame body.

Engineering consequence. The AV1 Dependency Descriptor RTP header extension: codec-agnostic layer info that lets an SFU do SVC layer selection without parsing codec payloads — and the reason E2EE SVC forwarding survives SFrame (bridge to L7.4). Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.

7.3.5SFU switching

The operational claim is precise: SFU switching: hysteresis, keyframe-request coalescing; uplink cost comparison. Treat it as an observable contract. Record the state transition or counter that proves it, test the failure path as well as the happy path, and keep units and clock domains explicit. That discipline is what separates a plausible WebRTC explanation from a production diagnosis.

Engineering consequence. SFU switching: hysteresis, keyframe-request coalescing; uplink cost comparison. Verify the behavior with a controlled trace before interpreting a live call: establish the expected state, change one variable, and compare deltas rather than isolated values. In production, attach the observation to a timestamp and media direction so the same evidence cannot be mistaken for a different failure.

7.3.AA worked reasoning trace

Start with the claim “Simulcast: rid/sendEncodings, scaleResolutionDownBy, per-layer bitrate/keyframes, a=simulcast SDP” and turn it into a falsifiable trace. Write down the input condition, the expected state transition, the counter or packet field that should change, and the deadline by which it must change. This prevents a familiar mistake: observing a symptom after several mechanisms have already reacted and assigning it to whichever mechanism has the most recognizable name. The causal order matters. A queue grows before loss; an ICE path fails before a restart selects a new pair; a decoder loses reference state before frames stop rendering. Preserve that order in the evidence.

Now hold every unrelated variable still. Use canned timestamps, SDP, stats-shaped records, or a permission-free loopback source so the same input can be replayed. Record both media directions separately. For cumulative counters, retain two samples and divide the difference by their timestamp interval. For state machines, retain the ordered event sequence rather than the last state alone. For negotiated features, compare the offer, answer, and live stats result. These three views answer different questions: what was proposed, what was accepted, and what the browser actually used.

The first lab, Three Layers, Live, applies this discipline to a bounded implementation problem. Create a 3-layer simulcast sender on loopback (rids q/h/f, scaleResolutionDownBy 4/2/1), implement layerReport(stats) grouping outbound-rtp by rid, then kill the top layer via setParameters. Assertions: 3 outbound-rtp entries with distinct rids, resolutions in 4:2:1 ratio, top-layer bitrate →0 within 3s of active:false; hints catch mutating a stale encodings object instead of getParameters(). Note: browsers do not answer a plain loopback offer with a=simulcast recv, so the harness provides a simulcast answer-munging shim (rewriting rid send→recv and adding a=simulcast:recv q;h;f before setRemoteDescription — exactly why this lesson also teaches a=simulcast SDP); without it the sender silently falls back to one encoding. All assertions are sender-side; the live path is scoped to Chromium, feature-detected with a canned-stats fallback on other engines. Before coding, identify the invariant hidden in that brief and the tempting shortcut the tests should reject. Then run the reference trace, perturb one boundary value, and explain why the result changes. If the code passes but the explanation cannot predict the perturbed result, the implementation is memorized rather than understood.

7.3.BVerification protocol

1. Name the budget. State whether the scarce quantity is latency, bitrate, CPU, memory, reliability, or negotiation time, and attach a unit.
2. Choose the observation point. Decide whether proof lives in SDP, an RTP/RTCP field, an API state, an event sequence, decoded output, or a joined stats record.
3. Build the clean control. Establish one trace in which the mechanism succeeds before injecting loss, delay, glare, a codec mismatch, or a network change.
4. Change one variable. Keep clocks, direction, source cadence, and topology fixed so the observed difference has a defensible cause.
5. Test both sides of the threshold. A deadline or bitrate rule is not demonstrated by one passing value; include a value just inside and just outside the boundary.
6. Close the loop. After acting, prove that the intended media outcome changed. Calling an API successfully is not evidence that frames flowed or quality recovered.

7.3.CProduction boundaries

A browser loopback is deliberately cleaner than the internet. It removes signaling latency, hostile NAT behavior, relay geography, wireless contention, device heat, and competing encoders. That makes it ideal for proving protocol and API invariants, but a dangerous basis for capacity claims. Carry the invariant into a second test with a canned hostile trace: reordered packets, a delayed candidate, a missing remote stats object, a bitrate step, or a state transition that recovers before the debounce timer. The implementation should remain correct even when optional fields are absent and events arrive more than once.

Feature detection is also part of correctness. Browser support for a codec, scalability mode, encoded transform, or transport API does not imply that the negotiated peer supports it or that hardware acceleration is active. Prefer capability checks, preserve a standards-based fallback, and label any simulation honestly. The graded result should measure the learner's algorithm; an optional live coda may demonstrate the real API without making a camera, operating-system codec, or network service a hidden test dependency.

Finally, connect the mechanism to the last topic in the lesson: “SFU switching: hysteresis, keyframe-request coalescing; uplink cost comparison”. Advanced WebRTC failures cross boundaries. Congestion control changes encoder output; layer selection changes keyframe demand; end-to-end encryption constrains what an SFU can inspect; a watchdog can collide with negotiation; observability can mislabel a path if its joins are wrong. At every boundary, define ownership and an idempotent action. A retry must not create duplicate work, a stale event must not overwrite newer state, and an escalation must have a terminal cleanup path.

Use the problem set as an executable notebook. Keep a short record beside each answer: the invariant, the decisive evidence, the rejected alternative, and the production caveat. That record is more valuable than a green checkmark because it can be reused when a real call presents the same mechanism through noisier evidence.

Problems for § 7.3Executable understanding

Implement the decision rules, run every assertion, then use the explanations to distinguish a wrong result from a wrong model.

Three Layers, Live. Create a 3-layer simulcast sender on loopback (rids q/h/f, scaleResolutionDownBy 4/2/1), implement layerReport(stats) grouping outbound-rtp by rid, then kill the top layer via setParameters. Assertions: 3 outbound-rtp entries with distinct rids, resolutions in 4:2:1 ratio, top-layer bitrate →0 within 3s of active:false; hints catch mutating a stale encodings object instead of getParameters(). Note: browsers do not answer a plain loopback offer with a=simulcast recv, so the harness provides a simulcast answer-munging shim (rewriting rid send→recv and adding a=simulcast:recv q;h;f before setRemoteDescription — exactly why this lesson also teaches a=simulcast SDP); without it the sender silently falls back to one encoding. All assertions are sender-side; the live path is scoped to Chromium, feature-detected with a canned-stats fallback on other engines.

The SVC Drop Set. Given a canned L3T3 frame-dependency graph, implement droppableFrames(targetSpatial, targetTemporal, frames): which frames can an SFU drop for an S1T1 subscriber without breaking decode. Assertions: keeps the full dependency chain, drops all higher layers, never drops a frame a kept frame references; hints catch dropping-by-layer-id without walking the graph.

Fix the Flapping SFU (Optional Stretch). Optional stretch exercise — not required for lesson completion; the m7 exam Part A contains the graded version of this task. The harness provides an Sfu skeleton with one simulcast publisher and subscribers with fluctuating canned BWE signals. Implement selectLayer(subscriber) with hysteresis plus onKeyframeNeeded rate limiting. Assertions: no subscriber above its BWE >500 ms, flap count bounded on the oscillating trace, keyframe requests ≤1/sec despite 10 switching subscribers (the war-story bug, trapped), starved subscriber recovers within 2s.