Simulcast

Overview

Simulcast is short for "simultaneous broadcast." It's used in WebRTC to improve the overall quality of a group conference.

Instead of sending just one video stream to the server, each participant can send multiple copies ("encodings") of the same video but encoded to different quality targets. Typically, a high-quality (full bitrate) and low-quality (reduced bitrate) stream are encoded.

Consider a group call with Alice, Bob, Carol, and Dan:

  • Alice is sending a video using a bitrate of 2 Mbps.
  • Bob and Dan have reliable internet connections and can receive Alice's video at 2 Mbps.
  • Carol, however, has a limited internet connection and can only sustain inbound video at 1 Mbps.

In this scenario, to keep Carol from dropping the call, you must coordinate with Alice to reduce her outbound bitrate to 1 Mbps. Unfortunately for Bob and Dan, they must also see Alice at the reduced bitrate. This negative side effect is amplified and becomes more and more noticeable as you add more participants. With simulcast, you can send Carol a low-quality encoding while allowing Bob and Dan to continue receiving the high-quality encoding.

Simulcast doesn't support P2P connections. In a two-way call, the bitrate simply adapts to each receiver. There are no side effects because there are no other receivers to impact. Only SFU and MCU connections can take advantage of simulcast.

The simulcast implementation in LiveSwitch is compatible with the IETF proposal, which uses RTP stream IDs (RIDs) to identify simulcast streams. Non-RID-based simulcast (using SSRCs only) is available for native platforms but is likely to be deprecated in the future.

How does Simulcast Improve Stream Quality?

Simulcast is a powerful tool for providing optimal video stream quality. It works by simultaneously sending multiple feeds of different quality, dynamically enabling and disabling them according to who is subscribed to them, and on the receiver side, switching quickly between them as network conditions change.

Simulcast, by itself, can't resolve all quality issues for all network conditions. However, you can use it to fix most of the quality issues you encounter. There are other mechanisms that also affect overall stream quality. An alternative to simulcast that improves overall stream quality is Adaptive Video Resolution, where you adjust the video stream's resolution, constraining it to the actual solution of the receivers' view. For example, if you send HD video at a resolution of 1280 x 720, but all the receivers display the downstream in a smaller view because the app layout dictates demoting participant views during a screen share, you can adjust the sender's resolution to not waste bandwidth.

LiveSwitch continually improves bandwidth management and adaptation policies and capabilities. Contact Support if you have a use case that you want to improve.

Note

About the code examples on this page:

  • For Unity, Xamarin Android, Xamarin iOS, and Xamarin macOS, use the C# code.
  • For Java, use the Android code.
  • For macOS, use the iOS code.

Use Simulcast

Simulcast is fully supported in all native platforms and partially supported in web browsers. See Feature Support for specific details of what is available.

To use simulcast, set the SimulcastMode for your VideoStream to RtpStreamId:

videoStream.SimulcastMode = SimulcastMode.RtpStreamId;

You can then use this video stream when creating your SFU or MCU upstream connection.

Bandwidth Adaptation

LiveSwitch supports simulcast that doesn't apply bandwidth adaptation to the encoding layers. The goal of bandwidth adaptation is to limit packet loss and delay by adjusting the bitrate to a level that is sustainable for all parties. This means that it causes all receiving parties to experience the same quality of media. What's perceived as sustainable by the clients on poor-performing networks can be considered as low-quality for those clients with good-performing networks.

Therefore, with simulcast, bandwidth adaptation is not applied to the encoding layers themselves as there should be sufficient encoding layers available to satisfy most client networks. Note that if a client has network bandwidth below the lowest encoding layer, they will experience worse quality due to packet loss. The best practice in this situation would be to consider disabling certain streams to ensure only the most important streams can still provide an acceptable experience.

Simulcast is most efficient when using bandwidth estimation to determine which simulcast encoding to send to which participant.

Bandwidth Adaptation Policy controls if clients automatically switch between encoding layers. By default, BandwidthAdaptationPolicy is enabled in both the Configuration Console and the client SDK for VideoStream.

  • When enabled, the Media Server informs the clients when to switch encoding layers.
  • When disabled, the layer switching won't happen automatically. The client app needs to add logic to decide when to switch encoding layers.

The matrix below explains the expected behavior of when bandwidth adaptation and simulcast are enabled and disabled.

Simulcast enabled Simulcast disabled
Bandwidth Adaptation enabled
  • Switching encoding layers is the form of bandwidth adaptation.
  • Bandwidth adaptation isn't applied to each encoding layer.
  • Media Server informs the clients when to switch encoding layers.
  • All receiving clients experience the same quality of media.
  • What's sustainable by the clients on poor-performing networks can be considered as low-quality for those clients with good-performing networks.
  • In multi-party calls, when one of the clients has a poor connection, it sends a bitrate adjustment request to the sender and the sender lowers the quality for everyone in the call.
Bandwidth Adaptation disabled
  • The layer switching won't happen automatically.
  • The client app needs to add logic to decide when to switch encoding layers.
  • No bitrate adjustment.
  • Clients on good-performing network get good-quality feed, but it would be problematic for clients on poor-performing network as there are no other options available for them to subscribe to.

Configure Simulcast

You can modify the following default simulcast configurations for your use cases:

Note
  • You must configure all simulcast on your LocalMedia instance.
  • In native apps, you must apply configuration changes before calling Initialize.
  • In web apps, you must apply configuration changes before calling Start.

Encoding Count

Use the VideoSimulcastEncodingCount property to set the number of encodings to initialize. The default value is 2.

_LocalMedia.VideoSimulcastEncodingCount = 4;

The maximum number of encodings is 4.

Bits per Pixel

Use the VideoSimulcastBitsPerPixel property to control the bitrates used by the video encoders for each encoding. The default value is 0.05.

_LocalMedia.VideoSimulcastBitsPerPixel = 0.07;

The number of bits per pixel (bpp) describes a video stream's desired quality independent of the frame size (width/height) and frame rate. Given a video source that raises frames of a particular size and at a particular rate, you can determine the target bitrate for a video encoder, such as VP8, H.264, VP9, using a simple formula:

kbps = width * height * frameRate * bpp / 1000

Bandwidth usage is optimized based on the video stream's actual characteristics. To ensure quality is consistent across devices and platforms, it is strongly recommended to use bpp for custom simulcast configurations.

For example, a full HD video camera producing 1920x1080 images at 60 frames per second (fps) using the default bpp of 0.05 would imply that the target bitrate for the video encoder should be:

1920 * 1080 * 60 * 0.05 / 1000 = 6221kbps

On the other hand, a camera producing 640x480 images at 30 fps using the same default bpp of 0.05 would imply that the target bitrate for the video encoder should be:

640 * 480 * 30 * 0.05 / 1000 = 461kbps

Degradation Preference

Use the VideoSimulcastDegradationPreference property to control the trade-offs that lower-quality encodings must make to achieve lower bitrates. The default value is Automatic.

_LocalMedia.VideoSimulcastDegradationPreference = VideoDegradationPreference.balanced;

The following tables describe the allowed values:

Value Description
Resolution Reduces the number of pixels in half for each secondary encoding. For example, a video source raising 1280x720 frames with three encodings results in 1280x720, 905x509, and 640x360.
FrameRate Reduces the number of frames in half for each secondary encoding. For example, a video source raising frames at 30 frames per second with three encodings results in frame rates of 30, 15, and 7.5.
Balanced Reduces both the number of pixels and the number of frames for each secondary encoding, but less than the reduction that would take place with either Resolution or FrameRate. For example, a video source raising 1280x720 frames at 30 frames per second with three encodings results in resolution and frame rate combinations of 1280x720@30fps, 1076x605@21fps, and 905x509@15fps.
Automatic Select the degradation preference from the preceding three options based on the VideoType of the video source. There are three possible video types, and each type results in a different degradation preference:
  • A VideoType of Camera translates to a degradation preference of Resolution.
  • A VideoType of Screen translates to a degradation preference of FrameRate.
  • A VideoType of Unknown translates to a degradation preference of Balanced.

For more information about configuring simulcast in the Console, see the following:

Control Simulcast Behavior

You can control the simulcast behavior for your app by the following:

Enable and Disable Send Encodings

Enabling and disabling send encodings is a three-step process:

  1. Get the array of VideoEncodings.
  2. Change the Deactivated flags.
  3. Set the array of VideoEncodings.

This round-trip process ensures that the number of encodings doesn't change and the correct encodings are updated.

var videoEncodings = _LocalMedia.VideoEncodings;
videoEncodings[1].Deactivated = true;
_LocalMedia.VideoEncodings = videoEncodings;

Select an Initial Receive Encoding

Set the RemoteEncoding property of the VideoStream before opening the connection to request a specific encoding from LiveSwitch. By default, LiveSwitch attempts the highest quality encoding and downgrades it if necessary.

_Channel.OnRemoteUpstreamConnectionOpen += (remoteConnectionInfo) =>
{
    ...
    if (remoteConnectionInfo.HasVideo)
    {
        var remoteEncodings = remoteConnectionInfo.VideoStream.SendEncodings;
        if (remoteEncodings != null && remoteEncodings.Length > 0)
        {
            videoStream.RemoteEncoding = remoteEncodings[remoteEncodings.Length - 1];
        }
    }
    ...
};

Switch the Current Receive Encoding

Use the Update method after opening a connection to request a different encoding from LiveSwitch in three steps:

  1. Get the connection Config.
  2. Change the RemoteVideoEncoding property.
  3. Update the connection Config.
var config = connection.Config;
config.RemoteVideoEncoding = remoteEncodings[0];
await connection.Update(config);

To set an upper bitrate constraint based on the size of a view in your app, set the MaxReceiveBitrate of your VideoStream:

videoStream.MaxReceiveBitrate = bitrateInKbps;

LiveSwitch automatically switches the inbound stream to a more suitable encoding if a better option is available to satisfy that constraint.

Feature Support

Client Feature Matrix

Warning

Client-side simulcast is an experimental feature that has known browser issues.

Feature Web Native
Chrome Firefox Safari Edge IE (ActiveX) iOS Android Windows macOS Linux
Sending multiple encodings Yes Yes No Yes Yes Yes Yes Yes Yes Yes
Deactivating a send encoding Yes No No Yes Yes Yes Yes Yes Yes Yes
Updating a send encoding bitrate Yes Yes No Yes Yes Yes Yes Yes Yes Yes
Updating a send encoding frame rate No1 No1 No No Yes Yes Yes Yes Yes Yes
Updating a send encoding scale (size) Yes Yes No Yes Yes Yes Yes Yes Yes Yes
Receiving an encoding Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Switching receive encoding Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Multiple VP8 encodings Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Multiple H.264 encodings Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Multiple VP9 encodings No No No No No Yes Yes Yes Yes Yes

1 https://www.chromestatus.com/feature/6216116758118400