Simulcast

Overview

Simulcast is short for "simultaneous broadcast." It's used in WebRTC to improve the overall quality of a group conference.

Instead of sending just one video stream to the server, each participant can send multiple copies ("encodings") of the same video but encoded to different quality targets. Typically, a high-quality (full bitrate) and low-quality (reduced bitrate) stream are encoded.

Consider a group call with Alice, Bob, Carol, and Dan:

Alice is sending a video using a bitrate of 2 Mbps.
Bob and Dan have reliable internet connections and can receive Alice's video at 2 Mbps.
Carol, however, has a limited internet connection and can only sustain inbound video at 1 Mbps.

In this scenario, to keep Carol from dropping the call, you must coordinate with Alice to reduce her outbound bitrate to 1 Mbps. Unfortunately for Bob and Dan, they must also see Alice at the reduced bitrate. This negative side effect is amplified and becomes more and more noticeable as you add more participants. With simulcast, you can send Carol a low-quality encoding while allowing Bob and Dan to continue receiving the high-quality encoding.

Simulcast doesn't support P2P connections. In a two-way call, the bitrate simply adapts to each receiver. There are no side effects because there are no other receivers to impact. Only SFU and MCU connections can take advantage of simulcast.

The simulcast implementation in LiveSwitch is compatible with the IETF proposal, which uses RTP stream IDs (RIDs) to identify simulcast streams. Non-RID-based simulcast (using SSRCs only) is available for native platforms but is likely to be deprecated in the future.

How does Simulcast Improve Stream Quality?

Simulcast is a powerful tool for providing optimal video stream quality. It works by simultaneously sending multiple feeds of different quality, dynamically enabling and disabling them according to who is subscribed to them, and on the receiver side, switching quickly between them as network conditions change.

Simulcast, by itself, can't resolve all quality issues for all network conditions. However, you can use it to fix most of the quality issues you encounter. There are other mechanisms that also affect overall stream quality. For example, within the bounds of a particular encoding layer, simulcast relies on bandwidth estimation and adaptation to adapt stream bandwidth as needed. Another mechanism that improves overall stream quality is Adaptive Video Resolution, where you adjust the video stream's resolution, constraining it to the actual solution of the receivers' view. For example, if you send HD video at a resolution of 1280 x 720, but all the receivers display the downstream in a smaller view because the app layout dictates demoting participant views during a screen share, you can adjust the sender's resolution to not waste bandwidth.

LiveSwitch continually improves bandwidth management and adaptation policies and capabilities. Contact us if you have a use case that you want to improve.

Note

About the code examples on this page:

For Unity, UWP, Xamarin Android, Xamarin iOS, and Xamarin macOS, use the C# code.
For Java, use the Android code.
For macOS, use the iOS code.

Use Simulcast

Simulcast is fully supported in all native platforms and partially supported in web browsers. See Feature Support for specific details of what is available.

To use simulcast, set the SimulcastMode for your VideoStream to RtpStreamId:

videoStream.SimulcastMode = SimulcastMode.RtpStreamId;

videoStream.setSimulcastMode(SimulcastMode.RtpStreamId);

videoStream?.setSimulcastMode(FMLiveSwitchSimulcastMode.rtpStreamId)

videoStream.setSimulcastMode(SimulcastMode.RtpStreamId);

You can then use this video stream when creating your SFU or MCU upstream connection.

Bandwidth Adaptation

Simulcast is most efficient when using bandwidth estimation to determine which simulcast encoding to send to which participant. By default, BandwidthAdaptationPolicy is enabled in both the Configuration Console and the client SDK for VideoStream.

Configure Simulcast

You can modify the following default simulcast configurations for your use cases:

Encoding Count
Bits per Pixel
Degradation Preference

Note

You must configure all simulcast on your LocalMedia instance.
In native apps, you must apply configuration changes before calling Initialize.
In web apps, you must apply configuration changes before calling Start.

Warning

As a client, do not modify the bitrate, resolution, or frame rate of your upstream connection when server-side simulcast is enabled.

Encoding Count

Use the VideoSimulcastEncodingCount property to set the number of encodings to initialize. The default value is 2.

_LocalMedia.VideoSimulcastEncodingCount = 4;

localMedia.setVideoSimulcastEncodingCount(4);

_localMedia?.setVideoSimulcastEncodingCount(4)

localMedia.setVideoSimulcastEncodingCount(4);

The maximum number of encodings is 4.

Bits per Pixel

Use the VideoSimulcastBitsPerPixel property to control the bitrates used by the video encoders for each encoding. The default value is 0.05.

_LocalMedia.VideoSimulcastBitsPerPixel = 0.07;

localMedia.setVideoSimulcastBitsPerPixel(0.07);

_localMedia?.setVideoSimulcastBitsPerPixel(0.07)

localMedia.setVideoSimulcastBitsPerPixel(0.07);

The number of bits per pixel (bpp) describes a video stream's desired quality independent of the frame size (width/height) and frame rate. Given a video source that raises frames of a particular size and at a particular rate, you can determine the target bitrate for a video encoder, such as VP8, H.264, VP9, using a simple formula:

kbps = width * height * frameRate * bpp / 1000

Bandwidth usage is optimized based on the video stream's actual characteristics. To ensure quality is consistent across devices and platforms, it is strongly recommended to use bpp for custom simulcast configurations.

For example, a full HD video camera producing 1920x1080 images at 60 frames per second (fps) using the default bpp of 0.05 would imply that the target bitrate for the video encoder should be:

1920 * 1080 * 60 * 0.05 / 1000 = 6221kbps

On the other hand, a camera producing 640x480 images at 30 fps using the same default bpp of 0.05 would imply that the target bitrate for the video encoder should be:

640 * 480 * 30 * 0.05 / 1000 = 461kbps

Degradation Preference

Use the VideoSimulcastDegradationPreference property to control the trade-offs that lower-quality encodings must make to achieve lower bitrates. The default value is Automatic.

_LocalMedia.VideoSimulcastDegradationPreference = VideoDegradationPreference.balanced;

localMedia.setVideoSimulcastDegradationPreference(VideoDegradationPreference.balanced);

_localMedia?.setVideoSimulcastDegradationPreference(FMLiveSwitchVideoDegradationPreference.balanced)

localMedia.setVideoSimulcastDegradationPreference(VideoDegradationPreference.balanced);

The following tables describe the allowed values:

Value	Description
Resolution	Reduces the number of pixels in half for each secondary encoding. For example, a video source raising 1280x720 frames with three encodings results in 1280x720, 905x509, and 640x360.
FrameRate	Reduces the number of frames in half for each secondary encoding. For example, a video source raising frames at 30 frames per second with three encodings results in frame rates of 30, 15, and 7.5.
Balanced	Reduces both the number of pixels and the number of frames for each secondary encoding, but less than the reduction that would take place with either Resolution or FrameRate. For example, a video source raising 1280x720 frames at 30 frames per second with three encodings results in resolution and frame rate combinations of 1280x720@30fps, 1076x605@21fps, and 905x509@15fps.
Automatic	Select the degradation preference from the preceding three options based on the VideoType of the video source. There are three possible video types, and each type results in a different degradation preference: * A VideoType of Camera translates to a degradation preference of Resolution. * A VideoType of Screen translates to a degradation preference of FrameRate. * A VideoType of Unknown translates to a degradation preference of Balanced.

For more information about configuring simulcast in the Console, see the following:

Control Simulcast Behavior

You can control the simulcast behavior for your app by the following:

Enable and disable send encodings
Select an initial receive encoding
Switch the current receive encoding

Enable and Disable Send Encodings

Enabling and disabling send encodings is a three-step process:

Get the array of VideoEncodings.
Change the Deactivated flags.
Set the array of VideoEncodings.

This round-trip process ensures that the number of encodings doesn't change and the correct encodings are updated.

var videoEncodings = _LocalMedia.VideoEncodings;
videoEncodings[1].Deactivated = true;
_LocalMedia.VideoEncodings = videoEncodings;

VideoEncodingConfig[] videoEncodings = localMedia.getVideoEncodings();
videoEncodings[1].setDeactivated(true);
localMedia.setVideoEncodings(videoEncodings);

let videoEncodings = _localMedia?.doGetVideoEncodings()
(videoEncodings?[1] as! FMLiveSwitchVideoEncodingConfig).setDeactivated(true)
_localMedia?.setVideoEncodings(videoEncodings)

var videoEncodings = localMedia.getVideoEncodings();
videoEncodings[1].setDeactivated(true);
localMedia.setVideoEncodings(videoEncodings);

Select an Initial Receive Encoding

Set the RemoteEncoding property of the VideoStream before opening the connection to request a specific encoding from LiveSwitch. By default, LiveSwitch attempts the highest quality encoding and downgrades it if necessary.

_Channel.OnRemoteUpstreamConnectionOpen += (remoteConnectionInfo) =>
{
    ...
    if (remoteConnectionInfo.HasVideo)
    {
        var remoteEncodings = remoteConnectionInfo.VideoStream.SendEncodings;
        if (remoteEncodings != null && remoteEncodings.Length > 0)
        {
            videoStream.RemoteEncoding = remoteEncodings[remoteEncodings.Length - 1];
        }
    }
    ...
};

channel.addOnRemoteUpstreamConnectionOpen(remoteConnectionInfo -> {
    ...
    if (remoteConnectionInfo.getHasVideo()) {
        EncodingInfo[] remoteEncodings = remoteConnectionInfo.getVideoStream().getSendEncodings();
        if (remoteEncodings != null && remoteEncodings.length > 0) {
            videoStream.setRemoteEncoding(remoteEncodings[remoteEncodings.length - 1]);
        }
    }
    ...
});

_channel?.addOnRemoteUpstreamConnectionOpen({ (obj: Any!) in
    ...
    let remoteConnectionInfo = obj as! FMLiveSwitchConnectionInfo
    if (remoteConnectionInfo.hasVideo()) {
        let remoteEncodings = remoteConnectionInfo.videoStream().sendEncodings()
        if (remoteEncodings != nil && remoteEncodings!.count > 0) {
            videoStream?.setRemoteEncoding((remoteEncodings![remoteEncodings!.count - 1] as! FMLiveSwitchEncodingInfo))
        }
    }
    ...
})

channel.addOnRemoteUpstreamConnectionOpen((remoteConnectionInfo) => {
    ...
    if (remoteConnectionInfo.getHasVideo()) {
        var remoteEncodings = remoteConnectionInfo.getVideoStream().getSendEncodings();
        if (remoteEncodings != null && remoteEncodings.length > 0) {
            videoStream.setRemoteEncoding(remoteEncodings[remoteEncodings.length - 1]);
        }
    }
    ...
});

Switch the Current Receive Encoding

Use the Update method after opening a connection to request a different encoding from LiveSwitch in three steps:

Get the connection Config.
Change the RemoteVideoEncoding property.
Update the connection Config.

var config = connection.Config;
config.RemoteVideoEncoding = remoteEncodings[0];
await connection.Update(config);

ConnectionConfig config = connection.getConfig();
config.setRemoteVideoEncoding(remoteEncodings[0]);
connection.update(config);

let config = connection?.config()
config?.setRemoteVideoEncoding((remoteEncodings![0] as! FMLiveSwitchEncodingInfo))
connection?.update(with: config)

var config = connection.getConfig();
config.setRemoteVideoEncoding(remoteEncodings[0]);
await connection.update(config);

To set an upper bitrate constraint based on the size of a view in your app, set the MaxReceiveBitrate of your VideoStream:

videoStream.MaxReceiveBitrate = bitrateInKbps;

videoStream.setMaxReceiveBitrate(bitrateInKbps);

videoStream?.setMaxReceiveBitrate(bitrateInKbps)

videoStream.setMaxReceiveBitrate(bitrateInKbps);

LiveSwitch automatically switches the inbound stream to a more suitable encoding if a better option is available to satisfy that constraint.

Feature Support

Client Feature Matrix

Warning

Client-side simulcast is an experimental feature that has known browser issues.

Feature	Web					Native
Feature	Chrome	Firefox	Safari	Edge	IE (ActiveX)	iOS	Android	Windows	macOS	Linux
Sending multiple encodings	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Deactivating a send encoding	Yes	No	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Updating a send encoding bitrate	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Updating a send encoding frame rate	No¹	No¹	No	No	Yes	Yes	Yes	Yes	Yes	Yes
Updating a send encoding scale (size)	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Receiving an encoding	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Switching receive encoding	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Multiple VP8 encodings	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Multiple H.264 encodings	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Multiple VP9 encodings	No	No	No	No	No	Yes	Yes	Yes	Yes	Yes

1 https://www.chromestatus.com/feature/6216116758118400