Perform Adaptive Video Resolution

Consider a typical use case. Your app streams 1080 x 720 HD video from a broadcaster to a large number of recipients. The individual audience members receive the HD downstream and display it in a prominent view, typically also 1080 x 720 or higher. Your broadcaster then decides to screen share. Now, your app displays the screen share in a prominent view and demotes the broadcaster's VideoStream view to a small view on the side, such as 1/5 of the original size (216 x 144). Continuing to stream HD video for a small display (216 x 144) is an unnecessary use of bandwidth and processing power.

How to optimize? The answer is Adaptive Video Resolution (AVR). With AVR, you change the resolution of the sender's video according to the recipients' constraints.

For example, if you have an array of participants and each participant has an SFU downstream connection, sessionBitrate is the downstream bitrate constraint. Your app can do the following to perform AVR:

Get all connections and count the number of streams.
Divide the sessionBitrate by the count to get the bitrate for each stream.
Apply the new bitrate to all streams.

Note

Setting maximum receive bitrate is not supported when server-side simulcast is in use.

Note

About the code examples on this page:

For .NET MAUI and Unity, use the C# code.
For macOS, use the iOS code.

List<VideoStream> videoStreams = new List<VideoStream>();

foreach (Participant participant in participants)
{
    if (participant == me)
    {
        continue;
    }

    SfuDownstreamConnection connection = participant.Connection;
    if (connection != null)
    {
        VideoStream videoStream = connection.VideoStream;
        if (videoStream != null)
        {
            videoStreams.Add(videoStream);
        }
    }
}

int videoStreamCount = videoStreams.Count;
if (videoStreamCount > 0)
{
    int videoStreamBitrate = sessionBitrate / videoStreamCount;

    Log.Info($"Setting max bitrate for inbound video to {videoStreamBitrate} kbps per stream ({videoStreamCount} streams).");
    videoStreams.ForEach(videoStream => videoStream.MaxReceiveBitrate = videoStreamBitrate);
}

List<VideoStream> videoStreams = new ArrayList<VideoStream>();

for (Participant participant : participants) {
    if (participant == me) {
        continue;
    }

    SfuDownstreamConnection connection = participant.getConnection();
    if (connection != null) {
        VideoStream videoStream = connection.getVideoStream();

        if (videoStream != null) {
            videoStreams.add(videoStream);
        }
    }
}

int videoStreamCount = videoStreams.size();
if (videoStreamCount > 0) {
    int videoStreamBitrate = sessionBitrate / videoStreamCount;

    Log.info(String.format("Setting max bitrate for inbound video to %d kbps per stream (%d streams).", videoStreamBitrate, videoStreamCount));
    for (VideoStream videoStream : videoStreams) {
        videoStream.setMaxReceiveBitrate(videoStreamBitrate);
    }
}

var videoStreams: [VideoStream] = []

for participant in participants {
    if participant == me {
        continue
    }

    let connection: FMLiveSwitchSfuDownstreamConnection? = participant.connection
    if connection != nil {
        let videoStream: FMLiveSwitchVideoStream? = connection.videoStream
        if videoStream != nil {
            videoStreams.append(videoStream)
        }
    }
}

let videoStreamCount: Int = videoStreams.count
if videoStreamCount > 0 {
    videoStreamBitrate: Int = sessionBitrate / videoStreamCount;

    FMLiveSwitchLog.info("Setting max bitrate for inbound video to \(videoStreamBitrate) kbps per stream (\(videoStreamCount) streams).")
    for videoStream in videoStreams {
        videoStream.setMaxReceiveBitrate(videoStreamBitrate)
    }
}

let videoStreams = []
for (let participant of participants) {
    if (participant === me) {
        continue;
    }

    let connection = participant.connection;
    if (connection) {
        let videoStream = connection.getVideoStream();
        if (videoStream) {
            videoStreams.push(videoStream);
        }
    }
}
let videoStreamCount = videoStreams.length;
if (videoStreamCount > 0) {
    let videoStreamBitrate =  Math.floor(sessionBitrate / videoStreamCount);

    console.info('Setting max bitrate for inbound video to ' + videoStreamBitrate + ' kbps per stream (' + videoStreamCount + ' streams).');
    for (let videoStream of videoStreams) {
        videoStream.setMaxReceiveBitrate(videoStreamBitrate);
    }
}

AVR can be re-applied each time a new connection is established or closed to maximize the sessionBitrate across all active connections.