SFU Connection

Note

About the code examples on this page:

For .NET MAUI and Unity, use the C# code.
For macOS, use the iOS code.

SFU is an acronym that stands for Selective Forwarding Unit. An SFU is an endpoint in a media session that enhances the scalability of video conferencing sessions by forwarding audio and video data that it receives from connected users. In an SFU configuration, each user only has one upstream connection to the server, substantially reducing the amount of upload bandwidth required to run a video conference. Running LiveSwitch in SFU mode consumes more bandwidth on the server than if you were only using peer-to-peer connections but it also uses far less CPU than running LiveSwitch in MCU mode. Because of this, an SFU is a good middle-ground scaling option when you have an excess of bandwidth and a limited amount of CPU power.

Create an SFU Connection

You must be registered with the LiveSwitch gateway and joined to a FM.LiveSwitch.Channel instance as well. This is covered in the previous Registering a Client section. You should also review the Handling Local Media, Handling Remote Media, and Creating Streams and Connections sections, which review the media components and streams that are used in this guide.

When creating an SFU session, there are two different types of connections to consider. The first type of connection is known as the upstream connection. This connection forwards your own audio and video data to the server, which then forwards it to other participants. Each participant in an SFU session has one upstream connection. The second type of connection is known as a downstream connection. These connections receive audio and video data that is forwarded from the server. Each participant has one downstream connection for every other participant in the video conference. This guide first describes how to create an upstream connection and then details how to create downstream connections for other participants.

Make an Upstream Connection

To create your upstream connection, invoke the CreateSfuUpstreamConnection method of your Channel instance. Note that you must specify an audio and video stream for this method. This, in itself, is not unusual - but you must create the audio and video streams as send-only. To do so, when you create the FM.LiveSwitch.AudioStream and FM.LiveSwitch.VideoStream instances, do not specify a RemoteMedia instance. Instead, use the constructor overload that takes only a single LocalMedia instance.

This returns an FM.LiveSwitch.SfuUpstreamConnection instance that is set up to send, but not receive any data. Once you have created this connection, assign it ICE servers as you normally would and then invoke the Openmethod of the connection instance. The Open method returns a promise, which you should inspect to verify that your upstream connection is established successfully. This is demonstrated below.

var audioStream = new FM.LiveSwitch.AudioStream(localMedia, null);
var videoStream = new FM.LiveSwitch.VideoStream(localMedia, null);
var connection = channel.CreateSfuUpstreamConnection(audioStream, videoStream);
connection.IceServers = ...
connection.Open().Then((result) =>
{
    Log.Info("Upstream connection established");
}).Fail((Exception ex) =>
{
    Log.Info("An error occurred");
});

fm.liveswitch.AudioStream audioStream = new fm.liveswitch.AudioStream(localMedia, null)
fm.liveswitch.VideoStream videoStream = new fm.liveswitch.VideoStream(localMedia, null)
fm.liveswitch.SfuUpstreamConnection connection = channel.createSfuUpstreamConnection(audioStream, videoStream);
connection.setIceServers(...);
connection.open().then((Object result) -> {
    System.out.println("upstream connection established");
}).fail((Exception ex) -> {
    System.out.println("an error occurred");
});

var audioStream = FMLiveSwitchAudioStream(localMedia: localMedia, remoteMedia: nil)
var videoStream = FMLiveSwitchVideoStream(localMedia: localMedia, remoteMedia: nil)
var connection:FMLiveSwitchSfuUpstreamConnection = channel.createSfuUpstreamConnection(audioStream: audioStream, videoStream: videoStream)
connection.setIceServers(...)
connection.open().then(resolveBlock: { (result:NSObject) in
    print("upstream connection established")
}).fail(rejectBlock: { (ex:NSException) in
    print("an error occurred")
})

var audioStream = new fm.liveswitch.AudioStream(localMedia, null);
var videoStream = new fm.liveswitch.VideoStream(localMedia, null);
var connection = channel.createSfuUpstreamConnection(audioStream, videoStream);
connection.setIceServers(...);
connection.open().then(function(result) {
    console.log("upstream connection established");
}).fail(function(ex) {
    console.log("an error occurred");
});

You now have an upstream connection and can send data to the server, which is then forwarded on to other participants. However, you now need to create a downstream connection for other participants that join the SFU session. The next section shows you how to do this.

Note that instead of using WebRTC, you may decide to use the Media-over-WebSockets protocol. Please refer to the Media-over-WebSockets section to find out more.

Make Downstream Connections

In an SFU session, you must create a new downstream connection every time that a peer joins the channel. You do this by adding an event handler for the OnRemoteUpstreamConnectionOpen event. This event is raised whenever a remote user opens an upstream connection on this channel. The event handler has two tasks:

add a UI element for the downstream connection
create and open a downstream connection

The first thing you'll need to do in this event handler is create a RemoteMedia instance and add its view to the layout manager. You can add it by invoking the AddRemoteViewmethod of your LayoutManager instance. You need to specify both an id and a view instance.

Start by updating the UI. Create a new instance of your RemoteMedia class. Next, add the RemoteMedia instance's associated view to your layout manager by invoking its AddRemoteView method. This method requires an id parameter and a view object, both of which can be accessed through properties on the RemoteMedia instance.

channel.OnRemoteUpstreamConnectionOpen += (remoteConnectionInfo) =>
{
    var remoteMedia = new RemoteMedia();
    layoutManager.AddRemoteView(remoteMedia.Id, remoteMedia.View);
    ...
};

channel.addOnRemoteUpstreamConnectionOpen((fm.liveswitch.ConnectionInfo remoteConnectionInfo) -> {
    RemoteMedia remoteMedia = new RemoteMedia();
    layoutManager.addRemoteView(remoteMedia.getId(), remoteMedia.getView());
    ...
});

channel.addOnRemoteUpstreamConnectionOpen { (remoteConnectionInfo:FMLiveSwitchConnectionInfo) in
    var remoteMedia:RemoteMedia = RemoteMedia()
    layoutManager.addRemoteView(id: remoteMedia.id(), view: remoteMedia.view())
    ...
}

channel.addOnRemoteUpstreamConnectionOpen(function(remoteConnectionInfo) {
    var remoteMedia = new fm.liveswitch.RemoteMedia();
    layoutManager.addRemoteView(remoteMedia.getId(), remoteMedia.getView());
    ...
});

Next, you must create an FM.LiveSwitch.SfuDownstreamConnection instance. You do this by invoking the CreateSfuDownstreamConnection method of your Channel instance. You must pass an instance of FM.LiveSwitch.ConnectionInfo into this method as the first parameter. An instance of this is available to you as one of the parameters of your event handler. Assign ICE servers to your newly created SfuDownstreamConnectioninstance and then invoke its Open method. This returns a promise, the result of which you should inspect to ensure that the downstream connection is established properly. Once complete, your app is now able to manage both upstream and downstream portions of an SFU session. The next section focuses on how to properly tear down the session.

channel.OnRemoteUpstreamConnectionOpen += (FM.LiveSwitch.ConnectionInfo remoteConnectionInfo) =>
{
    ...
    var audioStream = new FM.LiveSwitch.AudioStream(null, remoteMedia);
    var videoStream = new FM.LiveSwitch.VideoStream(null, remoteMedia);
    var connection = channel.CreateSfuDownstreamConnection(remoteConnectionInfo, audioStream, videoStream);
    connection.IceServers = ...
    connection.Open().Then((object result) =>
    {
        Log.Info("Downstream connection established");
    }.Fail((Exception ex) =>
    {
        Log.Info("An error occurred");
    });
};

channel.addOnRemoteUpstreamConnectionOpen((fm.liveswitch.ConnectionInfo remoteConnectionInfo) -> {
    ...
    fm.liveswitch.AudioStream audioStream = new fm.liveswitch.AudioStream(null, remoteMedia);
    fm.liveswitch.VideoStream videoStream = new fm.liveswitch.VideoStream(null, remoteMedia);
    fm.liveswitch.SfuDownstreamConnection connection = channel.createSfuDownstreamConnection(remoteConnectionInfo, audioStream, videoStream);
    connection.setIceServers(...);
    connection.open().then((Object result) -> {
        System.out.println("downstream connection established");
    }).fail((Exception ex) -> {
        System.out.println("an error occurred");
    });
});

channel.addOnRemoteUpstreamConnectionOpen { (remoteConnectionInfo:FMLiveSwitchConnectionInfo) in
    ...
    var audioStream = FMLiveSwitchAudioStream(localMedia: nil, remoteMedia: remoteMedia)
    var videoStream = FMLiveSwitchVideoStream(localMedia: nil, remoteMedia: remoteMedia)
    var connection:FMLiveSwitchSfuDownstreamConnection = channel.createSfuDownstreamConnection(remoteConnectionInfo: remoteConnectionInfo, audioStream: audioStream, videoStream: videoStream)
    connection.setIceServers(...)
    connection.open().then(resolveAction: { (result:NSObject) in
        print("downstream connection established")
    }).fail(rejectAction: { (ex:NSException) in
        print("an error occurred")
    })
}

channel.addOnRemoteUpstreamConnectionOpen(function(remoteConnectionInfo) {
    ...
    var audioStream = new fm.liveswitch.AudioStream(null, remoteMedia);
    var videoStream = new fm.liveswitch.VideoStream(null, remoteMedia);
    var connection = channel.createSfuDownstreamConnection(remoteConnectionInfo, audioStream, videoStream);
    connection.setIceServers(...);
    connection.open().then(function(result) {
        console.log("downstream connection established");
    }).fail(function(ex) {
        console.log("an error occurred");
    });
});

Note that instead of using WebRTC, you may decide to use the Media-over-WebSockets protocol. Please refer to the Media-over-WebSockets section to find out more.

Teardown

When a user leaves a session, you should remove the remote view associated with them. If you do not do so, the view remains frozen on screen indefinitely. To remove a view when a user leaves, you must be notified when a peer's upstream connection closes. This is done by adding an OnStateChange event handler to each SfuDownstreamConnection instance. In this handler, you inspect the state of the SfuDownstreamConnection instance. If the state is Closing or Failing, you remove the associated remote view by invoking the RemoveRemoteView instance of your layout manager.

You should only add this handler to the downstream connections that you create, as upstream connections do not have a remote view associated with them.

channel.OnRemoteUpstreamConnectionOpen += (FM.LiveSwitch.ConnectionInfo remoteConnectionInfo) =>
{
    ...
    connection.OnStateChange += (FM.LiveSwitch.ManagedConnection c) =>
    {
        if (c.State == FM.LiveSwitch.ConnectionState.Closing || c.State == FM.LiveSwitch.ConnectionState.Failing)
        {
            layoutManager.RemoveRemoteView(remoteMedia.Id);
        }
    }
};

channel.addOnRemoteUpstreamConnectionOpen((fm.liveswitch.ConnectionInfo remoteConnectionInfo) -> {
    ...
    connection.addOnStateChange((fm.liveswitch.ManagedConnection c) -> {
        if (c.getState() == fm.liveswitch.ConnectionState.Closing || c.getState() == fm.liveswitch.ConnectionState.Failing) {
            layoutManager.removeRemoteView(remoteMedia.getId());
        }
    });
});

channel.addOnRemoteUpstreamConnectionOpen { (remoteConnectionInfo:FMLiveSwitchConnectionInfo) in
    ...
    connection.addOnStateChange { (c:FMLiveSwitchManagedConnection) in
        if c.state() == FMLiveSwitchConnectionStateClosing || c.state() == FMLiveSwitchConnectionStateFailing {
            layoutManager.removeRemoteView(id: remoteMedia.id())
        }
    }
}

channel.addOnRemoteUpstreamConnectionOpen(function(remoteConnectionInfo) {
    ...
    connection.addOnStateChange(function(c) {
        if (c.getState() == fm.liveswitch.ConnectionState.Closing || c.getState() == fm.liveswitch.ConnectionState.Failing) {
            layoutManager.removeRemoteView(remoteMedia.getId());
        }
    });
});

If you need to close a connection manually, you can do so by invoking the Close method of an SfuDownstreamConnection instance or SfuUpstreamConnection instance. The Close method returns a promise, the result of which you can inspect to ensure the connection has been closed properly. Note that if you close your upstream connection, then you effectively disconnects from the session, because the other peers are going to see their downstream connections from you as dropped.

Closing a connection is straightforward - note though, that the result object in this promise is not used, which is why it is assigned the generic object type.

connection.Close().Then((object result) =>
{
    Console.WriteLine("connection closed");
}).Fail((Exception ex) =>
{
    Console.WriteLine("an error occurred");
});

connection.close().then((Object result) -> {
    System.out.println("connection closed");
}).fail((Exception ex) -> {
    System.out.println("an error occurred");
});

connection.close().then(resolveBlock: { (result:NSObject) in
    print("connection closed")
}).fail(rejectBlock: { (ex:NSException) in
    print("an error occurred")
})

connection.close().then(function(result) {
    console.log("connection closed");
}).fail(function(ex) {
    console.log("an error occurred");
});

You now know how to establish and tear down an SFU session. These sessions impose some extra bandwidth requirements on your server components but they allow you to significantly scale up the number of users in a session, due to lower upstream bandwidth requirements. If your server component cannot support the CPU-intensive requirements of an MCU, then an SFU is a good compromise that can still allow you to scale up noticeably.