MCU Connection

Note

About the code examples on this page:

For .NET MAUI and Unity, use the C# code.
For macOS, use the iOS code.

MCU is an acronym that stands for Multipoint Control Unit. An MCU is an endpoint in a media session that allows for large amounts of users to participate in a video conferencing session by mixing their audio and video streams together on the server. Each participant in a video conference connects directly to the MCU, and the MCU creates one combined audio and video stream, which it then sends back to everyone. For session participants, this mode uses the least amount of bandwidth and CPU, as each participant only has one upstream and one downstream connection. However, the downside of this is that this mode uses the largest amount of bandwidth and CPU resources on the server side, as the MCU is now responsible for performing all audio and video mixing.

Create an MCU Connection

Start by registering with the LiveSwitch gateway and obtaining an FM.LiveSwitch.Channel instance. This is covered in the previous Registering a Client section. You should also review the Handling Local Media, Handling Remote Media, and Creating Streams and Connections sections, which review the media components and streams that are used in this guide.

In some ways, creating an MCU connection is the simplest option, because there is only one connection for you to manage. In other ways, however, it is trickier, because the output of a video conferencing session is completely controlled on the server. This guide walks you through connecting to a LiveSwitch Server in MCU mode, and helps you navigate some of the gotchas involved in doing so.

Make a Mixed Connection

To create your mixed connection, invoke the CreateMcuConnection method of your Channel instance, and pass in your audio and video streams. This returns an FM.LiveSwitch.McuConnection instance. With this McuConnection instance, you should assign ICE servers as described in the Creating Streams and Connections section. Once you have set these servers and any other connection-specific properties, open the connection by invoking the Open method. This method returns a promise, which resolves when the connection is established and is rejected if the connection does not. The code samples below show how to accomplish this.

Once complete, you are now connected to the server with an MCU connection. Because this is an MCU connection, there are no other connections to establish. The only other time that you have to deal with your McuConnection object is when you leave the session, which is covered in the next section.

var remoteMedia = new RemoteMedia();
var audioStream = new FM.LiveSwitch.AudioStream(localMedia, remoteMedia);
var videoStream = new FM.LiveSwitch.VideoStream(localMedia, remoteMedia);
var connection = channel.CreateMcuConnection(audioStream, videoStream);
layoutManager.AddRemoteView(remoteMedia.Id, remoteMedia.View);
connection.OnStateChange += (FM.LiveSwitch.ManagedConnection c) =>
{
    if (c.State == FM.LiveSwitch.ConnectionState.Closing || c.State == FM.LiveSwitch.ConnectionState.Failing)
    {
        layoutManager.RemoveRemoteView(remoteMedia.Id);
    }
};
connection.IceServers = ...
connection.Open().Then((result) =>
{
    Log.Info("Mixed Connection Established");
}).Fail((ex) =>
{
    Log.Info("An error occurred");
});

RemoteMedia remoteMedia = new RemoteMedia();
fm.liveswitch.AudioStream audioStream = new fm.liveswitch.AudioStream(localMedia, remoteMedia)
fm.liveswitch.VideoStream videoStream = new fm.liveswitch.VideoStream(localMedia, remoteMedia)
fm.liveswitch.McuConnection connection = channel.createMcuConnection(audioStream, videoStream);
layoutManager.addRemoteView(remoteMedia.getId(), remoteMedia.getView());
connection.addOnStateChange((fm.liveswitch.ManagedConnection c) -> {
    if (c.getState() == fm.liveswitch.ConnectionState.Closing || c.getState() == fm.liveswitch.ConnectionState.Failing) {
        layoutManager.removeRemoteView(remoteMedia.getId());
    }
});
connection.setIceServers(...);
connection.open().then((Object result) -> {
    System.out.println("mixed connection established");
}).fail((Exception ex) -> {
    System.out.println("an error occurred");
});

var remoteMedia = RemoteMedia();
var audioStream = FMLiveSwitchAudioStream(localMedia: localMedia, remoteMedia: remoteMedia)
var videoStream = FMLiveSwitchVideoStream(localMedia: localMedia, remoteMedia: remoteMedia)
var connection:FMLiveSwitchMcuConnection = channel.createMcuConnection(audioStream: audioStream, videoStream: videoStream)
layoutManager.addRemoteView(id: remoteMedia.id(), view: remoteMedia.view())
connection.addOnStateChange { (c:FMLiveSwitchManagedConnection) in
    if c.state() == FMLiveSwitchConnectionStateClosing || c.state() == FMLiveSwitchConnectionStateFailing {
        layoutManager.removeRemoteView(id: remoteMedia.id())
    }
}
connection.setIceServers(...)
connection.open().then(resolveBlock: { (result:NSObject) in
    print("mixed connection established")
}).fail(rejectBlock: { (ex:NSException) in
    print("an error occurred")
})

var remoteMedia = new fm.liveswitch.RemoteMedia();
var audioStream = new fm.liveswitch.AudioStream(localMedia, remoteMedia);
var videoStream = new fm.liveswitch.VideoStream(localMedia, remoteMedia);
var connection = channel.createMcuConnection(audioStream, videoStream);
layoutManager.addRemoteView(remoteMedia.getId(), remoteMedia.getView());
connection.addOnStateChange(function(c) {
    if (c.getState() == fm.liveswitch.ConnectionState.Closing || c.getState() == fm.liveswitch.ConnectionState.Failing) {
        layoutManager.removeRemoteView(remoteMedia.getId());
    }
});
connection.setIceServers(...);
connection.open().then(function(result) {
    console.log("mixed connection established");
}).fail(function(ex) {
    console.log("an error occurred");
});

Close a Connection

You should close an MCU connection when you wish to leave a session. This is not strictly required, as LiveSwitch detects when a user has disconnected. However, it is generally a better experience for the end-users if you explicitly destroy the connection yourself because the server is notified immediately that someone has left the session.

To close an MCU connection is, invoke the Close method of the FM.LiveSwitch.McuConnection instance. Similar to the Open method, Close returns a promise. Again, you should inspect the result of the promise to ensure that teardown is performed successfully. This is demonstrated below.

When you have finished working through this code example, you will have successfully opened and closed a connection to the server. However, you haven't really done anything with this connection yet. The next section of this guide focuses on actually displaying the data you receive from the MCU.

connection.Close().Then((object result) =>
{
    Console.WriteLine("connection closed");
}).Fail((Exception ex) =>
{
    Console.WriteLine("an error occurred");
});

connection.close().then((Object result) -> {
    System.out.println("connection closed");
}).fail((Exception ex) -> {
    System.out.println("an error occurred");
});

connection.close().then(resolveBlock: { (result:NSObject) in
    print("connection closed")
}).fail(rejectBlock: { (ex:NSException) in
    print("an error occurred")
})

connection.close().then(function(result) {
    console.log("connection closed");
}).fail(function(ex) {
    console.log("an error occurred");
});

Manage the Layout

There are two main aspects of layout that you have to consider - the local view and remote views. The local view is a video preview of your own camera. This is common in video conferencing applications, as users often want to adjust their camera so that they are properly in view. Remote views, on the other hand, are the views that show everyone else. In the context of an MCU, there is only one remote view - the remote view that displays the video stream generated by the MCU. The next sections focus on managing the layout for this local and remote view.

Update the MCU Layout

The number of participants in a media session often changes over time. As this occurs, the MCU automatically changes how it lays out the video feeds. Sometimes, this also requires an update on the client side, to display the layout with proper margins and padding. When such an update is required, an OnMcuVideoLayout event is raised on the FM.LiveSwitch.Channel instance that is associated the MCU session. In your app, you must add an event handler to the Channel instance to handle this.

This event handler has two responsibilities. Its first responsibility is to call the Layout method of your layout manager. As a UI operation, this must take place on the main thread. The event handler's other responsibility is to cache the FM.LiveSwitch.VideoLayout instance that is returned as a parameter from the event handler. The use of the VideoLayout instance is covered in the next section - for now, know that you must store it somewhere.

channel.OnMcuVideoLayout += (FM.LiveSwitch.VideoLayout videoLayout) =>
{
    this.videoLayout = videoLayout;
  
    if (layoutManager != null)
    {
        Dispatcher.Invoke(new Action(() =>
        {
            layoutManager.Layout();
        }));
    }  
};

channel.addOnMcuVideoLayout((fm.liveswitch.VideoLayout videoLayout) -> {
    this.videoLayout = videoLayout;
    if (layoutManager != null) {
        myActivity.runOnUiThread(new Runnable() {
            @Override
            public void run() {
                layoutManager.layout();
            }
        });
    }
});

channel.addOnMcuVideoLayout { (videoLayout:FMLiveSwitchVideoLayout) in
    self.videoLayout = videoLayout
    if layoutManager != nil {
        DispatchQueue.main.async {
            layoutManager.layout()
        }
    }
}

channel.addOnMcuVideoLayout(function(videoLayout) {
    this.videoLayout  = videoLayout;
    if (layoutManager != null) {
        layoutManager.layout();
    }
});

The preceding code takes a new layout from the server and applies it to your layout manager. You may also wish to apply additional effects to the layout now, which is why you need the previously cached VideoLayout instance. One common effect is to float the local video preview, so that it appears off to the side, on top of the remote video feeds. To do this, first add an OnLayout event handler to your FM.LiveSwitch.LayoutManager instance. In this event handler, invoke the FloatLocalPreview method of the FM.LiveSwitch.LayoutUtility utility class. You must pass in the previously cached VideoLayout instance, the Layout instance that you are provided as a parameter for the OnLayoutevent handler and the id of your McuConnection instance. The example below shows how to do this.

layoutManager.OnLayout += (FM.LiveSwitch.Layout layout) =>
{
    if (this.mcuConnection != null)
    {
        FM.LiveSwitch.LayoutUtility.FloatLocalPreview(layout, this.videoLayout, this.mcuConnection.Id);
    }
};

layoutManager.addOnLayout((fm.liveswitch.Layout layout) -> {
    if (this.mcuConnection != null) {
        fm.liveswitch.LayoutUtility.floatLocalPreview(layout, this.videoLayout, this.mcuConnection.getId());
    }
});

layoutManager.addOnLayout { (layout:FMLiveSwitchLayout) in
    if self.mcuConnection != nil {
        FMLiveSwitchLayoutUtility.floatLocalPreview(layout: layout, videoLayout: self.videoLayout, localConnectionId: self.mcuConnection.id())
    }
}

layoutManager.addOnLayout(function(layout) {
    if (mcuConnection != null) {
        fm.liveswitch.LayoutUtility.floatLocalPreview(layout, videoLayout, mcuConnection.getId());
    }
});

You now know how to setup and teardown an MCU connection. As mentioned above, this connection method is the simplest from an app perspective, but is most reliant on the bandwidth and CPU capabilities of your server. In general, you want to save MCU mode for conferencing sessions that truly require it - conferences of 9 or more people, as this can provide you substantial savings on server costs.