Create an SFU Connection

In this tutorial, you'll learn how to create an SFU connection to stream video and audio in a conference.

Prerequisites

This tutorial requires the Handle Media app that you created earlier.

Create an Upstream Connection

To send audio and video data, create two streams: one for audio data and one for video data. These streams are represented by the AudioStream and VideoStream classes. To create a uni-directional upstream, we create an instance of each of these classes, and provide only the LocalMedia instance to make it a send-only stream.

To establish an SFU upstream connection:

  1. Invoke the CreateSfuUpstreamConnection method from the Channel instance, and specify the send-only audio and video streams. This returns an SfuUpstreamConnection instance that's set up to send but doesn't receive any data.
  2. Invoke the Open method of the SfuUpstreamConnection instance to establish an upstream connection. This returns a promise that verifies if the upstream connection is established successfully.

Paste the following code into the HelloWorldLogic class.

private SfuUpstreamConnection _UpstreamConnection;
// Creating a SFU upstream connection
private SfuUpstreamConnection OpenSfuUpstreamConnection(LocalMedia localMedia)
{
    AudioStream audioStream = (localMedia.AudioTrack != null) ? new AudioStream(localMedia) : null;
    VideoStream videoStream = (localMedia.VideoTrack != null) ? new VideoStream(localMedia) : null;
    SfuUpstreamConnection connection = Channel.CreateSfuUpstreamConnection(audioStream, videoStream);

    connection.OnStateChange += (conn) =>
    {
        Log.Info(string.Format("Upstream connection {0} is currently in a {1} state.", conn.Id, conn.State.ToString()));

        if (conn.State == ConnectionState.Closing || conn.State == ConnectionState.Failing)
        {
            if (conn.RemoteClosed)
            {
                Log.Info(string.Format("Upstream connection {0} was closed,", conn.Id));
            }

        }

        else if (conn.State == ConnectionState.Failed)
        {
            OpenSfuUpstreamConnection(localMedia);
        }
    };

    connection.Open();
    return connection;
}

Create a Downstream Connection

To receive audio and video data, create an AudioStream and a VideoStream, and provide only the RemoteMedia instance to make it as receive-only streams.

To establish an SFU downstream connection:

  1. Invoke the CreateSfuDownstreamConnection method from the Channel instance. This returns an SfuDownstreamConnection instance that only receives data.
  2. Add the RemoteMedia instance to the layout manager by invoking the AddRemoteView method of the LayoutManager instance.
  3. Invoke the Open method of the SfuDownstreamConnection instance to establish a downstream connection.

When a user leaves a session, we need to properly tear down the session by removing the remote view associated with them. To do so, add an OnStateChange event handler to each SfuDownstreamConnection instance. In the handler, inspect the state of the SfuDownstreamConnection instance. If the state is Closing or Failing, remove the associated remote view by invoking the RemoveRemoteView instance of the layout manager.

Paste the following code into the HelloWorldLogic class.

private Dictionary<string, SfuDownstreamConnection> _DownStreamConnections = new Dictionary<string, SfuDownstreamConnection>();

// Creating a SFU downstream connection
private SfuDownstreamConnection OpenSfuDownStreamConnection(ConnectionInfo remoteConnectionInfo)
{
    RemoteMedia remoteMedia = null;

    _Dispatcher.Invoke(() =>
    {
        remoteMedia = new RemoteMedia(false, false, _AecContext);
        if (remoteMedia.View != null)
        {
            remoteMedia.View.Name = "RemoteView_" + remoteMedia.Id;
        }
    });

    _LayoutManager.AddRemoteView(remoteMedia.Id, remoteMedia.View);

    AudioStream audioStream = (remoteConnectionInfo.HasAudio) ? new AudioStream(remoteMedia) : null;
    VideoStream videoStream = (remoteConnectionInfo.HasVideo) ? new VideoStream(remoteMedia) : null;
    SfuDownstreamConnection connection = Channel.CreateSfuDownstreamConnection(remoteConnectionInfo, audioStream, videoStream);
    _DownStreamConnections.Add(remoteMedia.Id, connection);

    connection.OnStateChange += (conn) =>
    {
        Log.Info(string.Format("Downstream connection {0} is currently in a {1} state.", conn.Id, conn.State.ToString()));

        if (conn.State == ConnectionState.Closing || conn.State == ConnectionState.Failing)
        {
            _LayoutManager.RemoveRemoteView(remoteMedia.Id);
            _Dispatcher.Invoke(() =>
            {
                remoteMedia.Destroy();
            });

            _DownStreamConnections.Remove(remoteMedia.Id);
        }
    };

    connection.Open();
    return connection;
}

Handle Stream Connection

In an SFU session, we must create a new downstream connection every time that a client joins the channel. To do so, add an event handler for the OnRemoteUpstreamConnectionOpen event. This event is raised whenever a remote client opens an upstream connection on this channel. We add the event handler to the OnClientRegistered class.

Update the OnClientRegistered function in the HelloWorldLogic class with the following:

// Procedure to run once client is registered
private void OnClientRegistered(Channel[] channels)
{
    Channel = channels[0];
    DisplayMessage($"Client {Client.Id} has successfully connected to channel {Channel.Id}, Hello World!");

    Channel.OnRemoteUpstreamConnectionOpen += (remoteConnectionInfo) =>
    {
        Log.Info("An upstream connection opened.");
        OpenSfuDownStreamConnection(remoteConnectionInfo);
    };

    _UpstreamConnection = OpenSfuUpstreamConnection(LocalMedia);

    foreach (var remoteConnectionInfo in Channel.RemoteUpstreamConnectionInfos)
    {
        OpenSfuDownStreamConnection(remoteConnectionInfo);
    }
}

Run Your App

Note

LiveSwitch recommends that you use a phone and not an emulator to run the mobile apps. It's possible to run the mobile apps on an emulator, but the camera doesn't work.

Run your app in your project IDE and click Join. You should see video streaming on your app window.

For an SFU connection, the Media Server only sends back other participant's views. If you run your app on two devices, you should see the local view is displayed on the corner of the window and the remote view is displayed on the main window.

Congratulations, you've built a LiveSwitch conference app!

Go to your LiveSwitch Console's homepage, and select the Hello World Application. You should see one client, one channel, and one connection.

Go to your LiveSwitch Cloud's Dashboard page, and select the Hello World Application. You should see one client, one channel, and one connection.