Media-Over-WebSockets

Feature Overview

Media-over-WebSockets is a beta feature that supports sending media over the WebSocket protocol and joining WebRTC-based conferences.

Media-over-WebSockets is helpful in circumstances where WebRTC connections cannot be established due to network policies. This allows end-clients to establish WebSocket-based connections to the WebSocket Media Server. The WebSocket Media server then establishes WebRTC-based media connections to the Media Server.

Currently, only Web and .NET clients are supported. The existing WebRTC Client SDK has been upgraded to support Media-over-WebSockets protocol. Application developers need to make minimal changes to switch between WebRTC and Media-over-WebSockets (see client documentation below).

The feature has been introduced in LiveSwitch version 1.19.0.

Components

WebSocket Media Server that is responsible for sending and receiving media over WebSocket connections. It converts media for WebRTC compatibility, and establishes connections to the WebRTC-based Media server.
WebSocket-enabled clients connect to the Gateway to conduct signaling and to the WebSocket Media Server to send media.

When to Use Media-Over-WebSockets

WebRTC is ideal for real-time communication and it remains to be the preferred choice; however, its implementation may not be viable for some applications facing restricted networks. This is because WebRTC primarily uses UDP for network communication and requires a high range of port numbers to be opened. These ports may be blocked to restrict vulnerabilities they present for network security. Using WebSocket connections to send and receive media allows the media to stream over a well-known TCP port, such as port 443, that can be more easily opened and monitored for security purposes.

To identify whether WebRTC communication is possible, try establishing a WebRTC upstream connection with only a Data Stream (no Audio or Video streams necessary). If successful, WebRTC is not blocked and Media-over-WebSockets is not needed. This check can be done at the application startup, prior to the user attempting to establish connections (to eliminate connection establishing latency). Please refer to an example on how to set up an Upstream WebRTC connection.

Application developers should attempt to use WebRTC as the primary mode of communication and only use WebSockets as a fallback in the case where WebRTC connections fail. In the event the listener for the WebRTC connection state catches the connection failure, another connection can be established by enabling the Media-over-WebSockets protocol.

Finally, application developers may have advanced knowledge about networks where their application is deployed. If advanced information about restrictive networks is available, application developers may choose to bypass all the checks and establish WebSocket-based connections from the start. Nevertheless, it is important to recognize that WebRTC connections should be used when possible.

Features Supported in Beta

This beta release supports the following:

Using WebSockets for SFU connections for audio and video.
The Opus codec is used for audio with a sample rate of 48,000 with 2 channels. No other audio encoding options are currently supported.
The VP8 codec is used for video. No other video encoding options are currently supported.
The bitrates for each media type are fixed for the duration of the connection but can be configured prior to the connection being made.
One conference per web browser end client is supported. Opening a web conference in the second browser tab may lead to audio artifacts.
Performance on mobile browsers and Safari on Intel-based machines may degrade significantly when exceeding 4 simultaneous media connections.

Clients Supported

The following clients are supported for using WebSockets:

Chrome, both desktop and mobile
Safari, both desktop and mobile. Safari on Intel-based desktops supports up to 4 simultaneous media streams.
.NET SDK (MAUI is not yet supported)

Beta Limitations:

Data channels are not supported.
MCU and peer-to-peer connections are not supported.
Opus is the only audio codec supported.
VP8 is the only video codec supported.
Bandwidth adaptation is currently not supported. Excessive CPU load on browser, high network jitter and/or high network loss may cause audio artifacts.
The Firefox browser is not currently supported.

Client Overview

General

The client creates a WebSocket connection to a WebSocket Media Server. It sends and receives encoded media through this WebSocket connection along with other messages for signaling media intent, connection state and key frame requests.

Usage

When using the Web SDKs, using WebSocket connections instead of WebRTC is a matter of passing a boolean at the end of the API method when creating SFU upstream and downstream connections.

CSharp
JavaScript

var upstreamConnection = channel.CreateSfuUpstreamConnection(audioStream, videoStream, true);
var downstreamConnection = channel.CreateSfuDownstreamConnection(audioStream, videoStream, true);

var upstreamConnection = channel.createSfuUpstreamConnection(audioStream, videoStream, true);
var downstreamConnection = channel.createSfuDownstreamConnection(audioStream, videoStream, true);

Note

The last boolean indicates that Media-over-WebSockets is being used.

Web SDK

When a WebSocket connection is used to send media in browsers, we use low-level built-in libraries to capture audio and video directly and forward them to the servers.

C#

For .NET, the client captures media the same way as in the WebRTC way and those encoded frames are sent through the WebSocket connection instead of a WebRTC-based connection. The one difference is that the media is not packetized or depacketized.

An additional step is required to stream media over a WebSocket connection in .NET: To replace the packetizer on upstream connections with an IdentityAudioPipe for the AudioStream and an IdentityVideoPipe for the VideoStream. This is because the media is not packetized or depacketized when sent over WebSockets, as it is not necessary. The IdentityAudioPipe and the IdentityVideoPipe elements only pass through frames and do not modify them, but they complete the pipeline for both sending and receiving media.

There are several ways to create an AudioStream and VideoStream. Below are instructions on how to configure the pipeline correctly in each scenario.

LocalMedia Extending RtcLocalMedia

RtcLocalMedia has a property PacketizerDisabled. When this value is set, useRtcLocalMedia’s factory method to create the Opus and VP8 packetizers create the previously mentioned IdentityAudioPipe and IdentityVideoPipe elements.

CSharp

protected virtual AudioPipe CreateOpusPacketizer(AudioConfig config)
{
    if (PacketizerDisabled)
    {
        return new IdentityAudioPipe(new AudioFormat(AudioFormat.OpusName, config) { IsPacketized = false });
    }
    else
    {
        return new Opus.Packetizer(config);
    }
}

protected virtual VideoPipe CreateVp8Packetizer()
{
    if (PacketizerDisabled)
    {
        return new IdentityVideoPipe(new VideoFormat(VideoFormat.Vp8Name) { IsPacketized = false });
    }
    else
    {
        return new Vp8.Packetizer();
    }
}

If you are extending RtcLocalMedia and intend to send media through a WebSocket connection, you need to set this value PacketizerDisabled to true. If you are overriding this factory method, you need to return an IdentityAudioPipe element instead of a packetizer with the correct audio format of opus/48000/2 and the correct video format of vp8/90000.

AudioTrack Adding IdentityAudioPipe Instead Of Packetizer

When creating an audio track that to be sent over a WebSocket connection, the final packetizer element needs to be replaced with the IdentityAudioPipe. For example:

CSharp

audioTrack.Next(new IdentityAudioPipe(Opus.Format));

VideoTrack Adding IdentityVideoPipe Packetizer

When creating a video track that to be sent over a WebSocket connection, the final packetizer element needs to be replaced with the IdentityVideoPipe. For example:

CSharp

videoTrack.Next(new IdentityVideoPipe(Vp8.Format));

RemoteMedia Extending RtcRemoteMedia

RtcRemoteMedia has a similar property DepacketizerDisabled that needs to be set to create the correct element in place of the Opus and Vp8 Depacketizers.

CSharp

protected virtual VideoPipe CreateVp8Depacketizer()
{
    if (DepacketizerDisabled)
    {
        return new IdentityVideoPipe(new VideoFormat(VideoFormat.Vp8Name));
    }
    else
    {
        return new Vp8.Depacketizer();
    }
}

protected virtual AudioPipe CreateOpusDepacketizer(AudioConfig config)
{
    if (DepacketizerDisabled)
    {
        return new IdentityAudioPipe(new AudioFormat(AudioFormat.OpusName, config));
    }
    else
    {
        return new Opus.Depacketizer(config);
    }
}

If you are extending RtcRemoteMedia and intend to send media thru a WebSocket connection, you need to set this value DepacketizerDisabled to true. If you are overriding this factory method you again need to return an IdentityAudioPipe element instead of a packetizer with the correct audio format of Opus/48000/2 and the correct video format of VP8/90000.

AudioTrack Adding Depacketizer

When creating an AudioTrack that receives audio over a WebSocket connection, the AudioTrack is usually created with a depacketizer for the codec, this is the first element to process the incoming frames. When the frames are received through a WebSocket connection, the AudioTrack should be created with the IdentityAudioPipe element with the Opus format.

CSharp

var audioTrack = new AudioTrack(new IdentityAudioPipe(Opus.Format));

VideoTrack Adding Depacketizer

When creating a VideoTrack that receives video over a WebSocket connection, the VideoTrack is usually created with a depacketizer for the codec, which is the first element to process the incoming frames. When the frames are received through a WebSocket connection, the VideoTrack should be created with the IdentityVideoPipe element with the Vp8 format.

CSharp

var videoTrack = new VideoTrack(new IdentityVideoPipe(Vp8.Format));

Client-Side Troubleshooting

Clients may encounter errors while trying to establish WebSocket-based media connections. If connections cannot be established, please examine the client DEBUG-level logs to identify the issue. Set Up Client Logging Typical errors include:

WebSocketMediaConnectionsNotAllowed: Indicates that the WebSocket Media Connections are not allowed or unavailable for this application or deployment. This may mean that the server does not support the Media-over-WebSockets protocol. Please examine the logs and ensure that the Client and the Gateway are both at least version 1.19.0. This may also indicate that the server administrator has not enabled access to the feature for this deployment or application config. Please contact the server administrator. When this error occurs, connections should not be re-attempted on failure.
WebSocketMediaServerToMediaServerError: Indicates that the WebSocket Media Server encountered a problem connecting to the Media Server over a WebRTC connection. This is a transient error, connections should be re-attempted if this error occurs.
WebSocketMediaClientFailure: Indicates that an error occurred while communicating with a WebSocket Media Server over a service connection. This is a transient error, connections should be re-attempted if this error occurs.
WebSocketMediaServerRejectedConnectionTerminally: Indicates that the WebSocket Media Server rejected the connection due to the capacity policy. The client should not retry. Server may restrict the number of media connections that can be served over a WebSocket connection. Application layer may choose to restrict the number of connections further for low powered mobile devices that may not be able to handle up to 10 media connections over WebSockets.

Server Side

How It Works

The WebSocket Media Server handles and manages the WebSocket connections. On the server, WebRTC connections are created that send and receive media via WebRTC with remote connections. The WebSocket clients connect to this WebSocket Media Server via a single WebSocket connection.

Installation, Setup, Configuration

Multiple WebSocket Media Servers can handle the media system. A load balancer can be set up to handle how the server is selected depending on the server capacity utilization. The following deployment configurations control WebSocket Media Server installation configuration:

WebSocketMediaServerConnectionsAllowed - indicates whether Media-over-WebSockets is allowed for this deployment.
WebSocketMediaServerUrl - URL of the WebSocket Media Server or a related loadbalancer.
WebSocketMediaServerPort - port of the WebSocket Media Server or a related loadbalancer. Defaults to 7001.

Application configuration allows to override the deployment setting indicating whether Media-over-WebSockets is allowed. This is useful when the feature should only be enabled for some applications.

WebSocketMediaServerConnectionsAllowedOverride - indicates whether Media-over-WebSockets is allowed for this application.

Capacity Controls

Availability thresholds that determine server utilization can be accessed via REST API and environment variables:

WssCpuThreshold - Indicates the CPU percentage used to determine if a WebSocket Media Server is over capacity.
WssMemoryThreshold - Indicates the memory usage (in bytes) used to determine if a WebSocket Media Server is over capacity.
WssConnectionsPerCpuThreshold - Indicates the threshold of the number of clients connected over WebSockets per CPU to determine if a WebSocket Media Server is over capacity. Note that this is not the number of media connections, but rather the number of clients that may send and receive multiple SFU-style connections. Currently set to 4 by default.
WssMediaConnectionsPerClientThreshold - Indicates the threshold of the number of media connections per a WebSocket client to determine if a client is over capacity. No more than this number of media connections are allowed to be handled by a WebSocket connection serving this client. Unset by default. Currently recommended to be set to 10. The application layer should restrict the number of connections further for mobile devices.

The Gateway is responsible for access control. Access to the Media-over-WebSockets feature can be enabled on the Deployment or Application (but not Channel) level. Use WebSocketMediaServerConnectionsAllowed in the Application on Channel configuration to allow Media-over-WebSockets. Disabled by default. If allowed, the Gateway sends the WebSocket Media Server credentials and address in the registration response. The address can be set by accessing WebSocketMediaServerUrl and WebSocketMediaServerPort properties in the Deployment Config.