Custom Codecs

LiveSwitch supports integrating custom codecs for audio to the Media Server. For example, you can use third-party custom media codecs to support different formats such as Advanced Audio Coding (AAC).

Limitations

• Media Server supports custom encoder and decoder only.
• Standard BasicAudioPacketizer and BasicAudioDepacketizer are used for composing custom tracks.
• Media Server supports code for custom codecs in CSharp only.

Implementation

Implement custom codes as a .NET assembly which is late-bounded to the Media Server process during its initialization. That is, the Media Server scans the subdirectory named Extentions to find DLLs that contain implementation of a specific duck typing interface. There is no .NET interface to implement; the functionality is picked up by method signature matching.

Note

Use duck typing to avoid unnecessary dependency on FM.LiveSwitch.dll.

The Media Server dynamically composes media processing tracks using methods found in the extension DLLs.

Important

Method signatures should use basic standard types such as byte, int, string, or arrays to simplify understanding, implementation, and interoperability with native libraries.

During Media Server initialization, the Media Server code searches for classes that contain methods matching the following signatures.

public class AacAudioCodec
{
    public string FormatName => "AAC";
    public int ClockRate => 48000;
    public int ChannelCount => 2;

    public AacEncoder CreateEncoder()
    {
        return new AacEncoder();
    }

    public AacDecoder CreateDecoder()
    {
        return new AacDecoder();
    }
}

Rules

• Class name must end with AudioCodec.
• Get properties: FormatName, ClockRate, ChannelCount
• Name matching: CreateEncoder, CreateDecoder
• Return type must contain constructor which can be implicit default.
• Return type must contain the Destroy, ProcessFrame, and ProcessSdpMediaDescription methods with the signatures as shown in the example below.

Note

You must restart the Media Server after making changes to your class.

Example

public class AacEncoder
{
    public void Destroy()
    {

    }

    public (byte[] data, int offset, int length) ProcessFrame(byte[] data, int offset, int length)
    {
        return (data, offset, length);
    }

    public string ProcessSdpMediaDescription(int payloadType, string mapAttribute, string formatAttribute, bool isOffer, bool isLocalDescription)
    {
        return null;
    }
}

• Initialize this object in the constructor. Parameters for the constructors are optional; the default constructor doesn't have parameters. It’s up to the CreateEncoder or CreateDecoder methods how to construct the object.

• LiveSwitch calls the Destroy method for cleanup and release of any resources or native library references.

• LiveSwitch calls the ProcessFrame method and depending on the encoder's specifications that you implement, there can be either one or many outbound compressed frames. You can define either of the two ProcessFrame methods in the codec:

  • (byte[] data, int offset, int length) ProcessFrame (byte[] data, int offset, int length) receives single array (raw frame) and returns single array (compressed frame)
    The result is a tuple containing processed byte array, offset, and length.
  • (byte[] data, int index, int length)[] ProcessFrame(byte[] data, int index, int length) receives single array (raw frame) and returns multiple arrays (compressed frames)
    You can use this alternate method when the encoder produces multiple frames from a single Pulse-code Modulation (PCM) frame.

• The ProcessSdpMediaDescription method is intended to analyze Session Description Protocol (SDP) media description and update format parameter attributes if needed. LiveSwitch parses the SDP and calls method with the RTP map attribute and related format parameters attribute. LiveSwitch expects back an updated format parameters attribute or null if no changes are required.

To implement custom codec, refer to the code example below:

• CSharp

var audioConfig = new AudioConfig(8000, 1);

var source = new NAudio.Source(audioConfig);
var sink = new NAudio.Sink(audioConfig);

var encoder = new PcmxEncoder(audioConfig);
var decoder = new PcmxDecoder(audioConfig);
var packetizer = new BasicAudioPacketizer(new PcmxFormat(audioConfig));
var depacketizer = new BasicAudioDepacketizer(new PcmxFormat(audioConfig));

// Utilize Custom Codec (Start/Stop SFU or MCU connection)
await source.Start();

var track1 = new AudioTrack(source).Next(encoder).Next(packetizer);
var track2 = new AudioTrack(depacketizer).Next(decoder).Next(sink);

var stream1 = new AudioStream(track1, null);
var stream2 = new AudioStream(null, track2);

var mediaId = "media-sfu";
var (client, token) = CreateClient(mediaId);
var channels = await client.Register(token);
var channel = channels.Single();

var connection1 = channel.CreateSfuUpstreamConnection(stream1, mediaId);
var connection2 = channel.CreateSfuDownstreamConnection(mediaId, stream2);

await connection1.Open();
await connection2.Open();

await source.Stop();

await client.Unregister();