Filtering instant messages with the Lync Server SDK

Recently I was answering a question about filtering instant messages between Lync users, and I decided that this would make a good topic for a post. It’s a fairly common use case, and one that is not too easy to implement because it requires the arcane and mysterious Lync Server SDK, which is written in hieroglyphics and can only be used by the light of a full moon.

Okay, that last part isn’t true, but it is true that the Lync Server SDK is one of the least well-known and most challenging APIs in the Lync development platform. It consists of two components, the Microsoft SIP Processing Language, a.k.a. MSPL, and the Managed SIP Application API. The latter of these is the one we’ll need to use to filter instant messages.

Let me start by defining exactly what I mean by instant message filtering. This can mean a few different things, but what I’m talking about here is inspecting IMs as they go through the Front End Server and “bleeping out” undesirable words, like the names of legacy PBX systems, with a placeholder like ****. This way, the recipient of the IM gets something like the following:

Hey, we just replaced our old **** system with Microsoft Lync!

Some organizations may have other types of words they want to filter out, but you get the picture.

I want to point out a couple of things here that you should know. First of all, don’t try the following example in production. It’s sample code, and does not have exception handling or a lot of other things you should have in a production-ready application. Second, it’s important to realize that using a managed SIP application like this in your Lync environment can have a performance impact. There are some ways you could optimize the application that I haven’t done here because they would make it very complex for a sample app, but in general you should be aware of the performance implications if you use a technique like this in your own application.

Oh, one other note: I used a Lync 2010 Front End Server to test this sample, but everything should work almost identically on Lync Server 2013.

Now that we’re clear on what this application will do, how do we build it?

The first step is to create something called a manifest file, which contains basic information in XML for Lync Server about the application and what types of messages it is interested in. This file needs to have a .am extension.

Here’s the manifest file we’re going to use. Take a look at it and then I’ll explain the different pieces.

<?xml version="1.0"?>
<r:applicationManifest
 r:appUri="http://mspl.greenl.ee/ServerSideIMFilter"
 xmlns:r="http://schemas.microsoft.com/lcs/2006/05">
<r:allowRegistrationBeforeUserServices action="true" />

<!--
  - Handle all IMs, including those with strict routes
  - and those with request URIs that don't match this server's
  - domain.
  -->
<r:requestFilter methodNames="INVITE,MESSAGE"
                 strictRoute="true"
                 domainSupported="false" />

<!--
  - Handle none of the responses.
  -->
<r:responseFilter reasonCodes="NONE"/>

<!-- Dispatch all MESSAGE requests and any INVITEs where the SDP indicates the IM media type
  -  to the managed code component.
  -->
<r:splScript><![CDATA[
if (sipRequest && (sipRequest.Method == "MESSAGE" || ContainsString(sipRequest.Content, "m=message", true)))
{
    Dispatch("OnRequest");
}

]]></r:splScript>
</r:applicationManifest>

The first element that we’re interested in is the <r:applicationManifest> at the top. Notice that it needs to have an r:appUri attribute which gives a sort of unique identifier for the application in the form of a URI. This doesn’t need to point to a real web URL; just pick a URI that will uniquely identify your application.

Next, we have the following:

<r:allowRegistrationBeforeUserServices action="true" />

This allows your application to register with Lync Server before the built-in services that Lync uses to route messages. Registering before those services means that the application can override some behaviour that is otherwise controlled by the built-in services.

The next item is a request filter:

<r:requestFilter methodNames="INVITE,MESSAGE"
                 strictRoute="true"
                 domainSupported="false" />

The most important bit that you need to look at here is the methodNames attribute. This is a comma-delimited list of SIP methods that the application will intercept. In this case, we’re only interested in INVITE requests and MESSAGE requests. You can also use the word ALL or the word NONE. Note that some methods won’t work here. For example, you don’t seem to be able to intercept CANCEL requests, probably because they’re a sort of special type of request.

After the request filter is a response filter.

<r:responseFilter reasonCodes="NONE"/>

We don’t need to worry about any of the responses, so I’ve put NONE here. You could also use ALL or a comma-delimited list of reason codes, like 100,180,200.

Finally, we have the actual script part of the manifest. With “script-only applications,” which are what most people mean when they talk about MSPL, the script is the entire application. In this case, the script doesn’t do much except grab all MESSAGE requests and any INVITEs that are for IM conversations and “dispatch” them to the managed code part of the application. Let’s take a quick look at the script.

<r:splScript><![CDATA[
if (sipRequest && (sipRequest.Method == "MESSAGE" || ContainsString(sipRequest.Content, "m=message", true)))
{
    Dispatch("OnRequest");
}

]]></r:splScript>

The way we identify INVITE requests for IMs is by looking for a specific bit of text in the SDP (Session Description Protocol) body of the INVITE. The text m=message means that the media type is IM. We don’t really care much about A/V calls or application sharing (although if you were being really careful you might also want to filter the subjects of those other types of calls, the string that appears in the little pop-up window and on the title bar of the conversation window).

For any messages that need filtering, we then send them off to the other part of the application using the Dispatch function. The parameter is the name of a method in our handler class (which we’ll talk about in a minute) that will handle the dispatched message.

That’s the manifest. Not too bad, right? Let’s move on to the rest of the application, which is a console app written in C# using .NET 3.5.

First of all, you’ll need to add a reference to ServerAgent.dll, which you can find by default at C:\Program Files\Microsoft Lync Server 2010\SDK\Bin\ServerAgent.dll assuming you’ve installed the Lync core components already. Also, make sure you are targeting x64 and .NET 3.5.

Next, we need a handler class to actually do the filtering of messages. Here’s one called ServerSideIMFilter:

using System;
using System.Configuration;
using Microsoft.Rtc.Sip;

namespace ServerSideIMFilter
{
    public class ServerSideIMFilter
    {
        public const string MsBody = "ms-body=";
        public const string MsImFormat = "ms-im-format";
        public const string MsTextFormat = "ms-text-format";
        public const string FilterReplacementString = 
            "****";

        string[] _wordsToFilter;
        readonly object _syncObject = new object();

        public void OnRequest(object sender,
            RequestReceivedEventArgs e)
        {
            // Enable simple proxy mode and disable forking.
            e.Request.SimpleProxy = true;
            e.ServerTransaction.EnableForking = false;

            // Load the filtered words if necessary.
            LoadWordsToFilter();

            // Get a collection of all headers 
            // on the request.
            HeaderCollection headers = 
                e.Request.AllHeaders;

            // Filter the message content.
            e.Request.Content = FilterString(
                e.Request.Content);

            if (e.Request.Method == "INVITE")
            {
                // Filter the initial message in an
                // INVITE request.
                FilterToastMessageHeaders(headers);
            }

            Console.WriteLine("Applied filter to a message.");
                
            // Add a header for debugging purposes.
            headers.Add(new Header("Filter-Applied", "yes"));

            // Send the request along.
            e.ServerTransaction.CreateBranch().SendRequest(
                e.Request);
        }

        private void FilterToastMessageHeaders(HeaderCollection headers)
        {
            foreach (Header header in headers)
            {
                // In the initial INVITE, the "toast message" 
                // is in Base64 form in one or more SIP headers.
                if (header.Type.ToLower() == MsImFormat ||
                    header.Type.ToLower() == MsTextFormat)
                {
                    int msBodyLocation = header.Value.IndexOf(
                        MsBody);
                    string textToDecode = header.Value.Substring
                        (msBodyLocation + MsBody.Length);

                    string decodedHeader =
                        Base64Utils.DecodeUtf8FromBase64(
                        textToDecode);

                    // Filter the decoded toast message.
                    decodedHeader = FilterString(decodedHeader);

                    // Put the toast message back into the 
                    // appropriate header.
                    header.Value = header.Value.Substring(0,
                        header.Value.Length - textToDecode.Length) +
                        Base64Utils.EncodeUtf8ToBase64(decodedHeader);
                }
            }
        }

        private void LoadWordsToFilter()
        {
            if (_wordsToFilter == null)
            {
                lock (_syncObject)
                {
                    if (_wordsToFilter == null)
                    {
                        string filteredWordsConfig =
                            ConfigurationManager.AppSettings[
                            "filteredWords"];
                        _wordsToFilter =
                            filteredWordsConfig.Split(',');
                    }
                }
            }
        }

        private string FilterString(string stringToFilter)
        {
            string returnValue = stringToFilter;

            // Replace each instance of a filtered 
            // word with a placeholder.
            foreach (string wordToFilter in 
                _wordsToFilter)
            {
                returnValue = returnValue.Replace(
                    wordToFilter, FilterReplacementString);
            }

            return returnValue;
        }
    }
}

Let’s go through it piece by piece. The key part is the OnRequest method. Note that the name of the method doesn’t have to be OnRequest – that’s just the method name I referred to in the .am file.

public void OnRequest(object sender,
    RequestReceivedEventArgs e)
{
...
}

It is important, though, to have the method signature right, with RequestReceivedEventArgs as the second parameter. This is how you’ll get the information about the message that has been dispatched.

The first thing the OnRequest method does is to enable some optimizations called Simple Proxy, which we can use because we aren’t doing anything with routing, or forking the messages. Dispatching messages to managed code can have a significant impact on performance, so we naturally want to optimize the message handling as much as possible.

// Enable simple proxy mode and disable forking.
e.Request.SimpleProxy = true;
e.ServerTransaction.EnableForking = false;

Next, we load in the collection of words to filter if that hasn’t already happened, grab the collection of SIP headers in the request, and filter the content of the message.

// Load the filtered words if necessary.
LoadWordsToFilter();

// Get a collection of all headers
// on the request.
HeaderCollection headers =
    e.Request.AllHeaders;

// Filter the message content.
e.Request.Content = FilterString(
    e.Request.Content);

That Content property on the request refers to the actual body of the message, the part after all of the SIP headers. That’s where the actual text of an instant message is located in all but the first message when you are starting the conversation. We’ll get to that first message in a minute.

I won’t get into the actual filtering code here, since it isn’t at all Lync-specific – it just loops through the list of bad words, and replaces them with a predefined placeholder string. I’m sure there are some ways this code could be optimized, but this is just an example, so I kept things simple.

Things get a little more complicated for INVITE messages, which are the ones that actually initiate the instant message conversation.

if (e.Request.Method == "INVITE")
{
    // Filter the initial message in an
    // INVITE request.
    FilterToastMessageHeaders(headers);
}

In Lync, the first message is sort of packaged into the INVITE request so that it can be shown in the pop-up (“toast”) on the recipient’s desktop. In the INVITE request, the text isn’t in the message body, which is already taken up by SDP. Instead, Lync sticks it into a few SIP headers in Base 64 form. That’s the weird-looking mash of numbers, letters, and occasional equal signs that looks something like this: U29ycnksIG5vIGludGVyZXN0aW5nIG1lc3NhZ2UgaGVyZSE=

This is the code I came up with to filter the contents of these headers:

private void FilterToastMessageHeaders(HeaderCollection headers)
{
    foreach (Header header in headers)
    {
        // In the initial INVITE, the "toast message" 
        // is in Base64 form in one or more SIP headers.
        if (header.Type.ToLower() == MsImFormat ||
            header.Type.ToLower() == MsTextFormat)
        {
            int msBodyLocation = header.Value.IndexOf(
                MsBody);
            string textToDecode = header.Value.Substring
                (msBodyLocation + MsBody.Length);

            string decodedHeader =
                Base64Utils.DecodeUtf8FromBase64(
                textToDecode);

            // Filter the decoded toast message.
            decodedHeader = FilterString(decodedHeader);

            // Put the toast message back into the appropriate header.
            header.Value = header.Value.Substring(0,
                header.Value.Length - textToDecode.Length) +
                Base64Utils.EncodeUtf8ToBase64(decodedHeader);
        }
    }
}

It goes through the headers looking for Ms-Text-Format or Ms-Im-Format headers, and when it finds one it grabs the part that represents the message body, which is the part after ms-body=, and decodes it, runs it through the filter, and encodes it again. Then it sticks it back after the ms-body= and sends the message on its way.

Here is the Base64Utils class, by the way:

namespace ServerSideIMFilter
{
    public static class Base64Utils
    {
        public static string DecodeUtf8FromBase64(string input)
        {
            byte[] inputBytes = System.Convert.FromBase64String(input);
            return System.Text.Encoding.UTF8.GetString(inputBytes);
        }

        public static string EncodeUtf8ToBase64(string input)
        {
            byte[] inputBytes = System.Text.Encoding.UTF8.GetBytes(input);
            return System.Convert.ToBase64String(inputBytes);
        }
    }
}

Finally, with all the filtering done, we write to the console, add a SIP header to the message to show it’s been filtered (useful if we’re debugging using the Lync Server Logging Tool later on) and send the request along by calling e.ServerTransaction.CreateBranch().SendRequest(e.Request).

That’s the handler class. The last two pieces are simple. First, we need a Program.cs to actually start up the application:

using System;
using System.Threading;
using Microsoft.Rtc.Sip;

namespace ServerSideIMFilter
{
    class Program
    {
        static void Main(string[] args)
        {
            ServerSideIMFilter serverApplication = new ServerSideIMFilter();

            try
            {
                // Try to connect to the server 5 times.
                ServerAgent.WaitForServerAvailable(5);
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }

            Environment.CurrentDirectory = System.AppDomain.CurrentDomain.BaseDirectory;

            // Load the app manifest from a file.
            ApplicationManifest manifest = ApplicationManifest.CreateFromFile("ServerSideIMFilter.am");
            try
            {
                // Try to compile the manifest.
                manifest.Compile();
            }
            catch (CompilerErrorException ex)
            {
                Console.WriteLine(ex);
            }

            ServerAgent agent = null;
            try
            {
                // Create the new server agent object, setting
                // the ServerSideIMFilter object as the handler for messages.
                agent = new ServerAgent(serverApplication, manifest);
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }

            if (agent != null)
            {
                Console.WriteLine("Server application started.");
                while (true)
                {
                    // Wait for a message to arrive and then handle it.
                    agent.WaitHandle.WaitOne();
                    ThreadPool.QueueUserWorkItem(new WaitCallback(agent.ProcessEvent));
                }
            }
            else
            {
                Console.WriteLine("Server application failed to start.");
            }
        }
    }
}

Also, we need a very small App.config with the words to filter:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="filteredWords" value="PBX,filter"/>
  </appSettings>
</configuration>

Once all of this is in place, the application is ready. Build it, move it to a test Lync environment, making sure the .am file is in there too, and install it using the New-CsServerApplication commandlet. You can find instructions on how to do this in my earlier blog post on the Managed SIP Application API.