Trace Filtering

One of the best practices for working with Phobos in high throughput Akka.NET systems is to use Phobos.Tracing's filtering capabilities to help reduce noise collected inside the tracing system. This article explains how to use those filtering capabilities.

A quick introducing to trace filtering by way of example:

    var config = @"
    akka.actor.provider = ""Phobos.Actor.PhobosActorRefProvider,Phobos.Actor""
    
    phobos.tracing{
        provider-type = test
        filter{
            mode = blacklist
            message-types = [
                ""Phobos.Docs.Samples.Tests.Tracing.IFilteredMessage, Phobos.Docs.Samples.Tests""
            ]
        }
    }
";

    sys = ActorSystem.Create("PhobosTest", config);
    var tracer = (MockTracer) ActorTracing.For(sys).Tracer;
    var actor = sys.ActorOf(Props.Create(() => new EchoActor()), "echo");

    // send a message that WON'T be filtered out by blacklist
    actor.Ask<NormalMessage>(new NormalMessage("hi"), TimeSpan.FromMilliseconds(100)).Wait();
    AwaitAssert(() => tracer.FinishedSpans().Count.Should().Be(1)); // no trace recorded
    tracer.Reset(); // reset the MockTracer 

    // send a message that WILL be filtered out by blacklist
    actor.Ask<FilteredMessage>(new FilteredMessage("bye"), TimeSpan.FromMilliseconds(100)).Wait();
    Task.Delay(100).Wait(); // wait for activity in other threads to stop
    tracer.FinishedSpans().Count.Should().Be(0);

In the sample above, Phobos uses a blacklist filter by default to exclude all messages that implement the interface Phobos.Docs.Samples.Tests.Tracing.IFilteredMessage from any new or ongoing traces inside Phobos. The assertions included in the code sample validate that messages which do not implement this interface, such as a System.String, will be received by the actor and a trace will be recorded.

This is the gist of how trace filtering works: trace the interesting messages and don't bother tracing the rest.

If you need to learn more about how to configure Phobos.Tracing and the filter system, please see the Phobos.Actor configuration settings.

Filtering Modes

Phobos.Tracing exposes two different methods for filtering data inside your Akka.NET applications:

• Blacklist filtering - this means that the tracing system excludes any messages which appear on this list. Any messages that don't appear on the blacklist are included in the trace.
• Whitelist filtering - this means that the tracing system ONLY STARTS NEW TRACES when a message type that appears on this list is received. From that point onward, any other messages downstream in the trace will also be included in the trace, but only the message types that appear on this list can cause a new trace to start.

Matching Message Types

An important note about how the message-types setting works in Phobos: in addition to matching message types exactly the filter will also include or exclude types that are assignable from the specified types on this list.

For instance, if you include IEnumerable<string> in the message-types list for a whitelist filter then any messages of type List<string> or HashSet<string> would also be included in the whitelist. This design is intentional and it's intended to make it easier for developers to target broad categories of messages that implement specific interfaces or base classes.

Whitelist Filtering

Whitelist filtering is primarily used for noise reduction inside busy Akka.NET applications, designed ensure that only specific message types can start a trace. Once a trace is started with whitelist filtering, however, all subsequent messages originating from the initially traced message will be included in the trace going forward.

                var config = @"
akka.actor.provider = ""Phobos.Actor.PhobosActorRefProvider,Phobos.Actor""
akka.actor.deployment {
    /echo1 {
        phobos{
            # all children of echo1 will use the same filtering settings
            propagate-settings-to-children = on
            tracing.filter{
                mode = whitelist
                message-types = [
                    ""System.String""
                ]
            }
        }
    }
}                

phobos.tracing{
    provider-type = test
}";

                sys = ActorSystem.Create("PhobosTest", config);
                var tracer = (MockTracer) ActorTracing.For(sys).Tracer;
                var actorWithFiltering = sys.ActorOf(Props.Create(() => new EchoActor()), "echo1");
                var actorWithoutFiltering = sys.ActorOf(Props.Create(() => new EchoActor()), "echo2");

// send a message that will be filtered out by whitelist for echo1
                actorWithFiltering.Ask<NormalMessage>(new NormalMessage("hi"), TimeSpan.FromMilliseconds(100)).Wait();
                Task.Delay(100).Wait(); // wait for activity in other threads to stop
                tracer.FinishedSpans().Count.Should().Be(0); // no trace recorded

// send the same message to echo2
                actorWithoutFiltering.Ask<NormalMessage>(new NormalMessage("hi"), TimeSpan.FromMilliseconds(100))
                    .Wait();
                AwaitAssert(() => tracer.FinishedSpans().Count.Should().Be(1)); // trace gets recorded

                tracer.Reset(); // reset the MockTracer

// send a message to echo1 that won't be filtered out by whitelist
                actorWithFiltering.Ask<string>("bye", TimeSpan.FromMilliseconds(100)).Wait();
                AwaitAssert(() => tracer.FinishedSpans().Count.Should().Be(1));

In the example above, the /user/echo1 actor is able to use its own whitelist filter to only trigger new traces whenever it receives a message of type System.String. The rest of the ActorSystem will continue to use whatever the global filtering settings are, if any, but this actor and all of its children will use this custom whitelist since /user/echo1 was deployed with the phobos.propagate-settings-to-children option enabled.

Blacklist Filtering

Blacklist filtering is primarily used for breaking up large traces into smaller ones, by treating specific types of messages as "terminating" events for the trace. Phobos uses this, internally, to break apart large Akka.Streams graph-processing into a series of smaller streams that are easier to digest.

                var config = @"
akka.actor.provider = ""Phobos.Actor.PhobosActorRefProvider,Phobos.Actor""
akka.actor.deployment {
    /echo1 {
        phobos{
            # all children of echo1 will use the same filtering settings
            propagate-settings-to-children = on
            tracing.filter{
                mode = blacklist
                message-types = [
                    ""System.String""
                ]
            }
        }
    }
}                

phobos.tracing{
    provider-type = test
}";

                sys = ActorSystem.Create("PhobosTest", config);
                var tracer = (MockTracer) ActorTracing.For(sys).Tracer;
                var actorWithFiltering = sys.ActorOf(Props.Create(() => new EchoActor()), "echo1");
                var actorWithoutFiltering = sys.ActorOf(Props.Create(() => new EchoActor()), "echo2");

// send a message that will be filtered out by blacklist for echo1
                actorWithFiltering.Ask<string>("hi", TimeSpan.FromMilliseconds(100)).Wait();
                Task.Delay(100).Wait(); // wait for activity in other threads to stop
                tracer.FinishedSpans().Count.Should().Be(0); // no trace recorded

// send the same message to echo2
                actorWithoutFiltering.Ask<string>("hi", TimeSpan.FromMilliseconds(100)).Wait();
                AwaitAssert(() => tracer.FinishedSpans().Count.Should().Be(1)); // trace gets recorded

                tracer.Reset(); // reset the MockTracer

// send a message to echo1 that won't be filtered out by blacklist
                actorWithFiltering.Ask<NormalMessage>(new NormalMessage("bye"), TimeSpan.FromMilliseconds(100)).Wait();
                AwaitAssert(() => tracer.FinishedSpans().Count.Should().Be(1));

In the example above, the /user/echo1 actor is able to use its own blacklist filter to exclude any messages of type System.String from all traces, whether they're ongoing or not. The rest of the ActorSystem will continue to use whatever the global filtering settings are, if any, but this actor and all of its children will use this custom blacklist since /user/echo1 was deployed with the phobos.propagate-settings-to-children option enabled.