Filtering Telemetry Data with Phobos

One of the major Phobos performance best practices is to limit the amount of noise produced by the tracing system. Busy ActorSystems can produce quite a lot of actor messaging interactions and each of those will be automatically captured by default - this can take up quite a bit of bandwidth, CPU, and may even result in billable units on your preferred APM provider.

Filtering Traces

Like many other OpenTelemetry components Phobos includes some built-in APIs to help users configure which types of activities should be traced and which ones should not. We refer to this feature as "noise control" for tracing.

`ITraceFilter`

In Phobos 1.5+ and Phobos 2.0+ we've introduced the ITraceFilter interface - which users should implement in order to define a trace filtration strategy:


/// <summary>
/// Configure filtration settings for entire <see cref="ActorSystem"/>
/// </summary>
public sealed class MyTraceFilter : ITraceFilter
{
    public bool ShouldTraceMessage(object message, bool alreadyInTrace)
    {
        switch (message)
        {
            case FilteredMessage _:
                // only start new traces for FilteredMessage
                return true;
            case int i:
                // never include ints in traces, under any circumstances
                return false;
            default:
                // otherwise - include messages that are already
                // part of a trace in the existing trace.
                return alreadyInTrace;
        }
    }
}

In this instance we're creating a subclass of the ITraceFilter, a simple class that gives users of Phobos full control over which messages produce trace data and which ones don't - as this is key to eliminating noise from the traces that Phobos produces in high volume systems.

The ITraceFilter takes two arguments:

The current message being processed by an actor and
Whether or not this message is currently part of an active trace.

Here's a real-world example of ITraceFilter being used in one of our applications:

public sealed class ScenarioTraceFilter : ITraceFilter
{
    public bool ShouldTraceMessage(object message, bool alreadyInTrace)
    {
        switch (message)
        {
            /*
             * Only start new traces for these commands
             */
            case IScenarioProtocolMessage _:
            case Petabridge.Cmd.Command _:
            case Petabridge.Cmd.CommandResponse _:
                return true;
            default:
                // otherwise, only allow messages already in trace to be included
                return alreadyInTrace;
        }
    }
}

This filter will allow new traces to be started by:

Any class that implements IScenarioProtocolMessage - a marker interface unique to the messages inside our application and
Petabridge.Cmd.Command / Petabridge.Cmd.CommandResponse - messaging classes from Petabridge.Cmd CLI.

This filter will also allow any other subsequent messages invoked as a result of the two cases above to be traced too - i.e. this is what the alreadyInTrace property does.

This filter works extremely well at limiting the amount of noise inside our application to only the pertinent operations that matter to us.

Installing the `ITraceFilter` into Your `ActorSystem`

Actually using the ITraceFilter is quite easy.

Using Phobos.Hosting

In Phobos.Hosting we can pass our ITraceFilter implementation as an argument to the WithPhobos method:

builder.WithPhobos(AkkaRunMode.AkkaCluster,
    new ScenarioTraceFilter());

And just for context, you can see where this method fits within the larger picture of using Akka.Hosting to launch our ActorSystem:

var host = WebHost.CreateDefaultBuilder(args)
    .ConfigureServices((hostContext, services) =>
    {
        services.AddControllers();
        services.AddLogging();
        services.AddPhobosApm();
        services.AddAkka("ClusterSys", (builder, provider) =>
        {
            builder.WithRemoting("localhost", 8882);
            builder.WithClustering(new ClusterOptions()
            {
                SeedNodes = new[]
                {
                   "akka.tcp://ClusterSys@localhost:8882"
                },
                Roles = new[] { "app" }
            });
            
            builder.WithPhobos(AkkaRunMode.AkkaCluster,
                new ScenarioTraceFilter());
            
            // turn on Petabridge.Cmd
            builder.StartActors((system, registry) =>
            {
                var pbm = PetabridgeCmd.Get(system);
                pbm.RegisterCommandPalette(ScenarioCliHandler.Instance);
                pbm.Start();
            });
        });
    })

Using `PhobosSetup`

If you're not using Phobos.Hosting you can still install your ITraceFilter into the ActorSystem using PhobosSetup.

The PhobosSetup type played a much bigger role in Phobos 1.x but it is now optional for Phobos 2.x. The one area where it can still be used, however, is to specify filter rules for controlling noise produced by the tracing system inside Phobos:


/// <summary>
/// Configure filtration settings for entire <see cref="ActorSystem"/>
/// </summary>
public sealed class MyTraceFilter : ITraceFilter
{
    public bool ShouldTraceMessage(object message, bool alreadyInTrace)
    {
        switch (message)
        {
            case FilteredMessage _:
                // only start new traces for FilteredMessage
                return true;
            case int i:
                // never include ints in traces, under any circumstances
                return false;
            default:
                // otherwise - include messages that are already
                // part of a trace in the existing trace.
                return alreadyInTrace;
        }
    }
}

To pass this ITraceFilter implementation into our ActorSystem for active use within Phobos, we need to create a PhobosSetup instance and combine it with our BootstrapSetup and then pass the combined setup into our ActorSystem.Create method.


var config = @"
    phobos.tracing.trace-actor-lifecycle = off #disable actor startup tracing
    akka.actor.provider = ""Phobos.Actor.PhobosActorRefProvider,Phobos.Actor""";

var phobosConfig = new PhobosConfigBuilder()
    .WithTracing(m =>
    {
        m.SetTraceFilter(new MyTraceFilter());
    });

// create PhobosSetup + BootstrapSetup for configuring ActorSystem
// using OpenTelemetry ActivitySource name "TestSource2"
var phobosSetup = PhobosSpecSetup.Create("TestSource2", phobosConfig)
    .WithSetup(BootstrapSetup.Create().WithConfig(config));

sys = ActorSystem.Create("PhobosTest", phobosSetup);

var actor = sys.ActorOf(Props.Create(() => new EchoActor()), "echo");
var actor2 = sys.ActorOf(act =>
{
    IActorRef sender = null;
    act.Receive<FilteredMessage>((f, ctx) =>
    {
        sender = ctx.Sender;
        actor.Tell(f.Message);
    });

    act.Receive<string>((str, ctx) =>
    {
        sender.Tell(str, ctx.Self);
    });
}, "secondActor");

// send a message that WILL be filtered out
await actor2.Ask<string>(new FilteredMessage("bye"), TimeSpan.FromMilliseconds(1000));
await Task.Delay(300); // wait for activity in other threads to stop

// force the completion of in-process spans
PhobosResults.For(sys).ForceFlush();
var spans = PhobosResults.For(sys).CompletedActivities;

spans.Count.Should().Be(3); // for the FilteredMessage, Ask, and string receive

Other Tracing Noise Control Settings

In addition to using an ITraceFilter Phobos has a number of other settings that can be used to control the automatic creation of TelemetrySpans / Activity inside your Akka.NET applications.

`Ask<T>` Tracing

By default all Ask<T> operations are traced in Akka.NET unless they originate from actors with tracing explicitly disabled (i.e. system actors that use Ask<T> internally.)

However, as of Phobos 2.1 we've made Ask<T> tracing more configurable:

Ask<T> tracing can now be filtered using the same ITraceFilter and other filter rules as regular akka.actor.recv statements and
Ask<T> tracing can now be disabled via the fluent C# APIs or via phobos.tracing.trace-ask = off.

To disable Ask<T> tracing via Phobos.Hosting, use the following:

collection.AddAkka("LocalSys", (builder, sp) =>
{
    builder.WithPhobos(AkkaRunMode.Local, 
        configBuilder => configBuilder.WithTracing(t => t.SetTraceAsk(false)));
    AddTestActors(builder);
});

Akka.Cluster.Sharding Trace Compression

When you send a message to an entity actor via Akka.Cluster.Sharding it creates traces that look like the following by default:

graph TD A[Actor1] --> B[ShardRegion1] -- remoting to Node2 --> C[ShardRegion] --> D[ShardId] --> E[EntityId]

This can create quite a bit of noise and drive up costs inside your preferred APM / tracing platform, so one of the features we introduced beginning in Phobos 2.1 is "trace compression" for Akka.Cluster.Sharding - which can be enabled when subscribing to Phobos' ActivitySource inside the OpenTelemetry APIs:

services.AddOpenTelemetry()
    .WithTracing(builder =>
    {
        builder
            .SetResourceBuilder(resource)
            .AddPhobosInstrumentation(compressShardTraces:true) // remove built-in sharding infrastructure from traces
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation(options =>
            {
                options.Filter = context => !context.
                    Request.Path.StartsWithSegments("/metrics");
            })
            .SetSampler(new TraceIdRatioBasedSampler(1.0d))
            .AddJaegerExporter(opt =>
            {
                opt.AgentHost = Environment.GetEnvironmentVariable(JaegerAgentHostEnvironmentVar) ?? "localhost";
            });
    })

Calling .AddPhobosInstrumentation(compressShardTraces:true) will enable trace compression for sharding. This value is set to false by default. Under the hood this works by using an OpenTelemetry BaseProcessor<Activity> to flatten the Cluster.Sharding trace structure to look like this instead:

graph TD A[Actor1] -- remoting to Node 2 --> E[EntityId]

In effect, trace compression simply hides all of the message "hops" that are built into the inner workings of Akka.NET's Cluster.Sharding infrastructure itself - it's a great way to reduce noise and cost when it comes to understanding the behavior of your Akka.NET applications.

Manual Tracing

As of Phobos 2.0.3 it's now possible to disable the automatic creation of TelemetrySpan / Activity upon receiving a message in Phobos for your entire ActorSystem all at once:

phobos.tracing.create-trace-upon-receive = off

Or using the fluent interface in Akka.Hosting / C#:

.WithPhobos(AkkaRunMode.AkkaCluster, configBuilder =>
{
    configBuilder.WithTracing(b => b.SetCreateTraceUponReceive(false));
})

Under this setting Phobos will no longer create spans automatically - you'll have to do that yourself via OpenTelemetery's C# APIs, but Phobos will still propagate those manually created traces for you between actors and over the network via Akka.Remote.

Manually Creating Spans

In order to create useful traces inside actors you can now access the IPhobosActorContext.UsableContext property - which contains a SpanContext? that can be used in the manual creation of OpenTelemetry spans:

public ManualTraceCreator()
{
    Receive<GetActiveSpan>(_ =>
    {
        // should be default(TelemetrySpan) when running with
        // phobos.tracing.create-trace-upon-receive = off
        Sender.Tell(new SpanResponse(Context.GetInstrumentation().ActiveSpan));
    });

    Receive<MyMsg>(m =>
    {
        using (var mySpan = Context.GetInstrumentation().Tracer.StartActiveSpan("MyMsg", SpanKind.Consumer,
                   Context.GetInstrumentation().UsableContext ?? default))
        {
            _logging.Info("My event");
            Sender.Tell(new MyReply("reply"));
        }
    });
}

This will allow the Phobos user full control over what appears in their spans and which actors / messages can produce spans.

Span propagation still works automatically inside Phobos with automatic span creation disabled - but now you're responsible for creating your own spans. This should significantly reduce the amount of noise produced by the tracing system - we're going to experiment using this feature inside our planned Akka.Streams support.

Use Cases

Manual tracing mode is the ultimate form of noise control - it puts the end user 100% in charge of what data gets captured during tracing, but Phobos still handles the tricky matter of propagation between actors and ActorSystems transparently for you.

Note

We don't recommend that you use manual tracing unless you have an extremely noisy system and you know exactly what data you need to capture.

Disabling Automatic Tracing in Select Actors and Hierarchies

We don't recommend doing disabling automatic span creation for your entire ActorSystem unless you know what you're doing as it's a potentially destructive action. Instead, it's better to disable automatic span creation in areas where you know the system is busy.

akka.actor.deployment{
    /manual1{
        phobos.tracing.create-trace-upon-receive = off
        phobos.tracing.propagate-settings-to-children = on
    }
}

This will allow you to disable automatic span creation for the entire hierarchy of actors beneath /user/manual1.

You can also accomplish the same thing via C# or F# via the Props extension methods introduced by Phobos:

var myManualSpanActor = Sys.ActorOf(Props.Create(() => new ManualTraceCreator())
    .WithInstrumentation(
        PhobosSettings.For(Sys).DefaultUserActorSettings
            .WithCreateTraceUponReceive(false)
            .WithPropagateToChildren(true)), "manual2");

This will work identically to the HOCON above.

When these settings are enabled, the affected actors will still have tracing enabled and will propagate any active trace data to other actors via IActorRef.Tell but will not automatically create any additional akka.msg.recv spans on their own any longer.

Filtering Metrics

OpenTelemetry.Metrics provides some convenient tools for filtering metrics produced by Phobos or any other OTel metrics instrumentation.

Using OpenTelemetry.Metrics `View`s to Remove Unneeded Metrics

The least expensive and most useful tool to filter out unnecessary or undesirable metrics data are OpenTelemetry metrics' Views.

These can be configured on the OpenTelemetry metrics pipeline like so:

services.AddOpenTelemetry()
    .WithMetrics(builder =>
    {
        builder
            .SetResourceBuilder(resource)
            .AddPhobosInstrumentation()
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddView("akka.messages.latency*", MetricStreamConfiguration.Drop) // removes all `akka.messages.latency` metrics from export
            .AddPrometheusExporter(opt => { });
    });

This specific code sample will filter out all of the akka.messages.latency metrics from being exported to our target APM system such as Prometheus or DataDog - and this can help reduce our data storage / processing costs on those platforms if the metrics aren't helpful to us.

Note

Use the "What Data Does Phobos Capture?" section on "Metrics" to find the names of all OpenTelemetry Akka.NET metrics are produced by Phobos.

Filtering Telemetry Data with Phobos

Filtering Traces

ITraceFilter

Installing the ITraceFilter into Your ActorSystem

Using Phobos.Hosting

Using PhobosSetup

Other Tracing Noise Control Settings

Ask<T> Tracing

Akka.Cluster.Sharding Trace Compression

Manual Tracing

Manually Creating Spans

Use Cases

Note

Disabling Automatic Tracing in Select Actors and Hierarchies

Filtering Metrics

Using OpenTelemetry.Metrics Views to Remove Unneeded Metrics

Note

`ITraceFilter`

Installing the `ITraceFilter` into Your `ActorSystem`

Using `PhobosSetup`

`Ask<T>` Tracing

Using OpenTelemetry.Metrics `View`s to Remove Unneeded Metrics