| Phobos - APM for Akka.NET

2.1.0-beta1 August 2 2022

Phobos 2.x BETA Release

Phobos 2.1.0-beta1 features some significant noise control improvements for tracing:

All IWithTimers messages are filtered out by default;
Inside Akka.Cluster.Sharding actors all internal messages and Akka.DistributedData messages are filtered out by default;
Resolved: Manually created trace spans don't seem to flow across the cluste; and
Resolved: How to filter out Ask<T> operations?

To disable Ask<T> tracing:

phobos.tracing.trace-ask = off

collection.AddAkka("LocalSys", (builder, sp) =>
{
    builder.WithPhobos(AkkaRunMode.Local, 
        configBuilder => configBuilder.WithTracing(t => t.SetTraceAsk(false)));
    AddTestActors(builder);
});

2.0.6 June 10 2022

Phobos 2.x Revision Release

Fixed: Stashing message may result in reset UsableContext
Upgraded to Akka.Hosting v0.3.1

2.0.5 June 06 2022

Phobos 2.x Revision Release

2.0.4 June 02 2022

Phobos 2.x Revision Release

Fixed: Custom span creation with Phobos 2.0.3-beta1 - fixed a bug when running Phobos with phobos.tracing.create-trace-upon-receive = off that allowed Phobos to continue creating spans when there was no active trace context available. This has been fixed and the setting now works as intended.
Fixed: Lots of noise in logging when using cluster sharding - introduced a new setting to Phobos configuration, phobos.tracing.log-message-events = on, which when set to off will disable the creation of waiting and message events from being added to each automatically-created akka.msg.recv TelemetrySpan. This will greatly reduce the data footprint for each TelemetrySpan produced by Phobos and will help reduce the amount of noise generated on platforms like Elastic APM.
phobos.tracing.log-message-events is now set to off by default for Akka.Cluster.Sharding and Akka.Cluster.Tools.PubSub actors.
Upgraded to Akka.NET v1.4.39

2.0.3 May 26th 2022

Phobos 2.x Revision Release

Fixed: Serializer exception when using Phobos and Akka.Hosting - this issue was caused by actor lifecycle tracing leaking Tracer.CurrentSpan instances across threads as a result of how actor spawning works on most systems. Specifically, when Akka.Cluster.Sharding was enabled this leaky mechanism made it possible for other /system actors responsible for clustering to suddenly have access to active traces, which in turn caused them to wrap built in system messages such as cluster heartbeats with Phobos' tracing envelopes.

We've fixed this in two separate ways:

Actor lifecycle traces now longer use the Tracer.CurrentSpan / System.Diagnostics.Activity.Current property, so the source of the leak has been removed;
Phobos.Actor now takes a dependency on Akka.Cluster and explicitly excludes all Akka.Cluster.IClusterMessage events from any kind of consideration during tracing.

2.0.3-beta1 May 12th 2022

Phobos 2.x BETA Release

This is a BETA release of a new propsoed feature for Phobos 2.0 - "macro" tracing: the ability to disable automatic akka.msg.recv span creation but still receive the SpanContext from incoming messages and still propagate SpanContext to other actors via TelemetrySpan / System.Diagnostic.Activity instances you create yourself.

Disabling Automatic Span Creation

You can disable automatic span creation for the entire ActorSystem via the following HOCON:

phobos.tracing.create-trace-upon-receive = off

We don't recommend doing this as it's a potentially destructive action. Instead, it's better to disable automatic span creation in areas where you know the system is busy.

akka.actor.deployment{
    /manual1{
        phobos.tracing.create-trace-upon-receive = off
        phobos.tracing.propagate-settings-to-children = on
    }
}

This will allow you to disable automatic span creation for the entire hierarchy of actors beneath /user/manual1.

You can also accomplish the same thing via C# or F# via the Props extension methods introduced by Phobos:

var myManualSpanActor = Sys.ActorOf(Props.Create(() => new ManualTraceCreator())
    .WithInstrumentation(
        PhobosSettings.For(Sys).DefaultUserActorSettings
            .WithCreateTraceUponReceive(false)
            .WithPropagateToChildren(true)), "manual2");

This will work identically to the HOCON above.

When these settings are enabled, the affected actors will still have tracing enabled and will propagate any active trace data to other actors via IActorRef.Tell but will not automatically create any additional akka.msg.recv spans on their own any longer.

Manually Creating Spans

So in order to create useful traces inside these areas, you can now access the IPhobosActorContext.UsableContext property - which contains a SpanContext? that can be used in the manual creation of OpenTelemetry spans:

public ManualTraceCreator()
{
    Receive<GetActiveSpan>(_ =>
    {
        // should be default(TelemetrySpan) when running with
        // phobos.tracing.create-trace-upon-receive = off
        Sender.Tell(new SpanResponse(Context.GetInstrumentation().ActiveSpan));
    });

    Receive<MyMsg>(m =>
    {
        using (var mySpan = Context.GetInstrumentation().Tracer.StartActiveSpan("MyMsg", SpanKind.Consumer,
                   Context.GetInstrumentation().UsableContext ?? default))
        {
            _logging.Info("My event");
            Sender.Tell(new MyReply("reply"));
        }
    });
}

This will allow the Phobos user full control over what appears in their spans and which actors / messages can produce spans. Span propagation still works automatically inside Phobos with automatic span creation disabled - but now you're responsible for creating your own spans. This should significantly reduce the amount of noise produced by the tracing system - we're going to experiment using this feature inside our planned Akka.Streams support.

2.0.2 April 19th 2022

Phobos 2.x Release

This release marks the official "release to market" of Phobos 2.0 - which includes the following changes (from 1.x and earlier betas of 2.x):

Migrated from App.Metrics and OpenTracing to OpenTelemetry - the future APM platform for all of .NET;
End-to-end message processing latency tracking in both Phobos metrics and traces;
Re-engineered the trace filtering system to be much easier to use and effective at reducing noise; and
Integrated with Akka.Hosting, a new HOCONless way of configuring, running, and integrating Akka.NET with the other parts of your .NET applications. See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here. This is accomplished through the new Phobos.Hosting NuGet package introduced in Phobos 2.0.

These changes all represent a major step forward for Phobos and Akka.NET telemetry.

For full details click here to read "What's New in Phobos 2.0?"

Phobos 1.x Support Lifetime Per our recent announcement, we will continue to support Phobos 1.x and 2.x concurrently. Phobos 1.x will exit support on December 31st, 2023.

1.5.1 April 12th 2022

Minor Version Release of Phobos 1.x

Phobos 1.5 is the next minor version release of Phobos. It contains two major, important changes:

Overhaul of the Trace Filtering System and
Integrated with Akka.Hosting, a new HOCONless way of configuring, running, and integrating Akka.NET with the other parts of your .NET applications. See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here. This is accomplished through the new Phobos.Hosting NuGet package introduced in Phobos 1.5.

And we've also fixed the following bugs:

New and Improved Trace Filtering

This release contains some breaking changes which we've introduced intentionally in order to get users concerned about noise control from the Phobos tracing system to migrate over to a new set of APIs that perform the job in a much more coherent fashion that what we have currently in Phobos 1.4.

You can see some examples of end-user negative experience reports with the current v1.x and 2.x trace filtering implementations here:

There have been many more like these over the past couple of years - separating noise from useful signals has been a persistent challenge with Phobos tracing and the changes we're introducing in Phobos v1.5 and 2.0 address this with a small new API:

/// <summary>
/// Each <see cref="ActorSystem"/> can implement exactly one of these to control
/// the decision-making around which traces to include and which ones to exclude.
///
/// When this type is configured, it will replace all of the <see cref="IFilterRule"/>s
/// that have been previously defined.
/// </summary>
/// <seealso cref="FilterDecision"/>
public interface ITraceFilter
{
    /// <summary>
    /// Evaluates a message processed by an actor and determines whether or not to include it in the current
    /// or a new trace.
    /// </summary>
    /// <param name="message">The message currently being processed by an actor.</param>
    /// <param name="alreadyInTrace">Whether or not this message is already part of an active trace.</param>
    /// <returns><c>true</c> if we are going to trace the message, <c>false</c> otherwise.</returns>
    /// <remarks>
    /// Should return <c>true</c> unless you are explicitly trying to filter a given message out of your trace.
    /// </remarks>
    bool ShouldTraceMessage(object message, bool alreadyInTrace);
}

This is the ITraceFilter - and there is exactly one of these for your entire ActorSystem. It allows you to decide whether or not to trace an actor's message processing activity based on:

Message type;
Message content; and
Whether or not we are inside an active trace or not.

This API is astonishingly succinct, much clearer than the previous TracingConfigBuilder API from before, extremely performant, and very effective at limiting noise inside your Phobos-enabled Akka.NET applications.

An example from one of our own stress testing applications:

public sealed class ScenarioTraceFilter : ITraceFilter
{
    public bool ShouldTraceMessage(object message, bool alreadyInTrace)
    {
        switch (message)
        {
            /*
             * Only start new traces for these commands
             */
            case IScenarioProtocolMessage _:
            case Petabridge.Cmd.Command _:
            case Petabridge.Cmd.CommandResponse _:
                return true;
            default:
                // otherwise, only allow messages already in trace to be included
                return alreadyInTrace;
        }
    }
}

This class allows new traces to only be started handful of Petabridge.Cmd types and any messages and our custom IScenarioProtocol messages. Any additional messages will not be included in a trace unless there is already an active trace present.

You can add this ITraceFilter to your applications like so:

var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
    .WithMetrics(m =>
        m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
    .WithTracing(t => t.SetTracer(tracer)
        .SetTraceFilter(new ScenarioTraceFilter())
    )) // binds Phobos to same tracer as ASP.NET Core
.WithSetup(BootstrapSetup.Create()
    .WithActorRefProvider(PhobosProviderSelection
        .Local)); // last line activates Phobos inside Akka.NET

Phobos still supports the old TracingConfigBuilder pattern and the WithIncludeMessage<T> rules - however, all of those rules will be overidden and replaced when the ITraceFilter is used instead.

We have broken the old TracingConfigBuilder method signatures in order to alert users to this important change.

You can see the full details here: "Filtering Tracing Data with Phobos"

Akka.Hosting Integration and Simplified Configuration

Phobos can now be fully configured without any HOCON or mangling of Setup classes using the brand new Phobos.Hosting NuGet package, built on top of Akka.Hosting.

Tip

Unfamiliar with Akka.Hosting? See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here.

See Configuring Phobos 2.0 - Using Phobos.Hosting for full details.

2.0.0-beta4 and 1.5.0-beta1 March 17th 2022

Major Update to Phobos 2.x and 1.x

You can see some examples of end-user negative experience reports with the current v1.x and 2.x trace filtering implementations here:

/// <summary>
/// Each <see cref="ActorSystem"/> can implement exactly one of these to control
/// the decision-making around which traces to include and which ones to exclude.
///
/// When this type is configured, it will replace all of the <see cref="IFilterRule"/>s
/// that have been previously defined.
/// </summary>
/// <seealso cref="FilterDecision"/>
public interface ITraceFilter
{
    /// <summary>
    /// Evaluates a message processed by an actor and determines whether or not to include it in the current
    /// or a new trace.
    /// </summary>
    /// <param name="message">The message currently being processed by an actor.</param>
    /// <param name="alreadyInTrace">Whether or not this message is already part of an active trace.</param>
    /// <returns><c>true</c> if we are going to trace the message, <c>false</c> otherwise.</returns>
    /// <remarks>
    /// Should return <c>true</c> unless you are explicitly trying to filter a given message out of your trace.
    /// </remarks>
    bool ShouldTraceMessage(object message, bool alreadyInTrace);
}

This is the ITraceFilter - and there is exactly one of these for your entire ActorSystem. It allows you to decide whether or not to trace an actor's message processing activity based on:

Message type;
Message content; and
Whether or not we are inside an active trace or not.

An example from one of our own stress testing applications:

public sealed class ScenarioTraceFilter : ITraceFilter
{
    public bool ShouldTraceMessage(object message, bool alreadyInTrace)
    {
        switch (message)
        {
            /*
             * Only start new traces for these commands
             */
            case IScenarioProtocolMessage _:
            case Petabridge.Cmd.Command _:
            case Petabridge.Cmd.CommandResponse _:
                return true;
            default:
                // otherwise, only allow messages already in trace to be included
                return alreadyInTrace;
        }
    }
}

You can add this ITraceFilter to your applications like so:

var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
    .WithMetrics(m =>
        m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
    .WithTracing(t => t.SetTracer(tracer)
        .SetTraceFilter(new ScenarioTraceFilter())
    )) // binds Phobos to same tracer as ASP.NET Core
.WithSetup(BootstrapSetup.Create()
    .WithActorRefProvider(PhobosProviderSelection
        .Local)); // last line activates Phobos inside Akka.NET

We have broken the old TracingConfigBuilder method signatures in order to alert users to this important change - however, we are going to keep these changes in a beta state for a short period in order to gather feedback from affected users. If you are concerned about trace noise inside your application, please try this beta and report on your experience on the Phobos Issue Tracker.

1.4.2 March 8th 2022

Maintenance Release for Phobos 1.4

Resolved: OpenTracing: Tracing breaks when using Ask().PipeTo()

To have tracing in Phobos 1.4 work with PipeTo(), you will have to use PipeTo(recipient, true) and call the Ask() method inside an async method even when the Ask() method is not awaited.

private class PipeToActor: ReceiveActor
{
    public PipeToActor(bool useAsync, IActorRef otherActor)
    {
        ReceiveAsync<int>(o =>
        {
            // Need to make sure that async context is preserved inside PipeTo
            otherActor.Ask("message").PipeTo(Self, true, Sender); 
            return Task.CompletedTask;
        });
    }
}

The boolean parameter in the PipeTo(IActorRef, true, IActorRef) method sets ConfigureAwait(true) inside the PipeTo() operation - and this allows us to preserve the current AsyncLocal<ISpan> context.
The ReceiveAsync method is also needed for proper async context propagation, which is what's needed in order to ensure that the ISpan created by the PipeTo method is properly correlated to the one created by the Ask method.

2.0.0-beta2 February 1st 2022

Maintenance Release for Phobos 2.0-beta1

Implemented a change where when phobos.tracing.trace-actor-lifecycle = off both parent actors and child actors will no longer create akka.actor.spawn operations.
Child actors will no longer log akka.actor.started operations if their Props have been updated via WithInstrumentation to disable actor lifecycle tracing.

1.4.1 February 1st 2022

Maintenance Release for Phobos 1.4

Implemented a change where when phobos.tracing.trace-actor-lifecycle = off both parent actors and child actors will no longer create akka.actor.spawn operations.
Child actors will no longer log akka.actor.started operations if their Props have been updated via WithInstrumentation to disable actor lifecycle tracing.

2.0.0-beta1 January 26th 2022

MAJOR RELEASE OF PHOBOS

In Phobos 2.0 we have migrated all of our tracing and monitoring capabilities to use OpenTelemetry.

As we mentioned in our recent blog post about Phobos 2.0 and its relationship with OpenTelemetry, we did this for the following reasons:

OpenTelemetry is now the official APM solution for the entire .NET platform, including other popular frameworks such as ASP.NET and Entity Framework;
All major APM vendors and OSS project such as Prometheus, Jaeger, DataDog, AWS, GCP, Azure, Elastic, and so on are all committed to supporting OpenTelemetry going forward and they implement their own first-party exporters that transmit data to those platforms;
The performance of the OpenTelemetry.Trace and OpenTelemetry.Metrics implementations, which Phobos uses, are identical no matter which combination of exporters you adopt; and
OpenTelemetry is much more decoupled and extensible than any of the previous solutions in market.

We have replaced all of our OpenTracing and App.Metrics dependencies with 2.x going forward.

Installation and Configuration To use Phobos 2.0.0-beta* in your solution, all you need to do is:

Install the appropriate Phobos.Actor.* package into your application using the 2.0.0-beta* version;
Install the appropriate OpenTelemetry.Exporter.* packages you want to use in combination with your preferred APM tools; and
Configure your IServiceCollection with the following (adjust as needed):

// This method gets called by the runtime. Use this method to add services to the container.
// For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
public void ConfigureServices(IServiceCollection services)
{
    // Prometheus exporter won't work without this
    services.AddControllers();

    var resource = ResourceBuilder.CreateDefault()
        .AddService(Assembly.GetEntryAssembly().GetName().Name, serviceInstanceId: $"{Dns.GetHostName()}");

    // enables OpenTelemetry for ASP.NET / .NET Core
    services.AddOpenTelemetryTracing(builder =>
    {
        builder
            .SetResourceBuilder(resource)
            .AddPhobosInstrumentation()
            .AddSource("Petabridge.Phobos.Web")
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddJaegerExporter(opt =>
            {
                opt.AgentHost = Environment.GetEnvironmentVariable(JaegerAgentHostEnvironmentVar);
            });
    });

    services.AddOpenTelemetryMetrics(builder =>
    {
        builder
            .SetResourceBuilder(resource)
            .AddPhobosInstrumentation()
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddPrometheusExporter(opt =>
            {
            });
    });

    // sets up Akka.NET
    ConfigureAkka(services);
}

The .AddPhobosInstrumentation() line will ensure that OTel picks up all of the signals produced by Phobos - and you need to call this line for both metrics and tracing!

Finally, you need to configure your ActorSystem to use Phobos and to leverage the TracerProvider and MeterProviders from OpenTelemetry:

public static void ConfigureAkka(IServiceCollection services)
{
    services.AddSingleton(sp =>
    {
        var metrics = sp.GetRequiredService<MeterProvider>();
        var tracer = sp.GetRequiredService<TracerProvider>();

        var config = ConfigurationFactory.ParseString(File.ReadAllText("app.conf"))
            .BootstrapFromDocker()
            .UseSerilog();

        var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
                .WithMetrics(m =>
                    m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
                .WithTracing(t => t.SetTracer(tracer))) // binds Phobos to same tracer as ASP.NET Core
            .WithSetup(BootstrapSetup.Create()
                .WithConfig(config) // passes in the HOCON for Akka.NET to the ActorSystem
                .WithActorRefProvider(PhobosProviderSelection
                    .Cluster)); // last line activates Phobos inside Akka.NET

        var sys = ActorSystem.Create("ClusterSys", phobosSetup);

        // create actor "container" and bind it to DI, so it can be used by ASP.NET Core
        return new AkkaActors(sys);
    });

    // this will manage Akka.NET lifecycle
    services.AddHostedService<AkkaService>();
}

This is fundamentally not all that different from how Phobos 1.x works.

We will update our official documentation and website with more Phobos 2.0-specific instructions as our releases mature, but for the time being please don't hesitate to submit an issue on the Phobos Issue Tracker if you run into trouble!

Phobos 1.x Support Lifetime Per our recent announcement, we will continue to support Phobos 1.x and 2.x concurrently. Phobos 1.x will exit support on December 31st, 2023.

1.4.0 January 19th 2022

New Minor Version Release for Phobos

Phobos 1.4 is a minor version release that introduces some exciting new capabilities for Akka.NET users, primarily:

All actor lifecycle activity, i.e. everything that occurs during PreStart, PostStop, PreRestart, and PostRestart is now traced by default. This will allow you to visualize everything that happens after an actor is spawned mid-flight. This setting does produce significantly more traces when enabled (naturally) so it can be disabled via the following HOCON setting: phobos.tracing.trace-actor-lifecycle = off.
Phobos 1.4 uses Akka.NET v1.4.32, which contains important changes to the Akka.Persistence Journal base classes that now allow us to properly trace activity that happens within it. As a result, when phobos.tracing.trace-actor-lifecycle = on (this is the default) you can now trace the entire recovery process of all PersistentActor implementations - so long as the Akka.Persistence plugin you're using is built against Akka.NET v1.4.32 or higher. In addition to tracing Akka.Persistence recoveries all Persist calls can now be traced end to end with Phobos.

1.3.3 January 3rd 2022

Patch for Phobos 1.3

Phobos v1.3.3 introduces a bugfix for a very edge-case regression that was introduced in Phobos v1.3.2:

Possible regression: System.NotImplementedException in Akka.Persistence.Sql.Common.Journal.BatchingSqlJournal

This was caused by some of our trace correlation software not fitlering remote IActorRefs that sent messages to actors outside of an actor's scheduled context, a rare case that could only really happen inside a Task closed over a remote IActorRef. Patch v1.3.3 resolves this issue.

1.3.2 December 21st 2021

Minor Version Release for Phobos

Phobos v1.3.2 includes some important tracing enhancements brought about as a result of upgrading Phobos to Akka.NET v1.4.31:

Phobos now supports tracing the output of PipeTo operations! This can be accomplished by making sure that the Sender property of the PipeTo operation contains a reference to the actor kicking of the Task<T> whose results will be piped:

var myActor = Sys.ActorOf(act =>
{
    act.ReceiveAny((o, ctx) =>
    {
        // won't capture trace - no "sender" included
        Task.Delay(TimeSpan.FromMilliseconds(100)).PipeTo(ctx.Sender, success: () => o);

        // will capture trace - "Sender" is specified
        Task.Delay(TimeSpan.FromMilliseconds(100)).PipeTo(ctx.Sender, success: () => o, sender:ctx.Self);
    });
});

1.3.1 December 9th 2021

Minor Version Release for Phobos

Phobos v1.3.1 includes some improvements for metrics support:

phobos.monitoring.monitor-mailbox-depth is now set to on by default as the performance impact is negligible for most actor types and can be found under the akka.actor.mailbox.depth metric header.
Phobos now records an App.Metrics Timer for every message received by an actor under the akka.messages.latency header. This is used to help measure how long an actor takes to process each message and these times will also be measured across an awaits that occur inside your actor's ReceiveAsync methods.

We also have updated all of our built-in Phobos dashboards to include the akka.messages.latency values. You can see an example below:

akka.messages.latency, new metrics introduced in Phobos v1.3.1, being displayed in Grafana

The addition of the akka.messages.latency metric to Phobos' captured Akka.NET data

In addition to these changes, we've also updated our performance benchmarks for Phobos to include the most recent data for v1.3.1. The addition of the akka.messages.latency reduces Phobos 1.3.1's total throughput by about 10% compared to our previous benchmarks, though usually this overhead pales in comparison to user-defined code. However, if this is a problem for you please file an issue with us here: https://github.com/petabridge/phobos-issues

1.3.0 November 18th 2021

Minor Version Release for Phobos

Phobos v1.3.0 includes some significant improvements in its tracing capabilities:

Phobos now properly handles await operations that occur within your actors - so any traces that begin when an actor starts processing a message will be carried all the way through until your ReceiveAsync methods exit. This is convenient when combined with other libraries that emit traces such as EntityFramework and OpenTracing.Contrib.
Phobos no longer records any akka.actor.ask object invocations during the startup of your ActorSystem, which were erroneously being attached to user-generated traces as well.
In some instance of using the WrappedTracer, which gets called whenver you use the WithScopeManager extension method on an existing ITracer object, it was possible for a null ISpan to be passed into the SpanBuilder.ChildOf method. This input is now null-checked and will no longer throw NullReferenceException errors.
All IScopes generated by Phobos are disposed of per their original invocation ordering, which reverses a major change we introduced in Phobos 1.2.0. We were able to introduce this change safely as a result of changing how we track traces around await calls.
The Phobos cluster-listener-actor will no longer log any Unhandled messages when akka.loglevel=DEBUG and akka.actor.debug.unhandled=on any longer.
Phobos has been upgraded to run on Akka.NET v1.4.28.

In addition to these changes, we've also updated our performance benchmarks for Phobos to include the most recent data for v1.3.0.