Automatically Captured Data

Phobos attempts to capture as much data as is relevant automatically, without the need for explicit instrumentation code inside your Akka.NET applications and most importantly: Phobos is designed to automatically connect the dots between Akka.NET and other parts of your application, like HttpClient, ASP.NET, ADO.NET, and more.

Trace Data

Trace data in Phobos is generated automatically whenever an actor processes a message:

Akka.NET actors generating trace data during message processing

The trace stops when actors stop producing new messages in response to ones they've previously received. So the example above will produced a distributed trace that looks like this:

Akka.NET distributed trace graph

If I have the following actor from the Quickstart Tutorial:

public sealed class RouterForwaderActor : ReceiveActor
{
    private readonly ILoggingAdapter _log = Context.GetLogger();
    private readonly IActorRef _routerActor;

    public RouterForwaderActor(IActorRef routerActor)
    {
        _routerActor = routerActor;
        Receive<string>(_ =>
        {
            _log.Info("Received: {0}", _);
            _routerActor.Forward(_);
        });
    }
}

And I see the actor a string message:

IActorRef myActor = sys.ActorOf(Props.Create(() => new RouterForwaderActor(otherActor)), "fwd")
myActor.Tell("foo");

This will generate a trace called akka.actor.recv String in our tracing system, whether it's Jaeger, Application Insights, DataDog, or something else:

Akka.NET actor tracing information in Jaeger

See a full-size version of this image

In this instance we're using Jaeger, one of the many tools Phobos supports, to capture our tracing data - but the data will be the more or less the same no matter which tracing or monitoring back-end you use.

Data Included in Actor Message Traces

All actor trace data includes the following pieces of information:

Events and Logs

Any information captured using the ILoggingAdapter in Akka.NET is automatically appended as a log event to the currently active span.
The .ToString() output of the current message is always appended to the active span.
In the event of an actor crash or an exception thrown while processing a message, all of the relevant error details are captured and tagged into that span.

Other Actor Events Captured During Tracing

We capture a number of other events during tracing too:

akka.msg.stash - recorded when any messages are stashed by an actor.
akka.msg.unstash - recorded when that message is unstashed.
akka.actor.start - recorded when an actor is spawned by its parent.
akka.actor.crash - recorded when an actor throws an unhandled exception during processing, and includes all of the error details.
akka.msg.deadletter - recorded when a message can't be delivered.
akka.msg.unhandled - recored when a message was received by an actor, but the actor wasn't programmed to process it.

Tracing Over the Network

Phobos.Tracing automatically registers a serializer behind the scenes that makes it possible to preserve SpanContext over Akka.Remote and Akka.Cluster connections transparently.

In a system like Jaeger, you'll see different nodes of the same service name but with different hostnames communicating when they're exchanging messages over the cluster.

Metric Data

Phobos uses App.Metrics to record several built-in metrics.

It's quite easy to take this data and construct it into a Grafana-friendly dashboard, like the one below:

Plotting Akka.NET data using Phobos and Prometheus on Grafana