Automatically Captured Data

Phobos attempts to capture as much data as is relevant automatically, without the need for explicit instrumentation code inside your Akka.NET applications and most importantly: Phobos is designed to automatically connect the dots between Akka.NET and other parts of your application, like HttpClient, ASP.NET, ADO.NET, and more.

Note

To see exactly what data Phobos captures visit the "Akka.NET Scenarios" area of our tutorials.

Trace Data

Trace data in Phobos is generated automatically whenever an actor processes a message:

The trace stops when actors stop producing new messages in response to ones they've previously received. So the example above will produced a distributed trace that looks like this:

If I have the following actor from the Quickstart Tutorial:

public sealed class RouterForwaderActor : ReceiveActor
{
    private readonly ILoggingAdapter _log = Context.GetLogger();
    private readonly IActorRef _routerActor;

    public RouterForwaderActor(IActorRef routerActor)
    {
        _routerActor = routerActor;
        Receive<string>(_ =>
        {
            _log.Info("Received: {0}", _);
            _routerActor.Forward(_);
        });
    }
}

And I see the actor a string message:

IActorRef myActor = sys.ActorOf(Props.Create(() => new RouterForwaderActor(otherActor)), "fwd")
myActor.Tell("foo");

This will generate a trace called akka.actor.recv String in our tracing system, whether it's Jaeger, Application Insights, DataDog, or something else:

See a full-size version of this image

In this instance we're using Jaeger, one of the many tools Phobos supports, to capture our tracing data - but the data will be the more or less the same no matter which tracing or monitoring back-end you use.

Data Included in Actor Message Traces

All actor trace data includes the following pieces of information:

Tags

• akka.actor.path - the absolute path of this actor;
• akka.actor.recv.msgType - the assembly + type name of this message;
• akka.actor.recv.sender - the absolute path of the actor who sent this message; and
• akka.actor.type - the implementation class of the actor who recorded this span.

Events and Logs

• Any information captured using the ILoggingAdapter in Akka.NET is automatically appended as a log event to the currently active span.
• The .ToString() output of the current message is always appended to the active span.
• In the event of an actor crash or an exception thrown while processing a message, all of the relevant error details are captured and tagged into that span.

Other Actor Events Captured During Tracing

We capture a number of other events during tracing too:

• akka.msg.stash - recorded when any messages are stashed by an actor.
• akka.msg.unstash - recorded when that message is unstashed.
• akka.actor.start - recorded when an actor is spawned by its parent.
• akka.actor.crash - recorded when an actor throws an unhandled exception during processing, and includes all of the error details.
• akka.msg.deadletter - recorded when a message can't be delivered.
• akka.msg.unhandled - recored when a message was received by an actor, but the actor wasn't programmed to process it.

Tracing Over the Network

Phobos.Tracing automatically registers a serializer behind the scenes that makes it possible to preserve SpanContext over Akka.Remote and Akka.Cluster connections transparently.

In a system like Jaeger, you'll see different nodes of the same service name but with different hostnames communicating when they're exchanging messages over the cluster.

Metric Data

Phobos uses App.Metrics to record several built-in metrics.

It's quite easy to take this data and construct it into a Grafana-friendly dashboard, like the one below:

See a full-size version of this image

Important

We have an entire area of the Phobos website dedicated to sharing free, ready-to-use dashboards that will visualize all of these metrics for you. You can use Grafana with Prometheus, Grafana with InfluxDb, DataDog, and more! Check out our Phobos dashboards.

Counter and Meter Data

• akka.logs - counts the number of logs of each log level into a collection. Also collections the Exception types raised in the error logs.
• akka.messages.recv - count the number of messages received by each message type / actor type pair.
• akka.messages.deadletter - counts the number of DeadLetter messages by message type.
• akka.messages.unhandled - counts the number of Unhandled messages by message type
• akka.actor.restarts - counts the number of actor restarts over time.
• akka.actor.created - counts the number of actors started.
• akka.actor.stopped - counts the number of actors stopped.

Cluster Data

As of Phobos 1.1, Phobos now also collects data about the members of the Akka.NET cluster:

• akka.cluster.members - includes the number of cluster members of role type and their MemberStatus. Used to track the overall health of the cluster from the perspective of each individual node.
• akka.cluster.reachable - tracks the number of reachable cluster members by their role.