Automatically Captured Data

Phobos attempts to capture as much data as is relevant automatically, without the need for explicit instrumentation code inside your Akka.NET applications and most importantly: Phobos is designed to automatically connect the dots between Akka.NET and other parts of your application, like HttpClient, ASP.NET, ADO.NET, and more.

Note

To see exactly what data Phobos captures visit the "Akka.NET Scenarios" area of our tutorials.

Trace Data

Trace data in Phobos is generated automatically whenever an actor processes a message:

Akka.NET actors generating trace data during message processing

The trace stops when actors stop producing new messages in response to ones they've previously received. So the example above will produced a distributed trace that looks like this:

Akka.NET distributed trace graph

If I have the following actor from the Quickstart Tutorial:

public sealed class RouterForwarderActor : ReceiveActor
{
    private readonly ILoggingAdapter _log = Context.GetLogger();
    private readonly IActorRef _routerActor;

    public RouterForwaderActor(IActorRef routerActor)
    {
        _routerActor = routerActor;
        Receive<string>(_ =>
        {
            _log.Info("Received: {0}", _);
            _routerActor.Forward(_);
        });
    }
}

And I see the actor a string message:

IActorRef myActor = sys.ActorOf(Props.Create(() => new RouterForwaderActor(otherActor)), "fwd")
myActor.Tell("foo");

This will generate a trace called akka.actor.recv String in our tracing system, whether it's Jaeger, Application Insights, DataDog, or something else:

Akka.NET actor tracing information in Jaeger

See a full-size version of this image

In this instance we're using Jaeger, one of the many tools Phobos supports, to capture our tracing data - but the data will be the more or less the same no matter which tracing or monitoring back-end you use.

Data Included in Actor Message Traces

All actor trace data includes the following pieces of information:

Tags

  • akka.actor.path - the absolute path of this actor;
  • akka.actor.recv.msgType - the assembly + type name of this message;
  • akka.actor.recv.sender - the absolute path of the actor who sent this message; and
  • akka.actor.type - the implementation class of the actor who recorded this span.

Events and Logs

  • Any information captured using the ILoggingAdapter in Akka.NET is automatically appended as a log event to the currently active span.
  • The .ToString() output of the current message is always appended to the active span.
  • In the event of an actor crash or an exception thrown while processing a message, all of the relevant error details are captured and tagged into that span.
  • All Ask<T> operations automatically record a trace on the sender side with an akka.actor.ask <ExpectedResponseType> operation name and a corresponding akka.msg.recv <InputMessageType> trace for the actor who is on the receiving side of the Ask<T> operation. This trace will also record the end-to-end duration it took to complete the Ask<T> as well any any timeouts or exceptions thrown by it.
  • Any time a message is stashed and subsequently unstashed, you will see akka.msg.stash akka.msg.unstash events appear attached to the akka.msg.recv <messageType> span that the actor originally processed.

Latency Tracking

Phobos tracks latency on all traced messages using the following syntax:

  1. Trace starts the moment IActorRef.Tell is called;
  2. A new event, called waiting, is recorded when the message starts being processed - the duration of the waiting event indicates how long the message was in-flight or sitting in a mailbox before it was processed; and
  3. An event called 'message' will start when the message begins processing and will complete upon the actor's receive method exiting.

The combination of these two events (both part of the same TelemetrySpan / Activity) will measure the total end-to-end processing time per each traced message.

Note

If this is too noisy for large volume applications, you can disable this feature via phobos.tracing.log-message-events = off for individual actors or for the entire ActorSystem. This can also be accomplished using the Props.WithInstrumentation extension method instead of HOCON. See "Phobos Configuration."

Other Actor Events Captured During Tracing

We capture a number of other events during tracing too:

  • akka.actor.spawn - recorded when an actor is spawned by its parent.
  • akka.actor.start - recorded during an actor's PreStart method.
  • akka.actor.restart - recorded during an actor's PreRestartMethod immediately following an akka.actor.crash event.
  • akka.actor.crash - recorded when an actor throws an unhandled exception during processing, and includes all of the error details.
  • akka.msg.deadletter - recorded when a message can't be delivered.
  • akka.msg.unhandled - recorded when a message was received by an actor, but the actor wasn't programmed to process it.
  • akka.actor.ask <ExpectedResponseType> - recorded when an Ask<T> operation begins when phobos.tracing.trace-ask=on (on by default.)
  • akka.actor.stop - as of Phobos 2.6.0, we also capture actor termination events in Phobos. This feature played a pivotal role in Petabridge's investigation of a very tricky [Heisenbug](https://en.wikipedia.org/wiki/Heisenbug) in Akka.Cluster.Sharding, which is why we decided to add it to Phobos for all customers.

Tracing Over the Network

Phobos.Tracing automatically registers a serializer behind the scenes that makes it possible to preserve SpanContext over Akka.Remote and Akka.Cluster connections transparently.

In a system like Jaeger, you'll see different nodes of the same service name but with different hostnames communicating when they're exchanging messages over the cluster.

Metric Data

Phobos uses OpenTelemetry to record several built-in metrics.

It's quite easy to take this data and construct it into a Grafana-friendly dashboard, like the one below:

Plotting Akka.NET data using Phobos and Prometheus on Grafana

See a full-size version of this image

Important

We have an entire area of the Phobos website dedicated to sharing free, ready-to-use dashboards that will visualize all of these metrics for you. You can use Grafana with Prometheus, Grafana with InfluxDb, DataDog, and more! Check out our Phobos dashboards.

Counter and Meter Data

  • akka.logs - counts the number of logs of each log level into a collection. Also collections the Exception types raised in the error logs.
  • akka.messages.recv - count the number of messages received by each message type / actor type pair.
  • akka.messages.latency - a timer that messages the latency for each message / actor type pair; uses the same end to end measurement that tracing does.
  • akka.messages.deadletters - counts the number of DeadLetter messages by message type.
  • akka.messages.unhandled - counts the number of Unhandled messages by message type.
  • akka.actor.mailbox.depth - a Histogram that measures the queue length of the actor. We would really like to change this back to a gauge again in the future, but we're waiting on https://github.com/open-telemetry/opentelemetry-specification/issues/2318 to be implemented in .NET 9.
  • akka.actor.restarts - counts the number of actor restarts over time.
  • akka.actor.created - counts the number of actors started.
  • akka.actor.stopped - counts the number of actors stopped.
  • akka.actor.live - a gauge that tracks the total number of live actors running right now.
Important

In order to use akka.actor.live you will need to set akka.actor.telemetry.enabled = on (it's off by default.)

If you use Phobos.Hosting to configure Phobos, this setting will be automatically enabled for you.

Cluster Data

Phobos also collects data about the members of the Akka.NET cluster:

  • akka.cluster.members - includes the number of cluster members of role type and their MemberStatus. Used to track the overall health of the cluster from the perspective of each individual node.
  • akka.cluster.reachable - tracks the number of reachable cluster members by their role.
  • akka.cluster.sharding.shards - keeps track of the number of shards per node per entity type.
  • akka.cluster.sharding.entities - keeps track of the number of entities per node per entity type.

Customizing Phobos Data Labels

In addition to Phobos' extensive filtering and configuration support for controlling what you collect from Akka.NET, Phobos also has some options for controlling what the collected output looks like.

Actor Type Names

One major issue we resolved in Akka.NET 2.6.0 is Actor type information and F# API, where Akkling users noticed that all of their F# actors all had identical names in Phobos's metrics and traces due to the way the F# type system works differently from C#.

In order to resolve this, added the PhobosActorName attribute, which will allow you to tell Phobos to use a different name for this actor type OTHER than its given CLI type.

[PhobosActorName("Foo")]
public class MyActor : UntypedActor
{
     // Implementation here
}

In Phobos' automatically collected metrics, the actortype attribute will now use Foo instead of Assembly.Namespace.MyActor.

In Phobos's automatically collected traces, the akka.actor.type attribute will now use Foo instead of Assembly.Namespace.MyActor.