Automatically Captured Data

Phobos attempts to capture as much data as is relevant automatically, without the need for explicit instrumentation code inside your Akka.NET applications.

The following article articulates all of the data that is captured automatically by individual actors as well as each ActorSystem that is instrumented with Phobos.

Captured Tracing Data

Tracing data is only captured when an actor is processing a message that either:

  1. Causes a new trace to be started; i.e it's the first operation that either hasn't been excluded via trace filtering or sampling. Or
  2. Is part of an going trace that was started by another actor.

Once a trace is active Phobos will automatically attempt to capture the following data each time an actor processes a message:

  1. The ActorPath of the actor processing the message will be recorded as the operation name of each individual ISpan;
  2. The .ToString() content of the received message, which will logged as an OpenTracing ISpan.Log() event;
  3. The fully qualified type name of the received message;
  4. The absolute ActorPath of the actor who sent this message.
  5. The fully qualified type name of the actor processing the message; and
  6. Any ILoggingAdapter calls made from within your actor will be recorded as ISpan.Log events.

Here's an example of what a single captured trace might look like inside Zipkin using the Phobos.Tracing.Zipkin driver package:

Akka.NET actor data and log events captured in Zipkin with Phobos Click here for a full-sized image.

The results for how this information will be displayed to you will vary a lot by the capabilities of each tracing back-end, but generally speaking this is how the data is captured via Phobos.

N.B. It's worth noting that all traces in Phobos are automatically correlated between actors and between actors running on different nodes via Akka.Remote. This is handled automatically behind the scenes provided that Phobos is installed on all nodes.

Other Trace Events

In addition to the data above, Phobos will also capture the other following types of activities automatically so long as there is an active trace available:

  1. Child actors being spawned;
  2. Actors crashing as a result of unhandled exceptions; in this instance we also automatically record relevant data about the crash such as the Exception and its stack trace; and
  3. Any unhandled messages or deadletters that occurred during an active trace.

Captured Monitoring Data

Monitoring data is captured when actors are processing messages, but data is also captured during other system events that occur independently of actors too.

Phobos will automatically monitor these events per-actor:

  1. Number of messages received per individual actor (i.e. how many messages has /user/foo received vs. /user/bar)?
  2. Number of messages received per actor implementation class (i.e. how many messages have actors of type MyFooActor received)?
  3. Number of unprocessed messages sitting in the mailbox per each individual actor (has to be explicitly enabled via Phobos.Monitoring configuration.)

Phobos also captures the following metrics per-ActorSystem:

  1. Number of log events (Info, Error, Debug, and Warning;)
  2. Number of unprocessed messages, i.e. DeadLetter and UnhandledMessage;
  3. Number of actor starts, stops, and restarts; and
  4. Total number of messages across all actors received (total ActorSystem throughput.)

Suppose you have a large Akka.NET Cluster - ideally you'd want to be able to monitor all of these values on a per-node basis and a cluster-wide basis. "I want to see how busy node 1 out of 13 is, but I also want to see how busy all 13 nodes are in aggregate." Phobos.Monitoring supports this as well by reporting all metrics on both a per-node and aggregate basis across the entire cluster.

If you want to capture your own custom metrics in addition to what we capture above, you can read about how to instrument your own Akka.NET monitoring metrics here.

Phobos automatically denormalizes all of these captured metrics into multiple dimensions, keeping track of metrics at both the cluster level and the node-specific level. You can read more about the dimensions actor metrics are recorded in here.