Automatically Captured Data
Phobos attempts to capture as much data as is relevant automatically, without the need for explicit instrumentation code inside your Akka.NET applications and most importantly: Phobos is designed to automatically connect the dots between Akka.NET and other parts of your application, like HttpClient, ASP.NET, ADO.NET, and more.
Note
To see exactly what data Phobos captures visit the "Akka.NET Scenarios" area of our tutorials.
Trace Data
Trace data in Phobos is generated automatically whenever an actor processes a message:
The trace stops when actors stop producing new messages in response to ones they've previously received. So the example above will produced a distributed trace that looks like this:
If I have the following actor from the Quickstart Tutorial:
public sealed class RouterForwaderActor : ReceiveActor
{
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly IActorRef _routerActor;
public RouterForwaderActor(IActorRef routerActor)
{
_routerActor = routerActor;
Receive<string>(_ =>
{
_log.Info("Received: {0}", _);
_routerActor.Forward(_);
});
}
}
And I see the actor a string
message:
IActorRef myActor = sys.ActorOf(Props.Create(() => new RouterForwaderActor(otherActor)), "fwd")
myActor.Tell("foo");
This will generate a trace called akka.actor.recv String
in our tracing system, whether it's Jaeger, Application Insights, DataDog, or something else:
In this instance we're using Jaeger, one of the many tools Phobos supports, to capture our tracing data - but the data will be the more or less the same no matter which tracing or monitoring back-end you use.
Data Included in Actor Message Traces
All actor trace data includes the following pieces of information:
Tags
akka.actor.path
- the absolute path of this actor;akka.actor.recv.msgType
- the assembly + type name of this message;akka.actor.recv.sender
- the absolute path of the actor who sent this message; andakka.actor.type
- the implementation class of the actor who recorded this span.
Events and Logs
- Any information captured using the
ILoggingAdapter
in Akka.NET is automatically appended as a log event to the currently active span. - The
.ToString()
output of the current message is always appended to the active span. - In the event of an actor crash or an exception thrown while processing a message, all of the relevant error details are captured and tagged into that span.
Other Actor Events Captured During Tracing
We capture a number of other events during tracing too:
akka.msg.stash
- recorded when any messages are stashed by an actor.akka.msg.unstash
- recorded when that message is unstashed.akka.actor.start
- recorded when an actor is spawned by its parent.akka.actor.crash
- recorded when an actor throws an unhandled exception during processing, and includes all of the error details.akka.msg.deadletter
- recorded when a message can't be delivered.akka.msg.unhandled
- recored when a message was received by an actor, but the actor wasn't programmed to process it.
Tracing Over the Network
Phobos.Tracing automatically registers a serializer behind the scenes that makes it possible to preserve SpanContext
over Akka.Remote and Akka.Cluster connections transparently.
In a system like Jaeger, you'll see different nodes of the same service name but with different hostnames communicating when they're exchanging messages over the cluster.
Metric Data
Phobos uses App.Metrics to record several built-in metrics.
It's quite easy to take this data and construct it into a Grafana-friendly dashboard, like the one below:
Counter and Meter Data
akka.logs
- counts the number of logs of each log level into a collection.akka.messages.recv
- count the number of messages received by each message type / actor type pair.akka.messages.deadletter
- counts the number ofDeadLetter
messages by message type.akka.messages.unhandled
- counts the number ofUnhandled
messages by message typeakka.actor.restarts
- counts the number of actor restarts over time.akka.actor.created
- counts the number of actors started.akka.actor.stopped
- counts the number of actors stopped.