Show / Hide Table of Contents

    Automatically Captured Data

    Phobos attempts to capture as much data as is relevant automatically, without the need for explicit instrumentation code inside your Akka.NET applications and most importantly: Phobos is designed to automatically connect the dots between Akka.NET and other parts of your application, like HttpClient, ASP.NET, ADO.NET, and more.

    Note

    To see exactly what data Phobos captures visit the "Akka.NET Scenarios" area of our tutorials.

    Trace Data

    Trace data in Phobos is generated automatically whenever an actor processes a message:

    Akka.NET actors generating trace data during message processing

    The trace stops when actors stop producing new messages in response to ones they've previously received. So the example above will produced a distributed trace that looks like this:

    Akka.NET distributed trace graph

    If I have the following actor from the Quickstart Tutorial:

    public sealed class RouterForwaderActor : ReceiveActor
    {
        private readonly ILoggingAdapter _log = Context.GetLogger();
        private readonly IActorRef _routerActor;
    
        public RouterForwaderActor(IActorRef routerActor)
        {
            _routerActor = routerActor;
            Receive<string>(_ =>
            {
                _log.Info("Received: {0}", _);
                _routerActor.Forward(_);
            });
        }
    }
    

    And I see the actor a string message:

    IActorRef myActor = sys.ActorOf(Props.Create(() => new RouterForwaderActor(otherActor)), "fwd")
    myActor.Tell("foo");
    

    This will generate a trace called akka.actor.recv String in our tracing system, whether it's Jaeger, Application Insights, DataDog, or something else:

    Akka.NET actor tracing information in Jaeger

    See a full-size version of this image

    In this instance we're using Jaeger, one of the many tools Phobos supports, to capture our tracing data - but the data will be the more or less the same no matter which tracing or monitoring back-end you use.

    Data Included in Actor Message Traces

    All actor trace data includes the following pieces of information:

    Tags

    • akka.actor.path - the absolute path of this actor;
    • akka.actor.recv.msgType - the assembly + type name of this message;
    • akka.actor.recv.sender - the absolute path of the actor who sent this message; and
    • akka.actor.type - the implementation class of the actor who recorded this span.

    Events and Logs

    • Any information captured using the ILoggingAdapter in Akka.NET is automatically appended as a log event to the currently active span.
    • The .ToString() output of the current message is always appended to the active span.
    • In the event of an actor crash or an exception thrown while processing a message, all of the relevant error details are captured and tagged into that span.

    Other Actor Events Captured During Tracing

    We capture a number of other events during tracing too:

    • akka.msg.stash - recorded when any messages are stashed by an actor.
    • akka.msg.unstash - recorded when that message is unstashed.
    • akka.actor.start - recorded when an actor is spawned by its parent.
    • akka.actor.crash - recorded when an actor throws an unhandled exception during processing, and includes all of the error details.
    • akka.msg.deadletter - recorded when a message can't be delivered.
    • akka.msg.unhandled - recored when a message was received by an actor, but the actor wasn't programmed to process it.

    Tracing Over the Network

    Phobos.Tracing automatically registers a serializer behind the scenes that makes it possible to preserve SpanContext over Akka.Remote and Akka.Cluster connections transparently.

    In a system like Jaeger, you'll see different nodes of the same service name but with different hostnames communicating when they're exchanging messages over the cluster.

    Metric Data

    Phobos uses App.Metrics to record several built-in metrics.

    It's quite easy to take this data and construct it into a Grafana-friendly dashboard, like the one below:

    Plotting Akka.NET data using Phobos and Prometheus on Grafana

    See a full-size version of this image

    Counter and Meter Data

    • akka.logs - counts the number of logs of each log level into a collection.
    • akka.messages.recv - count the number of messages received by each message type / actor type pair.
    • akka.messages.deadletter - counts the number of DeadLetter messages by message type.
    • akka.messages.unhandled - counts the number of Unhandled messages by message type
    • akka.actor.restarts - counts the number of actor restarts over time.
    • akka.actor.created - counts the number of actors started.
    • akka.actor.stopped - counts the number of actors stopped.
    Back to top Generated by DocFX