Introduction to Phobos.Tracing

One of the most popularly requested features when it comes to Akka.NET DevOps is being able to visualize and trace requests across actors, including actors that run in different ActorSystems via Akka.Remote. The Phobos.Tracing package introduces this capability to Akka.NET applications.

Read more about how Phobos works in general.

Tracing Concepts

Phobos.Tracing is built against the OpenTracing vendor-neutral standard for distributed tracing systems, and thus it adopts the same lexicon:

A Span represents a single operation in the course of fulfilling a request of some kind. A span could measure something as simple as an actor processing a single message, a single database query, or a single HTTP request. It's the unit of work commonly used to describe traces inside an OpenTracing system. When consuming Phobos.Tracing inside your own applications, Phobos is automatically creating spans each time your actor receives a message. Spans can contain valuable metadata such as relationships to other spans, tags, logs, and more.

A Trace is all of the spans that flowed together as part of one logical operation inside a distributed application. For instance, if your application caused a message to be sent to an actor and that actor sent a message to one of its child actors, each of those message-processing operations would show up as individual spans that are both connected together into a single trace.

For instance, take this diagram of two actors sending each other messages while being instrumented with Phobos:

Actors sending each other messages while instrumented with Phobos

If the parent actor (/user/recv) receives a message and then forwards that message to its child (/user/recv/aaron) who in turn does the real processing of the message, the sum of both of these message-processing operations will produce a trace that constitutes the entire operation. In fact, here's a screenshot of what this activity looks like when recorded using Phobos' Zipkin integration:

Actors sending each other messages as rendered on Zipkin

Click here for a full-sized image.

A Tracing Engine is the backend system responsible for collecting and visualizing all of the trace information so it can be used by developers to diagnose problems and visualize the flow of messages passing through an Akka.NET application. Zipkin is one such example of a tracing back-end.

Actor Tracing

The goal of Phobos.Tracing is to automatically produce complete traces for flows that result directly from actor message processing, we through Phobos.Actor we can accomplish this without the need for any additional code whatsoever: all of your tracing needs are automatically handled in the background by the Phobos infrastructure itself.

Here is, in a nutshell, what gets traced automatically by Phobos:

  1. Each individual message that is processed by an actor that has tracing enabled (see Phobos actor configuration for more details, but tracing is enabled by default;)
  2. The message itself (whatever it's object.ToString() representation is) and the address of the sender; and
  3. The previous span of the actor who sent us the message is automatically referenced as the "parent" span, hence why they show up as related on the trace.

A trace terminates when there are no more messages are produced in the context of processing a request.

A new trace begins when an actor receives a message for the very first time, such as:

  1. Actor receives a message from the Akka.NET IScheduler;
  2. Actor receives a message from an external, non-actor entity such as an ASP.NET controller or SignalR hub; or
  3. Actor receives a message from the EventStream.

Inter-process Tracing

Phobos.Tracing works transparently over Akka.Remote; it leverages Akka.NET's extensible serialization system and makes it possible to propagate ISpan (an OpenTracing interface used by Phobos.Tracing for defining spans) instances over the network without the need for any explicit configuration or instructions from Phobos end-users.

Supported Tracing Engines

Phobos boasts a growing list of service integrations for Akka.NET, but here is the set of supported tracing implementations we offer out of the box:

  1. Zipkin
  2. OpenTracing-compatible drivers

Please click through to those links to read more about Phobos' integration with these services.

Driver Lifecycles

When using the built-in HOCON configuration and first party drivers within Phobos, such as Phobos.Tracing.Zipkin, Phobos will manage the entire lifecycle of the driver. However, if you use a driver instantiated via Phobos' OpenTracing integration then Phobos makes no guarantees as to whether or not the driver will be properly terminated upon ActorSystem shutdown.

Next: how to configure and use Phobos.Tracing inside your Akka.NET applications.