Using Phobos with Datadog Tracing and Metrics

This tutorial is a modified version of the Phobos QuickStart Tutorial.

The Big Picture

This Datadog + Phobos tutorial is going to run inside a local Kubernetes cluster deployed on your local development machine - and this configuration could also be applied to a real production Kubernetes and Akka.Cluster deployment without significant modification.

Phobos + Akka.NET + Datadog tutorial

We're going to generate HTTP traffic via a web browser, which will propagate requests into our Akka.NET cluster that will bounce between all of the nodes - our goal is to visualize this activity inside Datadog so we can better observe how well our combined ASP.NET + Akka.NET system performs with traffic on it.

Phobos will transmit data from each individual pod to a local Datadog agent hosted as a DaemonSet - there will be one Datadog agent pod per node in Kubernetes. These agent pods will pre-aggregate the metrics reported by our application and then transmit them to Datadog's cloud hosted services - where you can view the output.

Note

As of Phobos 1.0.5, right now the metrics reported by the App.Metrics driver are reported directly to https://api.datadoghq.com/ via Datadog's HTTP API. Once https://github.com/AppMetrics/AppMetrics/pull/627 is merged in we can report metrics the same way we report traces using the DogStatsD interface.

Setup and Requirements

To run this tutorial you will need to do the following:

Buy a Phobos license and get access to your NuGet key;
Create a Datadog account and access your Datadog API keys;
Clone the Petabridge.Phobos.Web.DataDog repository to your local machine;
Install Kubernetes on your local computer - the easiest path for a local development machine is typically through Docker Desktop.
Install Helm 3, the package manager for Kubernetes - we're going to use Helm to install the datadog Helm chart, which will install a Datadog Agent as a DaemonSet in our local Kubernetes cluster.

Installing the Datadog Helm Repository

In order to install the datadog Helm package we need to add the Datadog repository to our local Helm package sources:

PS> helm repo add datadog https://helm.datadoghq.com
PS> helm repo update

Once that's done, we should be able to use the build scripts that come included with the Petabridge.Phobos.Web.DataDog repository we're going to use to run this sample.

Running the Sample

Open a shell in the directory where you cloned Petabridge.Phobos.Web.DataDog and run the following commands:

Windows

PS> ./build.cmd Docker
PS> ./k8s/deployAll.cmd [your Datadog API Key]

The first command will build Docker images from the Petabridge.Phobos.Web.csproj project and the second command will deploy Datadog and the petabridge.phobos.web Docker images into a dedicated Kubernetes namespace called phobos-web.

If you run the following command, kubectl -n phobos-web get all you will see output similar to the following:

NAME                                                    READY   STATUS    RESTARTS   AGE
pod/datadog-agent-kube-state-metrics-6f4cd6b45b-k4hzn   1/1     Running   0          7m34s
pod/datadog-agent-l2s8w                                 2/2     Running   0          7m34s
pod/phobos-web-0                                        1/1     Running   0          7m33s
pod/phobos-web-1                                        1/1     Running   0          7m32s
pod/phobos-web-2                                        1/1     Running   0          7m30s

NAME                                       TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/datadog-agent-kube-state-metrics   ClusterIP      10.102.75.157   <none>        8080/TCP         7m34s
service/phobos-web                         ClusterIP      None            <none>        4055/TCP         7m33s
service/phobos-webapi                      LoadBalancer   10.106.31.43    localhost     1880:32147/TCP   7m33s

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/datadog-agent   1         1         1       1            1           kubernetes.io/os=linux   7m34s

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/datadog-agent-kube-state-metrics   1/1     1            1           7m34s

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/datadog-agent-kube-state-metrics-6f4cd6b45b   1         1         1       7m34s

To view the application, visit http://localhost:1880

We deployed a Kubernetes loadbalancer that points to the petabridge.phobos.web application at http://localhost:1880 - if you visit this Url it will generate both ASP.NET and Akka.NET traces and metrics.

Datadog Output

What sorts of metrics and traces are produced by Phobos?

Traces

First, if we take a look on the Datadog dashboard to view our service list, you'll see petabridge.phobos.web:

Datadog Phobos Service Overview

If we drill into the petabridge.phobos.web service on Datadog, we'll be able to view individual traces that look like this one:

Datadog Phobos ASP.NET + Akka.NET trace

You can see the trace connects both the original ASP.NET request and the Akka.NET actors who processed the subsequent messages.

Metrics

Phobos also pushes metrics to Datadog, which can be viewed in the metrics area:

Datadog Phobos ASP.NET + Akka.NET metrics

These are the metrics that are produced automatically by Phobos - and we can drill down into those by clicking on an individual metric:

Detailed Datadog Akka.NET metrics

You can see that we successfully gathered this metric from all three of the separate nodes included inside the petabridge.phobos.web K8s pod instances. We can even view a time-series graph of this data by clicking on "Metrics Explorer" inside Datadog:

Time series of Akka.NET metrics via Datadog Metrics Explorer

Configuring Datadog and Phobos

So how can you configure Datadog to work alongside Phobos in your own Akka.NET applications?

First, we need to install the following NuGet packages:

Phobos.Actor.Cluster - or whichever core Phobos package you need;
App.Metrics.Datadog - this includes all of the App.Metrics drivers for Datadog; and
Datadog.Trace.OpenTracing - this is the official DataDog .NET tracing driver that targets the OpenTracing standard supported by Phobos.

One those are installed, we can go about configuring Datadog tracing and metrics inside the Startup.cs class using the Startup.ConfigureServices method:

public void ConfigureServices(IServiceCollection services)
{
    // enables OpenTracing for ASP.NET Core
    services.AddOpenTracing(o =>
    {
        o.ConfigureAspNetCore(a =>
            a.Hosting.OperationNameResolver = context => $"{context.Request.Method} {context.Request.Path}");
        o.AddCoreFx();
    });

    // sets up Prometheus + DataDog + ASP.NET Core metrics
    ConfigureAppMetrics(services);

    // sets up DataDog tracing
    ConfigureDataDogTracing(services);

    // sets up Akka.NET
    ConfigureAkka(services);
}

Tracer Setup

Let's explore what the ConfigureDataDogTracing method does:

public static void ConfigureDataDogTracing(IServiceCollection services)
{
    // Add DataDog Tracing
    services.AddSingleton<ITracer>(sp =>
    {
        return OpenTracingTracerFactory.CreateTracer()
            .WithScopeManager(new ActorScopeManager());
    });
}

We call Datadog's OpenTracingTracerFactory.CreateTracer() factory method to create a brand new tracer which will direct its traces to wherever the Datadog Agent Host is located. We set that location using the DD_AGENT_HOST environment variable in our Kubernetes configuration.

The second method - WithScopeManager is an extension method supported by Phobos itself used to help override the OpenTracing.IScopeManager inside the underlying tracer. This method is used to pass in an instance of Phobos' ActorScopeManager into Datadog's tracer.

Important

In order to provide the best possible data correlation between Akka.NET and other technologies such as ASP.NET, it's a best practice to always use the ActorScopeManager in combination with any tracing technologies you choose.

Metrics Setup

The ConfigureAppMetrics method is used to configure all of the metrics Phobos and App.Metrics will export to Datadog:

public static void ConfigureAppMetrics(IServiceCollection services)
{
    services.AddMetricsTrackingMiddleware();
    services.AddMetrics(b =>
    {
        var metrics = b.Configuration.Configure(o =>
            {
                o.GlobalTags.Add("host", Dns.GetHostName());
                o.DefaultContextLabel = "akka.net";
                o.Enabled = true;
                o.ReportingEnabled = true;
            })
            .Report.ToDatadogHttp(options => {
                options.Datadog.BaseUri = new Uri($"https://api.datadoghq.com/");
                options.Datadog.ApiKey = Environment.GetEnvironmentVariable("DD_API_KEY");
                options.HttpPolicy.BackoffPeriod = TimeSpan.FromSeconds(30);
                options.HttpPolicy.FailuresBeforeBackoff = 5;
                options.HttpPolicy.Timeout = TimeSpan.FromSeconds(10);
                options.FlushInterval = TimeSpan.FromSeconds(20);
            })
            .Build();
    });
    services.AddMetricsReportingHostedService();
}

The Phobos sample follows the usual App.Metrics best practices with Microsoft.Extensions.Hosting - the only really important thing to include here is your Datadog API key, which we pass in via a custom DD_API_KEY environment variable.

Phobos and Akka.NET Configuration

Once Datadog's tracing and metrics are both configured, we just need to configure Phobos:

public static void ConfigureAkka(IServiceCollection services)
{
    services.AddSingleton(sp =>
    {
        var metrics = sp.GetRequiredService<IMetricsRoot>();
        var tracer = sp.GetRequiredService<ITracer>();


        var config = ConfigurationFactory.ParseString(
                File.ReadAllText("app.conf")).BootstrapFromDocker();


        var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
                .WithMetrics(m =>
                    m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
                .WithTracing(t => t.SetTracer(tracer))) // binds Phobos to same tracer as ASP.NET Core
            .WithSetup(BootstrapSetup.Create()
                .WithConfig(config) // passes in the HOCON for Akka.NET to the ActorSystem
                .WithActorRefProvider(PhobosProviderSelection
                    .Cluster)); // last line activates Phobos inside Akka.NET


        var sys = ActorSystem.Create("ClusterSys", phobosSetup);


        // create actor "container" and bind it to DI, so it can be used by ASP.NET Core
        return new AkkaActors(sys);
    });


    // this will manage Akka.NET lifecycle
    services.AddHostedService<AkkaService>();
}

The PhobosConfigBuilder will allow us to pass in our IMetricsRoot and ITracer dependencies into Phobos, which we can then pass into our ActorSystem to launch Phobos.

And that's all we need - Phobos will start producing traces and metrics automatically.