Using Phobos with Datadog Tracing and Metrics
This tutorial is a modified version of the Phobos QuickStart Tutorial.
The Big Picture
This Datadog + Phobos tutorial is going to run inside a local Kubernetes cluster deployed on your local development machine - and this configuration could also be applied to a real production Kubernetes and Akka.Cluster deployment without significant modification.
We're going to generate HTTP traffic via a web browser, which will propagate requests into our Akka.NET cluster that will bounce between all of the nodes - our goal is to visualize this activity inside Datadog so we can better observe how well our combined ASP.NET + Akka.NET system performs with traffic on it.
Phobos will transmit data from each individual pod to a local Datadog agent hosted as a DaemonSet - there will be one Datadog agent pod per node in Kubernetes. These agent pods will pre-aggregate the metrics reported by our application and then transmit them to Datadog's cloud hosted services - where you can view the output.
Note
As of Phobos 1.0.5, right now the metrics reported by the App.Metrics driver are reported directly to https://api.datadoghq.com/ via Datadog's HTTP API. Once https://github.com/AppMetrics/AppMetrics/pull/627 is merged in we can report metrics the same way we report traces using the DogStatsD interface.
Setup and Requirements
To run this tutorial you will need to do the following:
- Buy a Phobos license and get access to your NuGet key;
- Create a Datadog account and access your Datadog API keys;
- Clone the Petabridge.Phobos.Web.DataDog repository to your local machine;
- Install Kubernetes on your local computer - the easiest path for a local development machine is typically through Docker Desktop.
- Install Helm 3, the package manager for Kubernetes - we're going to use Helm to install the
datadog
Helm chart, which will install a Datadog Agent as a DaemonSet in our local Kubernetes cluster.
Installing the Datadog Helm Repository
In order to install the datadog
Helm package we need to add the Datadog repository to our local Helm package sources:
PS> helm repo add datadog https://helm.datadoghq.com
PS> helm repo update
Once that's done, we should be able to use the build scripts that come included with the Petabridge.Phobos.Web.DataDog repository we're going to use to run this sample.
Running the Sample
Open a shell in the directory where you cloned Petabridge.Phobos.Web.DataDog and run the following commands:
Windows
PS> ./build.cmd Docker
PS> ./k8s/deployAll.cmd [your Datadog API Key]
The first command will build Docker images from the Petabridge.Phobos.Web.csproj
project and the second command will deploy Datadog and the petabridge.phobos.web
Docker images into a dedicated Kubernetes namespace called phobos-web
.
If you run the following command, kubectl -n phobos-web get all
you will see output similar to the following:
NAME READY STATUS RESTARTS AGE
pod/datadog-agent-kube-state-metrics-6f4cd6b45b-k4hzn 1/1 Running 0 7m34s
pod/datadog-agent-l2s8w 2/2 Running 0 7m34s
pod/phobos-web-0 1/1 Running 0 7m33s
pod/phobos-web-1 1/1 Running 0 7m32s
pod/phobos-web-2 1/1 Running 0 7m30s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/datadog-agent-kube-state-metrics ClusterIP 10.102.75.157 <none> 8080/TCP 7m34s
service/phobos-web ClusterIP None <none> 4055/TCP 7m33s
service/phobos-webapi LoadBalancer 10.106.31.43 localhost 1880:32147/TCP 7m33s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/datadog-agent 1 1 1 1 1 kubernetes.io/os=linux 7m34s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/datadog-agent-kube-state-metrics 1/1 1 1 7m34s
NAME DESIRED CURRENT READY AGE
replicaset.apps/datadog-agent-kube-state-metrics-6f4cd6b45b 1 1 1 7m34s
To view the application, visit http://localhost:1880
We deployed a Kubernetes loadbalancer that points to the petabridge.phobos.web
application at http://localhost:1880 - if you visit this Url it will generate both ASP.NET and Akka.NET traces and metrics.
Datadog Output
What sorts of metrics and traces are produced by Phobos?
Traces
First, if we take a look on the Datadog dashboard to view our service list, you'll see petabridge.phobos.web
:
If we drill into the petabridge.phobos.web
service on Datadog, we'll be able to view individual traces that look like this one:
You can see the trace connects both the original ASP.NET request and the Akka.NET actors who processed the subsequent messages.
Metrics
Phobos also pushes metrics to Datadog, which can be viewed in the metrics area:
These are the metrics that are produced automatically by Phobos - and we can drill down into those by clicking on an individual metric:
You can see that we successfully gathered this metric from all three of the separate nodes included inside the petabridge.phobos.web
K8s pod instances. We can even view a time-series graph of this data by clicking on "Metrics Explorer" inside Datadog:
Configuring Datadog and Phobos
So how can you configure Datadog to work alongside Phobos in your own Akka.NET applications?
First, we need to install the following NuGet packages:
- Phobos.Actor.Cluster - or whichever core Phobos package you need;
- App.Metrics.Datadog - this includes all of the App.Metrics drivers for Datadog; and
- Datadog.Trace.OpenTracing - this is the official DataDog .NET tracing driver that targets the OpenTracing standard supported by Phobos.
One those are installed, we can go about configuring Datadog tracing and metrics inside the Startup.cs
class using the Startup.ConfigureServices
method:
public void ConfigureServices(IServiceCollection services)
{
// enables OpenTracing for ASP.NET Core
services.AddOpenTracing(o =>
{
o.ConfigureAspNetCore(a =>
a.Hosting.OperationNameResolver = context => $"{context.Request.Method} {context.Request.Path}");
o.AddCoreFx();
});
// sets up Prometheus + DataDog + ASP.NET Core metrics
ConfigureAppMetrics(services);
// sets up DataDog tracing
ConfigureDataDogTracing(services);
// sets up Akka.NET
ConfigureAkka(services);
}
Tracer Setup
Let's explore what the ConfigureDataDogTracing
method does:
public static void ConfigureDataDogTracing(IServiceCollection services)
{
// Add DataDog Tracing
services.AddSingleton<ITracer>(sp =>
{
return OpenTracingTracerFactory.CreateTracer()
.WithScopeManager(new ActorScopeManager());
});
}
We call Datadog's OpenTracingTracerFactory.CreateTracer()
factory method to create a brand new tracer which will direct its traces to wherever the Datadog Agent Host is located. We set that location using the DD_AGENT_HOST
environment variable in our Kubernetes configuration.
The second method - WithScopeManager
is an extension method supported by Phobos itself used to help override the OpenTracing.IScopeManager
inside the underlying tracer. This method is used to pass in an instance of Phobos' ActorScopeManager
into Datadog's tracer.
Important
In order to provide the best possible data correlation between Akka.NET and other technologies such as ASP.NET, it's a best practice to always use the ActorScopeManager
in combination with any tracing technologies you choose.
Metrics Setup
The ConfigureAppMetrics
method is used to configure all of the metrics Phobos and App.Metrics will export to Datadog:
public static void ConfigureAppMetrics(IServiceCollection services)
{
services.AddMetricsTrackingMiddleware();
services.AddMetrics(b =>
{
var metrics = b.Configuration.Configure(o =>
{
o.GlobalTags.Add("host", Dns.GetHostName());
o.DefaultContextLabel = "akka.net";
o.Enabled = true;
o.ReportingEnabled = true;
})
.Report.ToDatadogHttp(options => {
options.Datadog.BaseUri = new Uri($"https://api.datadoghq.com/");
options.Datadog.ApiKey = Environment.GetEnvironmentVariable("DD_API_KEY");
options.HttpPolicy.BackoffPeriod = TimeSpan.FromSeconds(30);
options.HttpPolicy.FailuresBeforeBackoff = 5;
options.HttpPolicy.Timeout = TimeSpan.FromSeconds(10);
options.FlushInterval = TimeSpan.FromSeconds(20);
})
.Build();
});
services.AddMetricsReportingHostedService();
}
The Phobos sample follows the usual App.Metrics best practices with Microsoft.Extensions.Hosting
- the only really important thing to include here is your Datadog API key, which we pass in via a custom DD_API_KEY
environment variable.
Phobos and Akka.NET Configuration
Once Datadog's tracing and metrics are both configured, we just need to configure Phobos:
public static void ConfigureAkka(IServiceCollection services)
{
services.AddSingleton(sp =>
{
var metrics = sp.GetRequiredService<IMetricsRoot>();
var tracer = sp.GetRequiredService<ITracer>();
var config = ConfigurationFactory.ParseString(
File.ReadAllText("app.conf")).BootstrapFromDocker();
var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
.WithMetrics(m =>
m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
.WithTracing(t => t.SetTracer(tracer))) // binds Phobos to same tracer as ASP.NET Core
.WithSetup(BootstrapSetup.Create()
.WithConfig(config) // passes in the HOCON for Akka.NET to the ActorSystem
.WithActorRefProvider(PhobosProviderSelection
.Cluster)); // last line activates Phobos inside Akka.NET
var sys = ActorSystem.Create("ClusterSys", phobosSetup);
// create actor "container" and bind it to DI, so it can be used by ASP.NET Core
return new AkkaActors(sys);
});
// this will manage Akka.NET lifecycle
services.AddHostedService<AkkaService>();
}
The PhobosConfigBuilder
will allow us to pass in our IMetricsRoot
and ITracer
dependencies into Phobos, which we can then pass into our ActorSystem
to launch Phobos.
And that's all we need - Phobos will start producing traces and metrics automatically.