Using Phobos with Datadog Tracing and Metrics
This tutorial is a modified version of the Phobos QuickStart Tutorial.
Note
If you're interested in using DataDog with Phobos, please be sure to see our Phobos + DataDog Akka.Cluster Dashboard that you can immediately import into your DataDog account and visualize your Phobos data right away.
The Big Picture
This Datadog + Phobos tutorial is going to run inside a local Kubernetes cluster deployed on your local development machine - and this configuration could also be applied to a real production Kubernetes and Akka.Cluster deployment without significant modification.
We're going to generate HTTP traffic via a web browser, which will propagate requests into our Akka.NET cluster that will bounce between all of the nodes - our goal is to visualize this activity inside Datadog so we can better observe how well our combined ASP.NET + Akka.NET system performs with traffic on it.
Phobos will transmit data from each individual pod to a local Datadog agent hosted as a DaemonSet - there will be one Datadog agent pod per node in Kubernetes. These agent pods will pre-aggregate the metrics reported by our application and then transmit them to Datadog's cloud hosted services - where you can view the output.
Setup and Requirements
To run this tutorial you will need to do the following:
- Buy a Phobos license and get access to your NuGet key;
- Create a Datadog account and access your Datadog API keys;
- Clone the Petabridge.Phobos.Web.DataDog repository to your local machine;
- Install Kubernetes on your local computer - the easiest path for a local development machine is typically through Docker Desktop.
- Install Helm 3, the package manager for Kubernetes - we're going to use Helm to install the
datadogHelm chart, which will install a Datadog Agent as a DaemonSet in our local Kubernetes cluster.
Installing the Datadog Helm Repository
In order to install the datadog Helm package we need to add the Datadog repository to our local Helm package sources:
PS> helm repo add datadog https://helm.datadoghq.com
PS> helm repo update
Once that's done, we should be able to use the build scripts that come included with the Petabridge.Phobos.Web.DataDog repository we're going to use to run this sample.
Running the Sample
Open a shell in the directory where you cloned Petabridge.Phobos.Web.DataDog and run the following commands:
Windows
PS> ./build.cmd Docker
PS> ./k8s/deployAll.cmd [your Datadog API Key]
The first command will build Docker images from the Petabridge.Phobos.Web.DataDog.csproj project and the second command will deploy Datadog and the petabridge.phobos.web.datadog Docker images into a dedicated Kubernetes namespace called phobos-web.
If you run the following command, kubectl -n phobos-web get all you will see output similar to the following:
NAME READY STATUS RESTARTS AGE
pod/datadog-agent-kube-state-metrics-6f4cd6b45b-k4hzn 1/1 Running 0 7m34s
pod/datadog-agent-l2s8w 2/2 Running 0 7m34s
pod/phobos-web-0 1/1 Running 0 7m33s
pod/phobos-web-1 1/1 Running 0 7m32s
pod/phobos-web-2 1/1 Running 0 7m30s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/datadog-agent-kube-state-metrics ClusterIP 10.102.75.157 <none> 8080/TCP 7m34s
service/phobos-web ClusterIP None <none> 4055/TCP 7m33s
service/phobos-webapi LoadBalancer 10.106.31.43 localhost 1880:32147/TCP 7m33s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/datadog-agent 1 1 1 1 1 kubernetes.io/os=linux 7m34s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/datadog-agent-kube-state-metrics 1/1 1 1 7m34s
NAME DESIRED CURRENT READY AGE
replicaset.apps/datadog-agent-kube-state-metrics-6f4cd6b45b 1 1 1 7m34s
To view the application, visit http://localhost:1880
We deployed a Kubernetes load-balancer that points to the petabridge.phobos.web application at http://localhost:1880 - if you visit this Url it will generate both ASP.NET and Akka.NET traces and metrics.
Datadog Output
What sorts of metrics and traces are produced by Phobos?
Traces
First, if we take a look on the Datadog dashboard to view our service list, you'll see petabridge.phobos.web:
If we drill into the petabridge.phobos.web service on Datadog, we'll be able to view individual traces that look like this one:
You can see the trace connects both the original ASP.NET request and the Akka.NET actors who processed the subsequent messages.
Metrics
Phobos also pushes metrics to Datadog, which can be viewed in the metrics area:
These are the metrics that are produced automatically by Phobos - and we can drill down into those by clicking on an individual metric:
You can see that we successfully gathered this metric from all three of the separate nodes included inside the petabridge.phobos.web K8s pod instances. We can even view a time-series graph of this data by clicking on "Metrics Explorer" inside Datadog:
Configuring Datadog and Phobos
So how can you configure Datadog to work alongside Phobos in your own Akka.NET applications?
First, we need to install the following NuGet packages:
- Phobos.Actor.Cluster - or whichever core Phobos package you need;
- App.Metrics.Datadog - this includes all of the App.Metrics drivers for Datadog; and
- Datadog.Trace.OpenTracing - this is the official DataDog .NET tracing driver that targets the OpenTracing standard supported by Phobos.
One those are installed, we can go about configuring Datadog tracing and metrics inside the Startup.cs class using the Startup.ConfigureServices method:
public void ConfigureServices(IServiceCollection services)
{
// needed on .NET Core 3.1
// see https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/README.md#special-case-when-using-insecure-channel
AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport",
true);
var otelAgentAddress = Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT");
if (string.IsNullOrEmpty(otelAgentAddress))
{
// default local address
otelAgentAddress = "http://0.0.0.0:4317";
}
var resource = ResourceBuilder.CreateDefault()
.AddService(Assembly.GetEntryAssembly()!.GetName().Name,
serviceVersion: Assembly.GetEntryAssembly().GetName().Version.ToString(),
serviceInstanceId: $"{Dns.GetHostName()}");
services.AddOpenTelemetry()
.WithTracing(tracer =>
{
tracer.AddAspNetCoreInstrumentation()
.SetResourceBuilder(resource)
.AddSource(AppOtelSourceName)
.AddHttpClientInstrumentation()
.AddPhobosInstrumentation()
.AddOtlpExporter(options =>
{
options.Protocol = OtlpExportProtocol.Grpc;
options.Endpoint = new Uri(otelAgentAddress);
});
})
.WithMetrics(metrics =>
{
metrics.AddAspNetCoreInstrumentation()
.SetResourceBuilder(resource)
.AddMeter(AppOtelSourceName)
.AddHttpClientInstrumentation()
.AddPhobosInstrumentation()
.AddOtlpExporter((options, readerOptions) =>
{
readerOptions.TemporalityPreference = MetricReaderTemporalityPreference.Delta;
options.Protocol = OtlpExportProtocol.Grpc;
options.Endpoint = new Uri(otelAgentAddress);
});
});
// sets up Akka.NET
ConfigureAkka(services);
}
This setup leverages OpenTelemetry's flexible exporter pipeline and more specifically, uses the OpenTelemetry Line Protocol (OTLP) as its means of transmitting trace and metric data to DataDog. Normally this data would be routed to an OTLP Collector - however, DataDog's own agents also support receiving OTLP data, so that's how we're going to transmit our data using the OtlpExportProtocol.Grpc protocol and the address of the DataDog agent.
Important
All you need to do to ensure that this setup functions correctly is to ensure that the DataDog Agent is running with the OTLP ports exposed - you can read more on how to do this here: https://docs.datadoghq.com/tracing/setup_overview/open_standards/otlp_ingest_in_the_agent/?tab=host
Phobos and Akka.NET Configuration
Once Datadog's tracing and metrics are both configured, we just need to configure Phobos:
public static void ConfigureAkka(IServiceCollection services)
{
var config = ConfigurationFactory.ParseString(File.ReadAllText("app.conf")).BootstrapFromDocker();
services.AddAkka("ClusterSys", (builder, provider) =>
{
builder
.AddHocon(config)
.WithPhobos(AkkaRunMode.AkkaCluster)
.StartActors((system, registry) =>
{
var consoleActor = system.ActorOf(Props.Create(() => new ConsoleActor()), "console");
var routerActor = system.ActorOf(Props.Empty.WithRouter(FromConfig.Instance), "echo");
var routerForwarder =
system.ActorOf(Props.Create(() => new RouterForwarderActor(routerActor)), "fwd");
registry.TryRegister<RouterForwarderActor>(routerForwarder);
})
.StartActors((system, registry) =>
{
// start https://cmd.petabridge.com/ for diagnostics and profit
var pbm = PetabridgeCmd.Get(system); // start Pbm
pbm.RegisterCommandPalette(ClusterCommands.Instance);
pbm.RegisterCommandPalette(RemoteCommands.Instance);
pbm.Start(); // begin listening for PBM management commands
});
});
}
And that's all we need - Phobos will start producing traces and metrics automatically.