Using Phobos with Azure Application Insights Tracing and Metrics
This tutorial is a modified version of the Phobos QuickStart Tutorial.
Note
If you're interested in using Azure Application Insights with Phobos and Akka.NET, please be sure to see our Phobos + Azure Application Insights Akka.Cluster Dashboard that you can immediately import into your Application Insights account and visualize your Phobos data right away.
The Big Picture
This Azure Application Insights + Phobos tutorial is going to run inside a local Kubernetes cluster deployed on your local development machine - and this configuration could also be applied to a real production Kubernetes and Akka.Cluster deployment without significant modification.
We're going to generate HTTP traffic via a web browser, which will propagate requests into our Akka.NET cluster that will bounce between all of the nodes - our goal is to visualize this activity inside Application Insights so we can better observe how well our combined ASP.NET + Akka.NET system performs with traffic on it.
Phobos will leverage the following two libraries:
To report both metrics and traces to an Azure Application Insights account.
Setup and Requirements
To run this tutorial you will need to do the following:
- Buy a Phobos license and get access to your NuGet key;
- Create an Azure Application Insights resource and copy its Instrumentation Key into our demo script.
- Clone the Petabridge.Phobos.Web.ApplicationInsights repository to your local machine;
- Install Kubernetes on your local computer - the easiest path for a local development machine is typically through Docker Desktop.
Once that's done, we should be able to use the build scripts that come included with the Petabridge.Phobos.Web.ApplicationInsights repository we're going to use to run this sample.
Running the Sample
Open a shell in the directory where you cloned Petabridge.Phobos.Web.ApplicationInsights and run the following commands:
Windows
PS> ./build.cmd Docker
PS> ./k8s/deployAll.cmd [your Application Insights Instrumentation Key]
The first command will build Docker images from the Petabridge.Phobos.Web.ApplicationInsights.csproj project and the second command will deploy Datadog and the petabridge.phobos.web.applicationinsights Docker images into a dedicated Kubernetes namespace called phobos-web.
If you run the following command, kubectl -n phobos-web get all you will see output similar to the following:
NAME READY STATUS RESTARTS AGE
pod/phobos-web-0 1/1 Running 0 21h
pod/phobos-web-1 1/1 Running 0 21h
pod/phobos-web-2 1/1 Running 0 21h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/phobos-web ClusterIP None <none> 4055/TCP 21h
service/phobos-webapi LoadBalancer 10.108.218.100 localhost 1880:30014/TCP 21h
NAME READY AGE
statefulset.apps/phobos-web 3/3 21h
To view the application, visit http://localhost:1880
We deployed a Kubernetes loadbalancer that points to the petabridge.phobos.web application at http://localhost:1880 - if you visit this Url it will generate both ASP.NET and Akka.NET traces and metrics.
Application Insights Output
What sorts of metrics and traces are produced by Phobos?
Traces
First, if we take a look on the Application Insights dashboard to view our service, you'll see Petabridge.Phobos.Web.ApplicationInsights:
If we click on that we should see a set of our "slowest requests":
From there, we can drill into any one of these requests and pull some data about this request type over the observed lifetime of the application thus far - and I can even drill into specific samples collected by Phobos:
You can see the trace connects both the original ASP.NET request and the Akka.NET actors who processed the subsequent messages.
Metrics
Phobos also pushes metrics to Application Insights, which can be viewed in the metrics area:
Select "Akka.NET" and from there we can view any of the built-in metrics included inside Phobos. In this case we're viewing the changes in the rate of Akka.NET messages received on average over a 24 hour period.
Important
If you want detailed visiblity into all of Phobos' metrics inside Application Insights, please install the Akka.NET + Phobos Akka.Cluster Azure Monitor Workbook.
Configuring Application Insights and Phobos
So how can you configure Application Insights to work alongside Phobos in your own Akka.NET applications?
First, we need to install the following NuGet packages:
- Phobos.Actor.Cluster - or whichever core Phobos package you need;
- App.Metrics.Reporting.ApplicationInsights; and
- Petabridge.Tracing.ApplicationInsights - this library is maintained by Petabridge in order to make it possible to wrap the Application Insights tracing API inside the OpenTracing standard.
One those are installed, we can go about configuring Application Insights tracing and metrics inside the Startup.cs class using the Startup.ConfigureServices method:
public void ConfigureServices(IServiceCollection services)
{
// enables OpenTracing for ASP.NET Core
services.AddOpenTracing(o =>
{
o.ConfigureAspNetCore(a =>
{
a.Hosting.OperationNameResolver = context => $"{context.Request.Method} {context.Request.Path}";
// skip Prometheus HTTP /metrics collection from appearing in our tracing system
a.Hosting.IgnorePatterns.Add(x => x.Request.Path.StartsWithSegments(new PathString("/metrics")));
});
o.ConfigureGenericDiagnostics(c => { });
});
// add the Telemetry configuration to our DI dependencies if App Insights is available
if (IsAppInsightsEnabled)
{
var appKey = Environment.GetEnvironmentVariable(
AppInsightsInstrumentationKeyVariableName);
var config = new TelemetryConfiguration(appKey);
services.AddSingleton<TelemetryConfiguration>(config);
}
// sets up App Insights + ASP.NET Core metrics
ConfigureAppMetrics(services);
// sets up App Insights tracing
ConfigureTracing(services);
// sets up Akka.NET
ConfigureAkka(services);
}
Instrumentation Key
So in this sample we pass in our Application Insights Instrumentation Key using an environment variable:
- name: APP_INSIGHTS_INSTRUMENTATION_KEY
valueFrom:
secretKeyRef:
name: appinsights
key: APP_INSIGHTS_INSTRUMENTATION_KEY
And this is what gets read inside our application, which also creates a TelemetryConfiguration from the Application Insights SDK:
// add the Telemetry configuration to our DI dependencies if App Insights is available
if (IsAppInsightsEnabled)
{
var appKey = Environment.GetEnvironmentVariable(
AppInsightsInstrumentationKeyVariableName);
var config = new TelemetryConfiguration(appKey);
services.AddSingleton<TelemetryConfiguration>(config);
}
We'll use that value next to configure tracing.
Tracer Setup
Let's explore what the ConfigureDataDogTracing method does:
public static void ConfigureTracing(IServiceCollection services)
{
services.AddSingleton<ITracer>(sp =>
{
if (IsAppInsightsEnabled)
{
var telConfig = sp.GetRequiredService<TelemetryConfiguration>();
var endpoint = new Tracing.ApplicationInsights.Endpoint(Assembly.GetEntryAssembly()?.GetName().Name,
Dns.GetHostName(), null);
// name the service after the executing assembly
var tracer = new ApplicationInsightsTracer(telConfig, new ActorScopeManager(), new B3Propagator(),
new DateTimeOffsetTimeProvider(), endpoint);
return tracer;
}
// use the GlobalTracer, otherwise
return GlobalTracer.Instance;
});
}
If Application Insights is enabled, we'll fetch the TelemetryConfiguration from the IServiceProvider and use it to create a new ApplicationInsightsTracer.
However, it's important to note that the Tracing.ApplicationInsights.Endpoint needs to also be populated in order to correctly name each of our services inside Application Insights - so we pass in the name of our service (defined as the name of the entry assembly) and the host address of the service instance (Dns.GetHostName()).
Also notice that we're passing an instance of Phobos' ActorScopeManager into the Application Insights tracer.
Important
In order to provide the best possible data correlation between Akka.NET and other technologies such as ASP.NET, it's a best practice to always use the ActorScopeManager in combination with any tracing technologies you choose.
Metrics Setup
The ConfigureAppMetrics method is used to configure all of the metrics Phobos and App.Metrics will export to Application Insights:
public static void ConfigureAppMetrics(IServiceCollection services)
{
services.AddMetricsTrackingMiddleware();
services.AddMetrics(b =>
{
var metrics = b.Configuration.Configure(o =>
{
o.GlobalTags.Add("host", Dns.GetHostName());
o.DefaultContextLabel = "akka.net";
o.Enabled = true;
o.ReportingEnabled = true;
});
if (IsAppInsightsEnabled)
metrics = metrics.Report.ToApplicationInsights(opts =>
{
opts.InstrumentationKey =
Environment.GetEnvironmentVariable(
AppInsightsInstrumentationKeyVariableName);
opts.ItemsAsCustomDimensions = true;
opts.DefaultCustomDimensionName = "item";
});
else // report to console if AppInsights isn't enabled
metrics = metrics.Report.ToConsole();
metrics.Build();
});
services.AddMetricsReportingHostedService();
}
The Phobos sample follows the usual App.Metrics best practices with Microsoft.Extensions.Hosting - the only really important thing to include here is your Application Insights instrumentation key, which we pass in via a custom APP_INSIGHTS_INSTRUMENTATION_KEY environment variable.
Phobos and Akka.NET Configuration
Once Application Insights's tracing and metrics are both configured, we just need to configure Phobos:
public static void ConfigureAkka(IServiceCollection services)
{
services.AddSingleton(sp =>
{
var metrics = sp.GetRequiredService<IMetricsRoot>();
var tracer = sp.GetRequiredService<ITracer>();
var config = ConfigurationFactory.ParseString(
File.ReadAllText("app.conf")).BootstrapFromDocker();
var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
.WithMetrics(m =>
m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
.WithTracing(t => t.SetTracer(tracer))) // binds Phobos to same tracer as ASP.NET Core
.WithSetup(BootstrapSetup.Create()
.WithConfig(config) // passes in the HOCON for Akka.NET to the ActorSystem
.WithActorRefProvider(PhobosProviderSelection
.Cluster)); // last line activates Phobos inside Akka.NET
var sys = ActorSystem.Create("ClusterSys", phobosSetup);
// create actor "container" and bind it to DI, so it can be used by ASP.NET Core
return new AkkaActors(sys);
});
// this will manage Akka.NET lifecycle
services.AddHostedService<AkkaService>();
}
The PhobosConfigBuilder will allow us to pass in our IMetricsRoot and ITracer dependencies into Phobos, which we can then pass into our ActorSystem to launch Phobos.
And that's all we need - Phobos will start producing traces and metrics automatically.