Using Phobos 2.X with Azure Application Insights Tracing and Metrics

This tutorial is a modified version of the Phobos QuickStart Tutorial.

Note

If you're interested in using Azure Application Insights with Phobos and Akka.NET, please be sure to see our Phobos + Azure Application Insights Akka.Cluster Dashboard that you can immediately import into your Application Insights account and visualize your Phobos data right away.

The Big Picture

This Azure Application Insights + Phobos tutorial is going to run inside a local Kubernetes cluster deployed on your local development machine - and this configuration could also be applied to a real production Kubernetes and Akka.Cluster deployment without significant modification.

We're going to generate HTTP traffic via a web browser, which will propagate requests into our Akka.NET cluster that will bounce between all of the nodes - our goal is to visualize this activity inside Application Insights so we can better observe how well our combined ASP.NET + Akka.NET system performs with traffic on it.

Phobos will leverage the following libraries:

To report both metrics and traces to an Azure Application Insights account.

Setup and Requirements

To run this tutorial you will need to do the following:

  1. Buy a Phobos license and get access to your NuGet key;
  2. Create an Azure Application Insights resource and copy its *Connection String into our demo script.
  3. Clone the Petabridge.Phobos.Web.ApplicationInsights repository to your local machine;
  4. Install Kubernetes on your local computer - the easiest path for a local development machine is typically through Docker Desktop.

Once that's done, we should be able to use the build scripts that come included with the Petabridge.Phobos.Web.ApplicationInsights repository we're going to use to run this sample.

Running the Sample From Your IDE

Create a new launch setting by creating a "launchSettings.json" file inside the "Properties" folder:

{
  "profiles": {
    "Petabridge.Phobos.Web.ApplicationInsights": {
      "commandName": "Project",
      "launchBrowser": true,
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development",
        "APPLICATIONINSIGHTS_CONNECTION_STRING": "[your Application Insights connection string]",
        "CLUSTER__SEEDS__0": "akka.tcp://ClusterSys@localhost:4055"
      },
      "applicationUrl": "https://localhost:5001;http://localhost:5000"
    }
  }
}

Run the new launch setting from inside your IDE

To view the application, visit http://localhost:5000

Running the Sample Inside Kubernetes

Open a command prompt in the directory where you cloned Petabridge.Phobos.Web.ApplicationInsights and run the following commands:

Windows

C:\git\Petabridge.Phobos.Web.ApplicationInsights> build Docker
C:\git\Petabridge.Phobos.Web.ApplicationInsights> .\k8s\deployAll.cmd "[your Application Insights connection string]"

The first command will build Docker images from the Petabridge.Phobos.Web.ApplicationInsights.csproj project and the second command will deploy the petabridge.phobos.web.applicationinsights Docker images into a dedicated Kubernetes namespace called phobos-web.

If you run the following command, kubectl -n phobos-web get all you will see output similar to the following:

NAME               READY   STATUS    RESTARTS   AGE
pod/phobos-web-0   1/1     Running   0          21h
pod/phobos-web-1   1/1     Running   0          21h
pod/phobos-web-2   1/1     Running   0          21h

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/phobos-web      ClusterIP      None             <none>        4055/TCP         21h
service/phobos-webapi   LoadBalancer   10.108.218.100   localhost     1880:30014/TCP   21h

NAME                          READY   AGE
statefulset.apps/phobos-web   3/3     21h

To view the application, visit http://localhost:1880

We deployed a Kubernetes load balancer that points to the petabridge.phobos.web application at http://localhost:1880 - if you visit this Url it will generate both ASP.NET and Akka.NET traces and metrics.

Application Insights Output

What sorts of metrics and traces are produced by Phobos?

Traces

First, if we take a look on the Application Insights dashboard to view our application map, you'll see Petabridge.Phobos.Web.ApplicationInsights:

Application Insights Akka.NET Phobos Application Map

If we click on that we should see a set of our "slowest requests":

Akka.NET slowest requests Application Insights

From there, we can drill into any one of these requests and pull some data about this request type over the observed lifetime of the application thus far - and I can even drill into specific samples collected by Phobos:

Application Insights Akka.NET performance details Application Insights Akka.NET trace details

You can see the trace connects both the original ASP.NET request and the Akka.NET actors who processed the subsequent messages.

Metrics

Phobos also pushes metrics to Application Insights, which can be viewed in the metrics area:

Akka.NET custom metrics in Azure Application Insights

Select "azure.applicationinsights" and from there we can view any of the built-in metrics included inside Phobos. In this case we're viewing the changes in the rate of Akka.NET messages received on average over a 24 hour period.

Akka.NET message processing rate in Azure Application Insights

Important

If you want detailed visibility into all of Phobos' metrics inside Application Insights, please install the Akka.NET + Phobos Akka.Cluster Azure Monitor Workbook.

Configuring Application Insights and Phobos

So how can you configure Application Insights to work alongside Phobos in your own Akka.NET applications?

First, we need to install the following NuGet packages:

Once those are installed, we can go about configuring Application Insights tracing and metrics inside the Startup.cs class using the Startup.ConfigureServices method:

public void ConfigureServices(IServiceCollection services)
{
    var appInsightsConnectionString = Environment.GetEnvironmentVariable("APPLICATIONINSIGHTS_CONNECTION_STRING");
    if (appInsightsConnectionString is null)
        throw new Exception("Application Insights connection string was not found in environment variables");

    var resource = ResourceBuilder.CreateDefault()
        .AddService(Assembly.GetEntryAssembly()!.GetName().Name!, serviceInstanceId: $"{Dns.GetHostName()}");

    // enables OpenTelemetry for ASP.NET Core
    services
        .AddOpenTelemetry()
        .WithTracing(builder =>
        {
            builder
                .SetResourceBuilder(resource)
                .AddPhobosInstrumentation()
                .AddSource(OTelSourceName)
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation();
        })
        .WithMetrics(builder =>
        {
            builder
                .SetResourceBuilder(resource)
                .AddPhobosInstrumentation()
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation();
        })
        .UseAzureMonitor();

    // sets up Akka.NET
    ConfigureAkka(services);
}

Instrumentation Key

So in this sample we pass in our Application Insights Instrumentation Key using an environment variable:

- name: APPLICATIONINSIGHTS_CONNECTION_STRING
  valueFrom:
    secretKeyRef:
      name: appinsights
      key: APPLICATIONINSIGHTS_CONNECTION_STRING

And this is what gets read inside our application automatically by the Azure.Monitor.OpenTelemetry.AspNetCore package.

Tracer Setup

Let's explore what the ConfigureServices method does:

public void ConfigureServices(IServiceCollection services)
{
<...>
    var resource = ResourceBuilder.CreateDefault()
        .AddService(Assembly.GetEntryAssembly()!.GetName().Name!, serviceInstanceId: $"{Dns.GetHostName()}");

    // enables OpenTelemetry for ASP.NET Core
    services
        .AddOpenTelemetry()
        .WithTracing(builder =>
        {
            builder
                .SetResourceBuilder(resource)
                .AddPhobosInstrumentation()
                .AddSource(OTelSourceName)
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation();
        })
<...>
}

First, we create an Open Telemetry ResourceBuilder, this is a Resource factory that represents a set of attributes that describes the entity producing the telemetry. We mark the resource as a service with name equals to the assembly name and give it an instance id of the host name of the machine the application is running in.

Within the .WithTracing() builder method, we:

  1. pass the ResourceBuilder instance into the TraceProviderBuilder.SetResourceBuilder() method,
  2. add Phobos instrumentation to the tracer provider,
  3. name the tracer provider "Petabridge.Phobos.Web.AppInsight", which is stored inside the constant OTelSourceName, and
  4. add Http client and ASP.NET instrumentation.

Metrics Setup

We also use the ConfigureServices method to set up metrics collection.

public void ConfigureServices(IServiceCollection services)
{
<...>    
    var resource = ResourceBuilder.CreateDefault()
        .AddService(Assembly.GetEntryAssembly()!.GetName().Name!, serviceInstanceId: $"{Dns.GetHostName()}");

    // enables OpenTelemetry for ASP.NET Core
    services
        .AddOpenTelemetry()
<...>
        .WithMetrics(builder =>
        {
            builder
                .SetResourceBuilder(resource)
                .AddPhobosInstrumentation()
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation();
        })
<...>
}

As before, we're using the ResourceBuilder instance. Within the .WithMetrics() builder method, we:

  1. pass the ResourceBuilder instance into the MeterProviderBuilder.SetResourceBuilder() method,
  2. add Phobos instrumentation to the meter provider, and
  3. add Http client and ASP.NET instrumentation.

Phobos and Akka.NET Configuration

Once Application Insights' tracing and metrics are both configured, we just need to configure Phobos:

public static void ConfigureAkka(IServiceCollection services)
{
    services.AddAkka("ClusterSys", (builder, provider) =>
    {
        // use our legacy app.conf file
        var config = ConfigurationFactory.ParseString(File.ReadAllText("app.conf"));

        builder
            .BootstrapFromDocker(provider)
            .AddHocon(config, HoconAddMode.Prepend)
            .ConfigureLoggers(loggerBuilder =>
            {
                loggerBuilder.ClearLoggers();
                loggerBuilder.AddLogger<SerilogLogger>();
            })
            .WithPhobos(AkkaRunMode.AkkaCluster) // enable Phobos
            .StartActors((system, registry) =>
            {
                var consoleActor = system.ActorOf(Props.Create(() => new ConsoleActor()), "console");
                var routerActor = system.ActorOf(Props.Empty.WithRouter(FromConfig.Instance), "echo");
                var routerForwarderActor =
                    system.ActorOf(Props.Create(() => new RouterForwarderActor(routerActor)), "fwd");
                registry.TryRegister<RouterForwarderActor>(routerForwarderActor);
            })
            // start https://cmd.petabridge.com/ for diagnostics
            .AddPetabridgeCmd(
                new PetabridgeCmdOptions
                {
                    // default IP address used to listen for incoming petabridge.cmd client connections
                    // should be a safe default as it listens on "all network interfaces".
                    Host = "0.0.0.0",
                    // default port number used to listen for incoming petabridge.cmd client connections
                    Port = 9110
                }, 
                pbmConfig =>
                {
                    pbmConfig.RegisterCommandPalette(ClusterCommands.Instance);
                    pbmConfig.RegisterCommandPalette(new RemoteCommands());
                });
    });
}

We are using the Akka.Hosting extension inside Phobos.Hosting to set up Phobos and Akka.NET. To do this, all you need to do is invoke the .WithPhobos() method.

And that's all we need - Phobos will start producing traces and metrics automatically.