Phobos 1.x Quickstart Tutorial

This tutorial will help you understand how to configure and install Phobos 1.x inside an Akka.NET v1.4+ application.

Important

Don't start any new Phobos applications with Phobos 1.x - use Phobos 2.x instead.

Note

For this quickstart tutorial, we're going to clone the Petabridge.Phobos.Web repository from Github and run it. Make sure you git checkout 1.x in order to run the 1.x version of this sample.

This project is a ready-made solution for testing Phobos in a real production environment using the following technologies:

Video Overview

If you want to get a gist of how the installation process works, you can follow the video from our Phobos Installation Tutorial below.

For further reading, please also see "Phobos Configuration".

Note

The Phobos QuickStart Tutorial also uses the "Phobos + Prometheus Akka.Cluster Dashboard for Grafana," which can immediately visualize your Phobos data inside Grafana when you're using Prometheus (which this demo does.) You should check it out along with the other Phobos dashboards.

Build and Local Deployment

Start by cloning this repository to your local system.

Next - to build this solution you will need to purchase a Phobos license key. They cost $4,000 per year per organization with no node count or seat limitations and comes with a 30 day money-back guarantee.

Once you purchase a Phobos NuGet keys for your organization, you're going to want to open NuGet.config and add your key:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <solution>
    <add key="disableSourceControlIntegration" value="true" />
  </solution>
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="phobos" value="{your key here}" />
  </packageSources>
</configuration>

From there, run the following command on the prompt:

PS> build.cmd Docker

This will create the Docker images the solution needs to run inside Kubernetes: petabridge.phobos.web:0.1.0.

Deploying the K8s Cluster (with Telemetry Installed)

From there, everything you need to run the solution in Kubernetes is already defined inside the k8s/ - just run the following command to launch the Phobos-enabled application inside Kubernetes:

PS> ./k8s/deployAll.cmd

This will spin up a separate Kubernetes namespace, phobos-web, and you can view which services are deployed by running the following command:

PS> kubectl get all -n phobos-web

You should see the following or similar output:

NAME                                        READY   STATUS    RESTARTS   AGE
pod/grafana-5f54fd5bf4-wvdgw                1/1     Running   0          11m
pod/jaeger-578558d6f9-2xzdv                 1/1     Running   0          11m
pod/phobos-web-0                            1/1     Running   3          11m
pod/phobos-web-1                            1/1     Running   2          10m
pod/phobos-web-2                            1/1     Running   0          9m54s
pod/prometheus-deployment-c6d99b8b9-28tmq   1/1     Running   0          11m

NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                               AGE
service/grafana-ip-service      LoadBalancer   10.105.46.6      localhost     3000:31641/TCP                        11m
service/jaeger-agent            ClusterIP      None             <none>        5775/UDP,6831/UDP,6832/UDP,5778/TCP   11m
service/jaeger-collector        ClusterIP      10.109.248.20    <none>        14267/TCP,14268/TCP,9411/TCP          11m
service/jaeger-query            LoadBalancer   10.109.204.203   localhost     16686:30911/TCP                       11m
service/phobos-web              ClusterIP      None             <none>        4055/TCP                              11m
service/phobos-webapi           LoadBalancer   10.103.247.68    localhost     1880:30424/TCP                        11m
service/prometheus-ip-service   LoadBalancer   10.101.119.120   localhost     9090:31698/TCP                        11m
service/zipkin                  ClusterIP      None             <none>        9411/TCP                              11m

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana                 1/1     1            1           11m
deployment.apps/jaeger                  1/1     1            1           11m
deployment.apps/prometheus-deployment   1/1     1            1           11m

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-5f54fd5bf4                1         1         1       11m
replicaset.apps/jaeger-578558d6f9                 1         1         1       11m
replicaset.apps/prometheus-deployment-c6d99b8b9   1         1         1       11m
Note

The restarts from the phobos-web-* pods come from calling Dns.GetHostName() prior to the local K8s service allocating its hostnames. Nothing to worry about - it'll resolve itself in a few moments.

Once the cluster is fully up and running you can explore the application and its associated telemetry via the following Urls:

There's many more metrics exported by Phobos that you can use to create your own dashboards or extend the existing ones - you can view all of them by going to http://localhost:1880/metrics

Looking at the Code

So we can poke around and play with the solution now, but where's the real code?

It's all located inside the Startup.cs class.

public class Startup
{
    /// <summary>
    ///     Name of the <see cref="Environment" /> variable used to direct Phobos' Jaeger
    ///     output.
    ///     See https://github.com/jaegertracing/jaeger-client-csharp for details.
    /// </summary>
    public const string JaegerAgentHostEnvironmentVar = "JAEGER_AGENT_HOST";

    public const string JaegerEndpointEnvironmentVar = "JAEGER_ENDPOINT";

    public const string JaegerAgentPortEnvironmentVar = "JAEGER_AGENT_PORT";

    public const int DefaultJaegerAgentPort = 6832;

    // This method gets called by the runtime. Use this method to add services to the container.
    // For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
    public void ConfigureServices(IServiceCollection services)
    {
        // enables OpenTracing for ASP.NET Core
        services.AddOpenTracing(o =>
        {
            o.ConfigureAspNetCore(a =>
                a.Hosting.OperationNameResolver = context => $"{context.Request.Method} {context.Request.Path}");
            o.AddCoreFx();
        });

        // sets up Prometheus + ASP.NET Core metrics
        ConfigureAppMetrics(services);

        // sets up Jaeger tracing
        ConfigureJaegerTracing(services);

        // sets up Akka.NET
        ConfigureAkka(services);
    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment()) app.UseDeveloperExceptionPage();

        app.UseRouting();

        // enable App.Metrics routes
        app.UseMetricsAllMiddleware();
        app.UseMetricsAllEndpoints();

        app.UseEndpoints(endpoints =>
        {
            var actors = endpoints.ServiceProvider.GetService<AkkaActors>();
            var tracer = endpoints.ServiceProvider.GetService<ITracer>();
            endpoints.MapGet("/", async context =>
            {
                using (var s = tracer.BuildSpan("Cluster.Ask").StartActive())
                {
                    // router actor will deliver message randomly to someone in cluster
                    var resp = await actors.RouterForwarderActor
                        .Ask<string>($"hit from {context.TraceIdentifier}",
                        TimeSpan.FromSeconds(5));
                    await context.Response.WriteAsync(resp);
                }
            });
        });
    }
}

For the sake of brevity, the Startup methods that configure tracing, metrics, and Phobos aren't shown yet. This is to set the stage.

Configuring ASP.NET + Phobos Tracing

We call install the OpenTracing.Contrib.NetCore NuGet package so we can get built-in ASP.NET, HttpClient, and .NET Core tracing in our solution.

PS> Install-Package OpenTracing.Contrib.NetCore

With that package installed, we can call the following method during service setup:

// enables OpenTracing for ASP.NET Core
services.AddOpenTracing(o =>
{
    o.ConfigureAspNetCore(a =>
        a.Hosting.OperationNameResolver = context 
            => $"{context.Request.Method} {context.Request.Path}");
    o.AddCoreFx();
});

The services.AddOpenTracing call injects the OpenTracing instrumentation into the ASP.NET Core middleware, and the a.Hosting.OperationNameResolver is used to help us customize the operation names that appear in Jaeger for ASP.NET HTTP calls.

Next, we need to create a Jaeger ITracer that we can actually consume inside ASP.NET, Phobos, and other services.

PS> Install-Package Jaeger

We need to install the Jaeger NuGet package into our solution and add a method for configuring our ITracer.

public static void ConfigureJaegerTracing(IServiceCollection services)
{
    static ISender BuildSender()
    {
        if (string.IsNullOrEmpty(Environment.GetEnvironmentVariable(JaegerEndpointEnvironmentVar)))
        {
            if (!int.TryParse(Environment.GetEnvironmentVariable(JaegerAgentPortEnvironmentVar),
                out var udpPort))
                udpPort = DefaultJaegerAgentPort;
            return new UdpSender(
                Environment.GetEnvironmentVariable(JaegerAgentHostEnvironmentVar) ?? "localhost",
                udpPort, 0);
        }

        return new HttpSender(Environment.GetEnvironmentVariable(JaegerEndpointEnvironmentVar));
    }

    services.AddSingleton<ITracer>(sp =>
    {
        var loggerFactory = sp.GetRequiredService<ILoggerFactory>();

        var builder = BuildSender();
        var logReporter = new LoggingReporter(loggerFactory);

        var remoteReporter = new RemoteReporter.Builder()
            .WithLoggerFactory(loggerFactory) // optional, defaults to no logging
            .WithMaxQueueSize(100) // optional, defaults to 100
            .WithFlushInterval(TimeSpan.FromSeconds(1)) // optional, defaults to TimeSpan.FromSeconds(1)
            .WithSender(builder) // optional, defaults to UdpSender("localhost", 6831, 0)
            .Build();

        var sampler = new ConstSampler(true); // keep sampling disabled

        // name the service after the executing assembly
        var tracer = new Tracer.Builder(typeof(Startup).Assembly.GetName().Name)
            .WithReporter(new CompositeReporter(remoteReporter, logReporter))
            .WithSampler(sampler)
            .WithScopeManager(
                new ActorScopeManager()); // IMPORTANT: ActorScopeManager needed to properly correlate trace inside Akka.NET

        return tracer.Build();
    });
}

Jaeger supports multiple reporting modes - and in this case we're using some environment variables to choose between reporting directly to the Jaeger aggregation service or an individual Jaeger agent. You can read more about that from the official Jaeger C# driver documentation.

But once Jaeger is setup, we need to call services.AddSingleton<ITracer> so we can re-use this configured tracer inside both ASP.NET and Phobos.

Configuring App.Metrics, Prometheus, and ASP.NET

The App.Metrics base packages are already pre-installed into Phobos.Actor via the Phobos.Monitoring NuGet package, so we don't need to install those.

However, we do need to install the packages needed for ASP.NET integration and reporting to Prometheus:

PS> Install-Package App.Metrics.AspNetCore
PS> Install-Package App.Metrics.AspNetCore.Tracking
PS> Install-Package App.Metrics.Formatters.Prometheus

Once that's done, we need to configure everything inside the IServiceCollection:

public static void ConfigureAppMetrics(IServiceCollection services)
{
    services.AddMetricsTrackingMiddleware();
    services.AddMetrics(b =>
    {
        var metrics = b.Configuration.Configure(o =>
            {
                o.GlobalTags.Add("host", Dns.GetHostName());
                o.DefaultContextLabel = "akka.net";
                o.Enabled = true;
                o.ReportingEnabled = true;
            })
            .OutputMetrics.AsPrometheusPlainText()
            .Build();

        services.AddMetricsEndpoints(ep =>
        {
            ep.MetricsTextEndpointOutputFormatter = metrics.OutputMetricsFormatters
                .OfType<MetricsPrometheusTextOutputFormatter>().First();
            ep.MetricsEndpointOutputFormatter = metrics.OutputMetricsFormatters
                .OfType<MetricsPrometheusTextOutputFormatter>().First();
        });
    });
    services.AddMetricsReportingHostedService();
}

The services.AddMetricsTrackingMiddleware() call adds the middleware we need for recording metrics from ASP.NET HTTP traffic.

The services.AddMetrics() call configures the IMetricsRoot that Phobos and ASP.NET Core depend on for aggregating metrics.

services.AddMetricsEndpoints(ep =>
{
    ep.MetricsTextEndpointOutputFormatter = metrics.OutputMetricsFormatters
        .OfType<MetricsPrometheusTextOutputFormatter>().First();
    ep.MetricsEndpointOutputFormatter = metrics.OutputMetricsFormatters
        .OfType<MetricsPrometheusTextOutputFormatter>().First();
});

The above call ensures that all of the metrics will be available in Prometheus plain text format on the HTTP routes GET /metrics or GET /metrics-text. The Prometheus instance we spin up inside the K8s cluster is configured to periodically scrape the metrics data from this url on each instance of the Petabridge.Phobos.Web node.

Configuring Phobos and Starting the ActorSystem

Now that we've configured our ITracer , our IMetricsRoot, and registered both of these with the IServiceCollection - we can now configure Akka.NET and Phobos to consume them.

Important

We need to have the Phobos.Actor.Cluster NuGet package installed in order to complete instrumentation of this Akka.NET application.

public static void ConfigureAkka(IServiceCollection services)
{
    services.AddSingleton(sp =>
    {
        var metrics = sp.GetRequiredService<IMetricsRoot>();
        var tracer = sp.GetRequiredService<ITracer>();

        var config = ConfigurationFactory.ParseString(File.ReadAllText("app.conf")).BootstrapFromDocker();

        var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
                .WithMetrics(m =>
                    m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
                .WithTracing(t => t.SetTracer(tracer))) // binds Phobos to same tracer as ASP.NET Core
            .WithSetup(BootstrapSetup.Create()
                .WithConfig(config) // passes in the HOCON for Akka.NET to the ActorSystem
                .WithActorRefProvider(PhobosProviderSelection
                    .Cluster)); // last line activates Phobos inside Akka.NET

        var sys = ActorSystem.Create("ClusterSys", phobosSetup);

        // create actor "container" and bind it to DI, so it can be used by ASP.NET Core
        return new AkkaActors(sys);
    });

    // this will manage Akka.NET lifecycle
    services.AddHostedService<AkkaService>();
}

The key to setting up an ActorSystem with Phobos is to use the PhobosSetup class, which is derived from the Setup class in Akka.NET.

var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
                .WithMetrics(m =>
                    m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
                .WithTracing(t => t.SetTracer(tracer))) // binds Phobos to same tracer as ASP.NET Core
            .WithSetup(BootstrapSetup.Create()
                .WithConfig(config) // passes in the HOCON for Akka.NET to the ActorSystem
                .WithActorRefProvider(PhobosProviderSelection
                    .Cluster)); // last line activates Phobos inside Akka.NET

This allows us to take the same tracer and metrics that will be used by ASP.NET and pass it into Akka.NET - thus we can correlate activity that begins inside ASP.NET Core and uses the underlying Akka.NET cluster!

Important

Please note we call WithSetup and pass in an Akka.Actor.BootstrapSetup - this allows us to make sure that the HOCON Config gets passed into the ActorSystem as well. Additionally, we also need to pass in the WithActorRefProvider.PhobosProviderSelection.Cluster) method because that's what actually enables Phobos inside the ActorSystem.

That's all we need to capture the automatically collected Phobos data, but if we want to manually capture data there's more we can do below.

Calling Phobos APIs

If you want to call the Phobos APIs directly to capture additional data, this is something you can do by calling Context.GetInstrumentation() inside any actor:

public sealed class ConsoleActor : ReceiveActor
{
    private readonly ILoggingAdapter _log = Context.GetLogger();

    public ConsoleActor()
    {
        Receive<string>(_ =>
        {
            // use the local metrics handle to record a timer duration for how long this block of code takes to execute
            Context.GetInstrumentation().Monitor.Timer.Time(new TimerOptions {Name = "ProcessingTime"}, () =>
            {
                // start another span programmatically inside actor
                using (var newSpan = Context.GetInstrumentation().Tracer.BuildSpan("SecondOp").StartActive())
                {
                    _log.Info("Received: {0}", _);
                    Sender.Tell(_);
                }
            });
        });
    }
}

When we call Context.GetInstrumentation().Monitor, we are accessing the IMeasureMetrics instance used by the application - and we can create custom metrics, such as a Timing in this case - and these will all be picked up by Prometheus automatically.

Alternatively, when we call Context.GetInstrumentation().Tracer.BuildSpan("SecondOp").StartActive() - this is creating a new span operation that is a child span of the current one the actor created when it began processing the message, and logs that are recorded inside that using block will automatically be appended to the currently active ISpan.

Tearing Down the Cluster

When you're done exploring this sample, you can tear down the cluster by running the following command:

PS> ./k8s/destroyAll.cmd

This will delete the phobos-web namespace and all of the resources inside it.

Further Reading

For further reading, please see: