Introduction

Phobos is a product developed and maintained by Petabridge and it provides Akka.NET users with the ability to quickly and easily instrument their production Akka.NET and Akka.Cluster applications with monitoring, distributed tracing, and more.

Phobos 2.x is a library - and it automatically injects OpenTelemetry instrumentation inside your Akka.NET actors so you can record trace and metric data at runtime.

Important

If you're not familiar with OpenTelemetry - it is an industry-wide standard for instrumenting applications with tracing, metrics, and logging. As of .NET 6 it's now the "official" way .NET applications are instrumented and will be supported going forward.

You can learn more by watching our video: "OpenTelemetry in .NET Explained"

All major APM vendors such as DataDog, Microsoft, Google, Amazon, NewRelic, Elastic, and major OSS projects like Jaeger and Prometheus are behind the effort and have implemented OpenTelemetry.Exporter.{Vendor} NuGet packages you can use to export data from OTel to their service. Read more.

Phobos 1.0, which is deprecated but supported for maintenance fixes through 2023, uses OpenTracing and App.Metrics to export your data to various tracing and reporting systems that support those open source .NET standards - you can see a full list of supported tracing and metrics libraries here.

You can consume that data from either version of Phobos through dashboard tools such as Grafana, Jaeger, and more.

How it Works

Fundamentally, Phobos works by instrumenting Akka.NET actors behind the scenes. You can install Phobos via your organization's private Phobos NuGet feed via Sdkbin and install the appropriate Phobos.Actor.* NuGet package:

• Phobos.Actor for purely local Akka.NET applications;
• Phobos.Actor.Remote for Akka.Remote applications;
• Phobos.Actor.Cluster, by far the most popular option, for Akka.Cluster applications; or
• Phobos.Hosting for the most flexibility and ease of use - includes all of the other packages listed above.

Once you've installed the appropriate Phobos NuGet package you can activate Phobos with ease.

Phobos 2.x Setup

Using Phobos.Hosting

Please note that the sample below is a complete Akka.NET application running with Akka.Hosting.

private static async Task Main(string[] args)
{
	var resource = ResourceBuilder.CreateDefault()
		.AddService(Assembly.GetEntryAssembly().GetName().Name, serviceInstanceId: $"{Dns.GetHostName()}");

	var host = WebHost.CreateDefaultBuilder(args)
		.ConfigureServices((hostContext, services) =>
		{
			services.AddControllers();
			services.AddOpenTelemetry()
				.WithTracing(builder =>
				{
					builder
						.SetResourceBuilder(resource)
						.AddPhobosInstrumentation()
						.AddHttpClientInstrumentation()
						.AddAspNetCoreInstrumentation()
						.AddOtlpExporter();
				})
				.WithMetrics(builder =>
				{
					builder
						.SetResourceBuilder(resource)
						.AddPhobosInstrumentation()
						.AddHttpClientInstrumentation()
						.AddAspNetCoreInstrumentation()
						.AddPrometheusExporter(opt => { });
				});
			services.AddAkka("ClusterSys", (builder, provider) =>
			{
				builder.WithRemoting("localhost", 8882);
				builder.WithClustering(new ClusterOptions()
				{
					SeedNodes = new[]
					{
						"akka.tcp://ClusterSys@localhost:8882"
					},
					Roles = new[] { "app" }
				});

				// ENABLE PHOBOS
				builder.WithPhobos(AkkaRunMode.AkkaCluster);

				// turn on Petabridge.Cmd
				builder.StartActors((system, registry) =>
				{
					var echoActor = system.ActorOf(Props.Create(() => new EchoActor()), "echo");
					registry.TryRegister<EchoActor>(echoActor);
				});
			});
		})
		.Configure(app =>
		{
			app.UseRouting();
			app.UseOpenTelemetryPrometheusScrapingEndpoint();
			app.UseEndpoints(endpoints =>
			{
				endpoints.MapGet("/",
					async context => await context.RequestServices.GetRequiredService<ActorRegistry>()
						.Get<EchoActor>().Ask<string>(context.TraceIdentifier, TimeSpan.FromSeconds(3)));
				endpoints.MapControllers();
			});
		})
		.Build();

	await host.RunAsync();
}

The key line that enables Phobos are these two:

services.AddOpenTelemetry()
    .WithTracing(builder =>
    {
        builder
            .SetResourceBuilder(resource)
            .AddPhobosInstrumentation() // subscribes to the Phobos OTel source (same for metrics)
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddJaegerExporter(opt => { });
    })

and

 // enable Phobos inside Akka.NET
builder.WithPhobos(AkkaRunMode.AkkaCluster);

Using BootstrapSetup and Your ActorSystem

If you're not using Akka.Hosting in your Akka.NET applications you can still install Phobos with a minimal amount of code using the BootstrapSetup class:

var hocon = ConfigurationFactory.ParseString(File.ReadAllText("Config/app.conf"));

var tracer = Sdk.CreateTracerProviderBuilder()
    .AddSource(ActorTracing.PhobosActivitySourceName) // subscribe to built-in Phobos events
    .Build();
var metrics = Sdk.CreateMeterProviderBuilder()
    .AddMeter(ActorTracing.PhobosActivitySourceName)
    .Build();

// create an ActorSystemSetup class used to configure the ActorSystem
var phobosSetup = BootstrapSetup.Create()
        .WithConfig(hocon)
        .WithActorRefProvider(PhobosProviderSelection.Cluster);

// start ActorSystem
var actorSystem = ActorSystem.Create("myCluster", phobosSetup);

With that all of your Akka.NET actors using Phobos will automatically record monitoring and tracing events which will subsequently be published to your configured tracing and monitoring systems.

Tip

Read more about configuring and installing Phobos here.

Phobos at Scale

Phobos is specifically designed for use within highly available, large scale software systems built on top of Akka.NET and Akka.Cluster.

Phobos' tracing technology works transparently across Akka.Remote connections and Phobos' monitoring infrastructure automatically gathers metrics on both a per-node and cluster-wide basis. To leverage Phobos in a large Akka.NET Cluster all you have to do is instrument each node with OpenTelemetry exporters, which will automatically consume the metrics and traces produced by Phobos and subsequently export them to the APM platform of your choice.

How to Get Phobos

To get Phobos you will need to buy a Phobos license, which costs $4,000 per year per organization. This will allow you and your team to use Phobos throughout your organization's projects.