What's New in Phobos 2.x?

Phobos 2.x is a major upgrade from Phobos 1.x that introduces the following important improvements and changes:

  1. Migrated from App.Metrics and OpenTracing to OpenTelemetry - the future APM platform for all of .NET;
  2. End-to-end message processing latency tracking in both Phobos metrics and traces;
  3. Re-engineered the trace filtering system to be much easier to use and effective at reducing noise; and
  4. Integrated with Akka.Hosting, a new HOCONless way of configuring, running, and integrating Akka.NET with the other parts of your .NET applications. See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here.

These changes all represent a major step forward for Phobos and Akka.NET telemetry.

OpenTelemetry Support

We have written a great deal about bringing OpenTelemetry to Akka.NET and with Phobos 2.0 all users can take OTel into Akka.NET production with confidence.

Why OpenTelemetry?

OpenTelemetry is the official solution from Microsoft for all .NET instrumentation going forward - it will be intregated into the base class libraries, ASP.NET, SignalR, HttpClient, Azure SDKs, Entity Framework, and much more. All of the major APM vendors (Elastic, Jaeger, Prometheus, DataDog, New Relic, etc) are all onboard as OTel radically reduces their total cost of ownership for telemetry installations.

By working with OpenTelemetry in Phobos 2.0 we can ensure that all of your activity that happens outside your ActorSystem is seamlessly correlated with everything that happens inside your ActorSystem.

This will also radically expand the set of supported APM integrations with Phobos and Akka.NET as the OpenTelemetry ecosystem is vast and constantly expanding.

Important

Worth noting that the OpenTelemetry binaries from Microsoft are not out of Release Candidate status themselves. This is because the OpenTelemetry Metrics SDK is still being actively worked on but this is all expected to be relatively minor changes which should be finalized by the release of .NET 7 in November, 2022.

Configuration

To see how to configure your Akka.NET application to use OpenTelemetry, please see "Configuring Phobos 2.x".

End to End Message Processing Latency

One major change we introduced in Phobos 2.0 is how we measure per-message latency - specifically, we start measuring latency from the moment IActorRef.Tell is called, thus we can now measure how much time a message spent in-flight over Akka.Remote or waiting inside a busy actor's mailbox.

Previously, in Phobos 1.x we only measured latency from the point of view of the actor processing the message - we still measure that now too but it's treated as a separate discrete event on the TelemetrySpan / Activity that we record in Phobos.

In addition to tracking end-to-end latency during tracing we also capture this data inside Phobos' metrics and this will appear in some of the new built-in Phobos dashboards we have shipped, such as the Phobos + Prometheus 2.x Latency Tracking Dashboard

Radically Improved Trace Filtering

Trace filtering is now easier and more performant than ever using the new ITraceFilter construct available in Phobos 2.x. See "Filtering Tracing Data with Phobos" for details.

Akka.Hosting Integration and Simplified Configuration

Phobos can now be fully configured without any HOCON or mangling of Setup classes using the brand new Phobos.Hosting NuGet package, built on top of Akka.Hosting.

Tip

Unfamiliar with Akka.Hosting? See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here.

Here is an example of what an entire end-to-end Akka.NET program with Phobos.Hosting looks like:

private static async Task Main(string[] args)
{
	var resource = ResourceBuilder.CreateDefault()
		.AddService(Assembly.GetEntryAssembly().GetName().Name, serviceInstanceId: $"{Dns.GetHostName()}");

	var host = WebHost.CreateDefaultBuilder(args)
		.ConfigureServices((hostContext, services) =>
		{
			services.AddControllers();
			services.AddOpenTelemetry()
				.WithTracing(builder =>
				{
					builder
						.SetResourceBuilder(resource)
						.AddPhobosInstrumentation()
						.AddHttpClientInstrumentation()
						.AddAspNetCoreInstrumentation()
						.AddOtlpExporter();
				})
				.WithMetrics(builder =>
				{
					builder
						.SetResourceBuilder(resource)
						.AddPhobosInstrumentation()
						.AddHttpClientInstrumentation()
						.AddAspNetCoreInstrumentation()
						.AddPrometheusExporter(opt => { });
				});
			services.AddAkka("ClusterSys", (builder, provider) =>
			{
				builder.WithRemoting("localhost", 8882);
				builder.WithClustering(new ClusterOptions()
				{
					SeedNodes = new[]
					{
						"akka.tcp://ClusterSys@localhost:8882"
					},
					Roles = new[] { "app" }
				});

				// ENABLE PHOBOS
				builder.WithPhobos(AkkaRunMode.AkkaCluster);

				// turn on Petabridge.Cmd
				builder.StartActors((system, registry) =>
				{
					var echoActor = system.ActorOf(Props.Create(() => new EchoActor()), "echo");
					registry.TryRegister<EchoActor>(echoActor);
				});
			});
		})
		.Configure(app =>
		{
			app.UseRouting();
			app.UseOpenTelemetryPrometheusScrapingEndpoint();
			app.UseEndpoints(endpoints =>
			{
				endpoints.MapGet("/",
					async context => await context.RequestServices.GetRequiredService<ActorRegistry>()
						.Get<EchoActor>().Ask<string>(context.TraceIdentifier, TimeSpan.FromSeconds(3)));
				endpoints.MapControllers();
			});
		})
		.Build();

	await host.RunAsync();
}

See Configuring Phobos 2.x - Using Phobos.Hosting for full details.

Massively Improved Performance

Phobos 2.5 and higher, especially, are radically faster than their Phobos 1.x an 2.x predecessors. See Phobos Performance Impact on Akka.NET for details, and you can read about our improvement process in "How We Made Phobos 2.4's OpenTelemetry Usage 62% Faster."