Configuring Phobos 2.0

Note

If you are still using Phobos 1.x and need configuration instructions, you can find Phobos 1.x configuration instructions here.

There are two approaches to installing Phobos 2.0.

Use Phobos.Hosting

Phobos.Hosting is built on top of Akka.Hosting and makes the entire installation process very simple:

private static async Task Main(string[] args)
{
    var resource = ResourceBuilder.CreateDefault()
        .AddService(Assembly.GetEntryAssembly().GetName().Name, serviceInstanceId: $"{Dns.GetHostName()}");

    var host = WebHost.CreateDefaultBuilder(args)
        .ConfigureServices((hostContext, services) =>
        {
            services.AddControllers();
            services.AddOpenTelemetryTracing(builder =>
            {
                builder
                    .SetResourceBuilder(resource)
                    .AddPhobosInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddAspNetCoreInstrumentation()
                    .AddJaegerExporter(opt => { });
            });

            services.AddOpenTelemetryMetrics(builder =>
            {
                builder
                    .SetResourceBuilder(resource)
                    .AddPhobosInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddAspNetCoreInstrumentation()
                    .AddPrometheusExporter(opt => { });
            });
            services.AddAkka("ClusterSys", (builder, provider) =>
            {
                builder.WithRemoting("localhost", 8882);
                builder.WithClustering(new ClusterOptions()
                {
                    SeedNodes = new[]
                    {
                        "akka.tcp://[email protected]:8882"
                    },
                    Roles = new[] { "app" }
                });

                // ENABLE PHOBOS
                builder.WithPhobos(AkkaRunMode.AkkaCluster);

                // turn on Petabridge.Cmd
                builder.StartActors((system, registry) =>
                {
                    var echoActor = system.ActorOf(Props.Create(() => new EchoActor()), "echo");
                    registry.TryRegister<EchoActor>(echoActor);
                });
            });
        })
        .Configure(app =>
        {
            app.UseRouting();
            app.UseOpenTelemetryPrometheusScrapingEndpoint();
            app.UseEndpoints(endpoints =>
            {
                endpoints.MapGet("/",
                    async context => await context.RequestServices.GetRequiredService<ActorRegistry>()
                        .Get<EchoActor>().Ask<string>(context.TraceIdentifier, TimeSpan.FromSeconds(3)));
                endpoints.MapControllers();
            });
        })
        .Build();

    await host.RunAsync();
}

The bulk of this code is:

OpenTelemetry configuration via the OpenTelemetry.Extensions.Hosting NuGet package's extension methods. This tells our app which OTel-accessible data sources to gather information from and where to export that data. In this case we're going to export trace data to Jaeger and metric data to Promethus;
ASP.NET configuration - boilerplate for setting up some basic HTTP routes; and
Lastly, Akka.Hosting for configuring our ActorSystem, starting our actors, and launching Phobos!

The key line that enables Phobos are these two:

services.AddOpenTelemetry()
    .WithTracing(builder =>
    {
        builder
            .SetResourceBuilder(resource)
            .AddPhobosInstrumentation() // subscribes to the Phobos OTel source (same for metrics)
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddJaegerExporter(opt => { });
    });

and

 // enable Phobos inside Akka.NET
builder.WithPhobos(AkkaRunMode.AkkaCluster);

And that's all you need to start automatically capturing data using OpenTelemetry inside your Akka.NET applications - no HOCON, no more Setup classes, and no passing references around. Clean, simple, and fast.

Installation Procedure

Note

Not familiar with Akka.Hosting? See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video for a walkthrough of the library and how it can save you a tremendous amount of time and trouble.

To install Phobos in this fashion:

Port your Akka.NET initialization to use Akka.Hosting if you haven't already;
Install Phobos.Hosting;
Install the appropriate OpenTelemetry.Instrumentation.x and OpenTelemetry.Exporter.x NuGet packages;
Configure them using Microsoft.Extensions.DependencyInjection as shown above.

Classic Phobos Installation

If you're not using Akka.Hosting in your Akka.NET applications you can still install Phobos with a minimal amount of code using the BootstrapSetup class:

var hocon = ConfigurationFactory.ParseString(File.ReadAllText("Config/app.conf"));

var tracer = Sdk.CreateTracerProviderBuilder()
    .AddSource(ActorTracing.PhobosActivitySourceName) // subscribe to built-in Phobos events
    .Build();
var metrics = Sdk.CreateMeterProviderBuilder()
    .AddMeter(ActorTracing.PhobosActivitySourceName)
    .Build();

// create an ActorSystemSetup class used to configure the ActorSystem
var phobosSetup = BootstrapSetup.Create()
        .WithConfig(hocon)
        .WithActorRefProvider(PhobosProviderSelection.Cluster);

// start ActorSystem
var actorSystem = ActorSystem.Create("myCluster", phobosSetup);

To perform this installation all you have to do is:

Create your OpenTelemetry TracerProvider and MeterProvider while subscribing to the ActorTracing.PhobosActivitySourceName events produced by Phobos;
Create a new BootstrapSetup that passes in the appropriate PhobosProviderSelection a. ProviderSelection.Local for stand-alone Akka.NET; b. ProviderSelection.Remote for Akka.Remote; and c. ProviderSelection.Cluster for Akka.Cluster.

This will allow you to startup your ActorSystem with Phobos instrumentation installed from the get-go.

Additional Phobos HOCON Configuration

Some settings for classic Phobos configuration are done via HOCON rather than the BootstrapSetup or the PhobosSetup classes.

Note

Phobos.Hosting will let you specify all of this via the strongly typed builder interface. No HOCON required there.

Here's the full set of stand-alone HOCON settings that are specific to Akka.NET.

Tip

The phobos HOCON section is a stand-alone section; meaning that it does not get included inside the akka HOCON section that Akka.NET developers typically modify.

##################################
# Phobos Reference Config File   #
##################################

phobos{
	tracing{
		# Setting used to trigger tracing for all built-in /system actors by default.
		# When set to "on," all /system actors will automatically be outfitted with tracing
		# instrumentation.
		trace-all-system-actors = off

		# Setting used to trigger tracing for all /user actors by default.
		# When set to "on," all /user actors will automatically be outfitted with tracing
		# instrumentation.
		trace-all-user-actors = on
		
		# Setting is used to trace actor PreStart / PostRestart / PostStop / PreRestart activity prior to processing
		# messages. This setting is particularly important when attempting to trace Akka.Persistence
		# Recovery.
		trace-actor-lifecycle = on
		
		# Traces Akka.Persistence Persist and Recovery operations.
		# NOTE: Akka.Persistence recovery tracing also depends upon `trace-actor-lifecycle = on`.
		# However, if `trace-actor-lifecycle = on` but `trace-akka-persistence = on` means you can
		# measure Persist operations but not recoveries.
		trace-akka-persistence = on
		
		# Enables tracing of `Ask<T>` operations. When disabled, Phobos will not trace Ask<T> operations.
		trace-ask = on
		
		# Creates message-processing spans automatically when tracing is enabled.
		# When this setting is disabled users will be able to manually create traces
		# from SpanContexts received during tracing and traces will still be propagated.
		# REMARKS: this setting should really only be used when attempting to suppress
		# large amounts of noise from the tracing system by manually controlling when
		# traces are created.
		create-trace-upon-receive = on
		
		# In Phobos 2.0 we introduced latency tracking which includes logging two discrete events
		# on all automatically created Phobos spans:
		# - "waiting" - created when the message is first IActorRef.Tell'd
 		# - "message" - created when the message is first processed by the actor and
 		#               contains the message's content in stringified form.
 		# Turning this setting to "off" disables the productions of these events on all `akka.msg.recv`
 		# spans created by Phobos.
		log-message-events = on
		
		
		# Debugging settings used for testing the configuration of Phobos.Tracing
		debug{
			# When enabled: will log incoming tracing information
			# on all actors capable of receiving trace events.
			log-traces = off
		}

		# Setting used to instruct Phobos whether or not it should automatically append
		# all captured log messages to the active ISpan during which they were produced.
		#
		# Turning this setting off may help improve performance slightly in some instances,
		# but at the cost of missing useful debugging and diagnostic data.
		# 
		# If you want to throttle down the amount of log messages being appended to outgoing
		# spans, consider simply changing the `akka.loglevel` setting or others inside core Akka.NET.
		append-logs-to-trace = on
	}

	monitoring{
		# Setting used to trigger monitoring for all built-in /system actors by default.
		# When set to "on," all /system actors will automatically be outfitted with monitoring
		# instrumentation.
		monitor-all-system-actors = off

		# Setting used to trigger monitoring for all /user actors by default.
		# When set to "on," all /user actors will automatically be outfitted with monitoring
		# instrumentation.
		monitor-all-user-actors = on

		# Toggles monitoring of mailbox queue depth for any actors who have monitoring enabled
		monitor-mailbox-depth = on

		# When enabled, records all of the default events that pass through the eventstream
		# (log events, dead letters, and unhandled messages) and writes that out to a Global
		# counter for the ActorSystem as a whole.
		monitor-eventstream = on
	}
}

akka{
    # enable Akka.NET built-in telemetry by default in Phobos
    actor.telemetry.enabled = on
}

Configuring Trace Filtering

See "Filtering Tracing Data with Phobos" for details on how to do this.

Configuring Individual Actors via HOCON and `Props`

Should you need it, Phobos offers the ability to configure individual actors or groups of actors with specific tracing / monitoring settings.

Tip

Typically, configuring specific actors is most often used in contexts where Phobos users want to limit the amount of traces produced by specific areas of the actor hierarchy for "signal vs. noise" control or to reduce the total observability overhead of the system. For more tips on performance-optimizing your Phobos installation, please see "Performance Impact and Best Practices."

HOCON

Here's the full set of HOCON values that can be used inside the akka.actor.deployment HOCON namespace in Akka.NET - Phobos will look for these values and override the ActorSystem-wide settings for these groups of actors:

######################################################
# Phobos Default Actor-Specific Deployment Options   #
######################################################

akka.actor.deployment.default.phobos{
	# When set to "on" this setting ensures that all children, grandchildren
	# and so on of the actor deployed with this configuration receive the exact
	# same settings as this actor unless explicitly overridden by their own
	# deployment configuration.
	propagate-settings-to-children = off

	tracing{
		# Toggles the entire tracing functionality of Phobos.Tracing on or off for this actor.
		# If set to 'off' then this actor will not record any traces. However, traces can still
		# be propagated through this actor to others. 
		enabled = on
	}

	monitoring{
		# Toggles the entire monitoring functionality of Phobos.Monitoring on or off for this actor.
		# If set to 'off' then this actor will not record any metrics.
		enabled = on
		
		# Toggles mailbox depth monitoring on or off for this actor.
		monitor-mailbox-depth = off
	}
}

What do each of these settings do?

phobos.propagate-settings-to-children - when this setting is set to on, any settings used to configure this actor will also be applied recursively down to children and their children and so on. This is designed to help toggle tracing or monitoring off and on throughout an entire area of the actor hierarchy.
phobos.tracing.enabled - when this is set to on tracing will be enabled for this actor. When it's off tracing will be disabled for all messages processed by this actor.
phobos.monitoring.enabled - when this is set to on monitoring will be enabled for this actor. When it's off monitoring will be disabled for all messages processed by this actor.
phobos.monitoring.monitor-mailbox-depth - this value is set to off by default, as there is a small performance impact to measuring the mailbox depth of any pariticular actor. However, when this setting is enabled a new gauge will be recorded that includes the point-in-time queue length for the mailbox length of all actors of this particular type.

You can apply this HOCON to an actor inside the akka.actor.deployment namespace like this:

akka.actor{
	provider = "Phobos.Actor.PhobosActorRefProvider, Phobos.Actor"

	deployment{
		# use custom sample rates and filtering for high-throughput actors
		/coordinator{
			phobos{
				# ensure that all descendants of /user/coordinator have these settings
				propagate-settings-to-children = on

				tracing{
					enabled = on
				}

				monitoring{
					enabled = on
					monitor-mailbox-depth = off # don't monitor mailboxes
				}
			}
		}
	}
}

When an actor at location /user/coordinator gets deployed, it will utilize all of the settings found in this HOCON file.

`Props`

In addition to configuring Phobos through akka.actor.deployment you can also configure each actor's monitoring and tracing settings explicitly via some extensions Phobos adds to the Props class in Akka.NET.

/// <summary>
///     Creates a new <see cref="PhobosActorSettings" /> instance.
/// </summary>
/// <param name="trace">Toggles tracing on or off for this actor.</param>
/// <param name="monitor">Toggles monitoring on or off for this actor.</param>
/// <param name="filterSettings">Determines how this actor filters incoming messages for tracing purposes.</param>
/// <param name="customFiltering">When set to true, indicates that this actor is running with custom filtering.</param>
/// <param name="phobosNotInstalled">
///     Optional. Use to indicate when Akka.Remote is enabled that the actors on the other
///     side of the network don't have Phobos installed.
/// </param>
/// <param name="mailboxDepth">Optional. When enabled turns on monitoring of the mailbox depth of this actor.</param>
/// <param name="propagateSettingsToChildren">
///     When set to true, propagates the current monitoring settings down to
///     children.
/// </param>
/// <param name="traceActorLifecycle">When set to true, allow actors to trace their PreStart / PostRestart activities.</param>
/// <param name="createTraceUponReceive">When set to true, actors will create traces automatically upon processing messages.</param>
/// <param name="logMessageEvents">When set to true and when <see cref="CreateTraceUponReceive"/> is true, will automatically append `waiting` and `message`
/// events to all automatically created spans during actor message processing.</param>
public PhobosActorSettings(bool trace, bool monitor, TraceFilteringSettings filterSettings = null,
    bool customFiltering = false,
    bool phobosNotInstalled = false, bool mailboxDepth = false,
    bool propagateSettingsToChildren = false,
    bool traceActorLifecycle = true, bool createTraceUponReceive = true, bool logMessageEvents = true)
{
    Trace = trace;
    Monitor = monitor;
    LogMessageEvents = logMessageEvents;
    PropagateSettingsToChildren = propagateSettingsToChildren;
    PhobosNotInstalled = phobosNotInstalled;
    MonitorMailbox = mailboxDepth;
    TraceFiltering = filterSettings ?? TraceFilteringSettings.ExplicitFiltering;
    HasCustomTraceFilteringSettings = customFiltering;
    TraceActorLifecycle = traceActorLifecycle;
    CreateTraceUponReceive = createTraceUponReceive;
}

So to apply the PhobosActorSettings to a specific group of actors, we just need to call the WithInstrumentation extension method on the Props class.

For instance, here's an example of how to disable monitoring this way:


// get the current actor settings from Phobos and disable tracing
var defaultSettings = PhobosSettings.For(Sys).ActorSettings.WithMonitoring(false);

// pass in the Phobos settings to this actor
var phobosProps = Props.Create(() => new EchoActor()).WithInstrumentation(defaultSettings);
var myActor = Sys.ActorOf(phobosProps);

// won't generate any metrics
myActor.Tell("foo");
ExpectMsg("foo");
//ExpectNoMetricsFrom(myActor, TimeSpan.FromMilliseconds(500));

And an example of how to disable tracing:


// get the current actor settings from Phobos and disable tracing
var defaultSettings = PhobosSettings.For(Sys).ActorSettings.WithTracing(false);

// pass in the Phobos settings to this actor
var phobosProps = Props.Create(() => new EchoActor()).WithInstrumentation(defaultSettings);
var myActor = Sys.ActorOf(phobosProps);

// won't generate a trace
myActor.Tell("foo");
ExpectMsg("foo");
ExpectNoTracesFrom(myActor, TimeSpan.FromMilliseconds(500));

Any of the settings you can specify via HOCON can also be specified by changing the instrumentation on Props for any and all of your actors.

Configuring Phobos 2.0

Note

Use Phobos.Hosting

Installation Procedure

Note

Classic Phobos Installation

Additional Phobos HOCON Configuration

Note

Tip

Configuring Trace Filtering

Configuring Individual Actors via HOCON and Props

Tip

HOCON

Props

Configuring Individual Actors via HOCON and `Props`

`Props`