Configuring Phobos 1.x

Phobos 1 and later uses a split configuration setup:

For mandatory settings these are passed into the ActorSystem.Create method programmatically as part of a PhobosSetup class in C# or F#;
For optional / detailed settings - these are overridable via HOCON configuration. Any setting for configuring individual tracing / monitoring settings that can be configured inside akka.actor.deployment for individual actors can also bet set on the fluent interface of the Akka.NET Props class.

Note

If you are ready to jump in and see an example of Phobos being configured inside a ASP.NET Core + Akka.NET application, please see the Phobos QuickStart Tutorial.

Let's take a look at how to configure Phobos completely using both of these tools.

`PhobosSetup` and `PhobosConfigBuilder`

PhobosSetup takes advantage of the new Setup and ActorSystemSetup types introduced into Akka.NET in v1.4. These types exist to allow a degree of programmatic configuration to be specified prior to creating an ActorSystem. This type of programmatic setup is ideal for Phobos, where we want to externally configure application observability endpoints (metrics, logging, and tracing) once and re-use that configuration across Akka.NET, ASP.NET, SignalR, RabbitMQ, and everywhere else.

In order to create a PhobosSetup we need to initialize both of the following:

An OpenTracing ITracer - you can see a list of some of the .NET tracing libraries that support OpenTracing here and
An App.Metrics IMetricsRoot, which will be used to pump metrics computed via Phobos out to one of our preferred monitoring solutions. You can see a full list of metrics reporting systems supported by App.Metrics here.

Here's a small example of what a complete example of a fully configured PhobosSetup looks like:

var hocon = ConfigurationFactory.ParseString(File.ReadAllText("Config/app.conf"));

OpenTracing.ITracer tracer = new MockTracer();
App.Metrics.IMetricsRoot metrics = new MetricsBuilder().Configuration.Configure(o =>
    {
        o.GlobalTags.Add("host", Dns.GetHostName());
        o.DefaultContextLabel = "akka.net";
        o.Enabled = true;
        o.ReportingEnabled = true;
    }).Report.ToConsole().Build();

var phobosConfigBuilder = new PhobosConfigBuilder()
    .WithMetrics(m => m.SetMetricsRoot(metrics))
    .WithTracing(t => t.SetTracer(tracer));

// create an ActorSystemSetup class used to configure the ActorSystem
var phobosSetup = PhobosSetup.Create(phobosConfigBuilder)
    .And(BootstrapSetup.Create()
        .WithConfig(hocon)
        .WithActorRefProvider(PhobosProviderSelection.Cluster));

// start ActorSystem
var actorSystem = ActorSystem.Create("myCluster", phobosSetup);

Breaking this down into smaller parts, the first step is to configure an ITracer and an IMetricsRoot

ITracer tracer = new MockTracer();
IMetricsRoot metrics = new MetricsBuilder().Configuration.Configure(opt =>
{
    opt.DefaultContextLabel = "Test";
}).Report.Using(new ConsoleMetricsReporter()).Build();;

Note

Typically the ITracer and IMetricsRoot get bound as required services when setting up dependency injection inside Startup.cs when you're using .NET Core's Generic Host or ASP.NET Core. For a full example of this, please see how PhobosSetup gets constructed in concert with Microsoft.Extensions.DependencyIjnection in the Phobos QuickStart Tutorial.

The ITracer and the IMetricsRoot then need to be passed into an PhobosConfigBuilder, which binds the tracer and metrics to Phobos:

var phobosConfigBuilder = new PhobosConfigBuilder()
    .WithMetrics(m => m.SetMetricsRoot(metrics))
    .WithTracing(t => t.SetTracer(tracer));

From there we need to pass the PhobosConfigBuilder object into the PhobosSetup, which we will merge with an Akka.NET BootstrapSetup which accepts the HOCON configuration we would normally pass into an ActorSystem.Create method:

var bootstrap = BootstrapSetup.Create()
    .WithActorRefProvider(PhobosProviderSelection.Cluster) // configures Phobos for Akka.Cluster
    .WithConfig(HoconConfig);

// need this to launch our ActorSystem
ActorSystemSetup phobosSetup = PhobosSetup.Create(phobosConfigBuilder).And(bootstrap);
ActorSystem actorSystem = ActorSystem.Create("MySys", phobosSetup);

Warning

If the BoostrapSetup is omitted then Phobos will not run.

This is all we need to launch an Akka.NET ActorSystem that is configured using Phobos.

Enabling Phobos for Akka, Akka.Remote, and Akka.Cluster

Akka

To enable Phobos for stand-alone Akka.NET, i.e. in an environment where you're not using Akka.Remote or Akka.Cluster, then we need to change the call we make to the BootstrapSetup.WithActorRefProvider method:

 var bootstrap = BootstrapSetup.Create()
    .WithActorRefProvider(PhobosProviderSelection.Local) // configures Phobos for Akka only
    .WithConfig(HoconConfig);

The PhobosProviderSelection class includes toggles for Akka, Akka.Remote, and Akka.Cluster. We need to make sure we include the correct one for each environment.

Akka.Remote

To enable Phobos for Akka.Remote only, meaning without Akka.Cluster, we need to use the following code:

 var bootstrap = BootstrapSetup.Create()
    .WithActorRefProvider(PhobosProviderSelection.Remote) // configures Phobos for Akka.Remote
    .WithConfig(HoconConfig);

Akka.Cluster

To enable Phobos for Akka.Cluster only, meaning without Akka.Cluster, we need to use the following code:

 var bootstrap = BootstrapSetup.Create()
    .WithActorRefProvider(PhobosProviderSelection.Cluster) // configures Phobos for Akka.Cluster
    .WithConfig(HoconConfig);

Filtering Traced Messages with `PhobosConfigBuilder`

One of the other things you can do with the PhobosConfigBuilder is customize which messages can start new traces and which ones cannot. This can be really helpful in reducing the amount of tracing noise produced and memory traffic / bandwidth consumed.

All you need to do to filter trace messages is to pass in a handful of predicate functions to include and exclude messages.

Tip

By default all messages automatically initiate a new trace in Phobos if there isn't a current trace in progress.

If there's a trace already in progress all messages are added to it, even ones that are filtered out - the filtering controls are really there to prevent new traces from beginning as a result of messages that are considered to be "noise" rather than signal.

var phobosConfigBuilder = new PhobosConfigBuilder()
    .WithMetrics(m => m.SetMetricsRoot(metrics))
    .WithTracing(t => t.SetTracer(tracer)
        .AddIncludeMessageFilter<string>()
        .AddExcludeMessageFilter<int>()
        .AddIncludeMessageFilter(f => f is IActorRef actorRef && !actorRef.IsNobody()));

This configuration will enable Phobos to include string messages, exclude int messages, and exclude any messages that implement the IActorRef interface where IActorRef.IsNobody() is true.

var actor = actorSystem.ActorOf(act => act.ReceiveAny((obj, ctx) =>
{
    ctx.Sender.Tell(obj);
}), "echo");

// should be traced
var response1 = await actor.Ask<string>("foo", TimeSpan.FromMilliseconds(5000));

// should NOT be traced
var response2 = await actor.Ask<int>(1, TimeSpan.FromMilliseconds(5000));

// check both results
var finishedSpans = tracer.As<MockTracer>().FinishedSpans();
finishedSpans.Should().Contain(span => span.OperationName.Contains("akka.msg.recv String"));
finishedSpans.Should().NotContain(span => span.OperationName.Contains("akka.msg.recv int"));

Defining Filter Predicates

The PhobosConfigBuilder allows you to add an arbitrary number of Predicate<object> expressions to your trace filtering specification via the AddMessageFilter method:

.AddMessageFilter(f => f is IActorRef actorRef && !actorRef.IsNobody()));

When the Predicate<object> returns true, the message will be included in the trace;
When the Predicate<object> returns false the message will be excluded from the trace; and
If any of the configured filters return false for a given message, the message will be excluded from the trace.

Since Phobos' tracing filters can filter messages based on both their types and their content, Phobos doesn't cache any of the outputs from a tracing filter - all filters are computed for every message on actors that have tracing enabled.

Phobos HOCON Configuration

Some settings cannot be easily expressed via a stand-alone builder interface and there are also legacy settings from previous versions of Phobos that are still incorporated via HOCON.

Here's the full set of stand-alone HOCON settings that are specific to Akka.NET.

Tip

The phobos HOCON section is a stand-alone section; meaning that it does not get included inside the akka HOCON section that Akka.NET developers typically modify.

##################################
# Phobos Reference Config File   #
##################################

phobos{
	tracing{
		# Setting used to trigger tracing for all built-in /system actors by default.
		# When set to "on," all /system actors will automatically be outfitted with tracing
		# instrumentation.
		trace-all-system-actors = off

		# Setting used to trigger tracing for all /user actors by default.
		# When set to "on," all /user actors will automatically be outfitted with tracing
		# instrumentation.
		trace-all-user-actors = on
		
		# Setting is used to trace actor PreStart / PostRestart / PostStop / PreRestart activity prior to processing
		# messages. This setting is particularly important when attempting to trace Akka.Persistence
		# Recovery.
		trace-actor-lifecycle = on
		
		# Traces Akka.Persistence Persist and Recovery operations.
		# NOTE: Akka.Persistence recovery tracing also depends upon `trace-actor-lifecycle = on`.
		# However, if `trace-actor-lifecycle = on` but `trace-akka-persistence = on` means you can
		# measure Persist operations but not recoveries.
		trace-akka-persistence = on

		# Debugging settings used for testing the configuration of Phobos.Tracing
		debug{
			# When enabled: will log incoming tracing information
			# on all actors capable of receiving trace events.
			log-traces = off
		}

		# Setting used to instruct Phobos whether or not it should automatically append
		# all captured log messages to the active ISpan during which they were produced.
		#
		# Turning this setting off may help improve performance slightly in some instances,
		# but at the cost of missing useful debugging and diagnostic data.
		# 
		# If you want to throttle down the amount of log messages being appended to outgoing
		# spans, consider simply changing the `akka.loglevel` setting or others inside core Akka.NET.
		append-logs-to-trace = on
	}

	monitoring{
		# Setting used to trigger monitoring for all built-in /system actors by default.
		# When set to "on," all /system actors will automatically be outfitted with monitoring
		# instrumentation.
		monitor-all-system-actors = off

		# Setting used to trigger monitoring for all /user actors by default.
		# When set to "on," all /user actors will automatically be outfitted with monitoring
		# instrumentation.
		monitor-all-user-actors = on

		# Toggles monitoring of mailbox queue depth for any actors who have monitoring enabled
		monitor-mailbox-depth = on

		# When enabled, records all of the default events that pass through the eventstream
		# (log events, dead letters, and unhandled messages) and writes that out to a Global
		# counter for the ActorSystem as a whole.
		monitor-eventstream = on
	}
}

Configuring Individual Actors via HOCON and `Props`

Should you need it, Phobos offers the ability to configure individual actors or groups of actors with specific tracing / monitoring settings.

Tip

Typically, configuring specific actors is most often used in contexts where Phobos users want to limit the amount of traces produced by specific areas of the actor hierarchy for "signal vs. noise" control or to reduce the total observability overhead of the system. For more tips on performance-optimizing your Phobos installation, please see "Performance Impact and Best Practices."

HOCON

Here's the full set of HOCON values that can be used inside the akka.actor.deployment HOCON namespace in Akka.NET - Phobos will look for these values and override the ActorSystem-wide settings for these groups of actors:

######################################################
# Phobos Default Actor-Specific Deployment Options   #
######################################################

akka.actor.deployment.default.phobos{
	# When set to "on" this setting ensures that all children, grandchildren
	# and so on of the actor deployed with this configuration receive the exact
	# same settings as this actor unless explicitly overridden by their own
	# deployment configuration.
	propagate-settings-to-children = off

	tracing{
		# Toggles the entire tracing functionality of Phobos.Tracing on or off for this actor.
		# If set to 'off' then this actor will not record any traces. However, traces can still
		# be propagated through this actor to others. 
		enabled = on
	}

	monitoring{
		# Toggles the entire monitoring functionality of Phobos.Monitoring on or off for this actor.
		# If set to 'off' then this actor will not record any metrics.
		enabled = on
		
		# Toggles mailbox depth monitoring on or off for this actor.
		monitor-mailbox-depth = off
	}
}

What do each of these settings do?

phobos.propagate-settings-to-children - when this setting is set to on, any settings used to configure this actor will also be applied recursively down to children and their children and so on. This is designed to help toggle tracing or monitoring off and on throughout an entire area of the actor hierarchy.
phobos.tracing.enabled - when this is set to on tracing will be enabled for this actor. When it's off tracing will be disabled for all messages processed by this actor.
phobos.monitoring.enabled - when this is set to on monitoring will be enabled for this actor. When it's off monitoring will be disabled for all messages processed by this actor.
phobos.monitoring.monitor-mailbox-depth - this value is set to off by default, as there is a small performance impact to measuring the mailbox depth of any particular actor. However, when this setting is enabled a new gauge will be recorded that includes the point-in-time queue length for the mailbox length of all actors of this particular type.

You can apply this HOCON to an actor inside the akka.actor.deployment namespace like this:

akka.actor{
	provider = "Phobos.Actor.PhobosActorRefProvider, Phobos.Actor"

	deployment{
		# use custom sample rates and filtering for high-throughput actors
		/coordinator{
			phobos{
				# ensure that all descendants of /user/coordinator have these settings
				propagate-settings-to-children = on

				tracing{
					enabled = on
				}

				monitoring{
					enabled = on
					monitor-mailbox-depth = off # don't monitor mailboxes
				}
			}
		}
	}
}

When an actor at location /user/coordinator gets deployed, it will utilize all of the settings found in this HOCON file.

`Props`

In addition to configuring Phobos through akka.actor.deployment you can also configure each actor's monitoring and tracing settings explicitly via some extensions Phobos adds to the Props class in Akka.NET.

/// <summary>
///     Creates a new <see cref="PhobosActorSettings" /> instance.
/// </summary>
/// <param name="trace">Toggles tracing on or off for this actor.</param>
/// <param name="monitor">Toggles monitoring on or off for this actor.</param>
/// <param name="propagateSettingsToChildren">When set to true, proagates the current monitoring settings down to children.</param>
/// <param name="customFiltering">When set to true, indicates that this actor is running with custom filtering.</param>
/// <param name="phobosMonitorSampleRate">[DEPRECATED] The sample rate used to instrument monitoring inside of this actor.</param>
/// <param name="phobosNotInstalled">
///     Optional. Use to indicate when Akka.Remote is enabled that the actors on the other
///     side of the network don't have Phobos installed.
/// </param>
/// <param name="mailboxDepth">Optional. When enabled turns on monitoring of the mailbox depth of this actor.</param>
/// <param name="filterSettings">Determines how this actor filters incoming messages for tracing purposes.</param>
public PhobosActorSettings(bool trace, bool monitor, TraceFilteringSettings filterSettings = null,
    bool customFiltering = false,
    double phobosMonitorSampleRate = MonitoringSettings.DefaultSampleRate,
    bool phobosNotInstalled = false, bool mailboxDepth = false,
    bool propagateSettingsToChildren = false)
{
    Trace = trace;
    Monitor = monitor;
    PropagateSettingsToChildren = propagateSettingsToChildren;
    PhobosNotInstalled = phobosNotInstalled;
    MonitorSampleRate = phobosMonitorSampleRate;
    MonitorMailbox = mailboxDepth;
    TraceFiltering = filterSettings ?? TraceFilteringSettings.ExplicitFiltering;
    HasCustomTraceFilteringSettings = customFiltering;
}

Warning

Sample rate is not longer supported internally by Phobos. This is now a setting you must configure through your ITracer or IMetricsRoot implementation. Phobos doesn't perform any type of sampling independent of your configured tracers and metric reporters.

So to apply the PhobosActorSettings to a specific group of actors, we just need to call the WithInstrumentation extension method on the Props class.

For instance, here's an example of how to disable monitoring this way:


// get the current actor settings from Phobos and disable tracing
var defaultSettings = PhobosSettings.For(Sys).ActorSettings.WithMonitoring(false);

// pass in the Phobos settings to this actor
var phobosProps = Props.Create(() => new EchoActor()).WithInstrumentation(defaultSettings);
var myActor = Sys.ActorOf(phobosProps);

// won't generate any metrics
myActor.Tell("foo");
ExpectMsg("foo");
//ExpectNoMetricsFrom(myActor, TimeSpan.FromMilliseconds(500));

And an example of how to disable tracing:


// get the current actor settings from Phobos and disable tracing
var defaultSettings = PhobosSettings.For(Sys).ActorSettings.WithTracing(false);

// pass in the Phobos settings to this actor
var phobosProps = Props.Create(() => new EchoActor()).WithInstrumentation(defaultSettings);
var myActor = Sys.ActorOf(phobosProps);

// won't generate a trace
myActor.Tell("foo");
ExpectMsg("foo");
ExpectNoTracesFrom(myActor, TimeSpan.FromMilliseconds(500));

Any of the settings you can specify via HOCON can also be specified by changing the instrumentation on Props for any and all of your actors.

Configuring Phobos 1.x

Note

PhobosSetup and PhobosConfigBuilder

Note

Warning

Enabling Phobos for Akka, Akka.Remote, and Akka.Cluster

Akka

Akka.Remote

Akka.Cluster

Filtering Traced Messages with PhobosConfigBuilder

Tip

Defining Filter Predicates

Phobos HOCON Configuration

Tip

Configuring Individual Actors via HOCON and Props

Tip

HOCON

Props

Warning

`PhobosSetup` and `PhobosConfigBuilder`

Filtering Traced Messages with `PhobosConfigBuilder`

Configuring Individual Actors via HOCON and `Props`

`Props`