Configuring Phobos 2.x

There are two approaches to installing Phobos 2.x.

Use Phobos.Hosting

Phobos.Hosting is built on top of Akka.Hosting and makes the entire installation process very simple:

private static async Task Main(string[] args)
{
    var resource = ResourceBuilder.CreateDefault()
        .AddService(Assembly.GetEntryAssembly().GetName().Name, serviceInstanceId: $"{Dns.GetHostName()}");

    var host = WebHost.CreateDefaultBuilder(args)
        .ConfigureServices((hostContext, services) =>
        {
            services.AddControllers();
            services.AddOpenTelemetry()
                .WithTracing(builder =>
                {
                    builder
                        .SetResourceBuilder(resource)
                        .AddPhobosInstrumentation()
                        .AddHttpClientInstrumentation()
                        .AddAspNetCoreInstrumentation()
                        .AddOtlpExporter();
                })
                .WithMetrics(builder =>
                {
                    builder
                        .SetResourceBuilder(resource)
                        .AddPhobosInstrumentation()
                        .AddHttpClientInstrumentation()
                        .AddAspNetCoreInstrumentation()
                        .AddPrometheusExporter(opt => { });
                });
            services.AddAkka("ClusterSys", (builder, provider) =>
            {
                builder.WithRemoting("localhost", 8882);
                builder.WithClustering(new ClusterOptions()
                {
                    SeedNodes = new[]
                    {
                        "akka.tcp://ClusterSys@localhost:8882"
                    },
                    Roles = new[] { "app" }
                });

                // ENABLE PHOBOS
                builder.WithPhobos(AkkaRunMode.AkkaCluster);

                // turn on Petabridge.Cmd
                builder.StartActors((system, registry) =>
                {
                    var echoActor = system.ActorOf(Props.Create(() => new EchoActor()), "echo");
                    registry.TryRegister<EchoActor>(echoActor);
                });
            });
        })
        .Configure(app =>
        {
            app.UseRouting();
            app.UseOpenTelemetryPrometheusScrapingEndpoint();
            app.UseEndpoints(endpoints =>
            {
                endpoints.MapGet("/",
                    async context => await context.RequestServices.GetRequiredService<ActorRegistry>()
                        .Get<EchoActor>().Ask<string>(context.TraceIdentifier, TimeSpan.FromSeconds(3)));
                endpoints.MapControllers();
            });
        })
        .Build();

    await host.RunAsync();
}

The bulk of this code is:

1. OpenTelemetry configuration via the OpenTelemetry.Extensions.Hosting NuGet package's extension methods. This tells our app which OTel-accessible data sources to gather information from and where to export that data. In this case we're going to export trace data to Jaeger and metric data to Prometheus;
2. ASP.NET configuration - boilerplate for setting up some basic HTTP routes; and
3. Lastly, Akka.Hosting for configuring our ActorSystem, starting our actors, and launching Phobos!

The key lines that enable Phobos are these two:

services.AddOpenTelemetry()
    .WithTracing(builder =>
    {
        builder
            .SetResourceBuilder(resource)
            .AddPhobosInstrumentation() // subscribes to the Phobos OTel source (same for metrics)
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddOtlpExporter();
    });

and

 // enable Phobos inside Akka.NET
builder.WithPhobos(AkkaRunMode.AkkaCluster);

And that's all you need to start automatically capturing data using OpenTelemetry inside your Akka.NET applications - no HOCON, no more Setup classes, and no passing references around. Clean, simple, and fast.

Installation Procedure

Note

Not familiar with Akka.Hosting? See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video for a walkthrough of the library and how it can save you a tremendous amount of time and trouble.

To install Phobos in this fashion:

1. Port your Akka.NET initialization to use Akka.Hosting if you haven't already;
2. Install Phobos.Hosting;
3. Install the appropriate OpenTelemetry.Instrumentation.x and OpenTelemetry.Exporter.x NuGet packages;
4. Configure them using Microsoft.Extensions.DependencyInjection as shown above.

Installing Phobos Without Akka.Hosting

If you're not using Akka.Hosting in your Akka.NET applications (for example, in console applications, Windows Services, or other non-hosted scenarios), you can still configure Phobos using either the modern PhobosSetup with PhobosConfigBuilder or traditional HOCON configuration.

Modern Approach: PhobosSetup with PhobosConfigBuilder

Our recommended approach for non-hosting scenarios uses PhobosConfigBuilder for programmatic configuration:

// Create Phobos configuration using PhobosConfigBuilder
var phobosConfig = new PhobosConfigBuilder()
    .WithTracing(tracing => tracing
        .SetTraceUserActors(true)
        .SetTraceSystemActors(false)
        .SetTraceActorLifecycle(true)
        .VerboseTracing(false) // Only trace explicitly included messages
        .SetTraceFilter(new CustomTraceFilter()))
    .WithMetrics(metrics => metrics
        .SetMonitorUserActors(true)
        .SetMonitorSystemActors(false)
        .SetMonitorMailboxDepth(true)
        .SetMonitorEventStream(true));

// Create ActorSystem with Phobos using ActorSystemSetup
var phobosSetup = PhobosSetup.Create(phobosConfig);
var bootstrap = BootstrapSetup.Create()
    // ## IMPORTANT: THIS IS WHAT INJECTS PHOBOS INTO YOUR ACTORS ##
    // See also: PhobosProviderSelection.Remote (for Akka.Remote)
    //      and  PhobosProviderSelection.Cluster (for Akka.Cluster)
    .WithActorRefProvider(PhobosProviderSelection.Local);

// combine setups
var actorSystemSetup = phobosSetup.And(bootstrap);

var system = ActorSystem.Create("PhobosNoHosting", actorSystemSetup);

Key steps:

1. Configure OpenTelemetry SDK with AddPhobosInstrumentation() for both tracing and metrics
2. Create PhobosConfigBuilder to configure tracing and monitoring settings
3. Use PhobosSetup.Create() with your configuration
4. Set WithActorRefProvider(PhobosProviderSelection.X) where X is Local/Remote/Cluster

Traditional Approach: HOCON Configuration

You can also configure Phobos using HOCON with minimal code:

var hocon = ConfigurationFactory.ParseString(File.ReadAllText("Config/app.conf"));

var tracer = Sdk.CreateTracerProviderBuilder()
    .AddSource(ActorTracing.PhobosActivitySourceName) // subscribe to built-in Phobos events
    .Build();
var metrics = Sdk.CreateMeterProviderBuilder()
    .AddMeter(ActorTracing.PhobosActivitySourceName)
    .Build();

// create an ActorSystemSetup class used to configure the ActorSystem
var phobosSetup = BootstrapSetup.Create()
        .WithConfig(hocon)
        .WithActorRefProvider(PhobosProviderSelection.Cluster);

// start ActorSystem
var actorSystem = ActorSystem.Create("myCluster", phobosSetup);

This approach requires:

1. Creating OpenTelemetry providers with AddPhobosInstrumentation()
2. Using BootstrapSetup with the appropriate PhobosProviderSelection

Phobos HOCON Reference

Some settings for classic Phobos configuration are done via HOCON rather than the BootstrapSetup or the PhobosSetup classes.

Note

Phobos.Hosting will let you specify all of this via the strongly typed builder interface. No HOCON required there.

Here's the full set of stand-alone HOCON settings that are specific to Akka.NET.

Tip

The phobos HOCON section is a stand-alone section; meaning that it does not get included inside the akka HOCON section that Akka.NET developers typically modify.

##################################
# Phobos Reference Config File   #
##################################

phobos{
    tracing{
        # Setting used to trigger tracing for all built-in /system actors by default.
        # When set to "on," all /system actors will automatically be outfitted with tracing
        # instrumentation.
        trace-all-system-actors = off

        # Setting used to trigger tracing for all /user actors by default.
        # When set to "on," all /user actors will automatically be outfitted with tracing
        # instrumentation.
        trace-all-user-actors = on
        
        # Setting is used to trace actor PreStart / PostRestart / PostStop / PreRestart activity prior to processing
        # messages. Also used to enable the akka.actor.start / akka.actor.stop events.
        trace-actor-lifecycle = on
        
        # Traces Akka.Persistence Persist and Recovery operations.
        # NOTE: Akka.Persistence recovery tracing also depends upon `trace-actor-lifecycle = on`.
        # However, if `trace-actor-lifecycle = on` but `trace-akka-persistence = on` means you can
        # measure Persist operations but not recoveries.
        trace-akka-persistence = on
        
        # Enables tracing of `Ask<T>` operations. When disabled, Phobos will not trace Ask<T> operations.
        trace-ask = on
        
        # Creates message-processing spans automatically when tracing is enabled.
        # When this setting is disabled users will be able to manually create traces
        # from SpanContexts received during tracing and traces will still be propagated.
        # REMARKS: this setting should really only be used when attempting to suppress
        # large amounts of noise from the tracing system by manually controlling when
        # traces are created.
        create-trace-upon-receive = on
        
        # In Phobos 2.0 we introduced latency tracking which includes logging two discrete events
        # on all automatically created Phobos spans:
        # - "waiting" - created when the message is first IActorRef.Tell'd
        # - "message" - created when the message is first processed by the actor and
        #               contains the message's content in stringified form.
        # Turning this setting to "off" disables the productions of these events on all `akka.msg.recv`
        # spans created by Phobos.
        log-message-events = on
        
        
        # Debugging settings used for testing the configuration of Phobos.Tracing
        debug{
            # When enabled: will log incoming tracing information
            # on all actors capable of receiving trace events.
            log-traces = off
        }

        # Setting used to instruct Phobos whether or not it should automatically append
        # all captured log messages to the active ISpan during which they were produced.
        #
        # Turning this setting off may help improve performance slightly in some instances,
        # but at the cost of missing useful debugging and diagnostic data.
        # 
        # If you want to throttle down the amount of log messages being appended to outgoing
        # spans, consider simply changing the `akka.loglevel` setting or others inside core Akka.NET.
        append-logs-to-trace = on
    }

    monitoring{
        # Setting used to trigger monitoring for all built-in /system actors by default.
        # When set to "on," all /system actors will automatically be outfitted with monitoring
        # instrumentation.
        monitor-all-system-actors = off

        # Setting used to trigger monitoring for all /user actors by default.
        # When set to "on," all /user actors will automatically be outfitted with monitoring
        # instrumentation.
        monitor-all-user-actors = on

        # Toggles monitoring of mailbox queue depth for any actors who have monitoring enabled
        monitor-mailbox-depth = on

        # When enabled, records all of the default events that pass through the eventstream
        # (log events, dead letters, and unhandled messages) and writes that out to a Global
        # counter for the ActorSystem as a whole.
        monitor-eventstream = on
    }
}

akka{
    # enable Akka.NET built-in telemetry by default in Phobos
    actor.telemetry.enabled = on
}

Configuring Trace Filtering

See "Filtering Tracing Data with Phobos" for details on how to do this.

Configuring Individual Actors via HOCON and Props

Should you need it, Phobos offers the ability to configure individual actors or groups of actors with specific tracing / monitoring settings.

Tip

Typically, configuring specific actors is most often used in contexts where Phobos users want to limit the amount of traces produced by specific areas of the actor hierarchy for "signal vs. noise" control or to reduce the total observability overhead of the system. For more tips on performance-optimizing your Phobos installation, please see "Performance Impact and Best Practices."

HOCON

Here's the full set of HOCON values that can be used inside the akka.actor.deployment HOCON namespace in Akka.NET - Phobos will look for these values and override the ActorSystem-wide settings for these groups of actors:

######################################################
# Phobos Default Actor-Specific Deployment Options   #
######################################################

akka.actor.deployment.default.phobos{
    # When set to "on" this setting ensures that all children, grandchildren
    # and so on of the actor deployed with this configuration receive the exact
    # same settings as this actor unless explicitly overridden by their own
    # deployment configuration.
    propagate-settings-to-children = off
    
    # When this is set to any value other than 'off', this name will be used for this
    # actor's "akka.actor.type" attribute on all metrics and traces instead of the .NET FQN
    actor-type-name = off

    tracing{
        # Toggles the entire tracing functionality of Phobos.Tracing on or off for this actor.
        # If set to 'off' then this actor will not record any traces. However, traces can still
        # be propagated through this actor to others. 
        enabled = on
    }

    monitoring{
        # Toggles the entire monitoring functionality of Phobos.Monitoring on or off for this actor.
        # If set to 'off' then this actor will not record any metrics.
        enabled = on
        
        # Toggles mailbox depth monitoring on or off for this actor.
        monitor-mailbox-depth = off
    }
}

What do each of these settings do?

• phobos.propagate-settings-to-children - when this setting is set to on, any settings used to configure this actor will also be applied recursively down to children and their children and so on. This is designed to help toggle tracing or monitoring off and on throughout an entire area of the actor hierarchy.
• phobos.tracing.enabled - when this is set to on tracing will be enabled for this actor. When it's off tracing will be disabled for all messages processed by this actor.
• phobos.monitoring.enabled - when this is set to on monitoring will be enabled for this actor. When it's off monitoring will be disabled for all messages processed by this actor.
• phobos.monitoring.monitor-mailbox-depth - this value is set to off by default, as there is a small performance impact to measuring the mailbox depth of any particular actor. However, when this setting is enabled a new gauge will be recorded that includes the point-in-time queue length for the mailbox length of all actors of this particular type.

You can apply this HOCON to an actor inside the akka.actor.deployment namespace like this:

akka.actor{
    provider = "Phobos.Actor.PhobosActorRefProvider, Phobos.Actor"

    deployment{
        # use custom sample rates and filtering for high-throughput actors
        /coordinator{
            phobos{
                # ensure that all descendants of /user/coordinator have these settings
                propagate-settings-to-children = on

                tracing{
                    enabled = on
                }

                monitoring{
                    enabled = on
                    monitor-mailbox-depth = off # don't monitor mailboxes
                }
            }
        }
    }
}

When an actor at location /user/coordinator gets deployed, it will utilize all of the settings found in this HOCON file.

Props - Per-Actor Configuration

Phobos offers fine-grained control over individual actors through the Props.WithInstrumentation() extension method. This allows you to override system-wide settings for specific actors or actor hierarchies.

Simple Enable/Disable

The easiest way to control instrumentation is to completely disable it for high-volume actors:

// Disable all instrumentation for a specific actor
var props = Props.Create<InstrumentedActor>()
    .WithInstrumentation(PhobosActorSettings.Empty);

var myHighVolumeActor = Sys.ActorOf(props, "high-volume-actor");

Or selectively enable/disable tracing vs monitoring:

// Using the system defaults as a base
var props = Props.Create<InstrumentedActor>()
    .WithInstrumentation(Sys, settings => settings
        .WithTracing(false)
        .WithMonitoring(true));

var myActor = Sys.ActorOf(props, "selective-actor");

Detailed Configuration

For more control, configure specific aspects of tracing and monitoring:

// Fine-grained control over tracing behavior
var props = Props.Create<InstrumentedActor>()
    .WithInstrumentation(Sys, settings => settings
        .WithTracing(true)
        .WithCreateTraceUponReceive(true)
        .WithTraceActorLifecycle(true)
        .WithLogMessageEvents(true)
        .WithMonitoring(true)
        with { MonitorMailbox = true });

var myImportantActor = Sys.ActorOf(props, "important-actor");

Custom Actor Type Names

Control how actors appear in your observability dashboards. This is especially useful for F# users where actors are often implemented as functions rather than classes, which can result in all actors having identical generic names in traces and metrics (see issue #80).

For F# and Akkling users, custom actor type names allow you to assign meaningful names to function-based actors that would otherwise appear with non-descriptive type information:

// Customize how the actor appears in traces and metrics
var props = Props.Create<InstrumentedActor>()
    .WithInstrumentation(new PhobosActorSettings(
        trace: true,
        monitor: true,
        actorTypeName: "CriticalBusinessActor"));

var myActor = Sys.ActorOf(props, "business-actor");

Selective Trace Creation

Reduce noise by disabling automatic trace creation while keeping tracing enabled for manual spans:

// Disable automatic trace creation but keep tracing enabled
// This allows manual span creation while reducing noise
var props = Props.Create<InstrumentedActor>()
    .WithInstrumentation(Sys, settings => settings
        .WithTracing(true)
        .WithCreateTraceUponReceive(false)
        .WithTraceActorLifecycle(false));

var quietActor = Sys.ActorOf(props, "quiet-actor");

Any of the settings you can specify via HOCON can also be specified by changing the instrumentation on Props for any and all of your actors.