Search Results for

    Show / Hide Table of Contents

    2.1.0 January 16 2023

    Phobos 2.1 Minor Release

    Phobos 2.1 is a minor release that has been in the works for a long time and introduces several major features and improvements to the existing Phobos 2.0 experience.

    Akka.Cluster.Sharding Metrics and Dashboards

    Issue: https://github.com/petabridge/phobos-issues/issues/72

    One of the things we added support for in Phobos 2.1 is automatically tracking Akka.Cluster.Sharding metrics for each ShardRegion for each node capable of hosting any shards - these numbers can be aggregated together to form a complete picture of total shard allocation by type and node throughout your cluster.

    The new built-in metrics are:

    • akka.cluster.sharding.shards - keeps track of the number of shards per node per entity type.
    • akka.cluster.sharding.entities - keeps track of the number of entities per node per entity type.

    More Accurate "Live Actor" Counts

    Phobos 2.1 takes advantage of a new actor telemetry feature introduced in Akka.NET v1.4.47, whihc allows us to use an OpenTelemetry ObservableGauge to accurately track the total number of live actors by type and node throughout each of your ActorSystems.

    Important

    In order to use this feature, you will need to set akka.actor.telemetry.enabled = on (it's off by default.)

    If you use Phobos.Hosting to configure Phobos, this setting will be automatically enabled for you.

    This value is now tracked via the akka.actor.live metric and will show up alongside the normal akka.actor.created|stopped|restarted counters. We've already incorporated akka.actor.live into our Phobos dashboards.

    Updated All Phobos Dashboards

    We've incorporated all Phobos dashboards to incorporate the following new metric types:

    • akka.cluster.sharding.shards - keeps track of the number of shards per node per entity type.
    • akka.cluster.sharding.entities - keeps track of the number of entities per node per entity type.
    • akka.actor.live - keeps more accurate track of the number of live actors per node by type.

    Please see "Phobos Dashboards" for the most recent versions of each of these.

    Akka.Cluster.Sharding Trace Compression

    One of the largest feature areas we've worked on for Phobos 2.1 is curtailing the amount of noise produced by Phobos tracing, especially in regards to features like Akka.Cluster.Sharding - where each message to one of your actors produces 2-3 spans recorded by built-in system actors.

    For starters, we now have the ability to compress Cluster.Sharding traces via an OpenTelemetry BaseProcessor<Activity> - and this can be enabled when you're subscribing to Phobos telemetry in your OTel pipeline:

    services.AddOpenTelemetryTracing(builder =>
    {
    builder
        .SetResourceBuilder(resource)
        .AddPhobosInstrumentation(compressShardTraces:true) // eliminate sharding infrastructure from traces
        .AddHttpClientInstrumentation()
        .AddAspNetCoreInstrumentation(options =>
        {
            options.Filter = context => !context.
                Request.Path.StartsWithSegments("/metrics");
        })
        .SetSampler(new TraceIdRatioBasedSampler(1.0d))
        .AddJaegerExporter(opt =>
        {
            opt.AgentHost = Environment.GetEnvironmentVariable(JaegerAgentHostEnvironmentVar) ?? "localhost";
        });
    });
    

    Calling .AddPhobosInstrumentation(compressShardTraces:true) will enable trace compression for sharding. This value is set to false by default.

    In addition to these changes, we've also filtered out all internal messages and Akka.DistributedData messages by default.

    Ask<T> Tracing Improvements

    We've made many significant improvements to Ask<T> tracing in Phobos 2.1:

    • Ask<T> tracing can now be disabled via the fluent C# APIs or via phobos.tracing.trace-ask = off;
    • Ask<T> tracing can now be filtered using the same ITraceFilter and other filter rules as regular akka.actor.recv statements;
    • Ask<T> tracing has been restructured such that we now accurately capture both the input and output types separately - additionally, we are much more accurate in tracking start and stop times than we were previously.
    • Fixed: Ask<T> tracing does not properly follow Tracer.ActiveSpan

    To disable Ask<T> tracing via Phobos.Hosting, use the following:

    collection.AddAkka("LocalSys", (builder, sp) =>
    {
        builder.WithPhobos(AkkaRunMode.Local, 
            configBuilder => configBuilder.WithTracing(t => t.SetTraceAsk(false)));
        AddTestActors(builder);
    });
    

    Other General Improvements

    • Fixed: Phobos DeadLetter logging.
    • Fixed: Tracer.CurrentSpan and PhobosActorContext.ActiveSpan should be the same when actor is processing message, but currently are not.
    • All IWithTimers messages are filtered out by default.

    2.1.0-beta1 August 2 2022

    Phobos 2.x BETA Release

    Phobos 2.1.0-beta1 features some significant noise control improvements for tracing:

    • All IWithTimers messages are filtered out by default;
    • Inside Akka.Cluster.Sharding actors all internal messages and Akka.DistributedData messages are filtered out by default;
    • Resolved: Manually created trace spans don't seem to flow across the cluste; and
    • Resolved: How to filter out Ask<T> operations?

    To disable Ask<T> tracing:

    phobos.tracing.trace-ask = off
    

    or

    collection.AddAkka("LocalSys", (builder, sp) =>
    {
        builder.WithPhobos(AkkaRunMode.Local, 
            configBuilder => configBuilder.WithTracing(t => t.SetTraceAsk(false)));
        AddTestActors(builder);
    });
    

    2.0.6 June 10 2022

    Phobos 2.x Revision Release

    • Fixed: Stashing message may result in reset UsableContext
    • Upgraded to Akka.Hosting v0.3.1

    2.0.5 June 06 2022

    Phobos 2.x Revision Release

    • Upgraded to OpenTelemetry 1.3
    • Fixed: Span is created for akka.actor.ask even when create-trace-upon-receive is set to off

    2.0.4 June 02 2022

    Phobos 2.x Revision Release

    • Fixed: Custom span creation with Phobos 2.0.3-beta1 - fixed a bug when running Phobos with phobos.tracing.create-trace-upon-receive = off that allowed Phobos to continue creating spans when there was no active trace context available. This has been fixed and the setting now works as intended.
    • Fixed: Lots of noise in logging when using cluster sharding - introduced a new setting to Phobos configuration, phobos.tracing.log-message-events = on, which when set to off will disable the creation of waiting and message events from being added to each automatically-created akka.msg.recv TelemetrySpan. This will greatly reduce the data footprint for each TelemetrySpan produced by Phobos and will help reduce the amount of noise generated on platforms like Elastic APM.
    • phobos.tracing.log-message-events is now set to off by default for Akka.Cluster.Sharding and Akka.Cluster.Tools.PubSub actors.
    • Upgraded to Akka.NET v1.4.39

    2.0.3 May 26th 2022

    Phobos 2.x Revision Release

    Fixed: Serializer exception when using Phobos and Akka.Hosting - this issue was caused by actor lifecycle tracing leaking Tracer.CurrentSpan instances across threads as a result of how actor spawning works on most systems. Specifically, when Akka.Cluster.Sharding was enabled this leaky mechanism made it possible for other /system actors responsible for clustering to suddenly have access to active traces, which in turn caused them to wrap built in system messages such as cluster heartbeats with Phobos' tracing envelopes.

    We've fixed this in two separate ways:

    1. Actor lifecycle traces now longer use the Tracer.CurrentSpan / System.Diagnostics.Activity.Current property, so the source of the leak has been removed;
    2. Phobos.Actor now takes a dependency on Akka.Cluster and explicitly excludes all Akka.Cluster.IClusterMessage events from any kind of consideration during tracing.

    2.0.3-beta1 May 12th 2022

    Phobos 2.x BETA Release

    This is a BETA release of a new propsoed feature for Phobos 2.0 - "macro" tracing: the ability to disable automatic akka.msg.recv span creation but still receive the SpanContext from incoming messages and still propagate SpanContext to other actors via TelemetrySpan / System.Diagnostic.Activity instances you create yourself.

    Disabling Automatic Span Creation

    You can disable automatic span creation for the entire ActorSystem via the following HOCON:

    phobos.tracing.create-trace-upon-receive = off
    

    We don't recommend doing this as it's a potentially destructive action. Instead, it's better to disable automatic span creation in areas where you know the system is busy.

    akka.actor.deployment{
        /manual1{
            phobos.tracing.create-trace-upon-receive = off
            phobos.tracing.propagate-settings-to-children = on
        }
    }
    

    This will allow you to disable automatic span creation for the entire hierarchy of actors beneath /user/manual1.

    You can also accomplish the same thing via C# or F# via the Props extension methods introduced by Phobos:

    var myManualSpanActor = Sys.ActorOf(Props.Create(() => new ManualTraceCreator())
        .WithInstrumentation(
            PhobosSettings.For(Sys).DefaultUserActorSettings
                .WithCreateTraceUponReceive(false)
                .WithPropagateToChildren(true)), "manual2");
    

    This will work identically to the HOCON above.

    When these settings are enabled, the affected actors will still have tracing enabled and will propagate any active trace data to other actors via IActorRef.Tell but will not automatically create any additional akka.msg.recv spans on their own any longer.

    Manually Creating Spans

    So in order to create useful traces inside these areas, you can now access the IPhobosActorContext.UsableContext property - which contains a SpanContext? that can be used in the manual creation of OpenTelemetry spans:

    public ManualTraceCreator()
    {
        Receive<GetActiveSpan>(_ =>
        {
            // should be default(TelemetrySpan) when running with
            // phobos.tracing.create-trace-upon-receive = off
            Sender.Tell(new SpanResponse(Context.GetInstrumentation().ActiveSpan));
        });
    
        Receive<MyMsg>(m =>
        {
            using (var mySpan = Context.GetInstrumentation().Tracer.StartActiveSpan("MyMsg", SpanKind.Consumer,
                       Context.GetInstrumentation().UsableContext ?? default))
            {
                _logging.Info("My event");
                Sender.Tell(new MyReply("reply"));
            }
        });
    }
    

    This will allow the Phobos user full control over what appears in their spans and which actors / messages can produce spans. Span propagation still works automatically inside Phobos with automatic span creation disabled - but now you're responsible for creating your own spans. This should significantly reduce the amount of noise produced by the tracing system - we're going to experiment using this feature inside our planned Akka.Streams support.

    2.0.2 April 19th 2022

    Phobos 2.x Release

    This release marks the official "release to market" of Phobos 2.0 - which includes the following changes (from 1.x and earlier betas of 2.x):

    1. Migrated from App.Metrics and OpenTracing to OpenTelemetry - the future APM platform for all of .NET;
    2. End-to-end message processing latency tracking in both Phobos metrics and traces;
    3. Re-engineered the trace filtering system to be much easier to use and effective at reducing noise; and
    4. Integrated with Akka.Hosting, a new HOCONless way of configuring, running, and integrating Akka.NET with the other parts of your .NET applications. See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here. This is accomplished through the new Phobos.Hosting NuGet package introduced in Phobos 2.0.

    These changes all represent a major step forward for Phobos and Akka.NET telemetry.

    For full details click here to read "What's New in Phobos 2.0?"

    Phobos 1.x Support Lifetime Per our recent announcement, we will continue to support Phobos 1.x and 2.x concurrently. Phobos 1.x will exit support on December 31st, 2023.

    1.5.1 April 12th 2022

    Minor Version Release of Phobos 1.x

    Phobos 1.5 is the next minor version release of Phobos. It contains two major, important changes:

    1. Overhaul of the Trace Filtering System and
    2. Integrated with Akka.Hosting, a new HOCONless way of configuring, running, and integrating Akka.NET with the other parts of your .NET applications. See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here. This is accomplished through the new Phobos.Hosting NuGet package introduced in Phobos 1.5.

    And we've also fixed the following bugs:

    1. https://github.com/petabridge/phobos-issues/issues/54
    2. https://github.com/petabridge/phobos-issues/issues/53

    New and Improved Trace Filtering

    This release contains some breaking changes which we've introduced intentionally in order to get users concerned about noise control from the Phobos tracing system to migrate over to a new set of APIs that perform the job in a much more coherent fashion that what we have currently in Phobos 1.4.

    You can see some examples of end-user negative experience reports with the current v1.x and 2.x trace filtering implementations here:

    • https://github.com/petabridge/phobos-issues/issues/46
    • https://github.com/petabridge/phobos-issues/issues/49

    There have been many more like these over the past couple of years - separating noise from useful signals has been a persistent challenge with Phobos tracing and the changes we're introducing in Phobos v1.5 and 2.0 address this with a small new API:

    /// <summary>
    /// Each <see cref="ActorSystem"/> can implement exactly one of these to control
    /// the decision-making around which traces to include and which ones to exclude.
    ///
    /// When this type is configured, it will replace all of the <see cref="IFilterRule"/>s
    /// that have been previously defined.
    /// </summary>
    /// <seealso cref="FilterDecision"/>
    public interface ITraceFilter
    {
        /// <summary>
        /// Evaluates a message processed by an actor and determines whether or not to include it in the current
        /// or a new trace.
        /// </summary>
        /// <param name="message">The message currently being processed by an actor.</param>
        /// <param name="alreadyInTrace">Whether or not this message is already part of an active trace.</param>
        /// <returns><c>true</c> if we are going to trace the message, <c>false</c> otherwise.</returns>
        /// <remarks>
        /// Should return <c>true</c> unless you are explicitly trying to filter a given message out of your trace.
        /// </remarks>
        bool ShouldTraceMessage(object message, bool alreadyInTrace);
    }
    

    This is the ITraceFilter - and there is exactly one of these for your entire ActorSystem. It allows you to decide whether or not to trace an actor's message processing activity based on:

    • Message type;
    • Message content; and
    • Whether or not we are inside an active trace or not.

    This API is astonishingly succinct, much clearer than the previous TracingConfigBuilder API from before, extremely performant, and very effective at limiting noise inside your Phobos-enabled Akka.NET applications.

    An example from one of our own stress testing applications:

    public sealed class ScenarioTraceFilter : ITraceFilter
    {
        public bool ShouldTraceMessage(object message, bool alreadyInTrace)
        {
            switch (message)
            {
                /*
                 * Only start new traces for these commands
                 */
                case IScenarioProtocolMessage _:
                case Petabridge.Cmd.Command _:
                case Petabridge.Cmd.CommandResponse _:
                    return true;
                default:
                    // otherwise, only allow messages already in trace to be included
                    return alreadyInTrace;
            }
        }
    }
    

    This class allows new traces to only be started handful of Petabridge.Cmd types and any messages and our custom IScenarioProtocol messages. Any additional messages will not be included in a trace unless there is already an active trace present.

    You can add this ITraceFilter to your applications like so:

    var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
        .WithMetrics(m =>
            m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
        .WithTracing(t => t.SetTracer(tracer)
            .SetTraceFilter(new ScenarioTraceFilter())
        )) // binds Phobos to same tracer as ASP.NET Core
    .WithSetup(BootstrapSetup.Create()
        .WithActorRefProvider(PhobosProviderSelection
            .Local)); // last line activates Phobos inside Akka.NET
    

    Phobos still supports the old TracingConfigBuilder pattern and the WithIncludeMessage<T> rules - however, all of those rules will be overidden and replaced when the ITraceFilter is used instead.

    We have broken the old TracingConfigBuilder method signatures in order to alert users to this important change.

    You can see the full details here: "Filtering Tracing Data with Phobos"

    Akka.Hosting Integration and Simplified Configuration

    Phobos can now be fully configured without any HOCON or mangling of Setup classes using the brand new Phobos.Hosting NuGet package, built on top of Akka.Hosting.

    Tip

    Unfamiliar with Akka.Hosting? See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here.

    See Configuring Phobos 2.0 - Using Phobos.Hosting for full details.

    2.0.0-beta4 and 1.5.0-beta1 March 17th 2022

    Major Update to Phobos 2.x and 1.x

    This release contains some breaking changes which we've introduced intentionally in order to get users concerned about noise control from the Phobos tracing system to migrate over to a new set of APIs that perform the job in a much more coherent fashion that what we have currently in Phobos 1.4 / 2.0.0-beta3 and newer.

    You can see some examples of end-user negative experience reports with the current v1.x and 2.x trace filtering implementations here:

    • https://github.com/petabridge/phobos-issues/issues/46
    • https://github.com/petabridge/phobos-issues/issues/49

    There have been many more like these over the past couple of years - separating noise from useful signals has been a persistent challenge with Phobos tracing and the changes we're introducing in Phobos v1.5 and 2.0-beta4 address this with a small new API:

    /// <summary>
    /// Each <see cref="ActorSystem"/> can implement exactly one of these to control
    /// the decision-making around which traces to include and which ones to exclude.
    ///
    /// When this type is configured, it will replace all of the <see cref="IFilterRule"/>s
    /// that have been previously defined.
    /// </summary>
    /// <seealso cref="FilterDecision"/>
    public interface ITraceFilter
    {
        /// <summary>
        /// Evaluates a message processed by an actor and determines whether or not to include it in the current
        /// or a new trace.
        /// </summary>
        /// <param name="message">The message currently being processed by an actor.</param>
        /// <param name="alreadyInTrace">Whether or not this message is already part of an active trace.</param>
        /// <returns><c>true</c> if we are going to trace the message, <c>false</c> otherwise.</returns>
        /// <remarks>
        /// Should return <c>true</c> unless you are explicitly trying to filter a given message out of your trace.
        /// </remarks>
        bool ShouldTraceMessage(object message, bool alreadyInTrace);
    }
    

    This is the ITraceFilter - and there is exactly one of these for your entire ActorSystem. It allows you to decide whether or not to trace an actor's message processing activity based on:

    • Message type;
    • Message content; and
    • Whether or not we are inside an active trace or not.

    This API is astonishingly succinct, much clearer than the previous TracingConfigBuilder API from before, extremely performant, and very effective at limiting noise inside your Phobos-enabled Akka.NET applications.

    An example from one of our own stress testing applications:

    public sealed class ScenarioTraceFilter : ITraceFilter
    {
        public bool ShouldTraceMessage(object message, bool alreadyInTrace)
        {
            switch (message)
            {
                /*
                 * Only start new traces for these commands
                 */
                case IScenarioProtocolMessage _:
                case Petabridge.Cmd.Command _:
                case Petabridge.Cmd.CommandResponse _:
                    return true;
                default:
                    // otherwise, only allow messages already in trace to be included
                    return alreadyInTrace;
            }
        }
    }
    

    This class allows new traces to only be started handful of Petabridge.Cmd types and any messages and our custom IScenarioProtocol messages. Any additional messages will not be included in a trace unless there is already an active trace present.

    You can add this ITraceFilter to your applications like so:

    var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
        .WithMetrics(m =>
            m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
        .WithTracing(t => t.SetTracer(tracer)
            .SetTraceFilter(new ScenarioTraceFilter())
        )) // binds Phobos to same tracer as ASP.NET Core
    .WithSetup(BootstrapSetup.Create()
        .WithActorRefProvider(PhobosProviderSelection
            .Local)); // last line activates Phobos inside Akka.NET
    

    Phobos still supports the old TracingConfigBuilder pattern and the WithIncludeMessage<T> rules - however, all of those rules will be overidden and replaced when the ITraceFilter is used instead.

    We have broken the old TracingConfigBuilder method signatures in order to alert users to this important change - however, we are going to keep these changes in a beta state for a short period in order to gather feedback from affected users. If you are concerned about trace noise inside your application, please try this beta and report on your experience on the Phobos Issue Tracker.

    1.4.2 March 8th 2022

    Maintenance Release for Phobos 1.4

    • Resolved: OpenTracing: Tracing breaks when using Ask().PipeTo()

    To have tracing in Phobos 1.4 work with PipeTo(), you will have to use PipeTo(recipient, true) and call the Ask() method inside an async method even when the Ask() method is not awaited.

    private class PipeToActor: ReceiveActor
    {
        public PipeToActor(bool useAsync, IActorRef otherActor)
        {
            ReceiveAsync<int>(o =>
            {
                // Need to make sure that async context is preserved inside PipeTo
                otherActor.Ask("message").PipeTo(Self, true, Sender); 
                return Task.CompletedTask;
            });
        }
    }
    
    • The boolean parameter in the PipeTo(IActorRef, true, IActorRef) method sets ConfigureAwait(true) inside the PipeTo() operation - and this allows us to preserve the current AsyncLocal<ISpan> context.
    • The ReceiveAsync method is also needed for proper async context propagation, which is what's needed in order to ensure that the ISpan created by the PipeTo method is properly correlated to the one created by the Ask method.

    2.0.0-beta2 February 1st 2022

    Maintenance Release for Phobos 2.0-beta1

    • Implemented a change where when phobos.tracing.trace-actor-lifecycle = off both parent actors and child actors will no longer create akka.actor.spawn operations.
    • Child actors will no longer log akka.actor.started operations if their Props have been updated via WithInstrumentation to disable actor lifecycle tracing.

    1.4.1 February 1st 2022

    Maintenance Release for Phobos 1.4

    • Implemented a change where when phobos.tracing.trace-actor-lifecycle = off both parent actors and child actors will no longer create akka.actor.spawn operations.
    • Child actors will no longer log akka.actor.started operations if their Props have been updated via WithInstrumentation to disable actor lifecycle tracing.

    2.0.0-beta1 January 26th 2022

    MAJOR RELEASE OF PHOBOS

    In Phobos 2.0 we have migrated all of our tracing and monitoring capabilities to use OpenTelemetry.

    As we mentioned in our recent blog post about Phobos 2.0 and its relationship with OpenTelemetry, we did this for the following reasons:

    • OpenTelemetry is now the official APM solution for the entire .NET platform, including other popular frameworks such as ASP.NET and Entity Framework;
    • All major APM vendors and OSS project such as Prometheus, Jaeger, DataDog, AWS, GCP, Azure, Elastic, and so on are all committed to supporting OpenTelemetry going forward and they implement their own first-party exporters that transmit data to those platforms;
    • The performance of the OpenTelemetry.Trace and OpenTelemetry.Metrics implementations, which Phobos uses, are identical no matter which combination of exporters you adopt; and
    • OpenTelemetry is much more decoupled and extensible than any of the previous solutions in market.

    We have replaced all of our OpenTracing and App.Metrics dependencies with 2.x going forward.

    Installation and Configuration To use Phobos 2.0.0-beta* in your solution, all you need to do is:

    1. Install the appropriate Phobos.Actor.* package into your application using the 2.0.0-beta* version;
    2. Install the appropriate OpenTelemetry.Exporter.* packages you want to use in combination with your preferred APM tools; and
    3. Configure your IServiceCollection with the following (adjust as needed):
    // This method gets called by the runtime. Use this method to add services to the container.
    // For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
    public void ConfigureServices(IServiceCollection services)
    {
        // Prometheus exporter won't work without this
        services.AddControllers();
    
        var resource = ResourceBuilder.CreateDefault()
            .AddService(Assembly.GetEntryAssembly().GetName().Name, serviceInstanceId: $"{Dns.GetHostName()}");
    
        // enables OpenTelemetry for ASP.NET / .NET Core
        services.AddOpenTelemetryTracing(builder =>
        {
            builder
                .SetResourceBuilder(resource)
                .AddPhobosInstrumentation()
                .AddSource("Petabridge.Phobos.Web")
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation()
                .AddJaegerExporter(opt =>
                {
                    opt.AgentHost = Environment.GetEnvironmentVariable(JaegerAgentHostEnvironmentVar);
                });
        });
    
        services.AddOpenTelemetryMetrics(builder =>
        {
            builder
                .SetResourceBuilder(resource)
                .AddPhobosInstrumentation()
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation()
                .AddPrometheusExporter(opt =>
                {
                });
        });
    
        // sets up Akka.NET
        ConfigureAkka(services);
    }
    

    The .AddPhobosInstrumentation() line will ensure that OTel picks up all of the signals produced by Phobos - and you need to call this line for both metrics and tracing!

    Finally, you need to configure your ActorSystem to use Phobos and to leverage the TracerProvider and MeterProviders from OpenTelemetry:

    public static void ConfigureAkka(IServiceCollection services)
    {
        services.AddSingleton(sp =>
        {
            var metrics = sp.GetRequiredService<MeterProvider>();
            var tracer = sp.GetRequiredService<TracerProvider>();
    
            var config = ConfigurationFactory.ParseString(File.ReadAllText("app.conf"))
                .BootstrapFromDocker()
                .UseSerilog();
    
            var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
                    .WithMetrics(m =>
                        m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
                    .WithTracing(t => t.SetTracer(tracer))) // binds Phobos to same tracer as ASP.NET Core
                .WithSetup(BootstrapSetup.Create()
                    .WithConfig(config) // passes in the HOCON for Akka.NET to the ActorSystem
                    .WithActorRefProvider(PhobosProviderSelection
                        .Cluster)); // last line activates Phobos inside Akka.NET
    
            var sys = ActorSystem.Create("ClusterSys", phobosSetup);
    
            // create actor "container" and bind it to DI, so it can be used by ASP.NET Core
            return new AkkaActors(sys);
        });
    
        // this will manage Akka.NET lifecycle
        services.AddHostedService<AkkaService>();
    }
    

    This is fundamentally not all that different from how Phobos 1.x works.

    We will update our official documentation and website with more Phobos 2.0-specific instructions as our releases mature, but for the time being please don't hesitate to submit an issue on the Phobos Issue Tracker if you run into trouble!

    Phobos 1.x Support Lifetime Per our recent announcement, we will continue to support Phobos 1.x and 2.x concurrently. Phobos 1.x will exit support on December 31st, 2023.

    1.4.0 January 19th 2022

    New Minor Version Release for Phobos

    Phobos 1.4 is a minor version release that introduces some exciting new capabilities for Akka.NET users, primarily:

    • All actor lifecycle activity, i.e. everything that occurs during PreStart, PostStop, PreRestart, and PostRestart is now traced by default. This will allow you to visualize everything that happens after an actor is spawned mid-flight. This setting does produce significantly more traces when enabled (naturally) so it can be disabled via the following HOCON setting: phobos.tracing.trace-actor-lifecycle = off.
    • Phobos 1.4 uses Akka.NET v1.4.32, which contains important changes to the Akka.Persistence Journal base classes that now allow us to properly trace activity that happens within it. As a result, when phobos.tracing.trace-actor-lifecycle = on (this is the default) you can now trace the entire recovery process of all PersistentActor implementations - so long as the Akka.Persistence plugin you're using is built against Akka.NET v1.4.32 or higher. In addition to tracing Akka.Persistence recoveries all Persist calls can now be traced end to end with Phobos.

    1.3.3 January 3rd 2022

    Patch for Phobos 1.3

    Phobos v1.3.3 introduces a bugfix for a very edge-case regression that was introduced in Phobos v1.3.2:

    • Possible regression: System.NotImplementedException in Akka.Persistence.Sql.Common.Journal.BatchingSqlJournal

    This was caused by some of our trace correlation software not fitlering remote IActorRefs that sent messages to actors outside of an actor's scheduled context, a rare case that could only really happen inside a Task closed over a remote IActorRef. Patch v1.3.3 resolves this issue.

    1.3.2 December 21st 2021

    Minor Version Release for Phobos

    Phobos v1.3.2 includes some important tracing enhancements brought about as a result of upgrading Phobos to Akka.NET v1.4.31:

    Phobos now supports tracing the output of PipeTo operations! This can be accomplished by making sure that the Sender property of the PipeTo operation contains a reference to the actor kicking of the Task<T> whose results will be piped:

    var myActor = Sys.ActorOf(act =>
    {
        act.ReceiveAny((o, ctx) =>
        {
            // won't capture trace - no "sender" included
            Task.Delay(TimeSpan.FromMilliseconds(100)).PipeTo(ctx.Sender, success: () => o);
    
            // will capture trace - "Sender" is specified
            Task.Delay(TimeSpan.FromMilliseconds(100)).PipeTo(ctx.Sender, success: () => o, sender:ctx.Self);
        });
    });
    

    1.3.1 December 9th 2021

    Minor Version Release for Phobos

    Phobos v1.3.1 includes some improvements for metrics support:

    • phobos.monitoring.monitor-mailbox-depth is now set to on by default as the performance impact is negligible for most actor types and can be found under the akka.actor.mailbox.depth metric header.
    • Phobos now records an App.Metrics Timer for every message received by an actor under the akka.messages.latency header. This is used to help measure how long an actor takes to process each message and these times will also be measured across an awaits that occur inside your actor's ReceiveAsync methods.

    We also have updated all of our built-in Phobos dashboards to include the akka.messages.latency values. You can see an example below:

    akka.messages.latency, new metrics introduced in Phobos v1.3.1, being displayed in Grafana

    The addition of the akka.messages.latency metric to Phobos' captured Akka.NET data

    In addition to these changes, we've also updated our performance benchmarks for Phobos to include the most recent data for v1.3.1. The addition of the akka.messages.latency reduces Phobos 1.3.1's total throughput by about 10% compared to our previous benchmarks, though usually this overhead pales in comparison to user-defined code. However, if this is a problem for you please file an issue with us here: https://github.com/petabridge/phobos-issues

    1.3.0 November 18th 2021

    Minor Version Release for Phobos

    Phobos v1.3.0 includes some significant improvements in its tracing capabilities:

    • Phobos now properly handles await operations that occur within your actors - so any traces that begin when an actor starts processing a message will be carried all the way through until your ReceiveAsync methods exit. This is convenient when combined with other libraries that emit traces such as EntityFramework and OpenTracing.Contrib.
    • Phobos no longer records any akka.actor.ask object invocations during the startup of your ActorSystem, which were erroneously being attached to user-generated traces as well.
    • In some instance of using the WrappedTracer, which gets called whenver you use the WithScopeManager extension method on an existing ITracer object, it was possible for a null ISpan to be passed into the SpanBuilder.ChildOf method. This input is now null-checked and will no longer throw NullReferenceException errors.
    • All IScopes generated by Phobos are disposed of per their original invocation ordering, which reverses a major change we introduced in Phobos 1.2.0. We were able to introduce this change safely as a result of changing how we track traces around await calls.
    • The Phobos cluster-listener-actor will no longer log any Unhandled messages when akka.loglevel=DEBUG and akka.actor.debug.unhandled=on any longer.
    • Phobos has been upgraded to run on Akka.NET v1.4.28.

    In addition to these changes, we've also updated our performance benchmarks for Phobos to include the most recent data for v1.3.0.

    1.2.5 October 13th 2021

    Maintenance Release for Phobos 1.2

    Phobos v1.2.5 includes some fixes for the new Ask<T> tracing capabilities we added to Phobos v1.2.4:

    • Ask<T> spans now have an operation name that reads akka.actor.ask <ExpectedReplyType> - in order to make them more easily searchable on the UI and
    • Ask<T> tracing now runs when called from outside the ActorSystem, a very common use-case that is frequently used in Web APIs and elsewhere.

    1.2.4 October 11th 2021

    Maintenance Release for Phobos 1.2

    Phobos v1.2.4 is a mostly maintenance upgrade from Phobos v1.2.3, but includes a valuable new feature which allows Phobos to more easily trace Ask<T> operations in order to help diagnose timeouts and other potential problems which may occur:

    • Upgraded to Akka.NET v1.4.27;
    • Added the ability to trace Ask<T> operations directly, including timeouts and other errors that might occur during processing. Look for the akka.actor.ask operation inside your trace system.

    1.2.3 September 8th 2021

    Maintenance Release for Phobos 1.2

    Phobos v1.2.3 is a mostly maintenance upgrade from Phobos v1.2.2, but includes some important fixes to tracing messages that pass through routed actors.

    • Upgraded to Akka.NET v1.4.25 - you cannot upgrade to Akka.NET v1.4.25 without using at least Phobos v1.2.3 due to binary compatibiltiy issues;
    • Fixed an issue that prevented Akka.Cluster.Metrics weighted routers from being used correctly with Phobos v1.2.2; and
    • Fixed a rare issue that caused ActorScope.Dispose to throw a NullReferenceException.

    1.2.2 August 18th 2021

    Maintenance Release for Phobos 1.2

    Phobos v1.2.2 is a mostly maintenance upgrade from Phobos v1.2.1, but includes some important fixes to tracing messages that pass through routed actors.

    • Upgraded to Akka.NET v1.4.24.
    • Phobos now correctly unpacks any messages that are sent to a ConsistentHashRouter via ActorSelection or Akka.Remote, as those messages weren't properly being unpacked from their SpanEnvelope data structure used by Phobos.

    1.2.1 August 10th 2021

    Maintenance Release for Phobos 1.2

    Phobos v1.2.1 is a mostly maintenance upgrade from Phobos v1.2.0, but fixes many known issues with actor tracing in particular.

    • Upgraded to Akka.NET v1.4.23.
    • Phobos now supports version ranges for Google.Protobuf, supporting anything from v3.12.1 and higher.
    • Tracing of remotely deployed actors now works correctly in all cases.
    • Fixed a bug introduced in Phobos 1.2.0 that caused TraceConfigBuilder.IncludeMessagesAlreadyInTrace(bool includeInTrace) to not be honored correctly.

    1.2.0 June 17th 2021

    Maintenance Release for Phobos 1.2

    • Resolved some IScope memory leaks occurring inside Phobos - these occured as a result of IScope being lost inside async contexts and because some IScopes were not properly disposed the actor had a different IScope set as its active scope.
    • Added new PhobosSetup.Create method which creates its own PhobosConfigBuilder, rather than requiring users to create and pass one in first.

    Significantly improved trace filtering, with the new TraceConfigBuilder.IncludeMessagesAlreadyInTrace(bool includeInTrace) method, which allows users to coimpletely drop any messages inside an active trace that don't match your IFilterRules - this is designed for high throughput scenarios where users really want to greatly reduce the amount of data being created by their tracing system.

    Here's an example of it in action:

    var config = @"akka.actor.provider = ""Phobos.Actor.PhobosActorRefProvider,Phobos.Actor""";
    
    var tracer = new MockTracer(new ActorScopeManager());
    var phobosConfig = new PhobosConfigBuilder()
        .WithTracing(m =>
        {
            m.SetTracer(tracer);
    
            // don't include messages in trace that don't satisfy other filters
            m.IncludeMessagesAlreadyInTrace(false);
    
            // only accept FilteredMessage types
            m.AddIncludeMessageFilter<FilteredMessage>();
        });
    
    // create PhobosSetup + BootstrapSetup for configuring ActorSystem
    var phobosSetup = PhobosSetup.Create(phobosConfig)
        .WithSetup(BootstrapSetup.Create().WithConfig(config));
    
    sys = ActorSystem.Create("PhobosTest", phobosSetup);
    
    var actor = sys.ActorOf(Props.Create(() => new EchoActor()), "echo");
    var actor2 = sys.ActorOf(act =>
    {
        IActorRef sender = null;
        act.Receive<FilteredMessage>((f, ctx) =>
        {
            sender = ctx.Sender;
            actor.Tell(f.Message);
        });
    
        act.Receive<string>((str, ctx) =>
        {
            sender.Tell(str, ctx.Self);
        });
    }, "secondActor");
    
    // send a message that WILL be filtered out
    actor2.Ask<string>(new FilteredMessage("bye"), TimeSpan.FromMilliseconds(100)).Wait();
    Task.Delay(300).Wait(); // wait for activity in other threads to stop
    var spans = tracer.FinishedSpans();
    spans.Count.Should().Be(1); // for the FilteredMessage only
    

    1.1.3 May 26th 2021

    Maintenance Release for Phobos 1.1

    • Upgraded Phobos to target Akka.NET v1.4.20
    • Phobos.Actor.Cluster now takes a dependency on Akka.Cluster.Sharding, in order to filter out several of the built-in message types for it, Akka.Cluster.Tools, and Akka.DistributedData which are sent automatically in the background.
    • Resolved: When using akka.serialize-messages = on together with Phobos it sometimes throws this error during shutdown
    • Resolved: Need to cut down on the number of messages captured by DistributedPubSub actors
    • Added the ability to capture actor crashes / restarts even when actors fail during initialization, in order to make it easier to troubleshoot problems with Akka.DependencyInjection and actor constructor arguments.

    1.1.2 May 6th 2021

    Maintenance Release for Phobos 1.1

    • Upgraded Phobos to target Akka.NET v1.4.19
    • Upgraded Phobos to target App.Metrics v4.2.0 - this includes a new StatsD reporter that can be used in combination with DogStatsD for DataDog or stand-alone StatsD + Graphite.

    1.1.1 March 23 2021

    • Resolved: Phobos may not work with Akka.DependencyInjection
    • Upgraded Phobos to target Akka.NET v1.4.18

    1.1.0 January 25 2021

    Akka.Cluster Metrics Phobos now automatically records the number of nodes of each MemberStatus and the reachability of those cluster members. This happens without any need for manual code on the part of the Phobos user.

    We also now append the first akka.cluster.roles item as a tag to all exported metrics - which should make it easy to group metrics together by Akka.Cluster role type.

    We've shipped a default Akka.Cluster + Phobos Grafana dashboard that gathers the default metrics that are collected via Prometheus that you can install to visualize this data inside your own applications:

    https://grafana.com/grafana/dashboards/13775

    Minor Changes:

    • Metrics - Capture exception types inside log events and actor crashes;
    • Bugfix: Capture correct message name when tracing message that caused actor to crash; and
    • Upgraded to Akka.NET v1.4.16.

    1.0.5 December 07 2020

    • Upgraded Akka.Cluster.Tools from v1.4.11 to v1.4.12
    • Removed obsolete HOCON default configuration. SampleRate setting is obsolete at Phobos.Actor.Configuration.MonitoringSettings.
    • Fixed missing tags for akka.actor.created metrics counter

    1.0.4 November 16 2020

    • Upgraded to Akka.NET v1.4.12
    • Resolved: Tracing: Self.Forward causes entirely new trace to be made

    1.0.3 October 12 2020

    Added WrappedTracer type, designed to make it possible for OpenTracing.ITracer implementations which don't support custom IScopeManagers to play nicely with Phobos' ActorScopeManager, which makes correlation from ASP.NET Core (and other non-Akka.NET services) --> Akka.NET automatic and accurate.

    The WrappedTracer gets invoked automatically via the extension method .WithScopeManager present on the TracerExtensions class:

    public static void ConfigureDataDogTracing(IServiceCollection services)
    {
        // Add DataDog Tracing
        services.AddSingleton<ITracer>(sp =>
        {
            return OpenTracingTracerFactory.CreateTracer().WithScopeManager(new ActorScopeManager());
        });
    }
    

    In this case, the DataDog OpenTracing.ITracer returned by this method will be nested inside a WrappedTracer, which will proxy all of the real underlying tracing calls down to DataDog's client - but Phobos' ActorScopeManager will be used to perform all of the correlation activities.

    Speaking of DataDog, we've also added a DataDog branch to the Phobos Quickstart tutorial Github repository here: https://github.com/petabridge/Petabridge.Phobos.Web/blob/feature/datadog-integration/README.md#working-with-datadog

    1.0.2 June 19 2020

    • Fixed bug that caused actor crashes when complex generic types were logged during tracing.
    • Fixed issue that caused Unhandled and DeadLetter messages to not be logged by tracing or metrics correctly.

    1.0.1 June 19 2020

    • Uses the NoMetrics.Instance as the default metrics inside PhobosSetup when users don't explicitly set an IMetricsRoot.

    1.0.0 June 18 2020

    Phobos 1.0.0 is a major shift from Phobos v0.9, and you can read the full details about what's new in Phobos 1.0.0 here.

    The short version:

    1. Programmatic Configuration - Most configuration is now done programmatically, through the use of the PhobosSetup class. Phobos no longer requires any additional driver packages beyond what you configure programmatically.
    2. Easily Correlate Data between Akka.NET, ASP.NET, Kafka, SignalR, RabbitMQ, and more.
    3. Replacing Phobos.Monitoring with App.Metrics - As of Phobos 1.0.0, we now use the popular .NET monitoring library App.Metrics to handle all of the underlying monitoring and reporting for Phobos. This means that the number of monitoring services Phobos supports just got a lot larger.
    4. Significantly Improved Signal / Noise Ratio - Phobos now produces signiciantly less noise that it used to, which makes it much easier for you to understand what's going on inside your system.
    5. Simplified Pricing Model - Phobos no longer has any per-node licensing - and we've significantly reduced the price. Learn more about Phobos' new price and modeling here.

    We've also built a big, full-fledged example you can use to learn the ins and outs of Phobos instead a real Akka.NET + ASP.NET Core Kubernetes cluster.

    0.8.0 March 11 2020

    • Upgraded all interfaces to Akka.NET v1.4.1

    0.7.1 September 02 2019

    • Added support for tracing messages sent via Akka.Cluster.Tools.DistributedPubSub.

    0.7.0 August 06 2019

    • Added support for tracking stashed and unstashed messages.
    • Added support for tracing /system actors correctly.
    • Can now trace sharded entity actors instantiated via DI.
    • Upgraded to Akka.NET v1.3.14.

    Phobos.Tracing v0.6.0

    • Added Kafka Zipkin transport options.
    • Rolled back to Google.Protobuf v3.3.0.
    • Upgraded to Akka.NET v1.3.14.
    • Upgraded to Petabridge.Tracing.Zipkin v0.5.1.

    Phobos.Monitoring v0.8.0

    • Added Phobos.Monitoring.Prometheus package. It's still a work in progress but it's now being actively supported.
    • Upgraded to Akka.NET v1.3.14.

    0.6.0 September 27 2018

    Phobos v0.6.0 is a feature release for Phobos which includes support for new tracing and monitoring integrations:

    • Tracing: Jaeger - we've added first party support for the popular open source Jaeger tracing engine, which is now available via the Phobos.Tracing.Jaeger NuGet package.
    • Tracing: Microsoft Application Insights - we've added OpenTracing-compatible tracing support for Microsoft Azure's managed Application Insights service via the Phobos.Tracing.ApplicationInsights package.
    • Monitoring: Microsoft Application Insights - in addition to adding tracing support for Application Insights, we've also added corresponding monitoring support via the Phobos.Monitoring.ApplicationInsights NuGet package.

    All of these are now available and are ready for immediate use. Please click on the links above for more details on how each of these platforms work.

    Minor Updates and Changes

    • Upgraded to Phobos.Tracing v0.5.1 and Phobos.Monitoring v0.7.0;
    • Phobos.Actor: fixed a bug that caused instrumented actors created via Akka.DI.Core's System.DI().Props<TActor> methods to fail at startup. This has been fixed.
    • Phobos.Actor.Cluster: all entity actors created via Akka.Cluster.Sharding are now automatically tracked just like any other /user actor.
    • Monitoring: Removing the Timing.Start() method from the Timing class as it created issues with denormalization under the hood.
    • Monitoring: Changed the IMonitor implementation such that all denormalization of metrics now occurs through there. This will make it significantly easier for developers building custom MetricsRecorder implementations for custom Phobos.Monitoring plugins to have all of their data denormalized into both node-specific and cluster-wide metrics in the first place.

    0.5.1 August 17 2018

    • Exposed the PhobosSettings.For(ActorSystem) method so users can easily can access to the current PhobosSettings and the default PhobosActorSettings in order to easily customize Phobos configurations for specific actors using Props rather than just the akka.actor.deployment HOCON namespace.

    0.5.0 August 09 2018

    • Upgraded to Phobos.Tracing v0.4.6 and Phobos.Monitoring v0.6.1;
    • Significantly improved processing speeds for actors when Phobos is not enabled;
    • Added the ability to filter incoming messages using both blacklisting and whitelisting systems during tracing;
    • Added ability to automatically capture actor crashes and spawns inside traces - these traces will not be recorded if there is no ActiveSpan available at the time either of these events occurs.
    • All metrics recorded inside any Phobos-enabled actor are now automatically correlated across all supported dimensions, as outlined here: https://phobos.petabridge.com/articles/monitoring/index.html#global-node-specific-class-specific-and-actor-specific-metrics

    Phobos.Monitoring v0.6.0 Changes

    • Renamed MonitoringExtension to ActorMonitoring in order to make it more consistent with the ActorTracing APIs used by Phobos.Tracing.
    • Added overloads to IMonitor for passing in TimeSpan for recording all Timing operations.
    • Introduced the ActorMonitor, which can be acquired via an actor's Context.MonitorFor() extension method. The ActorMonitor automatically reports any metrics created using it using both the actor's ActorPath as well as its implementation type; when sampling is enabled both metrics will either be emitted at the same time or neither will.
    • Added the ability for any of the Metric objects, such as Gauge, Counter, or Timing, to support reporting gathered metrics under multiple names or aliases for denormalization purposes. When sampling is enabled, either all names will be included in the sample or none will.
    • Introduced the TimingScope mechansim, which can be used by Phobos users to automatically time the duration of an operation with a simple using block. Here's an example down below:
    using (Monitor.StartTiming("foo")) // creates a TimingScope
    {
        Task.Delay(100).Wait();
    } // completes and records the timing once the scope is disposed here.
    

    0.4.4 July 18 2018

    • Resolved an issue with /system tracing and monitoring causing some issues at startup when enabled. These have been resolved.
    • Upgraded to Phobos.Monitoring v0.5.6 and Phobos.Tracing v0.4.5.

    0.4.3 July 16 2018

    • Added full support for sampling across all traces; all child spans will automatically be recorded if a parent span is present, but otherwise we will use probabilistic sampling if you set the phobos.tracing.sample-rate value to anything other than the default (1.0, which means a 100% sample rate.)
    • Fixed a bug with mailbox-monitoring where we weren't properly honoring the HOCON setting. This has been fixed.
    • Fixed an issue with tracing causing deserialization errors during sampling. This has been fixed.
    • Upgraded to v0.3.1 Petabridge.Tracing.Zipkin driver.

    0.4.1 July 1 2018

    • Upgraded to OpenTracing v0.1.2 for all tracing components;
    • Upgraded to Akka.NET v1.3.8.

    0.4.0 April 27 2018

    First certified, production-ready release of Phobos.Actor and all of its counterparts.

    Phobos v0.4.0 introduces the following changes:

    1. Automatic correlation of all DeadLetter, UnhandledMessage, and LogEvent (i.e. Debug et al) messages types to the current active ISpan when tracing is enabled for the actors involved in the production of those messages.
    2. Automatic monitoring of DeadLetter, UnhandledMessage, and LogEvent (i.e. Debug et al) occurrences per ActorSystem.
    3. Automatic monitoring of the total number of received messages, actor starts, actor stops, and actor restarts per ActorSystem.
    4. Full support for ActorSelection tracing both locally and over Akka.Remote and Akka.Cluster.
    5. Full support for OpenTracing.IScopeManager and OpenTracing.IScope inside all ITracer instances used by Phobos.Tracing.
    6. Intellisense support for all Phobos packages.
    7. And lots of other small changes.

    Please report any issues you come across to our mailing list here: https://phobos.petabridge.com/

    In This Article
    Back to top Generated by DocFX