2.8.4 April 14th 2025

View Phobos 2.8.4 on Sdkbin.

Phobos 2.x Maintenance Release

2.8.3 February 24th 2025

View Phobos 2.8.3 on Sdkbin.

Phobos 2.x Maintenance Release

2.8.2 January 23rd 2025

View Phobos 2.8.2 on Sdkbin.

Phobos 2.x Maintenance Release

2.8.1 January 22nd 2025

View Phobos 2.8.1 on Sdkbin.

Phobos 2.x Maintenance Release

Phobos 2.8.1 fixes some bugs and issues with Phobos 2.8.x.

2.8.0 January 15th 2025

View Phobos 2.8.0 on Sdkbin.

Phobos 2.x Feature Release

Phobos 2.8.0 builds off what we did in Phobos 2.7.0 and makes tracing much more reliable and significantly less noisy than before.

Bug Fixes and Improvements

Lastly, we've added a convenience method to make it easier to set custom PhobosActorSettings on your Props for actors:

  var shardRegion = await ClusterSharding.Get(Sys)
    .StartAsync("entity", entityId => 
            Props.Create(() => new EntityActor(magicString))
            .WithInstrumentation(Sys, settings => settings.WithActorTypeName("Foober")),
                    ClusterShardingSettings.Create(Sys), new MsgExtractor(10));

The full method signature is:

public static Props WithInstrumentation(this Props props, ActorSystem system,
            Func<PhobosActorSettings, PhobosActorSettings> configurator)

And it should make it much simpler for users to customize actor tracing settings going forward.

2.7.0 January 8th 2025

Phobos 2.x Feature Release

Phobos 2.7.0 introduces a pair of important features and changes to help make its metrics and tracing more reliable - you don't need to do anything special other than upgrading your Phobos version to take advantage of this.

  1. Fixed: Phobos can't trace messages sent to sharded entity actors when ShardRegion is buffering - due to the way Akka.Cluster.Sharding is designed, it's been possible for traces to entity actors to be broken the first time those entities are contacted. That has been resolved in Phobos 2.7.0.
  2. Fixed: Custom actor type names are still not used in Phobos 2.6.2 - this has been fixed, so the PhobosActorSettings.WithActorTypeName(actorTypeName) value will now show up properly in the akka.actor.live metrics, whereas it did not before.
  3. Fixed: akka.actor.live gauge now includes the actor's namespace - this should make the charts clearer and easier to understand than before.
  4. Ensure that when TracingConfigBuilder.DisableAllTracing() is enabled, no traces are produced or propagated by Phobos.

2.6.2 December 20th 2024

Phobos 2.x Maintenance Release

var props = Props.Create(() => new PropsReporter())
                .WithInstrumentation(new PhobosActorSettings(true, true).WithActorTypeName("CustomName"))
                .WithStashCapacity(10);

You can now mix phobos HOCON for individual actors with PhobosActorSettings - HOCON will always override PhobosActorSettings, but any values that aren't specified in HOCON will be supplied with your custom PhobosActorSettings values instead.

In addition to this, we also added a phobos.actor-type-name property to HOCON:

akka.actor.deployment{
    /customactorname {
        phobos.actor-type-name = CustomName
    }
}

This will also set the akka.actor.typeName in OpenTelemetry tracing and metrics.

2.6.1 December 13th 2024

Phobos 2.x Feature Release

Phobos 2.6.1 includes several new features and improvements aimed at making Phobos data even more useful for Akka.NET developers.

Customize Actor Type Names

One major issue we resolved is Actor type information and F# API, where Akkling users noticed that all of their F# actors all had identical names in Phobos's metrics and traces due to the way the F# type system works differently from C#.

In 2.6.0, we've added a new attribute to Phobos - the PhobosActorName class attribute to address this; this does not address the original problem because F# uses behavior delegate and not class inheritance.

In order to resolve this, we are removing the PhobosActorName class attribute and replacing it with a new PhobosActorSettings.ActorTypeName property which will allow you to tell Phobos to use a different name for this actor type OTHER than its given CLI type that can be applied directly to Props.Deploy.

C# Sample:

public class MyActor : UntypedActor
{
     // Implementation here
}

var actorSettings = new PhobosActorSettings(trace: true, monitor: true)
    .WithActorTypeName("Foo");
var actorRef = Sys.ActorOf(Props.Create<MyActor>()
    .WithInstrumentation(actorSettings));

F# Sample:

let propsWithInstrumentation (props: Props<'T>) (trace: bool, monitor: bool, actorTypeName: string) : Props<'T> =
    let settings = PhobosActorSettings(trace, monitor).WithActorTypeName(actorTypeName)
    let newDeploy =
        match props.Deploy with
        | Some value -> value.WithInstrumentation(settings)
        | None -> Deploy.Local.WithInstrumentation(settings)

    { props with Deploy = Some newDeploy }

let behavior (m: Actor<_>) =
    let rec loop () =
        actor {
            let! msg = m.Receive()
            m.Sender() <! msg
            return! loop ()
        }
    loop ()

let customActor = spawnAnonymous system (propsWithInstrumentation (props behavior) (true, true, "Foo"))

In Phobos' automatically collected metrics, the actortype attribute will now use Foo instead of Assembly.Namespace.MyActor in C# or FunActor in F#.

In Phobos's automatically collected traces, the akka.actor.type attribute will now use Foo instead of Assembly.Namespace.MyActor in C# or FunActor in F#.

2.6.0 December 5th 2024

Phobos 2.x Feature Release

Phobos 2.6 includes several new features and improvements aimed at making Phobos data even more useful for Akka.NET developers.

Customize Actor Type Names

One major issue we resolved is Actor type information and F# API, where Akkling users noticed that all of their F# actors all had identical names in Phobos's metrics and traces due to the way the F# type system works differently from C#.

In order to resolve this, we've added a new attribute to Phobos - the PhobosActorName attribute, which will allow you to tell Phobos to use a different name for this actor type OTHER than its given CLI type.

[PhobosActorName("Foo")]
public class MyActor : UntypedActor
{
     // Implementation here
}

In Phobos' automatically collected metrics, the actortype attribute will now use Foo instead of Assembly.Namespace.MyActor.

In Phobos's automatically collected traces, the akka.actor.type attribute will now use Foo instead of Assembly.Namespace.MyActor.

akka.actor.stop trace event

If you have actor lifecycle tracing enabled (which it is, by default) you will now notice that Phobos automatically records akka.actor.stop events whenever an actor is terminated - whether that's done through PoisonPill.Instance or Context.Stop is irrelevant; Phobos will capture both.

This is designed to make it easier to trace what causes actors to shut down, and this feature played a pivotal role in Petabridge's investigation of a very tricky Heisenbug in Akka.Cluster.Sharding earlier this year. We've now made this available for all customers.

Other Fixes and Improvements

  • PhobosStash has been made public on an experimental basis - you probably won't need it and we may remove access to it in the future if customers foot-gun themselves with it. But for the time being, it's no longer internal to the Phobos SDK.
  • Akka.Persistence actors will no longer record traces unless the actors recovering / persisting initiate them first - this is an important noise control measure aimed at making sure that actors tagged with INeverTrace or INeverInstrumented don't create chatter during their interactions with Akka.Persistence.
  • Upgraded to Akka.NET v1.5.32
  • Upgraded to Akka.Hosting v1.5.32

2.5.4 June 27th 2024

Phobos 2.x Maintenance Release

  • Upgraded to Akka.NET v1.5.26
  • Metrics: we can now track all IWithTimer / IScheduledTell messages correctly, using their underlying types
  • Configuration: made it easier to completely disable metrics and / or tracing via the PhobosConfigBuilder

2.5.3 June 17th 2024

Phobos 2.x Maintenance Release

  • Upgraded Akka.NET and Akka.Hosting to v1.5.25.

2.5.2 June 11th 2024

Phobos 2.x Maintenance Release

  • Upgraded Akka.NET and Akka.Hosting to v1.5.24.
  • Fix CVE-2024-32028
    • Upgraded OpenTelemetry.Instrumentation.Http to 1.8.1
    • Upgraded OpenTelemetry.Instrumentation.AspNetCore to 1.8.1
  • Fix CVE-2018-8292
    • Upgraded Phobos.Actor.Common to 1.1.2

2.5.1 March 20th 2024

Phobos 2.x Maintenance Release

  • Filtered out messages and traces from all Akka.Streams built-in actors.
  • Upgraded to Akka.NET and Akka.Hosting v1.5.18.

2.5.0 February 13th 2024

Phobos 2.5 Minor Release

Phobos 2.5.0 is, once again, a massive performance upgrade over Phobos 2.4.5 and earlier. 41% faster and uses 83% less memory on the critical path when both tracing and metrics are enabled, in fact.

Performance Improvements

We followed more of the same techniques we described in "How We Made Phobos 2.4's OpenTelemetry Usage 62% Faster" and were able to move major heavy sources of allocations outside of Phobos' performance critical path.

Our throughput in v2.4.5:

Metric Units / s Max / s Average / s Min / s StdDev / s
TotalCollections [Gen0] collections 83.31 81.55 80.38 0.88
TotalCollections [Gen1] collections 5.80 5.08 4.27 0.49
TotalCollections [Gen2] collections 2.17 1.97 1.42 0.31
[Counter] MessageReceived operations 2,173,276.34 2,131,567.21 2,096,899.11 23,842.54

Here's what our performance looked like in v2.5, in terms of total instrumented throughput:

Metric Units / s Max / s Average / s Min / s StdDev / s
TotalCollections [Gen0] collections 20.43 19.65 18.86 0.48
TotalCollections [Gen1] collections 7.04 5.83 4.96 0.65
TotalCollections [Gen2] collections 3.06 2.68 1.98 0.47
[Counter] MessageReceived operations 3,064,477.84 2,994,416.62 2,891,043.79 53,773.16

In terms of total throughput, we're looking at an average improvement of 41% efficiency. But when it comes to hot-path memory consumption, the numbers are even more incredible (you can also tell by looking at the stark difference in GC activity between 2.4.5 and 2.5.0):


BenchmarkDotNet v0.13.10, Windows 11 (10.0.22621.3007/22H2/2022Update/SunValley2)
12th Gen Intel Core i7-1260P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 8.0.101
  [Host]     : .NET 7.0.15 (7.0.1523.57226), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.15 (7.0.1523.57226), X64 RyuJIT AVX2
Method MessageKind Mean Error StdDev Req/sec Gen0 Allocated
CreateNewRootSpan Primitive 139.7 ns 2.10 ns 1.86 ns 7,157,539.53 0.0305 288 B
CreateChildSpan Primitive 142.2 ns 2.35 ns 2.19 ns 7,030,112.29 0.0305 288 B
CreateChildSpanWithBaggage Primitive 147.8 ns 2.28 ns 2.13 ns 6,766,701.91 0.0305 288 B
CreateNewRootSpan Class 139.1 ns 2.34 ns 2.18 ns 7,191,051.43 0.0305 288 B
CreateChildSpan Class 141.8 ns 2.03 ns 1.90 ns 7,054,363.87 0.0305 288 B
CreateChildSpanWithBaggage Class 148.6 ns 1.44 ns 1.20 ns 6,731,252.60 0.0305 288 B
CreateNewRootSpan Record 141.1 ns 2.47 ns 2.43 ns 7,086,392.57 0.0305 288 B
CreateChildSpan Record 141.0 ns 1.68 ns 1.57 ns 7,090,756.01 0.0305 288 B
CreateChildSpanWithBaggage Record 149.7 ns 2.62 ns 2.45 ns 6,678,880.92 0.0305 288 B

in 2.4.5 Phobos allocated 288 bytes per span (in this fixed-size test case - real-world values could be much larger.)

But in 2.5.0...


BenchmarkDotNet v0.13.10, Windows 11 (10.0.22621.3007/22H2/2022Update/SunValley2)
12th Gen Intel Core i7-1260P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 8.0.101
  [Host]     : .NET 7.0.15 (7.0.1523.57226), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.15 (7.0.1523.57226), X64 RyuJIT AVX2
Method MessageKind Mean Error StdDev Req/sec Gen0 Allocated
CreateNewRootSpan Primitive 59.97 ns 0.637 ns 0.565 ns 16,676,180.98 0.0050 48 B
CreateChildSpan Primitive 67.24 ns 0.943 ns 0.836 ns 14,871,155.56 0.0050 48 B
CreateChildSpanWithBaggage Primitive 77.87 ns 0.871 ns 0.814 ns 12,842,151.57 0.0050 48 B
CreateNewRootSpan Class 59.70 ns 0.695 ns 0.616 ns 16,749,521.86 0.0050 48 B
CreateChildSpan Class 63.69 ns 0.715 ns 0.669 ns 15,701,818.74 0.0050 48 B
CreateChildSpanWithBaggage Class 71.53 ns 0.650 ns 0.576 ns 13,979,808.57 0.0050 48 B
CreateNewRootSpan Record 56.55 ns 0.258 ns 0.229 ns 17,684,262.41 0.0051 48 B
CreateChildSpan Record 64.35 ns 0.369 ns 0.288 ns 15,538,845.63 0.0050 48 B
CreateChildSpanWithBaggage Record 72.41 ns 0.655 ns 0.613 ns 13,809,634.69 0.0050 48 B

That same test case now only allocates 48 bytes per span - 83% less memory than we used before.

Reduced Noise from IStash

We've also made a significant noise control improvement: IStash no longer produces separate operations that appear in traces any longer.

The akka.msg.stash and akka.msg.unstash no longer appear as their own distinct Activity / TelemetrySpan operations during tracing; rather, when stashing and unstashing occur in 2.5 they will be appended as akka.msg.stash and akka.msg.unstash events on the original akka.msg.recv <messageType> operation - and that operation will now remain open until the stashed message is finally unstashed and processed.

2.4.5 January 10th 2024

Phobos 2.x Revision Release

2.4.4 December 6th 2023

Phobos 2.x Revision Release

  • Fixes incorrectly calculated value for message receive latency metrics

2.4.3 November 29th 2023

Phobos 2.x Revision Release

  • Upgraded to Akka.NET v1.5.14
  • Updated all IWithTrace messages that Phobos uses to propagate TelemetrySpans between actors to implement Akka.NET's IWrappedMessage interface - this will cause these messages to properly log their contents when sent to DeadLetters and will also ensure that serialization is handled correctly when akka.actor.serialize-messages=on (once https://github.com/akkadotnet/akka.net/pull/7010 is merged)

Major Performance Improvements

We've made major performance improvements to all areas of Phobos in 2.4.3. Here's what our in-memory PingPong looks like between versions:

2.4.2


BenchmarkDotNet v0.13.9+228a464e8be6c580ad9408e98f18813f6407fb5a, Windows 11 (10.0.22621.2715/22H2/2022Update/SunValley2)
AMD Ryzen 7 6800H with Radeon Graphics, 1 CPU, 16 logical and 8 physical cores
.NET SDK 8.0.100
  [Host]     : .NET 7.0.14 (7.0.1423.51910), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.14 (7.0.1423.51910), X64 RyuJIT AVX2
Method MonitoringEnabled TracingEnabled Mean Error StdDev Median Req/sec
Actor_ping_pong_single_pair_in_memory False False 264.6 ns 4.37 ns 4.09 ns 264.4 ns 3,779,726.78
Actor_ping_pong_single_pair_in_memory False True 1,027.5 ns 8.65 ns 8.09 ns 1,028.9 ns 973,214.83
Actor_ping_pong_single_pair_in_memory True False 345.1 ns 6.89 ns 16.77 ns 335.5 ns 2,897,787.55
Actor_ping_pong_single_pair_in_memory True True 1,066.8 ns 17.63 ns 16.49 ns 1,067.0 ns 937,377.97

versus

2.4.3


BenchmarkDotNet v0.13.10, Windows 11 (10.0.22621.2715/22H2/2022Update/SunValley2)
AMD Ryzen 7 6800H with Radeon Graphics, 1 CPU, 16 logical and 8 physical cores
.NET SDK 8.0.100
  [Host]     : .NET 7.0.14 (7.0.1423.51910), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.14 (7.0.1423.51910), X64 RyuJIT AVX2
Method MonitoringEnabled TracingEnabled Mean Error StdDev Req/sec Gen0 Allocated
Actor_ping_pong_single_pair_in_memory False False 222.1 ns 3.81 ns 3.56 ns 4,502,145.93 0.0055 48 B
Actor_ping_pong_single_pair_in_memory False True 892.4 ns 16.83 ns 14.92 ns 1,120,553.02 0.0781 652 B
Actor_ping_pong_single_pair_in_memory True False 383.2 ns 2.46 ns 2.30 ns 2,609,607.03 0.0055 48 B
Actor_ping_pong_single_pair_in_memory True True 923.8 ns 13.16 ns 11.66 ns 1,082,481.37 0.0781 652 B

This is a 15.4% improvement when both tracing and metrics are enabled.

We've also made Phobos' serializer at least twice as fast as it was before; the default filtering engine is 4x faster; and we've made Baggage handling much more efficient inside Phobos than it was previously.

If we compare some of our end-to-end Akka.Cluster.Sharding benchmarks over the network, the results speak for themselves:

2.4.2

Method StateMode Mean Error StdDev Median Req/sec
StreamingToLocalEntity Persistence 2,293.1 ns 45.71 ns 112.98 ns 2,253.6 ns 436,097.67
StreamingToRemoteEntity Persistence 25,087.4 ns 494.66 ns 929.09 ns 25,139.2 ns 39,860.64
StreamingToLocalEntity DData 2,731.5 ns 188.06 ns 554.50 ns 2,501.9 ns 366,094.21
StreamingToRemoteEntity DData 25,517.2 ns 503.23 ns 798.17 ns 25,546.8 ns 39,189.19

2.4.3

Method StateMode Mean Error StdDev Median Req/sec
StreamingToLocalEntity Persistence 2,239.2 ns 154.65 ns 455.99 ns 1,986.8 ns 446,597.04
StreamingToRemoteEntity Persistence 24,179.3 ns 472.33 ns 735.35 ns 24,319.6 ns 41,357.63
StreamingToLocalEntity DData 2,227.6 ns 143.00 ns 421.64 ns 1,982.1 ns 448,915.08
StreamingToRemoteEntity DData 24,615.0 ns 486.97 ns 813.62 ns 24,651.3 ns 40,625.66

There's a lot of things that touch this benchmark (Akka.Remote networking layer, serialization, dispatching, Phobos, etc) - but the overall impact of remote end to end tracing is improved by 4% solely through efficiency improvements in Phobos.

2.4.2 September 19th 2023

Phobos 2.x Revision Release

  • Upgraded to Akka.NET v1.5.13
  • Using the new IScheduledTellMsg API in Akka.NET v1.5.13+, Phobos will now automatically filter out all scheduled messages from tracing by default as a noise reduction mechanism.

2.4.1 July 20th 2023

Phobos 2.x Revision Release

  • Fixes a bug introduced by Phobos 2.4.0 where phobos.monitoring.monitor-mailbox-depth = on caused a lot of new dimensions to be added to this metric by accident. We've fixed this behavior to work the same as it did in prior versions of Phobos.

2.4.0 July 18th 2023

Phobos 2.4 Minor Release

Phobos 2.4.0 is a massive performance upgrade over Phobos 2.3.1 and earlier. 62% faster when both tracing and metrics are enabled, in fact.

Benchmarks taken with the following hardware specs:


BenchmarkDotNet v0.13.6, Windows 11 (10.0.22621.1992/22H2/2022Update/SunValley2)
12th Gen Intel Core i7-1260P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 7.0.302
  [Host]     : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2

Before

Per-second Totals

Metric Units / s Max / s Average / s Min / s StdDev / s
TotalCollections [Gen0] collections 42.11 38.22 36.28 2.32
TotalCollections [Gen1] collections 2.59 2.12 1.81 0.31
TotalCollections [Gen2] collections 1.11 0.79 0.73 0.11
TotalBytesAllocated bytes 44,022,308.64 39,794,803.00 36,425,137.52 2,596,475.60
[Counter] MessageReceived operations 1,263,277.84 1,145,856.34 1,088,397.58 70,098.66

~1.15 million messages per second.

After

Per-second Totals

Metric Units / s Max / s Average / s Min / s StdDev / s
TotalCollections [Gen0] collections 72.71 71.27 70.39 0.67
TotalCollections [Gen1] collections 4.36 3.91 3.68 0.28
TotalCollections [Gen2] collections 1.26 1.24 1.22 0.01
TotalBytesAllocated bytes 61,527,405.30 60,372,472.15 59,632,699.32 555,785.76
[Counter] MessageReceived operations 1,896,656.50 1,859,265.18 1,836,387.09 17,542.45

~1.86 million messages per second. You can see complete, full numbers for Phobos across all of its various configuration settings here: "Phobos Performance Impact."

Other Changes

  • Upgraded to Akka.NET v1.5.9 and Akka.Hosting v1.5.8.1
  • Upgraded to OpenTelemetry v1.5.1
  • Restored lost functionality from Phobos 1.x - all messages decorated with INeverTrace are never traced; messages decorated with INeverMonitor are not monitored; and messages decorated with INeverInstrumented (implements both of the previous interfaces) are never traced or monitored under any circumstances.

2.3.0 April 27 2023

Phobos 2.3 Minor Release

  • Upgraded OpenTelemetry from 1.3.2 to 1.4.0
  • Upgraded OpenTelemetry.Extensions.Hosting on all samples from 1.0.0-rc9.4 to 1.4.0

We upgraded all of our samples to use the new OpenTelemetry.Extensions.Hosting v1.4.0 API and we recommend that you update to this version.

  • For upgrading from Phobos 1.x to Phobos 2.3, follow the instructions in this upgrade page
  • For upgrading from Phobos 2.x to Phobos 2.3 and the new recommended OpenTelemetry.Extensions.Hosting API, follow the instruction in this upgrade page

1.6.0 April 20 2023

Phobos 1.x Minor Release

Minor Version Release of Phobos 1.x

This version of Phobos 1.x supports Akka.NET v1.5.

2.2.0 March 02 2023

Phobos 2.2 Minor Release

Phobos 2.2 targets Akka.NET v1.5.0 and now dual targets both .NET Standard 2.0 and .NET 6 in order to take advantage of native performance improvements in Akka.NET v1.5.

2.1.0 January 16 2023

Phobos 2.1 Minor Release

Phobos 2.1 is a minor release that has been in the works for a long time and introduces several major features and improvements to the existing Phobos 2.0 experience.

Akka.Cluster.Sharding Metrics and Dashboards

Issue: https://github.com/petabridge/phobos-issues/issues/72

One of the things we added support for in Phobos 2.1 is automatically tracking Akka.Cluster.Sharding metrics for each ShardRegion for each node capable of hosting any shards - these numbers can be aggregated together to form a complete picture of total shard allocation by type and node throughout your cluster.

The new built-in metrics are:

  • akka.cluster.sharding.shards - keeps track of the number of shards per node per entity type.
  • akka.cluster.sharding.entities - keeps track of the number of entities per node per entity type.

More Accurate "Live Actor" Counts

Phobos 2.1 takes advantage of a new actor telemetry feature introduced in Akka.NET v1.4.47, which allows us to use an OpenTelemetry ObservableGauge to accurately track the total number of live actors by type and node throughout each of your ActorSystems.

Important

In order to use this feature, you will need to set akka.actor.telemetry.enabled = on (it's off by default.)

If you use Phobos.Hosting to configure Phobos, this setting will be automatically enabled for you.

This value is now tracked via the akka.actor.live metric and will show up alongside the normal akka.actor.created|stopped|restarted counters. We've already incorporated akka.actor.live into our Phobos dashboards.

Updated All Phobos Dashboards

We've incorporated all Phobos dashboards to incorporate the following new metric types:

  • akka.cluster.sharding.shards - keeps track of the number of shards per node per entity type.
  • akka.cluster.sharding.entities - keeps track of the number of entities per node per entity type.
  • akka.actor.live - keeps more accurate track of the number of live actors per node by type.

Please see "Phobos Dashboards" for the most recent versions of each of these.

Akka.Cluster.Sharding Trace Compression

One of the largest feature areas we've worked on for Phobos 2.1 is curtailing the amount of noise produced by Phobos tracing, especially in regards to features like Akka.Cluster.Sharding - where each message to one of your actors produces 2-3 spans recorded by built-in system actors.

For starters, we now have the ability to compress Cluster.Sharding traces via an OpenTelemetry BaseProcessor<Activity> - and this can be enabled when you're subscribing to Phobos telemetry in your OTel pipeline:

services.AddOpenTelemetryTracing(builder =>
{
builder
    .SetResourceBuilder(resource)
    .AddPhobosInstrumentation(compressShardTraces:true) // eliminate sharding infrastructure from traces
    .AddHttpClientInstrumentation()
    .AddAspNetCoreInstrumentation(options =>
    {
        options.Filter = context => !context.
            Request.Path.StartsWithSegments("/metrics");
    })
    .SetSampler(new TraceIdRatioBasedSampler(1.0d))
    .AddJaegerExporter(opt =>
    {
        opt.AgentHost = Environment.GetEnvironmentVariable(JaegerAgentHostEnvironmentVar) ?? "localhost";
    });
});

Calling .AddPhobosInstrumentation(compressShardTraces:true) will enable trace compression for sharding. This value is set to false by default.

In addition to these changes, we've also filtered out all internal messages and Akka.DistributedData messages by default.

Ask<T> Tracing Improvements

We've made many significant improvements to Ask<T> tracing in Phobos 2.1:

  • Ask<T> tracing can now be disabled via the fluent C# APIs or via phobos.tracing.trace-ask = off;
  • Ask<T> tracing can now be filtered using the same ITraceFilter and other filter rules as regular akka.actor.recv statements;
  • Ask<T> tracing has been restructured such that we now accurately capture both the input and output types separately - additionally, we are much more accurate in tracking start and stop times than we were previously.
  • Fixed: Ask<T> tracing does not properly follow Tracer.ActiveSpan

To disable Ask<T> tracing via Phobos.Hosting, use the following:

collection.AddAkka("LocalSys", (builder, sp) =>
{
    builder.WithPhobos(AkkaRunMode.Local, 
        configBuilder => configBuilder.WithTracing(t => t.SetTraceAsk(false)));
    AddTestActors(builder);
});

Other General Improvements

2.1.0-beta1 August 2 2022

Phobos 2.x BETA Release

Phobos 2.1.0-beta1 features some significant noise control improvements for tracing:

To disable Ask<T> tracing:

phobos.tracing.trace-ask = off

or

collection.AddAkka("LocalSys", (builder, sp) =>
{
    builder.WithPhobos(AkkaRunMode.Local, 
        configBuilder => configBuilder.WithTracing(t => t.SetTraceAsk(false)));
    AddTestActors(builder);
});

2.0.6 June 10 2022

Phobos 2.x Revision Release

2.0.5 June 06 2022

Phobos 2.x Revision Release

2.0.4 June 02 2022

Phobos 2.x Revision Release

  • Fixed: Custom span creation with Phobos 2.0.3-beta1 - fixed a bug when running Phobos with phobos.tracing.create-trace-upon-receive = off that allowed Phobos to continue creating spans when there was no active trace context available. This has been fixed and the setting now works as intended.
  • Fixed: Lots of noise in logging when using cluster sharding - introduced a new setting to Phobos configuration, phobos.tracing.log-message-events = on, which when set to off will disable the creation of waiting and message events from being added to each automatically-created akka.msg.recv TelemetrySpan. This will greatly reduce the data footprint for each TelemetrySpan produced by Phobos and will help reduce the amount of noise generated on platforms like Elastic APM.
  • phobos.tracing.log-message-events is now set to off by default for Akka.Cluster.Sharding and Akka.Cluster.Tools.PubSub actors.
  • Upgraded to Akka.NET v1.4.39

2.0.3 May 26th 2022

Phobos 2.x Revision Release

Fixed: Serializer exception when using Phobos and Akka.Hosting - this issue was caused by actor lifecycle tracing leaking Tracer.CurrentSpan instances across threads as a result of how actor spawning works on most systems. Specifically, when Akka.Cluster.Sharding was enabled this leaky mechanism made it possible for other /system actors responsible for clustering to suddenly have access to active traces, which in turn caused them to wrap built in system messages such as cluster heartbeats with Phobos' tracing envelopes.

We've fixed this in two separate ways:

  1. Actor lifecycle traces now longer use the Tracer.CurrentSpan / System.Diagnostics.Activity.Current property, so the source of the leak has been removed;
  2. Phobos.Actor now takes a dependency on Akka.Cluster and explicitly excludes all Akka.Cluster.IClusterMessage events from any kind of consideration during tracing.

2.0.3-beta1 May 12th 2022

Phobos 2.x BETA Release

This is a BETA release of a new proposed feature for Phobos 2.0 - "macro" tracing: the ability to disable automatic akka.msg.recv span creation but still receive the SpanContext from incoming messages and still propagate SpanContext to other actors via TelemetrySpan / System.Diagnostic.Activity instances you create yourself.

Disabling Automatic Span Creation

You can disable automatic span creation for the entire ActorSystem via the following HOCON:

phobos.tracing.create-trace-upon-receive = off

We don't recommend doing this as it's a potentially destructive action. Instead, it's better to disable automatic span creation in areas where you know the system is busy.

akka.actor.deployment{
    /manual1{
        phobos.tracing.create-trace-upon-receive = off
        phobos.tracing.propagate-settings-to-children = on
    }
}

This will allow you to disable automatic span creation for the entire hierarchy of actors beneath /user/manual1.

You can also accomplish the same thing via C# or F# via the Props extension methods introduced by Phobos:

var myManualSpanActor = Sys.ActorOf(Props.Create(() => new ManualTraceCreator())
    .WithInstrumentation(
        PhobosSettings.For(Sys).DefaultUserActorSettings
            .WithCreateTraceUponReceive(false)
            .WithPropagateToChildren(true)), "manual2");

This will work identically to the HOCON above.

When these settings are enabled, the affected actors will still have tracing enabled and will propagate any active trace data to other actors via IActorRef.Tell but will not automatically create any additional akka.msg.recv spans on their own any longer.

Manually Creating Spans

So in order to create useful traces inside these areas, you can now access the IPhobosActorContext.UsableContext property - which contains a SpanContext? that can be used in the manual creation of OpenTelemetry spans:

public ManualTraceCreator()
{
    Receive<GetActiveSpan>(_ =>
    {
        // should be default(TelemetrySpan) when running with
        // phobos.tracing.create-trace-upon-receive = off
        Sender.Tell(new SpanResponse(Context.GetInstrumentation().ActiveSpan));
    });

    Receive<MyMsg>(m =>
    {
        using (var mySpan = Context.GetInstrumentation().Tracer.StartActiveSpan("MyMsg", SpanKind.Consumer,
                   Context.GetInstrumentation().UsableContext ?? default))
        {
            _logging.Info("My event");
            Sender.Tell(new MyReply("reply"));
        }
    });
}

This will allow the Phobos user full control over what appears in their spans and which actors / messages can produce spans. Span propagation still works automatically inside Phobos with automatic span creation disabled - but now you're responsible for creating your own spans. This should significantly reduce the amount of noise produced by the tracing system - we're going to experiment using this feature inside our planned Akka.Streams support.

2.0.2 April 19th 2022

Phobos 2.x Release

This release marks the official "release to market" of Phobos 2.0 - which includes the following changes (from 1.x and earlier betas of 2.x):

  1. Migrated from App.Metrics and OpenTracing to OpenTelemetry - the future APM platform for all of .NET;
  2. End-to-end message processing latency tracking in both Phobos metrics and traces;
  3. Re-engineered the trace filtering system to be much easier to use and effective at reducing noise; and
  4. Integrated with Akka.Hosting, a new HOCONless way of configuring, running, and integrating Akka.NET with the other parts of your .NET applications. See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here. This is accomplished through the new Phobos.Hosting NuGet package introduced in Phobos 2.0.

These changes all represent a major step forward for Phobos and Akka.NET telemetry.

For full details click here to read "What's New in Phobos 2.0?"

Phobos 1.x Support Lifetime Per our recent announcement, we will continue to support Phobos 1.x and 2.x concurrently. Phobos 1.x will exit support on December 31st, 2023.

1.5.1 April 12th 2022

Minor Version Release of Phobos 1.x

Phobos 1.5 is the next minor version release of Phobos. It contains two major, important changes:

  1. Overhaul of the Trace Filtering System and
  2. Integrated with Akka.Hosting, a new HOCONless way of configuring, running, and integrating Akka.NET with the other parts of your .NET applications. See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here. This is accomplished through the new Phobos.Hosting NuGet package introduced in Phobos 1.5.

And we've also fixed the following bugs:

  1. https://github.com/petabridge/phobos-issues/issues/54
  2. https://github.com/petabridge/phobos-issues/issues/53

New and Improved Trace Filtering

This release contains some breaking changes which we've introduced intentionally in order to get users concerned about noise control from the Phobos tracing system to migrate over to a new set of APIs that perform the job in a much more coherent fashion that what we have currently in Phobos 1.4.

You can see some examples of end-user negative experience reports with the current v1.x and 2.x trace filtering implementations here:

There have been many more like these over the past couple of years - separating noise from useful signals has been a persistent challenge with Phobos tracing and the changes we're introducing in Phobos v1.5 and 2.0 address this with a small new API:

/// <summary>
/// Each <see cref="ActorSystem"/> can implement exactly one of these to control
/// the decision-making around which traces to include and which ones to exclude.
///
/// When this type is configured, it will replace all of the <see cref="IFilterRule"/>s
/// that have been previously defined.
/// </summary>
/// <seealso cref="FilterDecision"/>
public interface ITraceFilter
{
    /// <summary>
    /// Evaluates a message processed by an actor and determines whether or not to include it in the current
    /// or a new trace.
    /// </summary>
    /// <param name="message">The message currently being processed by an actor.</param>
    /// <param name="alreadyInTrace">Whether or not this message is already part of an active trace.</param>
    /// <returns><c>true</c> if we are going to trace the message, <c>false</c> otherwise.</returns>
    /// <remarks>
    /// Should return <c>true</c> unless you are explicitly trying to filter a given message out of your trace.
    /// </remarks>
    bool ShouldTraceMessage(object message, bool alreadyInTrace);
}

This is the ITraceFilter - and there is exactly one of these for your entire ActorSystem. It allows you to decide whether or not to trace an actor's message processing activity based on:

  • Message type;
  • Message content; and
  • Whether or not we are inside an active trace or not.

This API is astonishingly succinct, much clearer than the previous TracingConfigBuilder API from before, extremely performant, and very effective at limiting noise inside your Phobos-enabled Akka.NET applications.

An example from one of our own stress testing applications:

public sealed class ScenarioTraceFilter : ITraceFilter
{
    public bool ShouldTraceMessage(object message, bool alreadyInTrace)
    {
        switch (message)
        {
            /*
             * Only start new traces for these commands
             */
            case IScenarioProtocolMessage _:
            case Petabridge.Cmd.Command _:
            case Petabridge.Cmd.CommandResponse _:
                return true;
            default:
                // otherwise, only allow messages already in trace to be included
                return alreadyInTrace;
        }
    }
}

This class allows new traces to only be started handful of Petabridge.Cmd types and any messages and our custom IScenarioProtocol messages. Any additional messages will not be included in a trace unless there is already an active trace present.

You can add this ITraceFilter to your applications like so:

var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
    .WithMetrics(m =>
        m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
    .WithTracing(t => t.SetTracer(tracer)
        .SetTraceFilter(new ScenarioTraceFilter())
    )) // binds Phobos to same tracer as ASP.NET Core
.WithSetup(BootstrapSetup.Create()
    .WithActorRefProvider(PhobosProviderSelection
        .Local)); // last line activates Phobos inside Akka.NET

Phobos still supports the old TracingConfigBuilder pattern and the WithIncludeMessage<T> rules - however, all of those rules will be overidden and replaced when the ITraceFilter is used instead.

We have broken the old TracingConfigBuilder method signatures in order to alert users to this important change.

You can see the full details here: "Filtering Tracing Data with Phobos"

Akka.Hosting Integration and Simplified Configuration

Phobos can now be fully configured without any HOCON or mangling of Setup classes using the brand new Phobos.Hosting NuGet package, built on top of Akka.Hosting.

Tip

Unfamiliar with Akka.Hosting? See the "Introduction to Akka.Hosting - HOCONless, "Pit of Success" Akka.NET Runtime and Configuration" video with details here.

See Configuring Phobos 2.0 - Using Phobos.Hosting for full details.

2.0.0-beta4 and 1.5.0-beta1 March 17th 2022

Major Update to Phobos 2.x and 1.x

This release contains some breaking changes which we've introduced intentionally in order to get users concerned about noise control from the Phobos tracing system to migrate over to a new set of APIs that perform the job in a much more coherent fashion that what we have currently in Phobos 1.4 / 2.0.0-beta3 and newer.

You can see some examples of end-user negative experience reports with the current v1.x and 2.x trace filtering implementations here:

There have been many more like these over the past couple of years - separating noise from useful signals has been a persistent challenge with Phobos tracing and the changes we're introducing in Phobos v1.5 and 2.0-beta4 address this with a small new API:

/// <summary>
/// Each <see cref="ActorSystem"/> can implement exactly one of these to control
/// the decision-making around which traces to include and which ones to exclude.
///
/// When this type is configured, it will replace all of the <see cref="IFilterRule"/>s
/// that have been previously defined.
/// </summary>
/// <seealso cref="FilterDecision"/>
public interface ITraceFilter
{
    /// <summary>
    /// Evaluates a message processed by an actor and determines whether or not to include it in the current
    /// or a new trace.
    /// </summary>
    /// <param name="message">The message currently being processed by an actor.</param>
    /// <param name="alreadyInTrace">Whether or not this message is already part of an active trace.</param>
    /// <returns><c>true</c> if we are going to trace the message, <c>false</c> otherwise.</returns>
    /// <remarks>
    /// Should return <c>true</c> unless you are explicitly trying to filter a given message out of your trace.
    /// </remarks>
    bool ShouldTraceMessage(object message, bool alreadyInTrace);
}

This is the ITraceFilter - and there is exactly one of these for your entire ActorSystem. It allows you to decide whether or not to trace an actor's message processing activity based on:

  • Message type;
  • Message content; and
  • Whether or not we are inside an active trace or not.

This API is astonishingly succinct, much clearer than the previous TracingConfigBuilder API from before, extremely performant, and very effective at limiting noise inside your Phobos-enabled Akka.NET applications.

An example from one of our own stress testing applications:

public sealed class ScenarioTraceFilter : ITraceFilter
{
    public bool ShouldTraceMessage(object message, bool alreadyInTrace)
    {
        switch (message)
        {
            /*
             * Only start new traces for these commands
             */
            case IScenarioProtocolMessage _:
            case Petabridge.Cmd.Command _:
            case Petabridge.Cmd.CommandResponse _:
                return true;
            default:
                // otherwise, only allow messages already in trace to be included
                return alreadyInTrace;
        }
    }
}

This class allows new traces to only be started handful of Petabridge.Cmd types and any messages and our custom IScenarioProtocol messages. Any additional messages will not be included in a trace unless there is already an active trace present.

You can add this ITraceFilter to your applications like so:

var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
    .WithMetrics(m =>
        m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
    .WithTracing(t => t.SetTracer(tracer)
        .SetTraceFilter(new ScenarioTraceFilter())
    )) // binds Phobos to same tracer as ASP.NET Core
.WithSetup(BootstrapSetup.Create()
    .WithActorRefProvider(PhobosProviderSelection
        .Local)); // last line activates Phobos inside Akka.NET

Phobos still supports the old TracingConfigBuilder pattern and the WithIncludeMessage<T> rules - however, all of those rules will be overidden and replaced when the ITraceFilter is used instead.

We have broken the old TracingConfigBuilder method signatures in order to alert users to this important change - however, we are going to keep these changes in a beta state for a short period in order to gather feedback from affected users. If you are concerned about trace noise inside your application, please try this beta and report on your experience on the Phobos Issue Tracker.

1.4.2 March 8th 2022

Maintenance Release for Phobos 1.4

To have tracing in Phobos 1.4 work with PipeTo(), you will have to use PipeTo(recipient, true) and call the Ask() method inside an async method even when the Ask() method is not awaited.

private class PipeToActor: ReceiveActor
{
    public PipeToActor(bool useAsync, IActorRef otherActor)
    {
        ReceiveAsync<int>(o =>
        {
            // Need to make sure that async context is preserved inside PipeTo
            otherActor.Ask("message").PipeTo(Self, true, Sender); 
            return Task.CompletedTask;
        });
    }
}
  • The boolean parameter in the PipeTo(IActorRef, true, IActorRef) method sets ConfigureAwait(true) inside the PipeTo() operation - and this allows us to preserve the current AsyncLocal<ISpan> context.
  • The ReceiveAsync method is also needed for proper async context propagation, which is what's needed in order to ensure that the ISpan created by the PipeTo method is properly correlated to the one created by the Ask method.

2.0.0-beta2 February 1st 2022

Maintenance Release for Phobos 2.0-beta1

  • Implemented a change where when phobos.tracing.trace-actor-lifecycle = off both parent actors and child actors will no longer create akka.actor.spawn operations.
  • Child actors will no longer log akka.actor.started operations if their Props have been updated via WithInstrumentation to disable actor lifecycle tracing.

1.4.1 February 1st 2022

Maintenance Release for Phobos 1.4

  • Implemented a change where when phobos.tracing.trace-actor-lifecycle = off both parent actors and child actors will no longer create akka.actor.spawn operations.
  • Child actors will no longer log akka.actor.started operations if their Props have been updated via WithInstrumentation to disable actor lifecycle tracing.

2.0.0-beta1 January 26th 2022

MAJOR RELEASE OF PHOBOS

In Phobos 2.0 we have migrated all of our tracing and monitoring capabilities to use OpenTelemetry.

As we mentioned in our recent blog post about Phobos 2.0 and its relationship with OpenTelemetry, we did this for the following reasons:

  • OpenTelemetry is now the official APM solution for the entire .NET platform, including other popular frameworks such as ASP.NET and Entity Framework;
  • All major APM vendors and OSS project such as Prometheus, Jaeger, DataDog, AWS, GCP, Azure, Elastic, and so on are all committed to supporting OpenTelemetry going forward and they implement their own first-party exporters that transmit data to those platforms;
  • The performance of the OpenTelemetry.Trace and OpenTelemetry.Metrics implementations, which Phobos uses, are identical no matter which combination of exporters you adopt; and
  • OpenTelemetry is much more decoupled and extensible than any of the previous solutions in market.

We have replaced all of our OpenTracing and App.Metrics dependencies with 2.x going forward.

Installation and Configuration To use Phobos 2.0.0-beta* in your solution, all you need to do is:

  1. Install the appropriate Phobos.Actor.* package into your application using the 2.0.0-beta* version;
  2. Install the appropriate OpenTelemetry.Exporter.* packages you want to use in combination with your preferred APM tools; and
  3. Configure your IServiceCollection with the following (adjust as needed):
// This method gets called by the runtime. Use this method to add services to the container.
// For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
public void ConfigureServices(IServiceCollection services)
{
    // Prometheus exporter won't work without this
    services.AddControllers();

    var resource = ResourceBuilder.CreateDefault()
        .AddService(Assembly.GetEntryAssembly().GetName().Name, serviceInstanceId: $"{Dns.GetHostName()}");

    // enables OpenTelemetry for ASP.NET / .NET Core
    services.AddOpenTelemetryTracing(builder =>
    {
        builder
            .SetResourceBuilder(resource)
            .AddPhobosInstrumentation()
            .AddSource("Petabridge.Phobos.Web")
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddJaegerExporter(opt =>
            {
                opt.AgentHost = Environment.GetEnvironmentVariable(JaegerAgentHostEnvironmentVar);
            });
    });

    services.AddOpenTelemetryMetrics(builder =>
    {
        builder
            .SetResourceBuilder(resource)
            .AddPhobosInstrumentation()
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddPrometheusExporter(opt =>
            {
            });
    });

    // sets up Akka.NET
    ConfigureAkka(services);
}

The .AddPhobosInstrumentation() line will ensure that OTel picks up all of the signals produced by Phobos - and you need to call this line for both metrics and tracing!

Finally, you need to configure your ActorSystem to use Phobos and to leverage the TracerProvider and MeterProviders from OpenTelemetry:

public static void ConfigureAkka(IServiceCollection services)
{
    services.AddSingleton(sp =>
    {
        var metrics = sp.GetRequiredService<MeterProvider>();
        var tracer = sp.GetRequiredService<TracerProvider>();

        var config = ConfigurationFactory.ParseString(File.ReadAllText("app.conf"))
            .BootstrapFromDocker()
            .UseSerilog();

        var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
                .WithMetrics(m =>
                    m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
                .WithTracing(t => t.SetTracer(tracer))) // binds Phobos to same tracer as ASP.NET Core
            .WithSetup(BootstrapSetup.Create()
                .WithConfig(config) // passes in the HOCON for Akka.NET to the ActorSystem
                .WithActorRefProvider(PhobosProviderSelection
                    .Cluster)); // last line activates Phobos inside Akka.NET

        var sys = ActorSystem.Create("ClusterSys", phobosSetup);

        // create actor "container" and bind it to DI, so it can be used by ASP.NET Core
        return new AkkaActors(sys);
    });

    // this will manage Akka.NET lifecycle
    services.AddHostedService<AkkaService>();
}

This is fundamentally not all that different from how Phobos 1.x works.

We will update our official documentation and website with more Phobos 2.0-specific instructions as our releases mature, but for the time being please don't hesitate to submit an issue on the Phobos Issue Tracker if you run into trouble!

Phobos 1.x Support Lifetime Per our recent announcement, we will continue to support Phobos 1.x and 2.x concurrently. Phobos 1.x will exit support on December 31st, 2023.

1.4.0 January 19th 2022

New Minor Version Release for Phobos

Phobos 1.4 is a minor version release that introduces some exciting new capabilities for Akka.NET users, primarily:

  • All actor lifecycle activity, i.e. everything that occurs during PreStart, PostStop, PreRestart, and PostRestart is now traced by default. This will allow you to visualize everything that happens after an actor is spawned mid-flight. This setting does produce significantly more traces when enabled (naturally) so it can be disabled via the following HOCON setting: phobos.tracing.trace-actor-lifecycle = off.
  • Phobos 1.4 uses Akka.NET v1.4.32, which contains important changes to the Akka.Persistence Journal base classes that now allow us to properly trace activity that happens within it. As a result, when phobos.tracing.trace-actor-lifecycle = on (this is the default) you can now trace the entire recovery process of all PersistentActor implementations - so long as the Akka.Persistence plugin you're using is built against Akka.NET v1.4.32 or higher. In addition to tracing Akka.Persistence recoveries all Persist calls can now be traced end to end with Phobos.

1.3.3 January 3rd 2022

Patch for Phobos 1.3

Phobos v1.3.3 introduces a bugfix for a very edge-case regression that was introduced in Phobos v1.3.2:

This was caused by some of our trace correlation software not fitlering remote IActorRefs that sent messages to actors outside of an actor's scheduled context, a rare case that could only really happen inside a Task closed over a remote IActorRef. Patch v1.3.3 resolves this issue.

1.3.2 December 21st 2021

Minor Version Release for Phobos

Phobos v1.3.2 includes some important tracing enhancements brought about as a result of upgrading Phobos to Akka.NET v1.4.31:

Phobos now supports tracing the output of PipeTo operations! This can be accomplished by making sure that the Sender property of the PipeTo operation contains a reference to the actor kicking of the Task<T> whose results will be piped:

var myActor = Sys.ActorOf(act =>
{
    act.ReceiveAny((o, ctx) =>
    {
        // won't capture trace - no "sender" included
        Task.Delay(TimeSpan.FromMilliseconds(100)).PipeTo(ctx.Sender, success: () => o);

        // will capture trace - "Sender" is specified
        Task.Delay(TimeSpan.FromMilliseconds(100)).PipeTo(ctx.Sender, success: () => o, sender:ctx.Self);
    });
});

1.3.1 December 9th 2021

Minor Version Release for Phobos

Phobos v1.3.1 includes some improvements for metrics support:

  • phobos.monitoring.monitor-mailbox-depth is now set to on by default as the performance impact is negligible for most actor types and can be found under the akka.actor.mailbox.depth metric header.
  • Phobos now records an App.Metrics Timer for every message received by an actor under the akka.messages.latency header. This is used to help measure how long an actor takes to process each message and these times will also be measured across an awaits that occur inside your actor's ReceiveAsync methods.

We also have updated all of our built-in Phobos dashboards to include the akka.messages.latency values. You can see an example below:

akka.messages.latency, new metrics introduced in Phobos v1.3.1, being displayed in Grafana

The addition of the akka.messages.latency metric to Phobos' captured Akka.NET data

In addition to these changes, we've also updated our performance benchmarks for Phobos to include the most recent data for v1.3.1. The addition of the akka.messages.latency reduces Phobos 1.3.1's total throughput by about 10% compared to our previous benchmarks, though usually this overhead pales in comparison to user-defined code. However, if this is a problem for you please file an issue with us here: https://github.com/petabridge/phobos-issues

1.3.0 November 18th 2021

Minor Version Release for Phobos

Phobos v1.3.0 includes some significant improvements in its tracing capabilities:

  • Phobos now properly handles await operations that occur within your actors - so any traces that begin when an actor starts processing a message will be carried all the way through until your ReceiveAsync methods exit. This is convenient when combined with other libraries that emit traces such as EntityFramework and OpenTracing.Contrib.
  • Phobos no longer records any akka.actor.ask object invocations during the startup of your ActorSystem, which were erroneously being attached to user-generated traces as well.
  • In some instance of using the WrappedTracer, which gets called whenver you use the WithScopeManager extension method on an existing ITracer object, it was possible for a null ISpan to be passed into the SpanBuilder.ChildOf method. This input is now null-checked and will no longer throw NullReferenceException errors.
  • All IScopes generated by Phobos are disposed of per their original invocation ordering, which reverses a major change we introduced in Phobos 1.2.0. We were able to introduce this change safely as a result of changing how we track traces around await calls.
  • The Phobos cluster-listener-actor will no longer log any Unhandled messages when akka.loglevel=DEBUG and akka.actor.debug.unhandled=on any longer.
  • Phobos has been upgraded to run on Akka.NET v1.4.28.

In addition to these changes, we've also updated our performance benchmarks for Phobos to include the most recent data for v1.3.0.

1.2.5 October 13th 2021

Maintenance Release for Phobos 1.2

Phobos v1.2.5 includes some fixes for the new Ask<T> tracing capabilities we added to Phobos v1.2.4:

  • Ask<T> spans now have an operation name that reads akka.actor.ask <ExpectedReplyType> - in order to make them more easily searchable on the UI and
  • Ask<T> tracing now runs when called from outside the ActorSystem, a very common use-case that is frequently used in Web APIs and elsewhere.

1.2.4 October 11th 2021

Maintenance Release for Phobos 1.2

Phobos v1.2.4 is a mostly maintenance upgrade from Phobos v1.2.3, but includes a valuable new feature which allows Phobos to more easily trace Ask<T> operations in order to help diagnose timeouts and other potential problems which may occur:

  • Upgraded to Akka.NET v1.4.27;
  • Added the ability to trace Ask<T> operations directly, including timeouts and other errors that might occur during processing. Look for the akka.actor.ask operation inside your trace system.

1.2.3 September 8th 2021

Maintenance Release for Phobos 1.2

Phobos v1.2.3 is a mostly maintenance upgrade from Phobos v1.2.2, but includes some important fixes to tracing messages that pass through routed actors.

  • Upgraded to Akka.NET v1.4.25 - you cannot upgrade to Akka.NET v1.4.25 without using at least Phobos v1.2.3 due to binary compatibiltiy issues;
  • Fixed an issue that prevented Akka.Cluster.Metrics weighted routers from being used correctly with Phobos v1.2.2; and
  • Fixed a rare issue that caused ActorScope.Dispose to throw a NullReferenceException.

1.2.2 August 18th 2021

Maintenance Release for Phobos 1.2

Phobos v1.2.2 is a mostly maintenance upgrade from Phobos v1.2.1, but includes some important fixes to tracing messages that pass through routed actors.

  • Upgraded to Akka.NET v1.4.24.
  • Phobos now correctly unpacks any messages that are sent to a ConsistentHashRouter via ActorSelection or Akka.Remote, as those messages weren't properly being unpacked from their SpanEnvelope data structure used by Phobos.

1.2.1 August 10th 2021

Maintenance Release for Phobos 1.2

Phobos v1.2.1 is a mostly maintenance upgrade from Phobos v1.2.0, but fixes many known issues with actor tracing in particular.

  • Upgraded to Akka.NET v1.4.23.
  • Phobos now supports version ranges for Google.Protobuf, supporting anything from v3.12.1 and higher.
  • Tracing of remotely deployed actors now works correctly in all cases.
  • Fixed a bug introduced in Phobos 1.2.0 that caused TraceConfigBuilder.IncludeMessagesAlreadyInTrace(bool includeInTrace) to not be honored correctly.

1.2.0 June 17th 2021

Maintenance Release for Phobos 1.2

Significantly improved trace filtering, with the new TraceConfigBuilder.IncludeMessagesAlreadyInTrace(bool includeInTrace) method, which allows users to coimpletely drop any messages inside an active trace that don't match your IFilterRules - this is designed for high throughput scenarios where users really want to greatly reduce the amount of data being created by their tracing system.

Here's an example of it in action:

var config = @"akka.actor.provider = ""Phobos.Actor.PhobosActorRefProvider,Phobos.Actor""";

var tracer = new MockTracer(new ActorScopeManager());
var phobosConfig = new PhobosConfigBuilder()
    .WithTracing(m =>
    {
        m.SetTracer(tracer);

        // don't include messages in trace that don't satisfy other filters
        m.IncludeMessagesAlreadyInTrace(false);

        // only accept FilteredMessage types
        m.AddIncludeMessageFilter<FilteredMessage>();
    });

// create PhobosSetup + BootstrapSetup for configuring ActorSystem
var phobosSetup = PhobosSetup.Create(phobosConfig)
    .WithSetup(BootstrapSetup.Create().WithConfig(config));

sys = ActorSystem.Create("PhobosTest", phobosSetup);

var actor = sys.ActorOf(Props.Create(() => new EchoActor()), "echo");
var actor2 = sys.ActorOf(act =>
{
    IActorRef sender = null;
    act.Receive<FilteredMessage>((f, ctx) =>
    {
        sender = ctx.Sender;
        actor.Tell(f.Message);
    });

    act.Receive<string>((str, ctx) =>
    {
        sender.Tell(str, ctx.Self);
    });
}, "secondActor");

// send a message that WILL be filtered out
actor2.Ask<string>(new FilteredMessage("bye"), TimeSpan.FromMilliseconds(100)).Wait();
Task.Delay(300).Wait(); // wait for activity in other threads to stop
var spans = tracer.FinishedSpans();
spans.Count.Should().Be(1); // for the FilteredMessage only

1.1.3 May 26th 2021

Maintenance Release for Phobos 1.1

1.1.2 May 6th 2021

Maintenance Release for Phobos 1.1

  • Upgraded Phobos to target Akka.NET v1.4.19
  • Upgraded Phobos to target App.Metrics v4.2.0 - this includes a new StatsD reporter that can be used in combination with DogStatsD for DataDog or stand-alone StatsD + Graphite.

1.1.1 March 23 2021

1.1.0 January 25 2021

Akka.Cluster Metrics Phobos now automatically records the number of nodes of each MemberStatus and the reachability of those cluster members. This happens without any need for manual code on the part of the Phobos user.

We also now append the first akka.cluster.roles item as a tag to all exported metrics - which should make it easy to group metrics together by Akka.Cluster role type.

We've shipped a default Akka.Cluster + Phobos Grafana dashboard that gathers the default metrics that are collected via Prometheus that you can install to visualize this data inside your own applications:

https://grafana.com/grafana/dashboards/13775

Minor Changes:

  • Metrics - Capture exception types inside log events and actor crashes;
  • Bugfix: Capture correct message name when tracing message that caused actor to crash; and
  • Upgraded to Akka.NET v1.4.16.

1.0.5 December 07 2020

  • Upgraded Akka.Cluster.Tools from v1.4.11 to v1.4.12
  • Removed obsolete HOCON default configuration. SampleRate setting is obsolete at Phobos.Actor.Configuration.MonitoringSettings.
  • Fixed missing tags for akka.actor.created metrics counter

1.0.4 November 16 2020

1.0.3 October 12 2020

Added WrappedTracer type, designed to make it possible for OpenTracing.ITracer implementations which don't support custom IScopeManagers to play nicely with Phobos' ActorScopeManager, which makes correlation from ASP.NET Core (and other non-Akka.NET services) --> Akka.NET automatic and accurate.

The WrappedTracer gets invoked automatically via the extension method .WithScopeManager present on the TracerExtensions class:

public static void ConfigureDataDogTracing(IServiceCollection services)
{
    // Add DataDog Tracing
    services.AddSingleton<ITracer>(sp =>
    {
        return OpenTracingTracerFactory.CreateTracer().WithScopeManager(new ActorScopeManager());
    });
}

In this case, the DataDog OpenTracing.ITracer returned by this method will be nested inside a WrappedTracer, which will proxy all of the real underlying tracing calls down to DataDog's client - but Phobos' ActorScopeManager will be used to perform all of the correlation activities.

Speaking of DataDog, we've also added a DataDog branch to the Phobos Quickstart tutorial Github repository here: https://github.com/petabridge/Petabridge.Phobos.Web/blob/feature/datadog-integration/README.md#working-with-datadog

1.0.2 June 19 2020

  • Fixed bug that caused actor crashes when complex generic types were logged during tracing.
  • Fixed issue that caused Unhandled and DeadLetter messages to not be logged by tracing or metrics correctly.

1.0.1 June 19 2020

  • Uses the NoMetrics.Instance as the default metrics inside PhobosSetup when users don't explicitly set an IMetricsRoot.

1.0.0 June 18 2020

Phobos 1.0.0 is a major shift from Phobos v0.9, and you can read the full details about what's new in Phobos 1.0.0 here.

The short version:

  1. Programmatic Configuration - Most configuration is now done programmatically, through the use of the PhobosSetup class. Phobos no longer requires any additional driver packages beyond what you configure programmatically.
  2. Easily Correlate Data between Akka.NET, ASP.NET, Kafka, SignalR, RabbitMQ, and more.
  3. Replacing Phobos.Monitoring with App.Metrics - As of Phobos 1.0.0, we now use the popular .NET monitoring library App.Metrics to handle all of the underlying monitoring and reporting for Phobos. This means that the number of monitoring services Phobos supports just got a lot larger.
  4. Significantly Improved Signal / Noise Ratio - Phobos now produces signiciantly less noise that it used to, which makes it much easier for you to understand what's going on inside your system.
  5. Simplified Pricing Model - Phobos no longer has any per-node licensing - and we've significantly reduced the price. Learn more about Phobos' new price and modeling here.

We've also built a big, full-fledged example you can use to learn the ins and outs of Phobos instead a real Akka.NET + ASP.NET Core Kubernetes cluster.

0.8.0 March 11 2020

  • Upgraded all interfaces to Akka.NET v1.4.1

0.7.1 September 02 2019

  • Added support for tracing messages sent via Akka.Cluster.Tools.DistributedPubSub.

0.7.0 August 06 2019

  • Added support for tracking stashed and unstashed messages.
  • Added support for tracing /system actors correctly.
  • Can now trace sharded entity actors instantiated via DI.
  • Upgraded to Akka.NET v1.3.14.

Phobos.Tracing v0.6.0

  • Added Kafka Zipkin transport options.
  • Rolled back to Google.Protobuf v3.3.0.
  • Upgraded to Akka.NET v1.3.14.
  • Upgraded to Petabridge.Tracing.Zipkin v0.5.1.

Phobos.Monitoring v0.8.0

  • Added Phobos.Monitoring.Prometheus package. It's still a work in progress but it's now being actively supported.
  • Upgraded to Akka.NET v1.3.14.

0.6.0 September 27 2018

Phobos v0.6.0 is a feature release for Phobos which includes support for new tracing and monitoring integrations:

  • Tracing: Jaeger - we've added first party support for the popular open source Jaeger tracing engine, which is now available via the Phobos.Tracing.Jaeger NuGet package.
  • Tracing: Microsoft Application Insights - we've added OpenTracing-compatible tracing support for Microsoft Azure's managed Application Insights service via the Phobos.Tracing.ApplicationInsights package.
  • Monitoring: Microsoft Application Insights - in addition to adding tracing support for Application Insights, we've also added corresponding monitoring support via the Phobos.Monitoring.ApplicationInsights NuGet package.

All of these are now available and are ready for immediate use. Please click on the links above for more details on how each of these platforms work.

Minor Updates and Changes

  • Upgraded to Phobos.Tracing v0.5.1 and Phobos.Monitoring v0.7.0;
  • Phobos.Actor: fixed a bug that caused instrumented actors created via Akka.DI.Core's System.DI().Props<TActor> methods to fail at startup. This has been fixed.
  • Phobos.Actor.Cluster: all entity actors created via Akka.Cluster.Sharding are now automatically tracked just like any other /user actor.
  • Monitoring: Removing the Timing.Start() method from the Timing class as it created issues with denormalization under the hood.
  • Monitoring: Changed the IMonitor implementation such that all denormalization of metrics now occurs through there. This will make it significantly easier for developers building custom MetricsRecorder implementations for custom Phobos.Monitoring plugins to have all of their data denormalized into both node-specific and cluster-wide metrics in the first place.

0.5.1 August 17 2018

  • Exposed the PhobosSettings.For(ActorSystem) method so users can easily can access to the current PhobosSettings and the default PhobosActorSettings in order to easily customize Phobos configurations for specific actors using Props rather than just the akka.actor.deployment HOCON namespace.

0.5.0 August 09 2018

  • Upgraded to Phobos.Tracing v0.4.6 and Phobos.Monitoring v0.6.1;
  • Significantly improved processing speeds for actors when Phobos is not enabled;
  • Added the ability to filter incoming messages using both blacklisting and whitelisting systems during tracing;
  • Added ability to automatically capture actor crashes and spawns inside traces - these traces will not be recorded if there is no ActiveSpan available at the time either of these events occurs.
  • All metrics recorded inside any Phobos-enabled actor are now automatically correlated across all supported dimensions, as outlined here: https://phobos.petabridge.com/articles/monitoring/index.html#global-node-specific-class-specific-and-actor-specific-metrics

Phobos.Monitoring v0.6.0 Changes

  • Renamed MonitoringExtension to ActorMonitoring in order to make it more consistent with the ActorTracing APIs used by Phobos.Tracing.
  • Added overloads to IMonitor for passing in TimeSpan for recording all Timing operations.
  • Introduced the ActorMonitor, which can be acquired via an actor's Context.MonitorFor() extension method. The ActorMonitor automatically reports any metrics created using it using both the actor's ActorPath as well as its implementation type; when sampling is enabled both metrics will either be emitted at the same time or neither will.
  • Added the ability for any of the Metric objects, such as Gauge, Counter, or Timing, to support reporting gathered metrics under multiple names or aliases for denormalization purposes. When sampling is enabled, either all names will be included in the sample or none will.
  • Introduced the TimingScope mechansim, which can be used by Phobos users to automatically time the duration of an operation with a simple using block. Here's an example down below:
using (Monitor.StartTiming("foo")) // creates a TimingScope
{
    Task.Delay(100).Wait();
} // completes and records the timing once the scope is disposed here.

0.4.4 July 18 2018

  • Resolved an issue with /system tracing and monitoring causing some issues at startup when enabled. These have been resolved.
  • Upgraded to Phobos.Monitoring v0.5.6 and Phobos.Tracing v0.4.5.

0.4.3 July 16 2018

  • Added full support for sampling across all traces; all child spans will automatically be recorded if a parent span is present, but otherwise we will use probabilistic sampling if you set the phobos.tracing.sample-rate value to anything other than the default (1.0, which means a 100% sample rate.)
  • Fixed a bug with mailbox-monitoring where we weren't properly honoring the HOCON setting. This has been fixed.
  • Fixed an issue with tracing causing deserialization errors during sampling. This has been fixed.
  • Upgraded to v0.3.1 Petabridge.Tracing.Zipkin driver.

0.4.1 July 1 2018

  • Upgraded to OpenTracing v0.1.2 for all tracing components;
  • Upgraded to Akka.NET v1.3.8.

0.4.0 April 27 2018

First certified, production-ready release of Phobos.Actor and all of its counterparts.

Phobos v0.4.0 introduces the following changes:

  1. Automatic correlation of all DeadLetter, UnhandledMessage, and LogEvent (i.e. Debug et al) messages types to the current active ISpan when tracing is enabled for the actors involved in the production of those messages.
  2. Automatic monitoring of DeadLetter, UnhandledMessage, and LogEvent (i.e. Debug et al) occurrences per ActorSystem.
  3. Automatic monitoring of the total number of received messages, actor starts, actor stops, and actor restarts per ActorSystem.
  4. Full support for ActorSelection tracing both locally and over Akka.Remote and Akka.Cluster.
  5. Full support for OpenTracing.IScopeManager and OpenTracing.IScope inside all ITracer instances used by Phobos.Tracing.
  6. Intellisense support for all Phobos packages.
  7. And lots of other small changes.

Please report any issues you come across to our mailing list here: https://phobos.petabridge.com/