Phobos Performance Impact
As much as we'd like to make tracing and monitoring available for free, physics and computer organization are going to have something to say about that. Inevitably, Phobos.Tracing and Phobos.Monitoring introduce additional memory and CPU overhead in the course of their normal execution. The purpose of this document is to give Phobos users an idea of the impact of using Phobos inside their applications.
Reference Metrics and CPU
All of these metrics were recorded on a 8-core AMD Ryzen 7 1700 CPU clocked at 3.2GHz. The machine was running other workloads at the time, so be it.
For reference purposes, vanilla Akka.NET runs at about 5 million messages per second on this computer on both .NET Core 3.1.
All of these metrics were gathered using an NBench 2.0 concurrent benchmark against a single actor - so we don't get any benefit from having multiple cores.
What These Numbers Mean
Based on extensive profiling and experimentation carried out by the Petabridge team, the lion's share of the performance impact comes from the tracing system calling ToString on the ActorPath of the actors involved in the sent message and the message content itself. Thus, you shouldn't read too much into these numbers - the bulk of the cost of Phobos comes from string allocation and formatting.
In all likelihood, your actual business and application code will still cost significantly more than the overhead of Phobos.
.NET Core 3.1 Metrics
The following metrics were all recorded using .NET Core 3.1, NBench 2.1, and Phobos v1.3.0.
Fully Instrumented Actor Throughput
These are the data collected when an actor is running with the most verbose actor tracing and monitoring options available.
Totals
| Metric | Units | Max | Average | Min | StdDev |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 619.00 | 618.46 | 618.00 | 0.52 |
| TotalCollections [Gen1] | collections | 10.00 | 8.92 | 8.00 | 0.64 |
| TotalCollections [Gen2] | collections | 3.00 | 3.00 | 3.00 | 0.00 |
| TotalBytesAllocated | bytes | 101,533,400.00 | 100,046,719.38 | 98,794,576.00 | 1,128,124.49 |
| [Counter] MessageReceived | operations | 3,000,000.00 | 3,000,000.00 | 3,000,000.00 | 0.00 |
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 91.79 | 91.22 | 89.48 | 0.58 |
| TotalCollections [Gen1] | collections | 1.49 | 1.32 | 1.18 | 0.10 |
| TotalCollections [Gen2] | collections | 0.45 | 0.44 | 0.43 | 0.00 |
| TotalBytesAllocated | bytes | 14,993,292.55 | 14,756,916.33 | 14,281,586.77 | 218,579.25 |
| [Counter] MessageReceived | operations | 445,589.10 | 442,490.56 | 433,675.23 | 2,910.74 |
Uninstrumented Actor Throughput
These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = off and phobos.monitoring.enabled = off for this actor or ActorSystem.
Totals
| Metric | Units | Max | Average | Min | StdDev |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 7.00 | 6.77 | 6.00 | 0.44 |
| TotalCollections [Gen1] | collections | 4.00 | 3.38 | 3.00 | 0.51 |
| TotalCollections [Gen2] | collections | 2.00 | 1.77 | 1.00 | 0.44 |
| TotalBytesAllocated | bytes | 96,583,456.00 | 88,515,491.08 | 83,662,712.00 | 4,105,541.70 |
| [Counter] MessageReceived | operations | 3,000,000.00 | 3,000,000.00 | 3,000,000.00 | 0.00 |
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 10.05 | 8.18 | 6.66 | 0.91 |
| TotalCollections [Gen1] | collections | 5.31 | 4.09 | 3.26 | 0.73 |
| TotalCollections [Gen2] | collections | 2.87 | 2.14 | 1.11 | 0.57 |
| TotalBytesAllocated | bytes | 126,135,965.13 | 106,925,260.97 | 94,067,940.19 | 10,628,945.93 |
| [Counter] MessageReceived | operations | 4,305,545.63 | 3,623,434.70 | 3,263,990.93 | 314,618.68 |
Tracing-only Actor Throughput
These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = on and phobos.monitoring.enabled = off for this actor.
Totals
| Metric | Units | Max | Average | Min | StdDev |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 287.00 | 286.38 | 285.00 | 0.77 |
| TotalCollections [Gen1] | collections | 9.00 | 7.00 | 6.00 | 1.08 |
| TotalCollections [Gen2] | collections | 2.00 | 1.85 | 1.00 | 0.38 |
| TotalBytesAllocated | bytes | 102,376,472.00 | 98,733,027.69 | 96,961,248.00 | 1,828,138.73 |
| [Counter] MessageReceived | operations | 3,000,000.00 | 3,000,000.00 | 3,000,000.00 | 0.00 |
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 60.70 | 60.19 | 59.75 | 0.31 |
| TotalCollections [Gen1] | collections | 1.90 | 1.47 | 1.25 | 0.23 |
| TotalCollections [Gen2] | collections | 0.42 | 0.39 | 0.21 | 0.08 |
| TotalBytesAllocated | bytes | 21,481,719.40 | 20,749,484.33 | 20,206,114.56 | 383,895.46 |
| [Counter] MessageReceived | operations | 634,486.58 | 630,478.97 | 624,550.47 | 2,969.96 |
Monitoring-only Actor Throughput
These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = off and phobos.monitoring.enabled = on for this actor.
Totals
| Metric | Units | Max | Average | Min | StdDev |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 331.00 | 329.62 | 328.00 | 0.96 |
| TotalCollections [Gen1] | collections | 10.00 | 8.46 | 8.00 | 0.66 |
| TotalCollections [Gen2] | collections | 2.00 | 1.77 | 1.00 | 0.44 |
| TotalBytesAllocated | bytes | 101,019,696.00 | 96,482,312.00 | 92,906,304.00 | 2,667,288.15 |
| [Counter] MessageReceived | operations | 3,000,000.00 | 3,000,000.00 | 3,000,000.00 | 0.00 |
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 107.89 | 106.66 | 104.25 | 1.07 |
| TotalCollections [Gen1] | collections | 3.25 | 2.74 | 2.52 | 0.22 |
| TotalCollections [Gen2] | collections | 0.65 | 0.57 | 0.33 | 0.14 |
| TotalBytesAllocated | bytes | 33,229,941.26 | 31,223,723.47 | 29,980,462.07 | 1,027,325.39 |
| [Counter] MessageReceived | operations | 986,835.52 | 970,791.38 | 944,909.15 | 11,662.62 |
Performance Optimization Best Practices
So how can you get the best possible performance out of Phobos?
Configure Sampling for your ITracer or IMetricsRoot
Since the ITracer is an object you configure programmatically on your own, doing this will be dependent on the tracing implementation you choose.
Jaeger, for instance, has a very detailed sampling options available in its C# driver.
App.Metrics uses sliding window reservoir sampling by default, but this too can be customized.
Use Filtering for Tracing
Phobos supports filtering of messages for tracing in Akka.NET - tracing is the most expensive part of Phobos since it requires "stringification" of many common data elements, so using filtering can help reduce the amount of areas where this occurs.
var config = @"akka.actor.provider = ""Phobos.Actor.PhobosActorRefProvider,Phobos.Actor""";
var tracer = new MockTracer(new ActorScopeManager());
var phobosConfig = new PhobosConfigBuilder()
.WithTracing(m =>
{
m.SetTracer(tracer);
// don't start traces for Filtered messages
m.AddExcludeMessageFilter<FilteredMessage>();
});
// create PhobosSetup + BootstrapSetup for configuring ActorSystem
var phobosSetup = PhobosSetup.Create(phobosConfig)
.WithSetup(BootstrapSetup.Create().WithConfig(config));
var sys = ActorSystem.Create("PhobosTest", phobosSetup);
var actor = sys.ActorOf(Props.Create(() => new EchoActor()), "echo");
// send a message that WON'T be filtered out
actor.Ask<NormalMessage>(new NormalMessage("hi"), TimeSpan.FromMilliseconds(100)).Wait();
AwaitAssert(() => tracer.FinishedSpans().Count.Should().Be(1)); // no trace recorded
tracer.Reset(); // reset the MockTracer
// send a message that WILL be filtered out
actor.Ask<FilteredMessage>(new FilteredMessage("bye"), TimeSpan.FromMilliseconds(100)).Wait();
Task.Delay(100).Wait(); // wait for activity in other threads to stop
tracer.FinishedSpans().Count.Should().Be(0);
The m.AddExcludeMessageFilter<FilteredMessage>(); method call tells the tracing system "don't create new traces for messages of type FilteredMessage" - this check gets performed by the actor each time it processes a message and if the message doesn't satisfy the filter predicate (i.e. IncludeInTrace == false) then none of the trace objects get allocated.
Disable Metrics / Tracing Where You Don't Need Them
One possibility that Phobos supports is the ability to turn tracing and monitoring off / on in specific areas of the actor heirarchy. That way, you can disable traces and metrics where you don't need them and maintain a lower level of resource utilization behind the scenes.
Read about how to leverage this functionality: