Phobos Performance Impact
As much as we'd like to make tracing and monitoring available for free, physics and computer organization are going to have something to say about that. Inevitably, Phobos.Tracing and Phobos.Monitoring introduce additional memory and CPU overhead in the course of their normal execution. The purpose of this document is to give Phobos users an idea of the impact of using Phobos inside their applications.
Reference Metrics and CPU
All of these metrics were recorded on a 8-core AMD Ryzen 7 1700 CPU clocked at 3.2GHz. The machine was running other workloads at the time, so be it.
For reference purposes, vanilla Akka.NET runs at about 5 million messages per second on this computer on .NET Core 3.1.
All of these metrics were gathered using an NBench 2.0 concurrent benchmark against a single actor - so we don't get any benefit from having multiple cores.
What These Numbers Mean
Based on extensive profiling and experimentation carried out by the Petabridge team, the lion's share of the performance impact comes from the tracing system calling ToString on the ActorPath of the actors involved in the sent message and the message content itself. Thus, you shouldn't read too much into these numbers - the bulk of the cost of Phobos comes from string allocation and formatting.
In all likelihood, your actual business and application code will still cost significantly more than the overhead of Phobos.
.NET 8.0 Metrics
These metrics were all recorded using .NET 8.0, NBench 2.1, and Phobos v2.5.0 using the following hardware:
BenchmarkDotNet v0.13.10, Windows 11 (10.0.22621.1992/22H2/2022Update/SunValley2)
12th Gen Intel Core i7-1260P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 7.0.302
[Host] : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2
DefaultJob : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2
Fully Instrumented Actor Throughput
These are the data collected when an actor is running with the most verbose actor tracing and monitoring options available.
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 23.40 | 22.62 | 21.83 | 0.47 |
| TotalCollections [Gen1] | collections | 8.08 | 6.56 | 5.74 | 0.67 |
| TotalCollections [Gen2] | collections | 3.51 | 3.01 | 2.30 | 0.55 |
| TotalBytesAllocated | bytes | 117,690,516.43 | 106,422,782.17 | 97,667,790.73 | 7,682,124.96 |
| [Counter] MessageReceived | operations | 3,574,092.97 | 3,460,154.40 | 3,318,312.07 | 75,748.22 |
Uninstrumented Actor Throughput
These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = off and phobos.monitoring.enabled = off for this actor or ActorSystem.
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 10.29 | 9.55 | 9.11 | 0.36 |
| TotalCollections [Gen1] | collections | 7.72 | 7.16 | 6.83 | 0.27 |
| TotalCollections [Gen2] | collections | 5.15 | 4.77 | 4.56 | 0.18 |
| TotalBytesAllocated | bytes | 214,611,870.70 | 201,390,502.77 | 187,084,185.39 | 10,238,454.02 |
| [Counter] MessageReceived | operations | 7,720,652.38 | 7,159,519.52 | 6,833,118.39 | 271,551.59 |
Tracing-only Actor Throughput
These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = on and phobos.monitoring.enabled = off for this actor.
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 29.56 | 28.46 | 27.56 | 0.63 |
| TotalCollections [Gen1] | collections | 7.78 | 7.49 | 7.25 | 0.17 |
| TotalCollections [Gen2] | collections | 3.11 | 3.00 | 2.90 | 0.07 |
| TotalBytesAllocated | bytes | 153,293,528.01 | 146,274,733.09 | 141,235,039.33 | 3,244,902.37 |
| [Counter] MessageReceived | operations | 4,666,873.88 | 4,494,032.32 | 4,350,858.45 | 99,994.71 |
Monitoring-only Actor Throughput
These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = off and phobos.monitoring.enabled = on for this actor.
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 7.65 | 7.43 | 7.25 | 0.11 |
| TotalCollections [Gen1] | collections | 5.73 | 5.57 | 5.44 | 0.08 |
| TotalCollections [Gen2] | collections | 3.82 | 3.71 | 3.63 | 0.06 |
| TotalBytesAllocated | bytes | 165,623,703.64 | 157,977,889.27 | 150,372,201.82 | 5,576,369.88 |
| [Counter] MessageReceived | operations | 5,733,889.06 | 5,570,531.48 | 5,439,830.22 | 82,984.44 |
Performance Optimization Best Practices
So how can you get the best possible performance out of Phobos?
Configure Sampling for your TracerProvider or MeterProvider
OpenTelemetry for .NET exposes a customizable Activity / Metric processing pipeline for all telemetry data produced from OTel sources - using sampling won't reduce the amount of events produced by Phobos, so you will still incur a CPU penalty, but it will prevent data from being exported to its final destination so you will save on bandwidth and possibly APM cost by doing so.
You can read more about using OpenTelemetry sampling here:
Use Filtering for Tracing
Phobos supports filtering of messages for tracing in Akka.NET - tracing is the most expensive part of Phobos since it requires "stringification" of many common data elements, so using filtering can help reduce the amount of areas where this occurs.
Disable Metrics / Tracing Where You Don't Need Them
One possibility that Phobos supports is the ability to turn tracing and monitoring off / on in specific areas of the actor hierarchy. That way, you can disable traces and metrics where you don't need them and maintain a lower level of resource utilization behind the scenes.
Read about how to leverage this functionality: