Phobos Performance Impact
As much as we'd like to make tracing and monitoring available for free, physics and computer organization are going to have something to say about that. Inevitably, Phobos.Tracing and Phobos.Monitoring introduce additional memory and CPU overhead in the course of their normal execution. The purpose of this document is to give Phobos users an idea of the impact of using Phobos inside their applications.
Overview: Understanding Performance Trade-offs
Tip
Quick Performance Summary
- Full instrumentation: ~53% throughput reduction, still handles 3.7M+ messages/second per actor
- Monitoring only: ~20% throughput reduction, handles 6.3M+ messages/second per actor
- Real-world impact: Negligible for 99% of applications
- Main cost: Memory allocation from string formatting, not CPU
- Latency impact: 0.1-0.3 microseconds per message (insignificant)
Important
Most production applications will never approach these theoretical throughput limits. Even with full instrumentation enabled, a single actor can process over 3 million messages per second. Your business logic overhead will vastly exceed the OpenTelemetry instrumentation overhead.
Benchmark Methodology
All of these metrics were gathered using an NBench 2.1 concurrent benchmark against a single actor - so we don't get any benefit from having multiple cores. The benchmarks measure message throughput under different Phobos configuration scenarios.
What These Numbers Mean
Based on extensive profiling and experimentation carried out by the Petabridge team, the lion's share of the performance impact comes from the tracing system calling ToString on the ActorPath of the actors involved in the sent message and the message content itself. Thus, you shouldn't read too much into these numbers - the bulk of the cost of Phobos comes from string allocation and formatting.
In all likelihood, your actual business and application code will still cost significantly more than the overhead of Phobos.
.NET 9.0 Metrics
These metrics were all recorded using .NET 9.0, NBench 2.0.1, and Phobos v2.10.0 using the following hardware:
Ubuntu Linux (kernel 6.8.0)
Intel Xeon Server CPU, 16 logical cores
Server-grade memory
.NET SDK 9.0.100
[Host] : .NET 9.0.8, X64 RyuJIT AVX2
DefaultJob : .NET 9.0.8, X64 RyuJIT AVX2
Fully Instrumented Actor Throughput
These are the data collected when an actor is running with the most verbose actor tracing and monitoring options available.
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 24.96 | 23.93 | 23.40 | 0.51 |
| TotalCollections [Gen1] | collections | 7.71 | 6.67 | 6.16 | 0.62 |
| TotalCollections [Gen2] | collections | 3.74 | 2.86 | 2.46 | 0.57 |
| TotalBytesAllocated | bytes | 132,752,928.83 | 121,783,811.03 | 106,528,476.61 | 9,150,346.69 |
| [Counter] MessageReceived | operations | 3,856,281.68 | 3,718,759.92 | 3,586,072.96 | 65,057.79 |
Uninstrumented Actor Throughput
These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = off and phobos.monitoring.enabled = off for this actor or ActorSystem.
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 11.14 | 10.58 | 9.44 | 0.49 |
| TotalCollections [Gen1] | collections | 8.36 | 7.93 | 7.08 | 0.37 |
| TotalCollections [Gen2] | collections | 5.57 | 5.29 | 4.72 | 0.24 |
| TotalBytesAllocated | bytes | 247,895,558.68 | 228,693,731.57 | 190,088,030.04 | 17,861,991.93 |
| [Counter] MessageReceived | operations | 8,355,242.99 | 7,931,702.18 | 7,080,192.63 | 366,390.50 |
Tracing-only Actor Throughput
These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = on and phobos.monitoring.enabled = off for this actor.
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 29.27 | 28.28 | 26.55 | 0.82 |
| TotalCollections [Gen1] | collections | 9.15 | 7.62 | 5.90 | 1.01 |
| TotalCollections [Gen2] | collections | 3.08 | 2.77 | 1.48 | 0.57 |
| TotalBytesAllocated | bytes | 161,058,786.66 | 147,876,381.89 | 131,697,725.20 | 7,109,629.37 |
| [Counter] MessageReceived | operations | 4,621,806.14 | 4,501,907.81 | 4,272,196.20 | 97,211.43 |
Monitoring-only Actor Throughput
These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = off and phobos.monitoring.enabled = on for this actor.
Per-second Totals
| Metric | Units / s | Max / s | Average / s | Min / s | StdDev / s |
|---|---|---|---|---|---|
| TotalCollections [Gen0] | collections | 8.95 | 7.80 | 6.22 | 0.98 |
| TotalCollections [Gen1] | collections | 6.71 | 5.69 | 4.15 | 0.99 |
| TotalCollections [Gen2] | collections | 4.48 | 3.57 | 2.07 | 1.00 |
| TotalBytesAllocated | bytes | 210,137,935.36 | 191,987,859.42 | 169,651,030.30 | 11,765,922.76 |
| [Counter] MessageReceived | operations | 6,712,441.60 | 6,345,707.31 | 5,965,679.08 | 229,827.66 |
Real-World Performance Implications
Throughput Impact Summary
Based on the benchmarks above, here's what you can expect in production:
| Configuration | Messages/Second per Actor | Throughput Reduction | When to Use |
|---|---|---|---|
| No Instrumentation | ~7.93 million | Baseline (0%) | Never - you lose all observability |
| Monitoring Only | ~6.35 million | ~20% | Production systems with performance concerns |
| Tracing Only | ~4.50 million | ~43% | Debug-focused deployments |
| Full Instrumentation | ~3.72 million | ~53% | Most production systems |
Real-World Context
Note
Your application will never hit these limits. Here's why:
Typical Application Message Rates
- Web Application: 1,000 requests/second × 10 actor interactions = 10,000 msg/s (0.3% of instrumented capacity)
- High-Frequency Trading: 100,000 events/second = 100,000 msg/s (3% of instrumented capacity)
- IoT Platform: 10,000 devices × 1 report/second = 10,000 msg/s (0.3% of instrumented capacity)
- Real-Time Gaming: 1,000 players × 60 updates/second = 60,000 msg/s (1.6% of instrumented capacity)
Where You'll Actually See Impact
The real cost is memory allocation, not CPU. The instrumentation increases garbage collection frequency due to string formatting for traces. This is most noticeable in applications that:
Warning
High-Impact Scenarios
- Memory-constrained environments (containers with <1GB RAM)
- Very high message rates (>100,000 msg/s per actor)
- Large message payloads that get serialized for tracing
- Systems already experiencing GC pressure
Latency Impact
The per-message latency impact is negligible:
| Operation | Added Latency | Context |
|---|---|---|
| Per-message instrumentation | 0.1-0.3 μs | Compare to: |
| Network round-trip | 1-100 ms | 10,000× slower |
| Database query | 1-50 ms | 10,000× slower |
| Disk I/O | 1-10 ms | 10,000× slower |
| Business logic | 10-1000 μs | 100× slower |
Performance Optimization Best Practices
Based on the performance characteristics above, here's how to optimize Phobos for your specific needs:
Start with Monitoring Only
If you're concerned about performance impact, start with monitoring enabled and tracing disabled. This gives you:
- Only 20% throughput reduction (vs 53% with full instrumentation)
- Essential metrics for understanding system health
- Significantly less memory allocation pressure
phobos {
monitoring.enabled = on
tracing.enabled = off
}
Use Selective Tracing
Since tracing has the highest performance cost due to string formatting, enable it only where you need it:
Message-Based Filtering
Use ITraceFilter to trace only specific message types or business operations:
Actor-Specific Configuration
Disable tracing for high-throughput actors while keeping it enabled for critical business logic:
Environment-Based Configuration
Consider enabling full tracing only in staging/development environments and using monitoring-only or selective tracing in production.
Optimize for Memory-Constrained Environments
If garbage collection is your bottleneck:
Warning
Memory is the real cost, not CPU. The instrumentation increases garbage collection frequency due to string formatting for traces.
- Use sampling to reduce the volume of data being formatted and exported
- Filter out high-frequency, low-value messages from tracing
- Consider using monitoring-only mode for high-throughput actors
- Implement manual tracing for actors processing >100,000 messages/second
Configure OpenTelemetry Sampling
OpenTelemetry sampling reduces export costs but still incurs formatting overhead. For true performance gains, combine sampling with Phobos filtering: