Search Results for

    Show / Hide Table of Contents

    Phobos Performance Impact

    As much as we'd like to make tracing and monitoring available for free, physics and computer organization are going to have something to say about that. Inevitably, Phobos.Tracing and Phobos.Monitoring introduce additional memory and CPU overhead in the course of their normal execution. The purpose of this document is to give Phobos users an idea of the impact of using Phobos inside their applications.

    Reference Metrics and CPU

    All of these metrics were recorded on a 8-core AMD Ryzen 7 1700 CPU clocked at 3.2GHz. The machine was running other workloads at the time, so be it.

    For reference purposes, vanilla Akka.NET runs at about 5 million messages per second on this computer on both .NET Core 3.1.

    All of these metrics were gathered using an NBench 2.0 concurrent benchmark against a single actor - so we don't get any benefit from having multiple cores.

    What These Numbers Mean

    Based on extensive profiling and experimentation carried out by the Petabridge team, the lion's share of the performance impact comes from the tracing system calling ToString on the ActorPath of the actors involved in the sent message and the message content itself. Thus, you shouldn't read too much into these numbers - the bulk of the cost of Phobos comes from string allocation and formatting.

    In all likelihood, your actual business and application code will still cost significantly more than the overhead of Phobos.

    .NET Core 3.1 Metrics

    The following metrics were all recorded using .NET Core 3.1, NBench 2.1, and Phobos v2.0.0.

    Fully Instrumented Actor Throughput

    These are the data collected when an actor is running with the most verbose actor tracing and monitoring options available.

    Totals

    Metric Units Max Average Min StdDev
    TotalCollections [Gen0] collections 751.00 750.15 749.00 0.55
    TotalCollections [Gen1] collections 12.00 9.54 8.00 1.05
    TotalCollections [Gen2] collections 3.00 3.00 3.00 0.00
    TotalBytesAllocated bytes 101,407,744.00 99,476,764.31 98,028,952.00 866,285.66
    [Counter] MessageReceived operations 3,000,000.00 3,000,000.00 3,000,000.00 0.00

    Per-second Totals

    Metric Units / s Max / s Average / s Min / s StdDev / s
    TotalCollections [Gen0] collections 97.87 97.40 96.93 0.28
    TotalCollections [Gen1] collections 1.56 1.24 1.04 0.14
    TotalCollections [Gen2] collections 0.39 0.39 0.39 0.00
    TotalBytesAllocated bytes 13,187,426.68 12,915,958.85 12,732,851.54 125,510.74
    [Counter] MessageReceived operations 391,460.73 389,515.13 387,491.50 1,153.69

    Uninstrumented Actor Throughput

    These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = off and phobos.monitoring.enabled = off for this actor or ActorSystem.

    Totals

    Metric Units Max Average Min StdDev
    TotalCollections [Gen0] collections 7.00 6.77 6.00 0.44
    TotalCollections [Gen1] collections 4.00 3.38 3.00 0.51
    TotalCollections [Gen2] collections 2.00 1.77 1.00 0.44
    TotalBytesAllocated bytes 96,583,456.00 88,515,491.08 83,662,712.00 4,105,541.70
    [Counter] MessageReceived operations 3,000,000.00 3,000,000.00 3,000,000.00 0.00

    Per-second Totals

    Metric Units / s Max / s Average / s Min / s StdDev / s
    TotalCollections [Gen0] collections 10.05 8.18 6.66 0.91
    TotalCollections [Gen1] collections 5.31 4.09 3.26 0.73
    TotalCollections [Gen2] collections 2.87 2.14 1.11 0.57
    TotalBytesAllocated bytes 126,135,965.13 106,925,260.97 94,067,940.19 10,628,945.93
    [Counter] MessageReceived operations 4,305,545.63 3,623,434.70 3,263,990.93 314,618.68

    Tracing-only Actor Throughput

    These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = on and phobos.monitoring.enabled = off for this actor.

    Totals

    Metric Units Max Average Min StdDev
    TotalCollections [Gen0] collections 287.00 286.31 285.00 0.63
    TotalCollections [Gen1] collections 9.00 7.08 5.00 1.04
    TotalCollections [Gen2] collections 2.00 1.92 1.00 0.28
    TotalBytesAllocated bytes 101,446,392.00 98,243,960.00 96,564,216.00 1,624,310.96
    [Counter] MessageReceived operations 3,000,000.00 3,000,000.00 3,000,000.00 0.00

    Per-second Totals

    Metric Units / s Max / s Average / s Min / s StdDev / s
    TotalCollections [Gen0] collections 60.64 59.76 58.34 0.65
    TotalCollections [Gen1] collections 1.84 1.48 1.03 0.21
    TotalCollections [Gen2] collections 0.42 0.40 0.21 0.06
    TotalBytesAllocated bytes 20,963,790.45 20,504,150.68 19,698,982.26 348,493.91
    [Counter] MessageReceived operations 633,836.97 626,141.45 611,996.34 6,124.75

    Monitoring-only Actor Throughput

    These are the data collected when Phobos is installed and setup correctly, but phobos.tracing.enabled = off and phobos.monitoring.enabled = on for this actor.

    Totals

    Metric Units Max Average Min StdDev
    TotalCollections [Gen0] collections 416.00 414.92 413.00 0.86
    TotalCollections [Gen1] collections 11.00 9.31 9.00 0.75
    TotalCollections [Gen2] collections 2.00 1.85 1.00 0.38
    TotalBytesAllocated bytes 100,448,504.00 96,775,756.92 94,636,344.00 1,748,214.58
    [Counter] MessageReceived operations 3,000,000.00 3,000,000.00 3,000,000.00 0.00

    Per-second Totals

    Metric Units / s Max / s Average / s Min / s StdDev / s
    TotalCollections [Gen0] collections 120.89 120.08 118.97 0.63
    TotalCollections [Gen1] collections 3.20 2.69 2.58 0.22
    TotalCollections [Gen2] collections 0.58 0.53 0.29 0.11
    TotalBytesAllocated bytes 29,352,913.92 28,008,037.57 27,413,996.68 548,941.41
    [Counter] MessageReceived operations 876,655.58 868,225.78 860,021.05 4,884.27

    Performance Optimization Best Practices

    So how can you get the best possible performance out of Phobos?

    Configure Sampling for your TracerProvider or MeterProvider

    OpenTelemetry for .NET exposes a customizable Activity / Metric processing pipeline for all telemetry data produced from OTel sources - using sampling won't reduce the amount of events produced by Phobos, so you will still incur a CPU penalty, but it will prevent data from being exported to its final destination so you will save on bandwidth and possibly APM cost by doing so.

    You can read more about using OpenTelemetry sampling here:

    • Using OpenTelemetry Trace Sampling
    • Using OpenTelemetry Metric Sampling

    Use Filtering for Tracing

    Phobos supports filtering of messages for tracing in Akka.NET - tracing is the most expensive part of Phobos since it requires "stringification" of many common data elements, so using filtering can help reduce the amount of areas where this occurs.

    Important

    Learn how to control tracing noise with ITraceFilter in Phobos.

    Disable Metrics / Tracing Where You Don't Need Them

    One possibility that Phobos supports is the ability to turn tracing and monitoring off / on in specific areas of the actor heirarchy. That way, you can disable traces and metrics where you don't need them and maintain a lower level of resource utilization behind the scenes.

    Read about how to leverage this functionality:

    1. Change actor-specific tracing / monitoring settings via HOCON or
    2. Change actor-specific tracing / monitoring settings via Props.
    In This Article
    Back to top Generated by DocFX