Upgrading from Phobos 1.0 to Phobos 2.3

For current Phobos 1.0 customers who wish to upgrade to Phobos 2.3, follow the steps below to complete your upgrade.

Step 1 - Replace OpenTracing and App.Metrics Packages with OpenTelemetry

In your Startup.cs or where ever you configure your IServiceCollection, we need to start by replacing any of our OpenTracing and App.Metrics instrumentation with corresponding OpenTelemetry instrumentation.

If we take our Phobos + DataDog installation tutorial as a starting point, here's what our 1.x instrumentation NuGet packages look like:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <Import Project="..\common.props" />


  <PropertyGroup>
    <TargetFramework>$(NetCoreFramework)</TargetFramework>
    <OutputType>Exe</OutputType>
    <LangVersion>8</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <Content Include="app.conf">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
  </ItemGroup>

  <ItemGroup>
    <!-- begin remove -->
    <PackageReference Include="Datadog.Trace.OpenTracing" Version="2.5.1" />
    <PackageReference Include="OpenTracing.Contrib.NetCore" Version="0.8.0" />
    <PackageReference Include="App.Metrics.Datadog" Version="$(AppMetricsVersion)" />
    <PackageReference Include="App.Metrics.StatsD" Version="$(AppMetricsVersion)" />
    <PackageReference Include="App.Metrics.AspNetCore" Version="$(AppMetricsVersion)" />
    <PackageReference Include="App.Metrics.AspNetCore.Tracking" Version="$(AppMetricsVersion)" />
    <PackageReference Include="App.Metrics.Reporting.Console" Version="$(AppMetricsVersion)" />
    <!-- end remove -->
    <PackageReference Include="Phobos.Actor.Cluster" Version="$(PhobosVersion)" />
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="3.1.0" />
    <PackageReference Include="Petabridge.Cmd.Cluster" Version="$(PbmVersion)" />
    <PackageReference Include="Petabridge.Cmd.Remote" Version="$(PbmVersion)" />
    <PackageReference Include="Akka.Bootstrap.Docker">
      <Version>0.5.3</Version>
    </PackageReference>

  </ItemGroup>

</Project>

We'll begin by removing these packages and replacing them with the following packages from OpenTelemetry:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <Import Project="..\common.props" />


  <PropertyGroup>
    <TargetFramework>$(NetCoreFramework)</TargetFramework>
    <OutputType>Exe</OutputType>
    <LangVersion>8</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <Content Include="app.conf">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
  </ItemGroup>

  <ItemGroup>
    <!-- begin new packages -->
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9.14" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.14" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.4.0" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.4.0" />
    <!-- end new packages -->
    <PackageReference Include="Phobos.Actor.Cluster" Version="$(PhobosVersion)" />
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="3.1.0" />
    <PackageReference Include="Petabridge.Cmd.Cluster" Version="$(PbmVersion)" />
    <PackageReference Include="Petabridge.Cmd.Remote" Version="$(PbmVersion)" />
    <PackageReference Include="Akka.Bootstrap.Docker">
      <Version>0.5.3</Version>
    </PackageReference>

  </ItemGroup>

</Project>

What do these OpenTelemetry NuGet packages do?

• OpenTelemetry.Instrumentation.* packages collect and propagate OTel data from various .NET frameworks - in this case we're using the HttpClient and ASP.NET Core instrumentation packages.
• OpenTelemetry.Exporter.* packages export aggregated logs, traces, and metrics to your preferred APM service for processing. In this instance we're using the generic OpenTelemetry Protocol Exporter and we'll eventually configure an OTel Collector to export this data to DataDog.
• OpenTelemetry.Extensions.Hosting contains some extension methods that make it easy to integrate OpenTelemetry with Microsoft.Extensions.Hosting.

Step 2 - Upgrade Your Phobos NuGet Packages

The next step is easy - we need to upgrade from a Phobos 1.x distribution to a 2.3 and above distribution. The easiest way to do this is to take advantage of Phobos.Hosting and upgrade to using that package instead:

<PackageReference Include="Phobos.Hosting" Version="2.*" />

This will allow you use Akka.Hosting behind the scenes, which will make the configuration of your ActorSystem much simpler and more robust than previous Akka.NET practices.

Step 3 - Configure OpenTelemetry and Remove Old OpenTracing / App.Metrics Code

Next, we need to configure OpenTelemetry and remove our old OpenTracing and App.Metrics code from our solution. Here's what our old code looks like in Startup.cs.

public void ConfigureServices(IServiceCollection services)
{
    // enables OpenTracing for ASP.NET Core
    services.AddOpenTracing(o =>
    {
        o.ConfigureAspNetCore(a =>
            a.Hosting.OperationNameResolver = context => $"{context.Request.Method} {context.Request.Path}");
        o.AddCoreFx();
    });

    // sets up Prometheus + DataDog + ASP.NET Core metrics
    ConfigureAppMetrics(services);

    // sets up DataDog tracing
    ConfigureDataDogTracing(services);

    // sets up Akka.NET
    ConfigureAkka(services);
}

To make this easy, we're just going to delete everything short of the ConfigureAkka method:

public void ConfigureServices(IServiceCollection services)
{

    // sets up Akka.NET
    ConfigureAkka(services);
}

Next, we need to add our OpenTelemetry metrics and tracing SDKs:

public void ConfigureServices(IServiceCollection services)
{
    // needed on .NET Core 3.1
    AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport",
        true);

    var otelAgentAddress = Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT");
    if (string.IsNullOrEmpty(otelAgentAddress))
    {
        // default local address
        otelAgentAddress = "0.0.0.0:4317";
    }

    var resource = ResourceBuilder.CreateDefault()
                .AddService(Assembly.GetEntryAssembly()!.GetName().Name, 
                    serviceVersion:Assembly.GetEntryAssembly().GetName().Version.ToString(), 
                    serviceInstanceId:$"{Dns.GetHostName()}");

    services.AddOpenTelemetry()
        .WithTracing(tracer =>
        {
            tracer.AddAspNetCoreInstrumentation()
                .SetResourceBuilder(resource)
                .AddSource(AppOtelSourceName)
                .AddHttpClientInstrumentation()
                .AddPhobosInstrumentation()
                .AddOtlpExporter(options =>
                {
                    options.Protocol = OtlpExportProtocol.Grpc;
                    options.Endpoint = new Uri(otelAgentAddress);
                });
        })
        .WithMetrics(metrics =>
        {
            metrics.AddAspNetCoreInstrumentation()
                .SetResourceBuilder(resource)
                .AddMeter(AppOtelSourceName)
                .AddHttpClientInstrumentation()
                .AddPhobosInstrumentation()
                .AddOtlpExporter(options =>
                {
                    options.Protocol = OtlpExportProtocol.Grpc;
                    options.Endpoint = new Uri(otelAgentAddress);
                });
        });

    // sets up Akka.NET
    ConfigureAkka(services);
}

We're going to grab the address of the OpenTelemetry Line Protocol (OTLP) Collector from an environment variable, OTEL_EXPORTER_OTLP_ENDPOINT, but otherwise everything in this configuration is straight forward. The AddPhobosInstrumentation() lines are how we inject Phobos' data into both our metrics and traces.

Tip

The ResourceBuilder is used to tell DataDog which service produced which trace, which version of the service, and which instance of that service. Make sure you add one before you publish your application!

Step 4 - Configure Akka.NET to Use Phobos 2.3

Finally, we need to configure Akka.NET to use Phobos 2.3 - let's take a look at our ConfigureAkka method from earlier:

public static void ConfigureAkka(IServiceCollection services)
{
    services.AddSingleton(sp =>
    {
        var metrics = sp.GetRequiredService<IMetricsRoot>();
        var tracer = sp.GetRequiredService<ITracer>();

        var config = ConfigurationFactory.ParseString(File.ReadAllText("app.conf")).BootstrapFromDocker();

        var phobosSetup = PhobosSetup.Create(new PhobosConfigBuilder()
                .WithMetrics(m =>
                    m.SetMetricsRoot(metrics)) // binds Phobos to same IMetricsRoot as ASP.NET Core
                .WithTracing(t => t.SetTracer(tracer))) // binds Phobos to same tracer as ASP.NET Core
            .WithSetup(BootstrapSetup.Create()
                .WithConfig(config) // passes in the HOCON for Akka.NET to the ActorSystem
                .WithActorRefProvider(PhobosProviderSelection
                    .Cluster)); // last line activates Phobos inside Akka.NET

        var sys = ActorSystem.Create("ClusterSys", phobosSetup);

        // create actor "container" and bind it to DI, so it can be used by ASP.NET Core
        return new AkkaActors(sys);
    });

    // this will manage Akka.NET lifecycle
    services.AddHostedService<AkkaService>();
}

We're going to delete our own AkkaService implementation and rely on the built-in one from Akka.Hosting instead - we also don't need to explicitly reference a shared ITracer or IMetricsRoot implementation anymore in Phobos 2.0.

public static void ConfigureAkka(IServiceCollection services)
{
    var config = ConfigurationFactory.ParseString(File.ReadAllText("app.conf")).BootstrapFromDocker();

    services.AddAkka("ClusterSys", (builder, provider) =>
    {
        builder
            .AddHocon(config)
            .WithPhobos(AkkaRunMode.AkkaCluster)
            .StartActors((system, registry) =>
            {
                var consoleActor = system.ActorOf(Props.Create(() => new ConsoleActor()), "console");
                var routerActor = system.ActorOf(Props.Empty.WithRouter(FromConfig.Instance), "echo");
                var routerForwarder =
                    system.ActorOf(Props.Create(() => new RouterForwarderActor(routerActor)), "fwd");
                registry.TryRegister<RouterForwarderActor>(routerForwarder);
            })
            .StartActors((system, registry) =>
            {
                // start https://cmd.petabridge.com/ for diagnostics and profit
                var pbm = PetabridgeCmd.Get(system); // start Pbm
                pbm.RegisterCommandPalette(ClusterCommands.Instance);
                pbm.RegisterCommandPalette(RemoteCommands.Instance);
                pbm.Start(); // begin listening for PBM management commands
            });
    });
}

This is all we need to do to set Phobos up now - simply call the WithPhobos extension method on the AkkaConfigurationBuilder from Akka.Hosting.

Step 5 - Convert Any Manual OpenTracing / App.Metrics Instrumentation to OpenTelemetry

Next, if you have any manual OpenTracing / App.Metrics telemetry in your application you'll need to convert it over to use the equivalent OpenTelemetry APIs instead - here's some custom Phobos 1.0 code inside our ConsoleActor implementation:

public sealed class ConsoleActor : ReceiveActor
{
    private readonly ILoggingAdapter _log = Context.GetLogger();

    public ConsoleActor()
    {
        Receive<string>(_ =>
        {
            // use the local metrics handle to record a timer duration for how long this block of code takes to execute
            Context.GetInstrumentation().Monitor.Timer.Time(new TimerOptions { Name = "ProcessingTime" }, () =>
            {
                // start another span programmatically inside actor
                using (var newSpan = Context.GetInstrumentation().Tracer.BuildSpan("SecondOp").StartActive())
                {
                    var child = Context.ActorOf(Props.Create(() => new ChildActor()));
                    _log.Info("Spawned {0}", child);

                    child.Forward(_);
                }
            });
        });
    }
}

We need to convert this to use the equivalent OpenTelemetry APIs:

public sealed class ConsoleActor : ReceiveActor
{
    private readonly ILoggingAdapter _log = Context.GetLogger();

    public ConsoleActor()
    {
        var processingTime = Context.GetInstrumentation().Monitor.CreateHistogram<double>("ProcessingTime", "milliseconds");
        Receive<string>(_ =>
        {
            // use the local metrics handle to record a timer duration for how long this block of code takes to execute
            var start = DateTime.UtcNow;
            // start another span programmatically inside actor
            using (var newSpan = Context.GetInstrumentation().Tracer.StartActiveSpan("SecondOp"))
            {
                var child = Context.ActorOf(Props.Create(() => new ChildActor()));
                _log.Info("Spawned {0}", child);

                child.Forward(_);
            }

            var totalTime = DateTime.UtcNow - start;
            processingTime.Record(totalTime.TotalMilliseconds);
        });
    }
}

The tracing APIs in OpenTelemetry are a direct evolution of the OpenTracing APIs, so those will be quite familiar to you.

App.Metrics to OpenTelemetry.Metrics Conversion Guide

It's the metrics portion that is a little less straightforward since not all of the metrics concepts implemented in App.Metrics match 1:1 to meters implemented in OpenTelemetry.Metrics.

App.Metrics API	OpenTelemetry API	Notes
Counter	Counter or AsynchronousCounter	Both perform similarly.
Meter	Counter or Histogram	Depends on what you need to measure - for instance, measuring message processing latencies we use histogram percentiles to do this. For measuring simple rates we recommend counters.
Gauge	ObservableGauge	WARNING: there is no such thing as a synchronous Gauge in OTel right now; ObservableGauges run in the background on a dedicated timer to sample values periodically. This may change in the future.
Timer	Histogram	There are no timers in OpenTelemetry, but the can be achieved by creating a Histogram<long> or Histogram<double> and recording the delta between two sampling times.

Step 6 - Validate Your OpenTelemetry Configuration with Live Exporter

Next, we need to validate that everything works with our live exporter. This will vary depending on what your preferred APM Integrations are, but in our case we're going to use an OTLP exporter that transmits its data to DataDog.

See "Phobos End to End Tutorial with Datadog Tracing and Metrics" to learn more.