Message Tracing #

Additionally to all metrics and context propagation that you can get with Kamon, you can also use the instrumentation to generate Spans for actor messages using Kamon’s Tracing API, effectively giving you distributed tracing for your Akka applications, wohoo!

Tracing Actors #

Tracing must be enabled on a per-actor basis using the actors.trace filter within the Akka configuration:

kamon.instrumentation.akka.filters {
  actors.trace {
    includes = [ "my-app/user/job-manager", "my-app/user/worker-*" ]
    excludes = [ "my-app/system/**", "my-app/user/worker-helper" ]
  }
}

With these settings, all actors matching the filter above will generate Spans for their messages, but only if there is an ongoing trace already! In most situations that is not a problem at all because traces are already started a previous step in the application logic. For example, if there is a thin Akka HTTP API on receiving requests that then get processed by actors, the Akka HTTP instrumentation will take care of starting a trace and the filtered actors will participate on it.

All Spans generated for actor messages will start when a message is sent to an actor and finish when the message processing has finished. Additionally, these Spans will get:

• A akka.actor.dequeued mark when the message is taken out of the mailbox to start processing.
• All these tags:
  • component=akka.actor: Component marker. It’s constant on all Spans.
  • akka.system: Actor system name.
  • akka.actor.path: Actor path.
  • akka.actor.class: Actor class.
  • akka.actor.message-class: Message class.

Starting Traces #

Additionally to participating in traces, it is possible to configure Actors to automatically start a trace while they are processing a message, if there is no ongoing trace to participate in. Again, this is configured with a filter! In this case the filter is called actors.start-trace:

kamon.instrumentation.akka.filters {
  actors.start-trace {
    includes = [ "my-app/user/job-manager-rare" ]
  }
}

Be careful with this filter since making too many actors start traces could end up in generating many irrelevant spans that bury any useful information behind spans for scheduled messages and other potentially irrelevant messages that get processed by actors.

Customizing Spans #

In case you would like to modify the Span automatically created by instrumentation, you can access it using the Kamon.currentSpan() shortcut and do anything you want with it! This example below adds a custom tag to the Span:

  def receive = {
    case anything =>
      Kamon.currentSpan().tag("my-tag", "awesome-value")

      // do your processing here.
  }