Documentation
Sponsored by Kamino

Actor System Metrics

Kamon is able to gather metrics from all core Akka components: actor systems, actors, dispatchers, routers, remoting channels and, one addition of ours, actor groups; all thanks to bytecode instrumentation shipping with Kamon modules.

You must start your application with the AspectJ Weaver agent if you want to collect any Akka metrics.

Base Metrics

All actor systems will get the following metrics:

  • akka.system.dead-letters (Counter). Number of dead letters seen for a actor system. Tags:
    • system: Actor system name.
  • akka.system.unhandled-messages (Counter). Number of unhandled messages seen for a actor system. Tags:
    • system: Actor system name.
  • akka.system.active-actors (Range Sampler). Number of active actors running on the actor system, regardless of them being tracked for metrics or not. Tags:
    • system: Actor system name.
  • akka.system.processed-messages (Counter). Number of processed messages in the actor system. Tags:
    • system: Actor system name.
    • tracked: Whether the count is for tracked (tracked=true) or non-tracked actors (tracked=false).

Filtered Metrics

Since your application might be creating thousands of different actors, routers or even several dispatchers, Kamon wont simply catch them all but rather rely on filters to decide which components to track. Filters are configured in the kamon.util.filters configuration key and these are the filters expected by this module:

  • akka.tracked-actor: Decides which actors should be instrumented for metrics collection.
  • akka.tracked-router: Decides which routers should be instrumented for metrics collection.
  • akka.tracked-dispatcher: Decides which dispatcher should be instrumented for metrics collection.

Here is how this would look like in your configuration file:

kamon.util.filters {

  "akka.tracked-actor" {
    includes = [ "my-app/user/job-manager", "my-app/user/worker-*" ]
    excludes = [ "my-app/system/**", "my-app/user/worker-helper" ]
  }

  "akka.tracked-dispatcher" {
    includes = [ "my-app/akka.actor.default-dispatcher", "my-app/database-dispatcher" ]
  }

  "akka.tracked-router" {
    includes = [ "my-app/user/some-router" ]
  }
}

In the example filters above, for an actor system named my-app, all system actors and the user/worker-helper actor are excluded but the user/job-manager and all the user/worker-* actors should be tracked. With regards to dispatchers, only the default dispatcher in the application’s actor system and the database-dispatcher are tracked. Finally, the user/some-router router is the only one to be tracked by Kamon.

Once your filters are all set, you will be getting these metrics:

Actor Metrics

  • akka.actor.time-in-mailbox (Histogram): Tracks the time measured from the moment a message was enqueued into an actor’s mailbox until the moment it was dequeued for processing.
  • akka.actor.processing-time (Histogram): Tracks how long did it take for the actor to process every message.
  • akka.actor.mailbox-size (Range Sampler): Tracks the size of the actor’s mailbox.
  • akka.actor.errors (Counter): Number of failures the actor has experienced.

All actor metrics will have the following tags:

  • path: Actor’s path in the system.
  • system: Actor system name.
  • dispatcher: Configured dispatcher for the actor.
  • class: Actor’s class.

Router Metrics

  • akka.router.routing-time (Histogram): Tracks how long did it take to the router itself to decide which of it’s routees will process the message.
  • akka.router.time-in-mailbox (Histogram): Tracks the combined time measured from the moment a message was enqueued into a routee’s mailbox until the moment it was dequeued for processing. The measurements from all routees are combined here.
  • akka.router.processing-time (Histogram): Tracks how long did it take for the routees to process incoming messages. This histogram includes measurements from all routees.
  • akka.router.errors (Counter): Number of failures of the routees in the router.

All router metrics will have the following tags:

  • path: Router’s path in the system.
  • system: Actor system name.
  • dispatcher: Configured dispatcher for the router.

Please note that actor and router metrics are independent from each other and each actor will be tracked as either a regular actor or a routee, but not both.

Dispatcher Metrics

Dispatcher metrics rely on the kamon-executors module to capture metrics on the underlying executor service for Akka dispatchers. The following metrics will be exposed:

  • executor.threads (Histogram). Samples the number of threads in the executor service. Tags:
    • state: Active (state=active) and total (state=total).
  • executor.tasks (Counter). Tracks the number of tasks processed by the executor service. Tags:
    • state: Submitted (state=submitted) and completed (state=completed).
  • executor.queue (Histogram). Samples the queue size for the executor service.

Additionally, all executor service metrics will also have the following tags:

  • name: Dispatcher name.
  • actor-system: Actor system name.
  • type: Fork Join Pool (type=fjp) or Thread Pool Executor (type=tpe).

Additional Metrics for Thread Pool Executors

  • executor.pool (Gauge). Tracks several configuration settings for the executor service . Tags:
    • setting=min: Minimum pool size.
    • setting=max: Maximum pool size.
    • setting=corePoolSize: Core pool size.

Additional Metrics for Fork Join Pools

  • executor.pool (Gauge). Tracks several configuration settings for the executor service . Tags:
    • setting=min: Minimum pool size.
    • setting=max: Maximum pool size.
    • setting=parallelism: Parallelism.

Actor Group Metrics

There are situations in which you don’t want to track how a specific actor instance is behaving but rather, how a group of similar actors are behaving. The typical use cases for actor groups include:

  • Worker actors.
  • Per-request actors.
  • Short lived actors that perform support work for other bigger pieces of the system.

Configuring actor groups is a two step process:

  1. Create a filter in kamon.util.filters that matches all actors you would like to be included in a given group. The name of the filter is up to you, just make sure the name doesn’t clash with any other predefined filter.
  2. Include your filter name in the kamon.akka.actor-groups setting.

Here is a example of how to configure a actor group:

kamon {
  util.filters {
    "worker-actors" {
      includes = [ "my-app/user/worker-*" ]
    }
  }

  akka.actor-groups = [ "worker-actors" ]
}

All actor groups will get the following metrics:

  • akka.group.time-in-mailbox (Histogram): Tracks the time measured from the moment a message was enqueued into an actor’s mailbox until the moment it was dequeued for processing.
  • akka.group.processing-time (Histogram): Tracks how long did it take for the actor to process every message.
  • akka.group.mailbox-size (Range Sampler): Tracks the combined mailbox size of all member actors.
  • akka.group.members (Range Sampler): Tracks the number of group members.
  • akka.group.errors (Counter): Number of failures experienced by group members.

Additionally, all actor group metrics will have the following tags:

  • group: Group name.
  • system: Actor system name.

Remoting Metrics

If you are including the kamon-akka-remote module you will also get the following metrics:

  • akka.remote.message-size (Histogram). Tracks the size of incoming and outgoing messages. Tags:
    • system: Actor system name.
    • direction: Incoming (direction=in) or outgoing (direction=out).
  • akka.remote.serialization-time (Histogram). Tracks the time spent serializing and deserializing messages. Tags:
    • system: Actor system name.
    • direction: Incoming message deserialization (direction=in) or outgoing message serialization (direction=out).