You are viewing documentation for an outdated version. Do you wish to see documentation for the latest version?

Actor System Metrics #

Kamon is able to gather metrics from all core Akka components: actor systems, actors, dispatchers, routers, remoting channels and, one addition of ours, actor groups; all thanks to bytecode instrumentation shipping with Kamon modules.

Base Metrics #

All actor systems will get the following metrics:

• akka.system.dead-letters (Counter). Number of dead letters seen for an actor system. Tags:
  • system: Actor system name.
• akka.system.unhandled-messages (Counter). Number of unhandled messages seen for an actor system. Tags:
  • system: Actor system name.
• akka.system.active-actors (Range Sampler). Number of active actors running on the actor system, regardless of them being tracked for metrics or not. Tags:
  • system: Actor system name.
• akka.system.processed-messages (Counter). Number of processed messages in the actor system. Tags:
  • system: Actor system name.
  • tracked: Whether the count is for tracked (tracked=true) or non-tracked actors (tracked=false).

Filtered Metrics #

Since your application might be creating thousands of different actors, routers or even several dispatchers, Kamon won’t simply catch them all but rather rely on filters to decide which components to track. Filters are configured in the kamon.util.filters configuration key and these are the filters expected by this module:

• akka.tracked-actor: Decides which actors should be instrumented for metrics collection.
• akka.tracked-router: Decides which routers should be instrumented for metrics collection.
• akka.tracked-dispatcher: Decides which dispatcher should be instrumented for metrics collection.

Here is how this would look like in your configuration file:

kamon.util.filters {

  "akka.tracked-actor" {
    includes = [ "my-app/user/job-manager", "my-app/user/worker-*" ]
    excludes = [ "my-app/system/**", "my-app/user/worker-helper" ]
  }

  "akka.tracked-dispatcher" {
    includes = [ "my-app/akka.actor.default-dispatcher", "my-app/database-dispatcher" ]
  }

  "akka.tracked-router" {
    includes = [ "my-app/user/some-router" ]
  }
}

In the example filters above, for an actor system named my-app, all system actors and the user/worker-helper actor are excluded but the user/job-manager and all the user/worker-* actors should be tracked. With regards to dispatchers, only the default dispatcher in the application’s actor system and the database-dispatcher are tracked. Finally, the user/some-router router is the only one to be tracked by Kamon.

Once your filters are all set, you will be getting these metrics:

Actor Metrics #

• akka.actor.time-in-mailbox (Histogram): Tracks the time measured from the moment a message was enqueued into an actor’s mailbox until the moment it was dequeued for processing.
• akka.actor.processing-time (Histogram): Tracks how long did it take for the actor to process every message.
• akka.actor.mailbox-size (Range Sampler): Tracks the size of the actor’s mailbox.
• akka.actor.errors (Counter): Number of failures the actor has experienced.

All actor metrics will have the following tags:

• path: Actor’s path in the system.
• system: Actor system name.
• dispatcher: Configured dispatcher for the actor.
• class: Actor’s class.

Router Metrics #

• akka.router.routing-time (Histogram): Tracks how long did it take to the router itself to decide which of it’s routees will process the message.
• akka.router.time-in-mailbox (Histogram): Tracks the combined time measured from the moment a message was enqueued into a routee’s mailbox until the moment it was dequeued for processing. The measurements from all routees are combined here.
• akka.router.processing-time (Histogram): Tracks how long did it take for the routees to process incoming messages. This histogram includes measurements from all routees.
• akka.router.pending-messages (Range Sampler): Tracks how many messages are waiting to be processed across all routees on the router.
• akka.router.members (Range Sampler): Tracks the number of routees in a router.
• akka.router.errors (Counter): Number of failures of the routees in the router.

All router metrics will have the following tags:

• path: Router’s path in the system.
• system: Actor system name.
• dispatcher: Configured dispatcher for the router.

Please note that actor and router metrics are independent from each other and each actor will be tracked as either a regular actor or a routee, but not both.

Dispatcher Metrics #

Dispatcher metrics rely on the kamon-executors module to capture metrics on the underlying executor service for Akka dispatchers. The following metrics will be exposed:

• executor.threads (Histogram). Samples the number of threads in the executor service. Tags:
  • state: Active (state=active) and total (state=total).
• executor.tasks (Counter). Tracks the number of tasks processed by the executor service. Tags:
  • state: Submitted (state=submitted) and completed (state=completed).
• executor.queue (Histogram). Samples the queue size for the executor service.

Additionally, all executor service metrics will also have the following tags:

• name: Dispatcher name.
• actor-system: Actor system name.
• type: Fork Join Pool (type=fjp) or Thread Pool Executor (type=tpe).

Additional Metrics for Thread Pool Executors #

• executor.pool (Gauge). Tracks several configuration settings for the executor service . Tags:
  • setting=min: Minimum pool size.
  • setting=max: Maximum pool size.
  • setting=corePoolSize: Core pool size.

Additional Metrics for Fork Join Pools #

• executor.pool (Gauge). Tracks several configuration settings for the executor service . Tags:
  • setting=min: Minimum pool size.
  • setting=max: Maximum pool size.
  • setting=parallelism: Parallelism.

Actor Group Metrics #

There are situations in which you don’t want to track how a specific actor instance is behaving but rather, how a group of similar actors are behaving. The typical use cases for actor groups include:

• Worker actors.
• Per-request actors.
• Short lived actors that perform support work for other bigger pieces of the system.

Configuring actor groups is a two step process:

1. Create a filter in kamon.util.filters that matches all actors you would like to be included in a given group. The name of the filter is up to you, just make sure the name doesn’t clash with any other predefined filter.
2. Include your filter name in the kamon.akka.actor-groups setting.

Here is an example of how to configure an actor group:

kamon {
  util.filters {
    "worker-actors" {
      includes = [ "my-app/user/worker-*" ]
    }
  }

  akka.actor-groups = [ "worker-actors" ]
}

All actor groups will get the following metrics:

• akka.group.time-in-mailbox (Histogram): Tracks the time measured from the moment a message was enqueued into an actor’s mailbox until the moment it was dequeued for processing.
• akka.group.processing-time (Histogram): Tracks how long did it take for the actor to process every message.
• akka.group.pending-messages (Range Sampler): Tracks how many messages are waiting to be processed across of all member actors. Previously was called akka.group.mailbox-size.
• akka.group.members (Range Sampler): Tracks the number of group members.
• akka.group.errors (Counter): Number of failures experienced by group members.

Additionally, all actor group metrics will have the following tags:

• group: Group name.
• system: Actor system name.

Remoting Metrics #

If you are including the kamon-akka-remote module you will also get the following metrics:

• akka.remote.message-size (Histogram). Tracks the size of incoming and outgoing messages. Tags:
  • system: Actor system name.
  • direction: Incoming (direction=in) or outgoing (direction=out).
• akka.remote.serialization-time (Histogram). Tracks the time spent serializing and deserializing messages. Tags:
  • system: Actor system name.
  • direction: Incoming message deserialization (direction=in) or outgoing message serialization (direction=out).