Check out the New Service Map! Check out the all-new Kamon APM Service Map!

Actor System Metrics #

Kamon is able to gather metrics from all core Akka components: actor systems, actors, dispatchers, routers, remoting channels and, one addition of ours, actor groups. All thanks to bytecode instrumentation shipping with Kamon modules.

You must start your application with the Kanela agent if you want to collect any Akka metrics.

Base Metrics #

These metrics are collected for all actor systems, without any special configuration or filtering:

Tracks the number of active Actors in an Actor System.

Instruments are tagged with:

  • system: The Actor System being tracked.
Counts the number of dead letters in an Actor System.

Instruments are tagged with:

  • system: The Actor System being tracked.
Counts the number of unhandled messages in an Actor System.

Instruments are tagged with:

  • system: The Actor System being tracked
Counts the number of processed messages in an Actor System.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • tracked: Whether the messages was processed by a tracked actor or not.

Filtered Metrics #

Since your application might be creating thousands of different actors, routers or even several dispatchers, Kamon won’t simply catch them all but rather rely on filters to decide which components to track. Filters are configured under the kamon.instrumentation.akka.filters configuration path. Within this path, these filters are expected:

  • actors.track: Decides which actors should be instrumented for dedicated metrics collection. This means that each actor matched by this filter will get its own set of Actor metrics.
  • routers: Decides which routers should be instrumented for metrics collection.
  • dispatchers: Decides which dispatcher should be instrumented for metrics collection.

Here is how this would look like in your configuration file:

kamon.instrumentation.akka.filters {

  actors.track {
    includes = [ "my-app/user/job-manager", "my-app/user/worker-*" ]
    excludes = [ "my-app/system/**", "my-app/user/worker-helper" ]
  }

  dispatchers {
    includes = [ "my-app/akka.actor.default-dispatcher" ]
  }

  routers {
    includes = [ "my-app/user/some-router" ]
  }
}

In the example filters above, for an actor system named my-app, all system actors and the user/worker-helper actor are excluded but the user/job-manager and all the user/worker-* actors should be tracked. With regards to dispatchers, only the default dispatcher in the application’s actor system and the database-dispatcher are tracked. Finally, the user/some-router router is the only one to be tracked by Kamon.

The Doomsday Wildcard #

It might be very tempting to track all actors by adding ** in the actors.track filter, which might be very useful for testing purposes but is definitely a bad practice for running in production where thousands of tracked actors could eat all your memory. To avoid this, Kamon does not allow using ** on the actors.track filter, but you can disable this safeguard by adding actors.doomsday-wildcard = on setting to your config. Think twice before doing it, since most likely all you will need is to create actor groups, use the auto-grouping feature or target specific actors.

Actor Metrics #

These metrics are recorded for all tracked actors:

Tracks the time taken for the actor to process the receive function.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • path: The Actor's path in the System.
  • dispatcher: The Dispatcher on which the actor is running.
  • class: The Actor's class name.
Tracks the time since the instant a message is enqueued in an Actor's mailbox until it is dequeued for processing.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • path: The Actor's path in the System.
  • dispatcher: The Dispatcher on which the actor is running.
  • class: The Actor's class name.
Tracks the behavior of an Actor's mailbox size.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • path: The Actor's path in the System.
  • dispatcher: The Dispatcher on which the actor is running.
  • class: The Actor's class name.
Counts the number of processing errors experienced by an Actor.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • path: The Actor's path in the System.
  • dispatcher: The Dispatcher on which the actor is running.
  • class: The Actor's class name.

Router Metrics #

When a router is instrumented, metrics are recorded both from the router itself (measuring how long it takes to process the routing logic) and from all the routees behind the router, recording the following metrics together:

Tracks the time taken by a router to process its routing logic.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • path: The Router's path in the System.
  • dispatcher: The Dispatcher on which the Router is running.
  • routerClass: The Router's class name.
  • routeeClass: The Routee's class name.
Tracks the time taken for a routee to process the receive function.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • path: The Router's path in the System.
  • dispatcher: The Dispatcher on which the Router is running.
  • routerClass: The Router's class name.
  • routeeClass: The Routee's class name.
Tracks the time since the instant a message is enqueued in a routee's mailbox until it is dequeued for processing.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • path: The Router's path in the System.
  • dispatcher: The Dispatcher on which the Router is running.
  • routerClass: The Router's class name.
  • routeeClass: The Routee's class name.
Tracks the number of messages waiting to be processed across all routees.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • path: The Router's path in the System.
  • dispatcher: The Dispatcher on which the Router is running.
  • routerClass: The Router's class name.
  • routeeClass: The Routee's class name.
Tracks the number of routees belonging to a router.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • path: The Router's path in the System.
  • dispatcher: The Dispatcher on which the Router is running.
  • routerClass: The Router's class name.
  • routeeClass: The Routee's class name.
Counts the number of processing errors experienced by the routees of a router.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • path: The Router's path in the System.
  • dispatcher: The Dispatcher on which the Router is running.
  • routerClass: The Router's class name.
  • routeeClass: The Routee's class name.

Dispatcher Metrics #

The Dispatchers instrumentation relies on the kamon-executors module to capture metrics on the underlying executor service for Akka dispatchers. The following metrics will be recorded:

Tracks executor minimum number of Threads.

Instruments are tagged with:

  • name: The name assigned to the Executor.
  • type: Either "fjp" for ForkJoinPool executors or "tpe" for ThreadPoolExecutors.
Tracks executor maximum number of Threads.

Instruments are tagged with:

  • name: The name assigned to the Executor.
  • type: Either "fjp" for ForkJoinPool executors or "tpe" for ThreadPoolExecutors.
Tracks executor parallelism. This metrics is only available for ForkJoinPool executors.

Instruments are tagged with:

  • name: The name assigned to the Executor.
  • type: Either "fjp" for ForkJoinPool executors or "tpe" for ThreadPoolExecutors.
Samples the number of active threads on the executor service.

Instruments are tagged with:

  • name: The name assigned to the Executor.
  • type: Either "fjp" for ForkJoinPool executors or "tpe" for ThreadPoolExecutors.
Samples the total number of threads on the executor service.

Instruments are tagged with:

  • name: The name assigned to the Executor.
  • type: Either "fjp" for ForkJoinPool executors or "tpe" for ThreadPoolExecutors.
Tracks the number of tasks that completed execution on the executor service.

Instruments are tagged with:

  • name: The name assigned to the Executor.
  • type: Either "fjp" for ForkJoinPool executors or "tpe" for ThreadPoolExecutors.
Tracks the number of tasks submitted to the executor service.

Instruments are tagged with:

  • name: The name assigned to the Executor.
  • type: Either "fjp" for ForkJoinPool executors or "tpe" for ThreadPoolExecutors.
Tracks the time that tasks spend on the executor service's queue.

Instruments are tagged with:

  • name: The name assigned to the Executor.
  • type: Either "fjp" for ForkJoinPool executors or "tpe" for ThreadPoolExecutors.
Samples the number of tasks queued for execution on the executor service.

Instruments are tagged with:

  • name: The name assigned to the Executor.
  • type: Either "fjp" for ForkJoinPool executors or "tpe" for ThreadPoolExecutors.

Besides the default tags added to all metrics be the Executors module, the Dispatchers instrumentation will add a system tag with the name of the Actor System to which the dispatcher belongs.

Actor Groups #

There are situations in which you don’t want to track how a specific actor instance is behaving but rather, how a group of similar actors are behaving. The typical use cases for actor groups include:

  • Worker actors.
  • Per-request actors.
  • Short lived actors that perform support work for other bigger pieces of the system.

Auto-grouping #

As part of the default instrumentation, Kamon will automatically create groups that contain all actors of the same type at the same level in the Actor System tree. For example, in a system that has the following actors:

  • myApp/user/parent with class ParentActor
  • myApp/user/parent/child-1 with class ChildActor
  • myApp/user/parent/child-2 with class ChildActor

The instrumentation will automatically create two actor groups: one for all the ParentActor instances in the first level and another for all ChildActor instances in the second level. If you want to control which groups are eligible for auto-grouping use the filters for the auto-grouping group.

Finally, take into account that auto-grouping will only pick up actors that are not being explicitly tracked in any other way (e.g. the actor is not explicitly tracked nor included in an explicit group).

Explicit Groups #

Besides auto-grouping, you can explicitly create groups be defining filters under the groups section in the configuration:

kamon.instrumentation.akka.filters.groups {
  worker-actors {
    includes = [ "my-app/user/worker-*" ]
  }
}

The name of the group will be the name of the configuration (worker-actors in the example above) and the group will include all actors matching the include/excludes patterns.

Actor Group Metrics #

Regardless of how the group was created (auto-grouping or explicitly) the following metrics are recorded for all groups:

Tracks the time taken for a member actor to process the receive function.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • group: The group's name.
Tracks the time since the instant a message is enqueued in a member's mailbox until it is dequeued for processing.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • group: The group's name.
Tracks the number of messages waiting to be processed across all members.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • group: The group's name.
Tracks the number of routees belonging to a group.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • group: The group's name.
Counts the number of processing errors experienced by the members of a group.

Instruments are tagged with:

  • system: The Actor System being tracked.
  • group: The group's name.

Remoting Metrics #

If you are using Akka Remote or Akka Cluster, the instrumentation will also record these metrics for the remoting infrastructure:

Tracks the distribution of inbound message sizes.

Instruments are tagged with:

  • system: The Actor System being tracked.
Tracks the distribution of outbound message sizes.

Instruments are tagged with:

  • system: The Actor System being tracked.
Tracks the distribution of outbound message sizes.

Instruments are tagged with:

  • system: Tracks the time taken to serialize outgoing messages.
Tracks the time taken to deserialize incoming messages.

Instruments are tagged with:

  • system: Tracks the time taken to serialize outgoing messages.