Metrics #

Kamon provides five instrument types that can be used for recording metrics:

• Counters: Track the number of times certain event happened. It can only be increased. Useful for counting errors, cache misses, etc.
• Gauges: Track a single value that can be increased, decreased or explicitly set. The common use case for gauges are slowly moving values like disk usage, number of loaded classes or configuration settings.
• Histograms: Record the distribution of values within a configurable range and precision. Typically used for recording latency, message sizes and so on.
• Timers: Are just sugar on top of histograms that make it simple to track latency.
• Range Samplers: Used to track variables that increase and decrease very quickly, the most notable use case being tracking the number of concurrent requests or the number of elements in a queue.

You can find more details in the Metric Instruments section.

Creating and Removing Metrics #

To record metrics you need to request the appropriate instrument from the Kamon companion object (or static members, if on Java), tag it (or run it withoutTags()) and then you are ready to record metrics. Let’s look at a example:

  // Simple metrics can be one-liners, without refining (tagging)
  Kamon.counter("app.orders.sent").withoutTags().increment() // (1)

  // If refining (tagging) is needed, define a metric first;
  // here is a metric called "app.error" with a counter instrument.
  val errors = Kamon.counter("app.error") // (2

  // Refine the metric with a specific set of tags
  val invalidUserErrors = errors.withTag("class", "InvalidUser") // (3)
  val invalidPassErrors = errors.withTag("class", "InvalidPassword") // (4)

  invalidPassErrors.increment() // (5)

  // Removing a metric from Kamon
  errors.remove(TagSet.of("class", "InvalidUser")) // (6)

1. Simple metrics that do not require tags can be looked up and used in one-liners. (1)
2. Defining a metric with a counter instrument (2)
3. Creating two “tagged” versions of the metric, one using the class=InvalidUser (3) tag and another using the class=InvalidPassword tag (4).
4. Incrementing one of the refined metrics (5)
5. Removing a specific instance of the app.error metric with the class=InvalidUser tag from Kamon. (6)

There are a few important things that you should keep in mind when using metrics:

• When possible, define and refine your metrics up front, as a static member or instance members on the instrumented components of your services. This saves a bit of time in looking up instruments. If you can’t do so, rest assured that Kamon will always return the same instrument instance if requested with the same arguments (metric name and tags) from anywhere in your code.
• Always provide a measurement unit when your metric is tracking either time (milliseconds, microseconds, etc) or information (byes, kilobytes, etc). This will ensure that reporting modules can scale the recorded data appropriately.
• Follow our metrics naming conventions: we use namespaced metric names that describe “what” is being measured and use tags to specify from “where” we are getting this information. For example, we use the span.processing-time metric to track Span latencies and we add an operation=login or operation=search tag (plus some more) to identify from where we are recording such measurements.
• Avoid high cardinality variables as metric tags since they will most likely overwhelm your monitoring systems or make your bill go through the roof. Things like user identifiers, IP addresses, session identifiers and so on are a big no no for metric tags.

Counters #

Counters can only do one thing: increment. You can either increment() by one or increment(times) a specific number of times at once.

  // One-liner
  Kamon.counter("app.orders.sent").withoutTags().increment()


  // Fully defined and refined (tagged) counter
  val sentBytes = Kamon.counter("network.traffic", information.bytes)
    .withTag("direction", "out").withTag("interface", "eth0")

  sentBytes.increment()
  sentBytes.increment(512)

Gauges #

A gauge can be update(value) to as specific value, increment(), increment(times), decrement() and decrement(times).

  // One-liner
  Kamon.gauge("users").withoutTags().update(42)

  // Fully defined and refined (tagged) gauge
  val onlineUsers = Kamon.gauge("users")
    .withTag("status", "online")

  onlineUsers.update(43)
  onlineUsers.decrement()
  onlineUsers.decrement(4)
  onlineUsers.increment()
  onlineUsers.increment(7)

Histograms #

Histograms record the entire distribution of values for a given metric. The only available operations are record(value) and record(value, times).

  // One-liner
  Kamon.histogram("messaging.payload-size").withoutTags()

  // Fully defined and refined (tagged) histogram
  val responseSizes = Kamon.histogram("http.response-size", information.bytes)
    .withTag("status-code", "200")

  responseSizes.record(2048)

Timers #

Timers assume that you are measuring latency so all you need to do is provide a name and start() a timer. You will get back a Timer.Started instance that can be stop()ed whenever the operation you are measuring finishes.

  // One-liner
  val startedTimer = Kamon.timer("operations").withoutTags().start()
  // do some work
  startedTimer.stop()

  // Fully defined and refined (tagged) timer
  val operationLatency = Kamon.timer("operation-latency")
    .withTag("operation", "login")

  val timer = operationLatency.start()
  // do some work
  timer.stop()

Range Samplers #

A range sampler is always paired with a component or piece of state that starts at zero and can increment and decrement, like a message queue size or the number of concurrent requests being processed by a service. The exposed operations are similar to those offered by gauges, namely increment(), increment(times), decrement() and decrement(times).

  // One-liner
  Kamon.rangeSampler("http.in-flight").withoutTags().increment()


  // Fully defined and refined (tagged) timer
  val mailboxSize = Kamon.rangeSampler("actor.mailbox-size")
    .withTag("actor", "user/test-actor")

  mailboxSize.increment()
  mailboxSize.increment(12)
  mailboxSize.decrement(12)
  mailboxSize.decrement()