Documentation

Reporting Metrics to StatsD

[OpenTSDB] is a high performance open source database for handling time series data. It uses hbase to allow it to store and serve massive amounts of time series data without losing granularity.

Installation

Add the kamon-opentsdb dependency to your project and ensure that it is in your classpath at runtime, that’s it. Kamon’s module loader will detect that the OpenTSDB module is in the classpath and automatically start it.

Follow OpenTSDB installation instruction if you don’t already have an instance running.

Configuration

Basic

You will need to set the kamon.opentsdb.direct.quorum configuration to point at your ZooKeeper servers. Additionally to that, you can configure the metric categories to which this module will subscribe using the kamon.opentsdb.subscriptions key. By default, the following subscriptions are included:

kamon.opentsdb {
  subscriptions {
    histogram       = [ "**" ]
    min-max-counter = [ "**" ]
    gauge           = [ "**" ]
    counter         = [ "**" ]
    trace           = [ "**" ]
    trace-segment   = [ "**" ]
    akka-actor      = [ "**" ]
    akka-dispatcher = [ "**" ]
    akka-router     = [ "**" ]
    system-metric   = [ "**" ]
    http-server     = [ "**" ]
  }
}

If you are interested in reporting additional entities to OpenTSDB please ensure that you include the categories and name patterns accordingly.

Metric Name

Names are generated by concatenating the results of several rules together

kamon.opentsdb.default.name = [application, category, entity, metric, stat]

This design allows you to easily customize the metric name by reordering, adding, and removing rules as you see fit. kamon.opentsdb.name.separator is inserted between each rule who result is non-empty. Empty rules are removed from the metric name

Metric Tags

All tags associated with the kamon metric will be passed through to OpenTSDB. Additional tags may be added by mapping tag names to rules. A tag will be named the exact value listed in the config, if you would like a different name, create a new rule (See Extensibility)

kamon.opentsdb.default.tags = [ application, host ]

Stats

Many statistics can be calculated for each Kamon metric, and ,by default, each statistic will be stored as a separate OpenTSDB metric.

kamon.opentsdb.default.stats = [ count, rate, min, max, median, mean, 90, 99 ]

Most statistics can only be used with histograms, however assigning them to a counter is harmless.

Counter stats

  • count: The value of the counter
  • rate: The value of the counter divided by the tick length in seconds

Histogram Stats

  • count: The number of values stored in the histogram
  • rate: The number of values stored in the histogram divided by the tick length in seconds
  • mean: The average of all values stored during the tick
  • max: The largest value stored during the tick
  • min: The smallest value stored during the tick
  • median: The median value (synonym for “50”)

Additionally, numeric values in the stat list, will generate percentile statistics in the OpenTSDB database.

  • 50: 50th percentile
  • 70.5: 70.5th percentile
  • 90: 90th percentile

See Extensibility to learn how to create your own stats.

Rules

Rules can either be constant values or dynamical generated. All the provided rules are dynamically generated except for “application”, which is empty by default. To set the application name, provide set kamon.opentsdb.rules.application.value

  • category: The entity’s category.
  • entity: The entity’s name.
  • metric: The metric name assigned in the entity recorder.
  • stat: The name of the statistic
  • host: The local host name
  • application : The name of the application. Undefined by default.

See Extensibility to learn how to create your own rules.

Idle metrics

if kamon.opentsdb.default.filterIdle is true, values will not be sent to opentsdb for inactive metrics, as opposed to sending 0.

Timestamps

Setkamon.opentsdb.default.timestamp to ‘seconds’ or ‘milliseconds’.

Advanced configuration

Metrics can be configured at the global, category, and individual metric level.

Altering the configuration values under kamon.opentsdb.default will impact all metrics.

Category specific configuration can be achieved by creating entries under kamon.opentsdb.category.<category_name> Any values not set on the category level, will be inherited from the defaults.

kamon.opentsdb.category.counter = { name = [ metric ], stats = [count ] }

This configuration will change the name and stats recorded for counter metrics, but the tags and timestamp from kamon.opentsdb.default will be used.

Individual metric can be configured by add entries to kamon.opentsdb.metrics. Any values not set on the metric level, will be inherited from first the category and then defaults.

kamon.opentsdb.metrics."my.metric.name" = { stats = [ 90, 95, 99, 99.9, 99.999 ], filterIdle = false }

Here the name, tags, and timestamp from the defaults will be used, unless the metric is a “counter”, in which case the name and stats from above will be used.

Extensibility

You can add static values to your metric names and tags by adding entries of the format kamon.opentsdb.rules.<rule-name>.value = "some string" These values can be referenced in name and tags by using rule-name

EX.

kamon.opentsdb.rules.cluster.value = "EC2"
kamon.opentsdb.default.name = [ cluster, application, category, entity, metric, stat]

You can create your own dynamic rules by subclassing kamon.opentsdb.Rule and adding an entry of the format kamon.opentsdb.rules.<name>.generator = "fully.qualified.class.name"

EX.

kamon.opentsdb.rules.timezone.generator = "leider.ken.application.TimezoneRule"
kamon.opentsdb.tags = { host = host, tz = timezone }

You can create you own stats by subclassing kamon.opentsdb.Stat and adding an entry of the format kamon.opentsdb.stats.<name> = "fully.qualified.class.name"

EX.

kamon.opentsdb.stats.integral = "leider.ken.application.IntegralStat"
kamon.opentsdb.counter.stats = [ count, rate, integral ]

Visualization and Fun

OpenTSDB comes with a web service that provides a REST API and basic visualization. For our internal testing we choose Grafana to create beautiful dashboards. Check out how your metrics data might look like in Grafana with the screenshots below.