Skip to content

Usage

Jump to bottom
Ryan Smith edited this page Aug 2, 2013 · 26 revisions

This page contains information regarding the usage of l2met. Usage includes how to drain an apps logs into l2met and what sort of data can be extracted from the logs. If you are interested in how l2met works, checkout the architecture page. If you are interested in the system administration of l2met, checkout the administration page.

L2met has two basic inputs: the drain URL and the log line. The log lines are delivered to the drain URL over HTTPs. Thus, there are 3 topics related to l2met usage:

  • Logging Convention
  • Drain URLS
  • Log Delivery

Logging Convention

Logging is defined to be a stream of bytes emitted by a process to STDOUT. Ultimately, l2met expects the log lines to be delivered in RFC5424 frames, but understanding framing is not necessary in understanding the logging convention. More details about framing can be found in the section on Log Delivery. Thus, the following ruby snippet is an example of a log:

$stdout.puts("hello world")

As an application developer, this is your interface into l2met. Simply write a string to STDOUT, and you will be producing metrics in no time. Here is an example of a ruby process emitting a counter metric:

$stdout.puts("count#bottles-of-beer-on-the-wall=1")

In the string printed to STDOUT, you will notice a key=value pair. The key contains a metric type, followed by a # followed by the metric name. The value simply contains a number. Optionally you can attach a unit to the number. E.g.

$stdout.puts("count#beers-on-the-wall=1bottle")

There are 3 metrics types: counters, samples, and measurements.

# A counter.
$stdout.puts("count#user.clicks=4")
# A Sample.
$stdout.puts("sample#database.size=40.9MB")
# A Measurement.
$stdout.puts("measure#database.query=200ms")

Counters will emit the sum of the collected values. Samples will emit the last value seen. Measurements will emit the min, median, 95th & 99th percentile, max, mean, sum, and count.

Features

Multi-metrics

We want to be able to specify multiple measurements on a single line so as not to have to pay the (albeit low) overhead of writing to stdout. However, we don't want to take every k=v under the sun. L2met has always forced you to think about the things you are measuring and this feature does not regress in that regard.

Example:

echo 'measure.hello=10 measure.world=10' | log-shuttle

This will result in 2 buckets:

  1. {name=hello, vals=[10], ...}
  2. {name=world, vals=[10], ...}

Thus you can measure multiple things provided the key is prefixed with measure..

Heroku Router

The Heroku router has a log line convention described here.

This feature will read the User field in the syslog packet looking for the string "router." If a match is had, we will parse the structured data and massage it to match the l2met convention. Furthermore, we will prefix the measurement name with the parsed host field in the log line.

Example:

path=/logs host=test.l2met.net connect=1ms service=2ms bytes=0

Would produce the following buckets:

  1. {name=router.connect, source=test.l2met.net.connect, vals=[1]}
  2. {name=router.service, source=test.l2met.net.service, vals=[2]}
  3. {name=router.bytes, source=test.l2met.net.bytes, vals=[0]}

Drain Parameters

There are several configuration options that can be specified at the drain level. These options will be applied to all of the metrics coming into the drain.

  • Resolution
  • Prefix

High Resolution Buckets

By default, l2met will hold measurements in 1 minute buckets. This means that data visible in Librato and Graphite have a granularity of 1 minute. However, it is possible to achieve a greater level of resolution. For example, you can get 1 second level resolution by appending a query parameter to your drain url. Note that the resolution is specified in seconds.

The supported resolutions are: 1, 5, 30, 60.

Drain URL:

https://user:[email protected]/logs?resolution=1

Drain Prefix

It can be useful to prepend a string to all metrics going into a drain. For instance, say you want all of your metrics to include the environment or app name. You can achieve this by adding a drain prefix.

Drain URL:

https://[email protected]/logs?prefix=myapp.staging

Bucket Attrs

Most outlet providers allow meta-data to be associated with measurements. Bucket attrs is the l2met feature that allows you to interact with this meta-data.

Units

L2met supports associating units with numbers by appending a non-digit sequence after the digits. L2met will assign all measurements a units of y unless specified. You can specify a unit by prepending /[a-zA-z]+/ after the digit. For instance:

measure.db.get=1ms measure.web.get=1

This will create the following buckets:

  1. {name=db.get, vals=[1], units=ms}
  2. {name=web.get, vals=[1], units=y}

Min Y Value

Librato allows charts to have a min y value. L2met sets this to 0. It cannot be overridden at this time. Open a GH issue if this is a problem for you.

HTTP Outlet

The HTTP Outlet is a read API for your metrics. You can query metrics by id. Id construction is a bit rough at this stage, but if you know what you are looking for, reading metrics can be quite simple. For example, if the following logs are emitted:

measure.db.get=10ms

The metrics can be accessed by the following request:

$ curl http://[email protected]/metrics?name=db.get&resolution=60&units=ms&limit=1&offset=1

Depending on the resolution of the drain, you will get the last bucket offset by one. The offset implies that you are reading the previous bucket with respect to time. For instance, if your resolution is 60 (1 minute) then an offset of 1 will produce the last minute's bucket.

Metric Assertion

You can also assert what you expect the metrics to be. This is useful in that you can point pingdom directly at your l2met instance and create alerts on metrics. For example, if an alert must be made when user signups fall below 5 per minute, we could construct the following request:

$ curl http://[email protected]/metrics?name=user.signup&resolution=60&limit=1&offset=1&count=5

If there were only 4 counts of user.signup, then the query would return a 404 which would cause pingdom to send the alert.

The following metrics can be asserted:

  • count
  • mean
  • sum

Tolerance can also be applied to all of the assertions. This compensates for the lack of inequality operators in the URL. If the tolerance is not present, it is assumed to be 0. If the tolerance is specified, it applies to all metric assertions. You must construct separate requests if you want unique tolerance to metric assertions.

For example, if we wanted to test if the count was 5 +/- 2, the following request would suffice:

$ curl http://[email protected]/metrics?name=user.signup&resolution=60&limit=1&offset=1&count=5&tol=2
Clone this wiki locally