If you’re building a web-based app, it’s always a good idea to build some instrumentation into your app. That way, you can see how your app is behaving, and how your users are interacting with your app over time.

I’m sure everyone who reads my blog is familiar with Google Analytics for tracking page hits. But what about what’s happening inside your app? Right now? Do you know?

Graphite is one way to graph the stats that you add to your app. Combine it with (say) statsd from Etsy, and adding any stats you want is easy. (Read this blog post from Etsy if you want to learn more about measuring your app, and how to add support for Statsd to your app).

Normally, you’ll probably be interested in looking at graphs that show your stats over a period of hours or days (for trend analysis), and both Graphite and Statsd are sensibly tuned for that. But what if you want to see what’s happening now, in real time? I couldn’t find any clear instructions on how to do that elsewhere, so here’s my take on how to do it. I’m assuming you’re already familiar with installing both Statsd and Graphite, and that you’ve already had both up and running successfully with their default configurations.

Making Statsd Forward Data In Real Time

By default, Etsy’s Statsd collects the data sent from your apps, and forwards it on to Graphite’s data collector (known as Carbon) every 10 seconds. For real time, we need the data forwarded on every second. To do that, edit your Statsd config, adding the ‘flushInterval’ value:

{
  graphitePort: 2003
, graphiteHost: "localhost"
, port: 8123
, flushInterval: 1000
}

A value of 1000 tells Statsd to forward data on every second.

Making Graphite Store Data At One Second Resolution

Graphite’s default / sample configuration tells it to store incoming data at 60-second resolution; that allows us to look at the total stats recorded minute by minute, but we can’t drill down to see what happens second by second. To do that, we have to tell Graphite to store the data on a second-by-second basis.

Edit /opt/graphite/conf/storage-schemas.conf, and add the following clause:

[real_time]
priority = 200
pattern = ^stats.*
retentions = 1:34560000

This tells Graphite that we want all the data received from Statsd to be kept on a second-by-second basis for 400 days … plenty long enough for any sort of comparison you might need to do.

I found that, to get Graphite to start using this new storage definition, I had to go and delete the existing data files by doing:

$ rm -rf /opt/graphite/storage/whisper/stats*

Getting Graphite To Draw Real-Time Graphs

Now all we need to do is to get Graphite showing you all the collected data in real-time. By default, Graphite will happily plot the data onto a graph, but will only generate an updated graph every 60 seconds. That’s perfect for an ops team looking for trends over hours, but it isn’t real-time.

If you’re using Memcache with Graphite, you’ll need to add this to your /opt/graphite/webapp/graphite/local_settings.py file, to tell Graphite to only cache data in Memcache for 1 second:

MEMCACHE_DURATION = 1

Is it worth caching the data at all at this resolution? Honestly, I don’t know. I guess that depends on how many people need to watch the data in real-time or not. Ideally, it would be better if Graphite dynamically set the Memcache timeout based on the data stored in the particular key, but for now, you need to either stop using Memcache, or set the cache duration to 1 second.

This now gives you graphs with 1-second resolution … now we just need to change the Graphite web-app’s auto-refresh feature to load a new graph every second. By default, it will only generate an updated graph every 60 seconds. To change this, we have to edit some of the app’s Javascript code.

  • Open the file /opt/graphite/webapp/content/js/composer_widgets.js, and locate the function ‘toggleAutoRefresh’.
  • Locate the ‘interval’ variable inside that function. Change its value from ’60′ to ’1′.
  • Save the file, then refresh your web browser page.

Et voila. If you switch on auto refresh, you should now be able to see your app’s data being plotted second by second, giving you a real-time view of what your app is doing.

5 Comments

  1. Graphs says:
    September 22nd, 2011 at 4:01 am

    An informative hub, must say. Keep writing.

  2. Graham says:
    November 24th, 2011 at 10:28 pm

    You don’t need to delete the .wsp files to get the new retention settings to work; you just need to run /opt/graphite/whisper/bin/whisper-resize.py on them with the new retention settings and it will resize the databases for you.

  3. Felix says:
    February 9th, 2012 at 9:55 pm

    Have you had any trouble with Graphite (with data fed from statsd) not displaying accurate information for counters?

  4. Stuart Herbert says:
    February 18th, 2012 at 3:31 pm

    @felix: I haven’t inspected the underlying data to check either way, tbh.

  5. Scaling-out WordPress – Performance Measures – Using Graphite | Speed your IT says:
    August 24th, 2012 at 3:18 pm

    [...] Instructions from http://blog.stuartherbert.com/php/2011/09/21/real-time-graphing-with-graphite/ [...]