Graphite: Getting derivative to work with empty data points

While trying to calculate the number of servers added per day, was having an issue today.

We have a running metric that holds the total number of servers in our network, so I wanted to calculate the difference between data points (one per day) to know how many servers we add per day. It seemed derivative function was coming to rescue:

derivative(seriesList)
This is the opposite of the integral function. This is useful for taking a running total metric and calculating the delta between subsequent data points.

 

So equipped with this, I quickly build the formula to get the daily servers added:

alias(summarize(derivative(heartbeat.cng_full_servers_total), '1day', 'sum', true), 'Servers Added')

But this gave just a blank graph ... 

While checking more in depth, I found out that the resolution of the heartbit metrics is every hour (in conf/storage-schemas.conf):

[heartbeat]
pattern = ^heartbeat\.
retentions = 3600s:1825d

And when checking the contents of the cng_full_servers_total metric, we can see that there are empty data points (as metric is sent just once per day):

# whisper-fetch.py cng_full_servers_total.wsp 

1455807600None
1455811200None
1455814800None
1455818400None
1455822000None
1455825600None
1455829200None
1455832800552317.000000
1455836400None
1455840000None
1455843600None
1455847200None
1455850800None
1455854400None

Derivative doesn't like empty data points, so we need to find a workaround for this. keepLastValue to the rescue. As per graphite:

keepLastValue(seriesListlimit=inf)
Takes one metric or a wildcard seriesList, and optionally a limit to the number of ‘None’ values to skip over. Continues the line with the last received value when gaps (‘None’ values) appear in your data, rather than breaking your line.

So, we build a new formula that fixes the issue:

alias(summarize(derivative(keepLastValue(heartbeat.cng_full_servers_total)), '1day', 'sum', true), 'Servers Added')

Hope this helps someone else!