While trying to calculate the number of servers added per day, was having an issue today.
We have a running metric that holds the total number of servers in our network, so I wanted to calculate the difference between data points (one per day) to know how many servers we add per day. It seemed derivative function was coming to rescue:
This is the opposite of the integral function. This is useful for taking a running total metric and calculating the delta between subsequent data points.
So equipped with this, I quickly build the formula to get the daily servers added:
alias(summarize(derivative(heartbeat.cng_full_servers_total), '1day', 'sum', true), 'Servers Added')
But this gave just a blank graph ...
While checking more in depth, I found out that the resolution of the heartbit metrics is every hour (in conf/storage-schemas.conf):
[heartbeat] pattern = ^heartbeat\. retentions = 3600s:1825d
And when checking the contents of the cng_full_servers_total metric, we can see that there are empty data points (as metric is sent just once per day):
# whisper-fetch.py cng_full_servers_total.wsp 1455807600None 1455811200None 1455814800None 1455818400None 1455822000None 1455825600None 1455829200None 1455832800552317.000000 1455836400None 1455840000None 1455843600None 1455847200None 1455850800None 1455854400None
Derivative doesn't like empty data points, so we need to find a workaround for this. keepLastValue to the rescue. As per graphite:
Takes one metric or a wildcard seriesList, and optionally a limit to the number of ‘None’ values to skip over. Continues the line with the last received value when gaps (‘None’ values) appear in your data, rather than breaking your line.
So, we build a new formula that fixes the issue:
alias(summarize(derivative(keepLastValue(heartbeat.cng_full_servers_total)), '1day', 'sum', true), 'Servers Added')
Hope this helps someone else!