Metrics are how we assess aspects of our business, applications, and infrastructure. They allow us to measure and report on key indicators, and we can then use that information for making decisions, or to adjust for improved performance, stability and overall quality.
While collecting and analyzing metrics can be hard work, in many respects, the most challenging aspect of working with metrics is identifying which ones you should collect in the first place. You want to spend your time collecting metrics that matter, and avoid “vanity metrics”, by which I mean those that don’t accurately reflect your environment, or provide extraneous information that might seem useful but does not actually improve outcomes.
In this article I explain why vanity metrics pose a challenge and how to avoid them so that you collect and analyze only those metrics that help to drive positive change.
Vanity metrics might seem attractive, but they don’t deliver true value.
There is an argument to be made for picking metrics which make your product look good, but it’s not advisable. Sure, they’ll look great to stakeholders, but it will also lull you into a false sense of security.
Another problem with vanity metrics is that if you are using them to drive action, you might be set up to do a whole lot of work maximizing the wrong things.
Metrics should drive action and should highlight aspects of your system that are critical to its health and continued performance, so it is vital that you identify the right metrics to track.
Now that you know why vanity metrics are not useful, let’s discuss how to identify the metrics that do matter.
I’d like to introduce you to the four A’s of effective metrics.
Here’s what each of these types of metrics mean.
Before you measure anything, determine who the owner is. If the ownership of a metric is ambiguous or shared between multiple teams, you may encounter problems tying the success or failure of the metric to a particular individual or group; or worse, the success of one team may be eclipsed by the sub-optimal performance of another.
Establish a clear owner for each metric.The importance of this will become more apparent as we continue with the rest of the characteristics of metrics that matter.
Any metrics which you track should be actionable and should be tied to specific performance targets. If there isn’t an action you can take in response to a given metric, the value of that particular metric becomes questionable.
You can have the greatest minds in your organization curate a report with every possible permutation of raw and derived metrics, but if the info isn’t in an easily digestible format, it’s not going to be implementable by those in the trenches. Metrics and the reports containing them should be easy to understand for anyone who might read them.
Finally, ensure that you can validate the accuracy of metrics. Using metrics as a foundation from which to drive progress is only relevant if the improvement can be measured, and if the particular measurements being instrumented are accurate.
Monitoring is the process of collecting metrics from an application in real time or near-real time. The metrics are then used to determine the current state and health of the application and the infrastructure on which it installed.
When implemented correctly, with the right collection of metrics, an effective monitoring plan can ensure that your applications are running efficiently and effectively.
Additionally, if things take a turn for the worse, developers and production support personnel should be able to use the data available from your monitoring system to quickly isolate the cause and formulate a way to resolve it.
In addition to ensuring that the metrics you choose meet the requirements outlined by the four A’s, there are a couple more guidelines which will ensure you’re making the most of your metrics system.
First, avoid the temptation to monitor total counts or run total metrics. Total counts are a big part of the vanity metric problem. What you want to monitor instead is a specific rate over time. Calculating the number of transactions being performed over a distinct period will be significantly more useful to developers and stakeholders alike. You can even measure the rate of change over time to highlight whether your system is improving or deteriorating.
Second, make your metrics visible. The right collection of metrics should highlight both successes and failures. Ensure your metrics are displayed prominently to the owners of the metrics and other key stakeholders within the organization. Emphasize that a downturn in the metrics represents an opportunity for improvement, whereas good metrics should spur on the process of continuous improvement forward.
Finally, if you need help identifying, analyzing and displaying metrics related to your system, seek out the help of an expert, and investigate tools built specifically to help you expose useful and actionable metrics. Retrace by Stackify is one such tool, and you can try it free for 14 days. You can sign up for the trial or request a demo.
Stackify's APM tools are used by thousands of .NET, Java, PHP, Node.js, Python, & Ruby developers all over the world.
Explore Retrace's product features to learn more.
