How to Select Agile Metrics and Some Examples

One of the main tenets of agile and lean is a short feedback loop. Sprint reviews, retrospectives, and other communication give us important qualitative feedback. For trending and comparison against goals, metrics can be very valuable. But metrics can be dangerous as well. The wrong metrics can give false data or even have a negative impact. It is important to select the right metrics and to use them correctly.

There are several considerations when selecting metrics. Some are universal but most depend on the context including company culture, the maturity of the team and the type of project the team is working on. You, therefore, must regularly review your metrics as the context will change.

First of all, define your goals. Have clarity around the desired outcome the metric is tracking. Vanity metrics that appear impressive but do not provide actionable feedback should normally be avoided [27]. Sometimes vanity metrics can provide leading indicators or valuable in other ways [3, 18, 29].

Measure only what matters. Too many metrics don’t only waste time they also weaken the signal of the metrics we should act on.

Know the audience of your metric. I have squandered a significant amount of time tracking decent data the intended audience did not care about. Maybe they should have been interested but they were not. This effectively made my metrics not better than vanity metrics because the information was ignored. Some metrics should stay internal to the team. This is not a lack of transparency but a precaution against sharing data out of context where it may be misunderstood or even abused. E.g. velocity is an excellent internal team metric for planning but is an anti-pattern for reporting productivity to management [1, 10, 14, 22].

To allow us to timely react to the metric we should aim for leading rather than lagging metrics. Leading metrics are difficult to measure but provide more opportunity to adjust. Lagging metrics are easier to measure but delay the impact of any improvements [30]. Leading indicators can be a challenge, especially when trying to measure added value.

If we focus too much on the metric rather than the desired outcome we are trying to measure, we may get the wrong results [1]. To avoid single-loop learning caused by simplifying complexity at the cost of losing sight of the real end goal, explicitly link metrics to goals. Providing this context also allows people to better challenge their relevance or suggest better metrics [2].

When you turn a qualitative and highly interpretive issue (think of productivity, quality, and usability) into a number, any figure is relative and arbitrary. It often is better to look at trends rather than absolute numbers. Trends show whether the team is moving in the right direction.

When running a program or a portfolio it may be a challenge to get meaningful program-level data while allowing the teams the freedom to organize the way that works best for them. See [4]  for some ideas.

The average over a series works well for some metrics. For metrics that do not have a normal distribution or a high standard deviation an average indicating good health may hide problems. Consider distributions, mean, or percentiles [17].

Suggested Metrics

The suggested metrics below are a selection from the references at the end of this blog and my own experience.

Agile Maturity and process health

• Adherence to team working agreements (e.g. DoD, DoR, decision process)
• Focus – time spent on team goal vs external goals
• Time Blocked per Work Item [1, 24]
• Sprint review metrics [7]
• Lead Time [1, 19, 20, 24, 3]
• Cycle Time [1, 19, 8, 9, 24, 3]
• Throughput [24]
• Velocity
• Predictability, story/story points completion ratio [8, 3]
• WIP (work in  progress) [24]
• Acceleration [15]
• Duration of open pull requests [3]
• Comments per pull requests [3]

Technical

• Test Coverage [1, 20]
• Code complexity [1]
• Escape rate [1, 8, 9]
• Crash rate [1]
• Build time [1]
• PR size distribution
• Good versus failed builds [3]
• Duplicate code [20]
• MTTR (mean time to repair)
• MTBF (mean time between failures)
• Recidivism (ratio of user stories that bounce back to development) [3]
• Defect count by sprint/story count /story points [3]

Product Health

• Release / feature adoption
• Release success rate [1]
• User Analytics
• Time spent on new features planned tech debt reduction and unplanned defect resolution [9]

Team Health

• Happiness [1, 8, 20] or Meaning [11], See also [21]
• Team Tenure [1]
• Time to onboard new team members
• Stats on the need to consult team members who have left the team [1]
• Collaboration statistics. Percentage of tasks executed by one vs multiple team members [1]
• Sustainable pace [22]
• Employee net promoter score [23]

Continuous Improvement

• Learning Log [1]
• Time spent on learning training [22]
• Number of improvements identified in retrospectives and number implemented

References

[1] Understanding Agile Team Metrics: Measure Many Things by Andy Cleff
[2] An Appropriate Use of Metrics by Martin Fowler
[3] Agile Metrics: the Ultimate Guide by Leon Tranter
[4] Apply Consistent Metrics Categories Across An Agile Portfolio  by Scott Ambler
[5] Agile Metrics Madness by Teri Christian
[6] Faking Agile Metrics or Cooking the Agile Books by Stefan Wolpers
[7] Seven Simple Metrics for the Sprint Review by Tanner Wortham
[8] 4 Balanced Metrics for Tracking Agile Teams Joel Bancroft-Connors
[9] Metrics for Agile by Esther Derby
[10] Agile Metrics — The Good, the Bad, and the Ugly by Stefan Wolpers
[11] Why You Should Stop Trying to Be Happy at Work by Susan Peppercorn
[12] Metrics by Scaled Agile
[13] Scrum checklist by Henrik Kniberg
[14] Velocity is Killing Agility by Jim Highsmith
[15] Acceleration: An Agile Productivity Measure by Scott Ambler
[16] The Lean Startup by Eric Ries
[17] The DevOps Handbook by Gene Kim
[18] What Metrics Should Innovation Programs Measure? By Tristan Kromer
[19]] Towards a Useful Set of Agile Metrics
[20] The Happiness metric and a few others by Anders Laestadius
[21] The surprising truth about what motivates us by Dan Pink
[22] Measuring Team Performance by Ken Rubin
[23] Unlock Excellence with Agile Metrics by Kathryn Kuhn
[24] 6 Agile Metrics to Boost Your Project Management by Kanbanize
[25] Understanding Agile Metrics by Bob Ellis
[26] We can’t measure Programmer Productivity … or can we? by Jim Bird
[27] Vanity Metrics vs. Actionable Metrics by Eric Reis
[28] Are Vanity Metrics Really All That Bad? by Michael Cohn
[29] Don’t Be Fooled By Vanity Metrics by Erick Schonfeld
[30] Leading and Lagging Indicators by Derek Huether