Agile Metrics

Overview

There are no “standard” agile metrics, because what's easy to measure tends to distract us, and what's important to measure is hard to quantify.  The most important thing about Agile metrics is that we need to have a clear objective for using them--normally start with a hypothesis that, say, if defect count drops then lead time will go down too. Many of the metrics below would be used for a short period of time and then dropped when the objective is reached.

With that being said, I've seen clients dovetail push-and-pull metrics, like lead-time & defect count, so that they can be used for a longer period of time, and that if someone starts gaming one number they get penalized on the other.

The following may also be useful:

·       Lead Time

·       Defect count (at various phases; what’s a bug?)

·       Work in Progress

·       Code coverage

·       Unplanned Changes

·       Velocity (story points or story count per sprint)

·       Return on investment

·       Innovations per sprint

·       Artifacts generated

·       Slack time

·       Failure Load (firefighting time)

·       Iteration Burn-Down

·       Unfinished Stories

·       Customer Satisfaction

·       LOC (lines of code)

·       Un-deployed Stories

·       # Blocks

·       Budget/Schedule Compliance

·       Flow Efficiency (lead time / touch time)

·       Release Burn-Up

Definitions

Definitions and cross-references for all these metrics follow.

Lead Time

Defined:            Time from “concept to cash”, the total time it takes to develop an idea and sell it to a paying customer.

Caution:             It may be difficult to measure actual lead time, and many teams approximate lead time by capturing the time a request enters the development process and capturing the time it reaches the definition of done. This approximation may be a reasonable place to start measuring, but it may cause micro-optimization (changes that actually detract from corporate goals) or reduce customer discovery (learning what the customer would pay more for)

Side Effects:     If we blindly push this metric to a minimum, we may see:

·       increased defect count

·       reduced code coverage

·       increased failure load

Benefits:           customer satisfaction, flow efficiency, un-deployed stories, work in progress

Defect Count

Defined:            Total count of surprises, unexpected behavior, flaws, and shortcomings of the product identified during or after an iteration demo.

Caution:             Aggressive definitions of “defect” help everyone focus on customer satisfaction; anything short of the definition above will

Side Effects:     If we blindly push this metric to a minimum, we may see:

·       reduced velocity

·       reduced innovation

·       more unfinished stories

·       more blocks

Benefits:           code coverage, unplanned changes, customer satisfaction, lines of code

Work In Progress (WIP)

Defined:            Number of items we are actively working on. The higher the WIP, the more multi-tasking hurts our efficiency.

Caution:             While a WIP limit of 1 per person may seem ideal, research suggests it’s closer to 2 per person in the event a block prevents us from working the highest priority item.

Side effects:     If we blindly decrease this metric, we may see:

·       excessive slack time

Benefits:           lead time, defect count, velocity, ROI, unfinished stories, un-deployed stories, blocks, budget/schedule compliance, flow efficiency

Code Coverage

Defined:            Percentage of production code tested by the automated regression suite.

Caution:             Static and dynamic code coverage evaluation tools cannot tell us if the code just happened to be executed or if it was verified for proper behavior. The only strategy for full coverage of behavior is Test-Driven Design / TDD / BDD. Short of automation, we cannot find regressions fast enough to keep up with development.

Side effects:     If we blindly increase this metric, we may see:

·       increased failure load

·       decreased velocity

·       reduced innovation

Benefits:           lead time, defect count

Unplanned Changes

Defined:            Number of unanticipated change requests we were able to include in this product increment. Since Agile is all about being more responsive, this is a metric that shows how adaptive we’ve become.

Caution:             Tracking this metric could be burdensome—what counts as a change request? A font style change on the UI? An increase in scope? Pick a granularity to track and stick with it.

Side effects:     If we blindly increase this metric, we may see:

·       decreased velocity (churn)

·       excessive innovation (lack of focus)

Benefits:           customer satisfaction, return on investment, lead time

Velocity

Defined:            Abstract quantity of work that can be completed in a given iteration. Velocity automatically accounts for regular meeting overhead and business-as-usual activities. Velocity is often reported in units of Story Points, Ideal Days, Ideal Hours, or Story Count. Story Points tend to encompass effort, doubt & complexity, so they’re packed with more information than a simple estimate.

Caution:             For large organizations, it helps to normalize Story Points on approximately 1 Ideal Day to simplify strategic & roadmap level planning. Story Points should not used to evaluate past performance—they’re only intended for forward planning.

Side effects:     If we blindly increase this metric, we may see:

·       reduced customer satisfaction

·       increased failure load

·       reduced artifacts generated

·       reduced innovation

·       reduced slack time

Benefits:           lead time, budget/schedule compliance, flow efficiency, release burn-up

Return On Investment

Defined:            Percent earnings based on revenue, capital investment and operational cost.

Caution:             Many teams don’t have access to this data, or don’t track it long enough to see the impact of their work on ROI. Yet it’s key to justifying investment in software.

Side effects:     If we blindly increase this metric, we may see:

·       reduced innovation

·       reduced customer satisfaction

·       increased failure load

Benefits:           lead time, budget/schedule compliance

Innovations per sprint

Defined:            As an agile team becomes more cross-functional, the whole team gains a greater appreciation for what the customer finds valuable. When this results in feature ideas that the Product Owner selects for the backlog, we consider this a success of the whole team.

Caution:             Innovation must be customer-centric—in Kano’s terms, either a linear feature or an exciter/delighter.

Side effects:     If we blindly increase this metric, we may see:

·       reduced release burn-up

·       excessive unplanned changes

·       increased lead time

Benefits:           customer satisfaction, return on investment

Artifacts Generated

Defined:            Any document or non-source-code electronic file generated as a result of the software development process is an artifact. We may want to track help files generated to get a sense of whether our development is sustainable.

Caution:             Some artifacts were historically created for visibility into a long development cycle. If you can rely on automated customer tests instead, this type of “executable specification” will be demonstrably current.

Side effects:     If we blindly increase this metric, we may see:

·       increased lead time

·       increased work in progress

·       reduced budget/schedule compliance

Benefits:           n/a

Slack Time

Defined:            Buffer, maintenance, or creative work that is tangentially related to prioritized product backlog items. Just as a highway has serious congestion at 80% utilization, we see software teams loaded above 80% see serious performance bottlenecks.

Caution:             Slack time is not vacation or goofing off. It is one of the only steps in an agile SDLC that consistently reduces technical debt.

Side effects:     If we blindly increase this metric, we may see:

·       reduced velocity

·       more unfinished stories

·       more un-deployed stories

Benefits:           lead time, failure load, innovations per sprint, customer satisfaction

Failure Load

Defined:            Percent of time spent fixing defects. Failure load is waste; it’s forcing our customers to pay for features twice. We want to avoid failure load whenever practical. You can’t go fast without high quality!

Caution:             n/a

Side effects:     If we blindly decrease this metric, we may see:

·       reduced velocity

·       reduced innovation

·       more unfinished stories

·       more blocks

Benefits:           code coverage, unplanned changes, customer satisfaction, lines of code

Iteration Burn-Down

Defined:            Bar chart showing hours or story points remaining per day of the iteration. The trajectory of the bars shows whether we’re on schedule or not.

Caution:             Without small enough stories, teams will see a “clumping” effect where most of the work tends to get finished at the end of the iteration. This is not desirable—find ways to get to done earlier so there is time to make unforeseen adjustments.

Side effects:     If we blindly improve this metric, we may see:

·       increased blocks

Benefits:           unfinished stories, work in progress, return on investment, customer satisfaction, budget/schedule compliance

Unfinished Stories

Defined:            Any story that did not reach the “definition of done” in the same iteration in which it was begun is an unfinished story. A product owner may cancel, re-schedule, split, re-scope, or defer such a story.

Caution:             Unfinished stories come from a lack of discipline. There’s always a way to negotiate a good story so that it can be split or completed this iteration.

Side effects:     If we blindly decrease this metric, we may see:

·       n/a

Benefits:           customer satisfaction, lead time, release burn-up

Customer Satisfaction

Defined:            increased customer retention or increased revenue

Caution:             Learning about customer retention is slow, and we need safe sandboxes in which to experiment and learn more quickly (e.g., pilot markets or beta tests).

Side effects:     If we blindly increase this metric, we may see:

·       reduced innovation

·       reduced slack time

Benefits:           return on investment

Lines of Code (LOC)

Defined:            One source-code line; from an agile perspective, a line of code increases the risk of system failure and increases the cost of maintenance. We seek elegance, clean code, and avoid duplication in the code base.

Caution:             Mature software shouldn’t always grow. At some point, re-factoring will keep the LOC count stable while we continue to add features. At the same time, if we make code difficult to read or understand, we’ll introduce additional risk for system maintainers.

Side effects:     If we blindly decrease this metric, we may see:

·       increased defect count

Benefits:           lead time, failure load

Un-deployed Stories

Defined:            stories that have reached a team’s definition of done but are not yet actually earning money or being used by a customer

Caution:             until a paying customer uses our product increment, there is risk that delivery teams will need to get involved in supporting it

Side effects:     If we blindly decrease this metric, we may see:

·       decreased customer satisfaction (a product that changes too often?)

Benefits:           lead time, defect count, work in progress, unplanned changes, innovations

# Blocks

Defined:            The number of impediments that development teams have asked for help on.

Caution:             A large number of blocks may mean teams aren’t being as proactive as they could be, or they don’t have an adequate “definition of ready” before accepting work.

Side effects:     If we blindly decrease this metric, we may see:

·       unfinished stories

·       undeployed stories

Benefits:           lead time

Budget/Schedule Compliance

Defined:            compare the estimate of a strategic or roadmap level portfolio item with the team-level estimates (for completed stories only)

Caution:             until a product increment is considered deployable (a minimally marketable feature), we cannot make any assessment on its cost

Side effects:     If we blindly optimize this metric, we may see:

·       reduced innovation

·       fewer unplanned changes

·       reduced customer satisfaction

Benefits:           increased return on investment, reduced lead time, reduced work in progress

Flow Efficiency

Defined:            flow efficiency = lead time / touch time; that is, the amount of time to go through the whole system divided by the actual amount of time someone is actively working on it.

Caution:             Flow efficiency highlights wait time in the existing process, though we really need to focus on value added time. Use this to identify red flags but only as a secondary method to value-added optimization.

Side effects:     If we blindly decrease this metric, we may see:

·       excessive slack time

·       excessively limited WIP

Benefits:           lead time, return on investment

Feedback

What's missing? What would you change?