Metrics and Management

Even metrics proponents note that metric based development is hard¹. DoubleLoop is an app for building metrics that translate into strategy. Bundling a metric / progress tracking plan with a strategy plan is a powerful idea².

Metrics and Pressure

Counter points to managing by metrics

Goodhart’s Law implies that any metric you create will lead to perverse incentives. Scrum evangelists particularly abhor management by metrics³.

• In order to manage to objectives, we must assume that incremental progress can by made. If things are changing dramatically, an easy baseline will be difficult to establish.
• Individual managers will optimize locally while ignore global contribution.
• Metrics can fail to capture the real utility. E.g. example of a metric for weight of nails produced in a day results in one gigantic heavy nail being made⁴.

Metrics should be used when they are causal factors are well understood, or when an experiment can be designed to understand them better. For other scenarios, you have to run data science or econometrics. Richard Marmorstein argues that these situations appropriate for metrics are rare, not common. Therefore, metrics must back a good argument, but a good argument does not require metrics⁵.

Metric Management Anti-patterns :anti_patterns:management-by-metrics:metrics:

Will Larson offers a handful of anti-patterns in metrics⁶:

• Measuring individuals rather than the group
• Trying to get perfect data: data collection is difficult, settle for the level that allows your to make accurate decisions. This in itself is difficult to know, but don’t let perfect be the enemy of good.
• Worrying about how metrics will be miss-used. Certainly, you should be diligent about the metrics you present, but treat miss-understandings as a chance to bring the outsider into alignment
• Operating in isolation: bring in stakeholders and team members to metric creation
• Using optimization metrics (e.g. SDLC metrics) as a report for value creation. Use value creation metrics for stakeholders that need it. Don’t report your internal improvements as if these are valuable in and of themselves to the org. There is particular danger in this mistake with other executives or higher ups (board).
• The number one measurement risk is measuring nothing because you’re trying to measure everything

Executive Dashboards

Too often dashboards focus on what has happened in a business (e.g. revenue, contracts signed, etc). These are referred to as output metrics or results. Input metrics, or metrics that drive those results, are much more useful to learn about the business².

• Dashboards are incentive design and also a source of power struggle (it frames the focus of the business)²
• Dashboards need to provide context and differentiate from valuable information to make decisions⁷.
• Good dashboards show performance against goals in the simplest possible terms. It “draws the shortest possible line from ‘I see it’ to ‘I get it.’”. A good dashboard is opinionated and defines terms⁸.

Engineering Metrics

⁶

• Look to metrics to align discussion on specific contracts with other parties. There may be initial friction in how the metrics are used or understood, but work through those discussions towards a shared understanding.
• Measure for planning your goal is to support project selection and prioritization
• Measure for operation, what do you need to know to be confident that your

software and teams are operating effectively on a day-to-day basis?

• tracking the number of incidents (each connected to an incident review writeup), minutes of user-facing API and website downtime, latency of user-facing APIs and websites, engineering costs normalized against a core business metric (for example, cost to serve your most important API, calculated simply by dividing API requests per month by engineering spend for that month

• Measure to optimize what do you need to know to effectively invest

time into improving engineering productivity

• Measure to inspire and aspire: how can you concisely communicate engineering’s transformational impact to the business?
  • When you speak at company meetings, anchor on these kinds of generational improvements, and try to include at least one such improvement into every annual planning cycle.
• Finance metrics and capitalization
• Get alignment with finance on the way engineering costs allocated across business priorities.

Metrics selection process

Some things are difficult to measure, only measure those if you’ll actually incorporate that data into your decision making. Some things are easy to measure, be willing to measure those to build trust with your stakeholders, even if you don’t find them exceptionally accurate. Whenever possible, only take on one new measurement task at a time .⁶.

• Review the data on a weekly cadence. Ensure that review looks at how the data has changed over the last month, quarter or year
• Maintain a hypothesis for why the data changes. If the data changes in a way that violates your hypothesis (“cost per request should go down when requests per second go up, but in this case the cost went up as requests per second went up, why is that?”)
• Avoid spending too much time alone with the data, instead bring others with different points of view to push on the data together
• Segment data to capture distinct experiences
• Discuss how the objective measurement does or does not correspond with the subjective experience of whatever the data represents.

References