Engineering Metrics1
- Look to metrics to align discussion on specific contracts with other parties. There may be initial friction in how the metrics are used or understood, but work through those discussions towards a shared understanding.
- Measure for planning your goal is to support project selection and prioritization
- Measure for operation, what do you need to know to be confident that your software and teams are operating effectively on a day-to-day basis?
- tracking the number of incidents (each connected to an incident review writeup), minutes of user-facing API and website downtime, latency of user-facing APIs and websites, engineering costs normalized against a core business metric (for example, cost to serve your most important API, calculated simply by dividing API requests per month by engineering spend for that month
- Measure to optimize what do you need to know to effectively invest
time into improving engineering productivity
- Measure to inspire and aspire: how can you concisely communicate
engineeringâs transformational impact to the business?
- When you speak at company meetings, anchor on these kinds of generational improvements, and try to include at least one such improvement into every Annual Planning.
- Finance metrics and capitalization
- Get alignment with finance on the way engineering costs allocated across business priorities.
Metrics selection process
Dan Davies 2 Numbers are collected for a purpose, and itâs often surprisingly hard to use them for any other purpose
Some things are difficult to measure, only measure those if youâll actually incorporate that data into your decision making. Some things are easy to measure, be willing to measure those to build trust with your stakeholders, even if you donât find them exceptionally accurate. Whenever possible, only take on one new measurement task at a time .1.
- Review the data on a weekly cadence. Ensure that review looks at how the data has changed over the last month, quarter or year
- Maintain a hypothesis for why the data changes. If the data changes in a way that violates your hypothesis (âcost per request should go down when requests per second go up, but in this case the cost went up as requests per second went up, why is that?â)
- Avoid spending too much time alone with the data, instead bring others with different points of view to push on the data together
- Segment data to capture distinct experiences
- Discuss how the objective measurement does or does not correspond with the subjective experience of whatever the data represents. Even metrics proponents note that metric based development is hard3. DoubleLoop is an app for building metrics that translate into strategy. Bundling a metric / progress tracking plan with a strategy plan is a powerful idea4.
Counter points to managing by metrics
Quote
Not everything that counts is countable. Not everything that can be counted counts.
Goodhartâs Law implies that any metric you create will lead to perverse incentives. Scrum evangelists particularly abhor management by metrics5.
- In order to manage to objectives, we must assume that incremental progress can by made. If things are changing dramatically, an easy baseline will be difficult to establish.
- Individual managers will optimize locally while ignore global contribution.
- Metrics can fail to capture the real utility. E.g. example of a metric for weight of nails produced in a day results in one gigantic heavy nail being made6.
Metrics should be used when they are causal factors are well understood, or when an experiment can be designed to understand them better. For other scenarios, you have to run data science or econometrics. Richard Marmorstein argues that these situations appropriate for metrics are rare, not common. Therefore, metrics must back a good argument, but a good argument does not require metrics7.
Antipatterns Will Larson offers a handful of 1:
- Measuring individuals rather than the group
- Trying to get perfect data: data collection is difficult, settle for the level that allows your to make accurate decisions. This in itself is difficult to know, but donât let perfect be the enemy of good.
- Worrying about how metrics will be miss-used. Certainly, you should be diligent about the metrics you present, but treat miss-understandings as a chance to bring the outsider into alignment
- Operating in isolation: bring in stakeholders and team members to metric creation
- Using optimization metrics (e.g. SDLC metrics) as a report for value creation. Use value creation metrics for stakeholders that need it. Donât report your internal improvements as if these are valuable in and of themselves to the org. There is particular danger in this mistake with other executives or higher ups (board).
- The number one measurement risk is measuring nothing because youâre trying to measure everything
Executive Dashboards
Engineering Metrics1
- Look to metrics to align discussion on specific contracts with other parties. There may be initial friction in how the metrics are used or understood, but work through those discussions towards a shared understanding.
- Measure for planning your goal is to support project selection and prioritization
- Measure for operation, what do you need to know to be confident that your software and teams are operating effectively on a day-to-day basis?
- tracking the number of incidents (each connected to an incident review writeup), minutes of user-facing API and website downtime, latency of user-facing APIs and websites, engineering costs normalized against a core business metric (for example, cost to serve your most important API, calculated simply by dividing API requests per month by engineering spend for that month
- Measure to optimize what do you need to know to effectively invest
time into improving engineering productivity
- Measure to inspire and aspire: how can you concisely communicate
engineeringâs transformational impact to the business?
- When you speak at company meetings, anchor on these kinds of generational improvements, and try to include at least one such improvement into every Annual Planning.
- Finance metrics and capitalization
- Get alignment with finance on the way engineering costs allocated across business priorities.
Metrics selection process
Some things are difficult to measure, only measure those if youâll actually incorporate that data into your decision making. Some things are easy to measure, be willing to measure those to build trust with your stakeholders, even if you donât find them exceptionally accurate. Whenever possible, only take on one new measurement task at a time .1.
- Review the data on a weekly cadence. Ensure that review looks at how the data has changed over the last month, quarter or year
- Maintain a hypothesis for why the data changes. If the data changes in a way that violates your hypothesis (âcost per request should go down when requests per second go up, but in this case the cost went up as requests per second went up, why is that?â)
- Avoid spending too much time alone with the data, instead bring others with different points of view to push on the data together
- Segment data to capture distinct experiences
- Discuss how the objective measurement does or does not correspond with the subjective experience of whatever the data represents.
1. Larson, W. Measuring an engineering organization. https://lethain.com/measuring-engineering-organizations/ (2023).
2. Davies, D. The Unaccountability Machine: Why Big Systems Make Terrible Decisions â and How the World Lost Its Mind. (Profile Books Ltd, 2024).
3. Schmidt, D. Metrics-driven product development is hard. https://blog.doubleloop.app/metrics-driven-product-development-is-hard/ (2021).
4. Critchlow, T. Some notes on executive dashboards. https://tomcritchlow.com/2022/05/06/executive-dashboards/ (2022).
5. Holub, A. KPIs, Velocity, and Other Destructive Metrics. https://holub.com/kpis-velocity-and-other-destructive-metrics/ (2018).
6. DeMarco, T. Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency. (Currency, New York; London, 2002).
7. Marmorstein, R. Be good-argument-driven, not data-driven. http://twitchard.github.io/posts/2022-08-26-metrics-schmetrics.html (2022).