For years, software engineering teams have chased after a way to measure their efficiency with hard metrics that truly help them improve—without making developers feel spied upon. Finally, we’re getting somewhere.
Any developer knows the pain or potential pain of being measured against dubious metrics like lines of code written or number of pull requests merged that our industry has been known for historically. And any engineering manager knows the backlash and distrust such measures can instill in their team.
But when boards of directors, engineering leaders, and developers alike all want to know if a process is working, if the team is efficient, and how to be better, we need a way to measure the work that’s being done.
Several sets of metrics, frameworks, and best practices have arisen to accomplish this. Inevitably, some do it better than others. The holy grail is measuring work with the tools and systems that developers already work with daily. DORA metrics can do this, and it’s partly why they’re becoming the industry standard.
We’ll dive into that more, but first, let’s understand the other types of metrics out there.
Busyness metrics
Busyness metrics can be thought of as measuring how much flow time a developer has. If your flow is interrupted two or three times a day, you know it’s next to impossible to get things done.
In an attempt to guard developers’ time, a whole category of engineering efficiency tools were developed that connect to HR systems and calendars. They try to assess if a developer has too many context switches, meetings, and time-sucking processes to follow.
Ultimately, these metrics try to prevent burnout by looking at the human side of coding, which certainly matters, but these metrics are not very actionable.
If you know developers are in too many meetings, how do you structure an environment where the necessary meetings take place but the flow is also more efficient? The busyness metrics don’t come with a set of potential improvements to guide you.
SPACE framework
Nicole Forsgren, one of the founders of DORA (DevOps Research Assessment), developed this framework, which aims to understand developer productivity. But rather than hard metrics, the SPACE framework focuses on developers’ state of mind and physical well-being, which are no doubt important factors in developers’ overall enjoyment of their work and a team’s engineering performance.
The SPACE framework gauges five dimensions of developer productivity:
- Satisfaction: Satisfaction with their work, tooling, team, and culture; well-being
- Performance: The outcome of a system or process
- Activity: The quantity of work done, measured in terms of its outputs and actions
- Communication and collaboration: The collaborative process and support that represents software development teams
- Efficiency and flow: The degree to which software developers can progress in their tasks
Like Busyness metrics, the SPACE framework captures valid information, but it’s hard to act on. Think of it mostly as best practices that are hard to measure from the work being done. It’s missing succinctness and goal-oriented outcomes.
Old school metrics
These are the hard measures that are easy to game and don’t capture real developer effort—things like lines of code written, number of pull requests merged, number of hours spent coding. These measures came out of the punch card programming days, where the developer who accomplished the task with the least amount of instructions was the leader.
But developers know that they don’t actually measure anything important. I can write the most important five lines of code in the application that are so complex it would take me two weeks to make sure they’re the right five lines of code. Or, I could write five million lines of code that aren’t very useful. It’s the same with measuring the number of pull requests merged. This can tell you a little about your overall batch size, but that’s not incredibly insightful or useful for helping a team improve.
If you judge developers against these measures, they’ll know you don’t understand them or their work. Furthermore, measuring these things on an individual scale is toxic. Devs will feel spied on and judged, and they’ll dig in their heels.
Value stream metrics
The goal of value stream metrics is to know the distribution of engineering investments, i.e. where those investments are going. That’s especially useful in cases where organizations get a research and development credit from the government and need to classify how much work was R&D, fixing bugs, keeping the lights on, etc. These metrics are more about learning what teams are investing in than figuring out how to help them improve.
DORA metrics
Clearly, the above metrics haven’t all stuck. But why are so many teams and organizations embracing DORA metrics instead? Six key reasons come to mind.
- They’re backed by research, which shows a statistically significant correlation between positive DORA metrics and positive organizational performance. DORA metrics are not a gut feeling.
- DORA metrics are a crystallization of the DevOps practices we’ve been applying for many years but in a succinct way. The DORA metrics show how well your team is doing at continuous improvement and learning. For example, we’ve understood through practice that reducing batch size was effective because it allowed us to get work done quickly. DORA put those things into categories of metrics—deploy frequency, change lead time, change failure rate, and mean time to recovery— and showed how they relate to each other. From a practitioner’s perspective, DORA metrics have named the things we’ve always done.
- DORA metrics keep it simple. Organizations often get bogged down when deciding what to measure in terms of engineering. DORA allows teams to start with metrics that are well defined with industry benchmarks and have the wisdom of the crowd behind them.
- DORA metrics are team metrics and therefore don’t create the same fears and worries that individual metrics bring up for developers. DORA metrics can still be weaponized, but they recognize that software development is a team sport. If you read about DORA and the State of DevOps reports, they’re all about teams.
- DORA metrics distill complex activities into simple, hard measures. They can take data from source control, source review systems, issue trackers, incident management providers, and metrics tools and turn them into four key measures. This makes it possible to compare DORA metrics from one team to the next, even though not all teams are equal. The DORA research allows teams to bucket themselves into low, medium, and high performance categories based on how they perform across the four key metrics mentioned above. This allows teams to draw wide conclusions about how they perform compared to other teams.
- DORA metrics cover a broad swath, including the developer process and how well that process is delivering to customers. DORA metrics look at the process from the time a developer starts coding to the time the team delivers something to production. They recognize that no one wants to take the “move fast and break stuff” approach. DORA metrics encourage the healthier approach of “move fast, responsibly.”
DORA metrics are not the silver bullet to be the best engineering team—no set of metrics is. But DORA metrics have helped the software industry rally around a scientific system of measuring software delivery and operational performance in a way that developers actually don’t mind. Maybe they even like it.
Dylan Etkin is CEO and co-founder of Sleuth, the leading deployment-based metrics tracker. As one of the first 20 employees at Atlassian, Dylan was a founding engineer and the first architect of Jira. He has led engineering for products at scale in Bitbucket and Statuspage. He has a master’s degree in computer science from ASU.
—
New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to newtechforum@infoworld.com.