“Debt is like any other trap, easy enough to get into, but hard enough to get out of.”
—Josh Billings (American humorist)
It’s one of the dirtiest words in tech. Just as in life, the very mention of debt conjures feelings of being weighed down, under stress. And getting out of debt is a chore.
In software engineering specifically, technical debt generally refers to a system that is ageing and eating up engineers’ valuable time. Technical debt is something to be managed, maintained, minimized. It’s the last thing on the backlog. It will eventually sink you.
But does it have to be this way?
A growing school of thought among progressive engineering organizations says that technical debt should be a core part of the job of all software developers, and that by proactively managing technical debt, those teams may not only avoid sinking, but can actually outpace the competition.
What is technical debt?
Originally coined by computer scientist Ward Cunningham in 1992, technical debt is the idea that building short-term solutions to technical systems incurs a set of trade-offs, resulting in future obligations that must be repaid in the form of engineering work.
As software developer Martin Fowler described it in 2003:
In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development.
The average software developer spends more than 13 hours a week addressing technical debt, according to Stripe’s Developer Coefficient report from 2018. Now, as applications become increasingly complex, managing that debt has never been more critical. “Any code you have decided is a liability is tech debt,” Alexandre Omeyer, CEO of Stepsize, told InfoWorld.
Technical debt takes a great variety of forms. “On the lower end, it can be a small area of code that requires refactoring, libraries that need upgrading, or unit testing that needs fixing,” InfoWorld columnist Isaac Sacolick wrote last year. “On the higher end, technical debt includes reengineering complex monolithic applications, porting outdated web service protocols, consolidating multiple platforms to one standard, cleansing data debt issues, modernizing infrastructure, introducing observability practices, or automating a backlog of manual test cases. The worst type of technical debt is a ‘burning platform,’ meaning a platform with recurring outages and incidents that impact the business.”
While working on anything described as a burning platform may seem less appealing than shipping shiny new features, only by attacking technical debt as a team, in a proactive and ongoing manner, can developers avoid pain in the longer term.
“Addressing technical debt often gets short shrift, because doing so rarely addresses an urgent business need and, especially for nonurgent cases, the ROI is unclear and thus perceived as deferrable,” Sacolick wrote. “It’s a classic issue for anything involving maintenance, whether code or houses.”
Peering into the technical debt abyss
For Mik Kersten, author of Project to Product and founder of Tasktop, “Technical debt needs to be a first-class thing to tackle proactively. Unfortunately, in many cases, that is a novel concept.”
For many engineering teams, especially those in fast-growing organizations, technical debt may feel like it is in tension with the important work of pumping out new features. But for Honeycomb CTO and cofounder Charity Majors, tech debt itself is “a sign of success, it means that you are still alive.”
Only by “making sure the little things are taken care of can you ensure they don’t grow up to be big gremlins,” Majors told InfoWorld.
While that may be easy to say, there is a cultural shift that needs to happen across an entire organization to ensure that technical debt is taken seriously.
“To be able to have a true backlog that is prioritized is a challenge for a lot of enterprises, especially those that get to a point where they have incentives to ship new things, but tend to not have any kind of specific performance-based incentive to manage their tech debt at the same time,” RedMonk analyst Rachel Stephens told InfoWorld.
Or, as Tasktop’s Kersten said, “By only incentivizing features you will put yourself in a tech debt death spiral.”
How to take charge of your technical debt
John Kodumal, CTO and cofounder of LaunchDarkly, told InfoWorld earlier this year that while “technical debt is inevitable in software development,” being proactive about reducing debt over time is “much healthier than stopping other work and trying to dig out from a mountain of debt.”
This proactive approach to contending with technical debt involves treating anything you might consider as technical debt as another piece of feature work to be included in your normal agile process.
“The secret to maintaining applications and avoiding, or at least delaying, legacy status, lies in how organizations and teams manage technical debt,” Sacolick wrote. The key is being proactive, which means: “Establishing policy, convention, and processes to amortize the cost of reducing debt over time.”
Many teams will be tempted to devote a certain amount of capacity to tackling technical debt, such as 20% of each sprint. However, Tasktop’s Kersten advises taking a more dynamic approach that considers the context of upcoming deadlines or team capacity.
“You should measure the business outcome and the investment in tech debt and ensure that those balance out,” Kersten said. “Make technical debt visible so you always know how much you are managing. Once it is visible, set a target delivered percentage, which must be dynamic over time.”
For Ben Kus, CTO at cloud storage company Box, paying down technical debt is something all engineering teams need to include in their backlog. “Of course, that gets pushed back, but there should be an ongoing sense that you are constantly tackling those things,” Kus said.
Kus doesn’t recommend assigning certain engineers to focus on technical debt, though. “Don’t call it that, that is where attrition will come from,” he said. “No one wants to work on tech debt, or refactoring, any tasks like that.”
Instead, at Box they look to divide work evenly across engineering teams and surface issues during the sprint process, in postmortems, and when on call. “We have a rigid postmortem process and we identify things to fix to prevent the same issues occurring again,” Kus said. “We aren’t as presumptive to say ‘drop everything to fix something,’ but we do make it clear that if an issue happens again, that becomes an accountability issue. That is extremely unpleasant if it is the second time something happens.”
The importance of on-call rotations
That on-call element is increasingly important as engineering teams look to effectively unearth and measure the technical debt that is slowing them down.
Engineering managers like Honeycomb’s Majors are proponents of regularly pulling engineers from feature work to be on-call and focus on fixing, refactoring, and automating away that debt.
“Having an engineer who is responsible primarily for the little things is important. And as part of your on-call, you should be actively discouraged from doing product work. This introduces flex into a system that usually has very little,” Majors said.
Chris Evans is the founder of Incident.io, a software startup specializing in incident response. “The whole point of tracking things is weeding out that tacit knowledge,” Evans told InfoWorld. “You will be paged for things that you are not best placed to deal with.”
While that might sound scary at first, issues will get fixed, and then, by emphasizing what went wrong during post-incident washups or postmortems, the importance of tackling that technical debt can become more apparent.
“By taking on the operational responsibility for the work we do, we tighten the feedback loops between the shipping and running. This helps us to make pragmatic engineering decisions and provide a healthy tension between shipping new code and supporting and improving what we have,” Evans wrote in a December blog post.
For example, Incident.io engineers had recently been slowed down by interactions with one of its databases. “With a week of investment, we think we can build a better way to interact with the database, which will have a compounding effect on how we build every feature in the future,” Evans said.
And those successes should be celebrated as significantly as a major incident being resolved, or a cool new feature landing, whether that takes the form of Charity Majors’ Tiara of Technical Debt or Mik Kersten’s celebrating the “slaying of a big chunk of technical debt like winning a new customer.”
Rethinking technical debt at the Financial Times
The Financial Times has spent the past six years reshaping its approach to technical debt. Back in 2015, the British newspaper’s website was powered by a monolithic app called Falcon. In 2016, the company’s developers converted Falcon into a set of microservices, now called Next in its entirety. The 332 code repositories that resulted is managed by a set of durable teams with defined responsibilities, ranging from applications, content discovery, and ads, to central platforms, which is responsible for 72 repos alone.
“Within about a year, things had started to not go so well,” Anna Shipman, technical director for customer products at the Financial Times, said during the QCon conference in London in April.
Teams had lost sight of the overall tech strategy and who owned which services. This led to a growing pile of technical debt, ‘haunted forests,’ i.e. orphaned codebases no one wanted to touch, and a dwindling pool of engineers willing to be on-call out of hours.
As one of Shipman’s colleagues told her: “It doesn’t feel like we’re owning or guiding the system. We’re just jamming bits in.”
Fighting back required a conscious effort to move towards order, removing those haunted forests and accepting complexity so that it could be more effectively managed. Only when teams had clear ownership of their technology stacks could the organization start to more consciously attack those technical debts and haunted forests.
“It’s not something to do alongside regular feature delivery, it’s something you need to properly set aside time and schedule time to do,” Shipman said. “It’s not one and done. You don’t just do a bit of tidying up and everything’s fine.”
While there is no central mandate to assign, say, 20% of all engineering time to removing haunted forests and managing technical debt, Shipman believes teams are now better empowered to balance feature delivery versus technical debt.
Great, now convince your boss
Once you have reassessed your relationship with technical debt across engineering, and the developers understand the value of “slowing down” to address technical debt on an ongoing basis, the challenge doesn’t end there. You still have to communicate that value to senior management.
“Product and engineering managers allocate their time to adding business value, as if more bells and whistles is the only value, but sometimes the biggest value is tuning the car up,” Honeycomb’s Majors said.
Addressing technical debt may be the first thing to get deprioritized in the pursuit of business goals, but it’s imperative that engineering managers change that narrative.
“One of the most common complaints I hear when I talk with engineers is that they feel they have to continually work on features, while the software and tools they work with become more brittle, inconsistent, and frustrating, and it becomes harder and harder for them to get their job done,” David Van Couvering, senior principal architect at eBay, wrote in a blog post earlier this year.
Translating the risks of those brittle systems to the business often requires speaking their language, by emphasizing how attacking technical debt now can enable engineering to move faster in the future, ensure that software is more secure, and keep engineers happy or stop them from walking out the door.
“When you learn how to talk like a suit, your company, your team, and your career benefit. Your company benefits because they avoid the failures that can come from engineering work piling up,” Van Couvering wrote.
“Other engineers will understand how important this work is. Tell a story with your business partner as the hero and you as a trusted guide. You need to really tie into business metrics, like turnaround time, performance, and quality,” the Financial Times’ Shipman advises.
Don’t take the risk
Successfully managing technical debt will require putting a lot of effort into changing ingrained cultures and ways of working, as well as improving communication between engineering and the broader business. The incentives developers are working towards may need to change, but the risks of ignoring growing piles of technical debt are potentially existential.
“Your argument against technical debt will be strengthened if you focus on helping your business counterpart understand how decisions today increase future risk. Talk about the loss of predictability in the project. Show how compromises now lead to performance degradation later,” RedMonk analyst Rachel Stephens wrote in 2017.
Yes, shiny new features keep customers and executives happy, but debt-laden systems can bring everything to a shuddering halt, and climbing out of the debris doesn’t sound like much fun at all.