Debt Management for Architects – and everyone else

Debt — most people are familiar with it; most people also don't like talking about it. But today, we're talking about debt.

The most well-known kind of debt of course is financial debt. Loans, mortgages, leases, etcetera. They're all built around some form of deferred payment, with conditions formalized in some form of contract. You get something now, someone else paid for it, and you get to pay off your debt over time. Simple, efficient.

Also, typically, there are external guardrails in place to both ensure that a debtor pays off their debts, and to ensure that people don't accumulate debts they cannot afford to. As architects we deal with different kinds of debt, even if they essentially still come down to the same thing: you get something now, but it has a cost, and you need to pay off sooner or later (and probably with interest).

However, those guardrails that I mentioned don't really exist here.

Not all debt is bad

In the financial world, it can make sense to take out a loan. Not just to buy something you can't afford directly, but to invest. If you take out a loan with a 10% interest rate but your investment yields a 50% profit, it can be a very good idea to take on a little debt.

Tech again is similar. If we take on some tech debt that increases conversion by 30%, it's probably smart business-wise. Similarly, borrowing some staff from 'less important projects' or having teams work overtime to ensure some other important project ships on time, can make a lot of sense business-wise.

Not all debt is technical

If you're in Tech, you're probably familiar with tech debt. Temporary workarounds (that turned out to be permanent), quick-'n-dirty first implementations (that never were revisited because they 'worked'). But what to think of outdated or abandoned libraries you've built your entire world on top of? Or those performance issues that snuck in over time and turned your lean service into a massive memory hog? That complex part of the deployment pipeline that we "were going to document later", but now the person who built it left the company?

Tech debt comes in many forms, and I would argue that some tech debt is unavoidable. You didn't plan for that library to be abandoned, after all. Also, that choice for a monolithic architecture was the right decision at the time. But time has passed, and now it's no longer the right decision. Now it's tech debt.

But there's more. You can have organizational debt (or 'org debt') too!

Org debt again has many faces, and you're probably familiar with a few.

• Your team is understaffed? Org debt.

• Your company plays 'Team Tetris' by constantly moving people between projects as different deadlines need to be hit? Org debt again.

• Your DevOps organization has a Platform Team that's hidden behind some Jira wall? Org debt.

• Your old CI/CD platform has been replaced by a new, totally different one and nobody knows how it works? That's org debt too.

• Your Platform Team recently 'inherited' a data platform that was built by some external consultants? More org debt.

Death by a thousand tradeoffs

Architecture exists in a world where debt is seemingly unavoidable. Everything is a tradeoff. Avoid one kind of debt, just to get another.

An example: Company X decided to move a part of their landscape to AWS, so they could profit from the scalability and availability it offers, as well as the high-abstraction solutions that allow them to build better solutions, faster. Except they have no in-house experience on how to set up AWS properly, so they outsourced it to Supplier ABC which is a market leader in mission-critical cloud-based infrastructure.

Resulting debt:

• Skill: Their in-house staff is as untrained and inexperienced as it was before, but now they actually have AWS. The debt is essentially unchanged, the timeline for paying off the debt has been improved by outsourcing, but the interest has gone up.

• Quality and Control: There's no direct control over the quality that the outsourcing partner is delivering, and since they lack the in-house skill, they are unable to gauge the quality of what the outsourcing partner is delivering.

• Processes: They now have to deal with an outsourcing partner which works differently and operates outside of their organization. Their organization now has to adapt to how the outsourcing partner works.

• Integration and alignment: only part of the landscape went to AWS. Integrating the different parts of the landscape, or designing complex solutions across different parts of the landscape just got harder, and more expensive.

This is not to say Company X shouldn't have outsourced. If they hadn't, they might have ended up with a terrible AWS setup, which could have resulted in security issues, data breaches, downtime, poor performance, or unforeseen costs when external support needed to be brought in on short notice. It would probably have taken much longer to get to the same maturity. Maybe even years. Integration might have ended up just as cumbersome — this time due to a lack of knowledge.

Every choice has consequences, and a big part of getting your architecture right is being aware of those consequences, and managing them accordingly.

Debt Management for Architects — and everyone else

At the end of the day, there are 3 different categories of debt:

1. Unavoidable debt: the kind that simply happens because the world doesn't stand still, and we don't create things in a vacuum. Changing paradigms, changing priorities, architecture decisions that age like milk, dependencies that break or become obsolete, etcetera.  


2. Consciously accumulated and managed debt: the result of tradeoffs, or simply the reality of your current situation (e.g. 'we are currently underskilled for XYZ'). Third party tools, libraries, or platforms we decide to build our solutions on top of. Solutions or services we buy. Outsourcing. But also dysfunctional processes, misalignment with business, poor feedback loops, staffing issues, poor tools, employee morale, etcetera. 


3. Dark debt: when you have debt, but you don't know you have it. This should be avoided at all costs.

We need to make sure that most of our debt can be filed under the second category. Not just as architects, but any company that values good stewardship, on all levels. This means we should be explicitly considering the consequences of every potential (architectural) decision, document them, and act upon them. This requires two main things to be present:

1. Structure for Architecture Decision-making: Having a good structure for deliberation and a good structure for documenting the decision, its context, the discourse, and the consequences is essential. It's not just about having ADRs, or using the 'correct' ADR template; but I do recommend looking at this Github repo from Joel Parker Henderson for inspiration. In the end, your structure needs to match your organization and your needs.


2. Structure for Debt Governance: It's also about having governance and processes that extend beyond tech which are tailored to managing debt and risk, and which result in real-world actions being taken.

If that extended structure is lacking, you'll just end up with a lot of well-documented debt that nobody is managing.

For unavoidable debt and dark debt, we primarily need a straightforward means of documenting the debt once identified, and a process to bring these kinds of debt into Debt Management. We might not have chosen to take on these kinds of debt, but here we are — now we have to deal with them. It requires similar solutions, but smaller:

1. Accurate Documentation: this is perhaps the key ingredient to prevent dark debt as well as identifying unavoidable debt early, especially when it comes to dependencies. We often don't know what we depend on, so we can't (or don't) really manage those dependencies at all, until something breaks. Implementing tools to automatically generate various types of BOM (Bill Of Materials) in standardized formats can help you get those insights. You can even feed that information directly into your security tooling to monitor for CVEs!
You can find various BOM examples in this Github repo from CycloneDX. 


2. Debt Records: these Debt Records can be small pieces of documentation or issues in an issue tracker. It doesn't matter, as long as your company agrees on a central means of documenting, and actually uses it. A usable example can be found in this blog post by Patrick Roos.

3. Debt Identification: as part of Debt Identification, you work through Debt Records for further assessment, deduplication, and refinement. You want to get the clearest possible view of what kind of debt is there, how it affects your organization and/or your landscape, and potential mitigation strategies and consequences.
 Once identified, your debt is now effectively 'category 2 debt', and can be managed through your existing Debt Management processes and governance.

Bonus: Debt Limitation

How do you prevent taking on too much debt? In the financial world we have guardrails that, for example, make sure you can't take out a loan if you're deemed unable to pay it back. Or when you've proven to be an unreliable debtor. But in our case, what we need is some form of self-control or governance structure to prevent us from taking on too much debt.

At the bare minimum, it should be wise to monitor DORA metrics as well as regular developer satisfaction surveys. Usually when an organization is acumulating too much debt (and not paying it off accordingly), software quality and developer morale are the first things that suffer. However, as important as these sorts of monitoring are, they are — at best — smoke detectors. They'll tell you something is amiss, but not what's wrong, why, or how to address it.

If you already have a centralized overview of all identified debt, and some form of governance process, you probably came up with a means of scoring debt. If not, it can be as simple as this: 


• High Impact/Low Effort: Quick wins worth addressing early

• High Impact/High Effort: Plan these as longer-term improvements

• Low Impact/Low Effort: Tackle when there’s capacity

• Low Impact/High Effort: Usually low priority

Given the example above, a very simple way of directly monitoring your organization's ability to 'pay off debt' is to monitor the following for the top 2 categories:

1. Backlog: if your Debt Board is already showing a long list of high impact debt that you're not already trying to fix, you may need to reconsider taking on more high impact debt. 


2. Lead time to resolution: especially for the high impact/low effort kind of debt, this should be low. If it isn't, are you even taking your debt seriously?


3. Work in progress: if you have dozens of high impact debt items 'in progress', are they really in progress? Or did you end up with progress theater instead?

In an ideal world a company would impose hard limits on itself, but in a realistic world sometimes taking on more debt 'makes business sense'. So is it all smoke and mirrors? No, or at least it shouldn't be. Even if a hard Debt Limitation isn't feasible, leadership should be committed to minimizing and strategically addressing debt.

And while 'strategically addressing' and 'strategically accumulating' debt might feel like a weird paradox, your monitoring data can be exactly what management needs to approve a budget increase which allows you to get both done.