A developer joins a team. The task is simple: update how a recommendation service handles user preferences.
They open the codebase. They find this:
"Why does this service call an external embedding API?"
Nobody knows. The person who added it left a year ago.
"Is the cache still needed? The model changed twice since it was added."
"Probably not, but let's not risk it."
The fix takes thirty minutes. Figuring out what would break takes two days.
Now, you might ask: why not just use an LLM to understand the code? Point Claude or Copilot at the repository, ask it to explain the architecture, and move on.
Here is the thing. The developer could understand what the code does. Any capable engineer, with or without AI assistance, can trace through a service and figure out its behavior. What they could not figure out is why it was built this way. Why was this particular API chosen over alternatives? Was the cache a deliberate performance decision or a quick fix that became permanent? Is the feature pipeline still serving its original purpose, or is it an artifact of a model that no longer exists?
That reasoning is not in the code. It is not in the comments. It walked out the door with the people who made those decisions. Researchers call this architectural knowledge evaporation, and it is one of the most persistent problems in software engineering. No matter how good our tools for reading code become, the reasoning behind the code remains a fundamentally human artifact that disappears when people leave.
Every change made sense. The result doesn't.
None of those components appeared by accident. The cache was added because Model v2 was slow. The feature pipeline was built because the original model needed handcrafted features. The external API solved a real problem eighteen months ago.
Every change was reasonable. Every change worked.
But a system shaped by three hundred reasonable changes over two years is not a system anyone designed. It emerged. And the reasoning behind each piece left with the person who added it.
Studies consistently show that practitioners recognize this problem but lack systematic ways to address it. In surveys, the most commonly reported tool for understanding the relationship between architecture and code is simply personal knowledge and experience, not dedicated tooling or formal methods [3]. When someone who holds that knowledge leaves the team, the understanding goes with them. And the most frequently cited barrier to maintaining this understanding? Cost and effort: teams know it matters but cannot justify the time under delivery pressure [3].
You have seen this outside software. A house renovated over fifteen years: someone adds a room, another closes a doorway, someone converts the garage, another reroutes the plumbing. Every renovation made sense. Every contractor did competent work.
Ask anyone to draw the current floor plan from memory.
Nobody can.
What Happens When No One Remembers Why the System Was Built This Way
If you ask someone to draw the system's architecture on a whiteboard, they will draw something clean. Boxes. Arrows. Services. Databases. It will look perfectly reasonable.
But that diagram is a snapshot. It tells you what exists. It tells you nothing about why.
And the snapshot is often wrong. Research has found that a majority of inconsistencies between architecture descriptions and actual code trace back to documentation that simply was not kept up to date [1]. The code evolved. The documentation did not. This gap between the intended design and the implemented system is what researchers call architectural drift, a gradual divergence driven not by bad decisions but by the accumulation of reasonable ones made without a shared record of their reasoning [1].
Consider how a real system evolves:
Look only at 2025. You cannot tell whether the feature store is still needed. You cannot tell why both a cache and a vector DB exist. You cannot tell whether the embedding service replaced something or was added on top of it.
To understand today's architecture, you need to understand yesterday's architecture. The snapshot tells you the structure. The history tells you the intent. And without the intent, the structure can mislead you.
ML systems have the same problem, with more dimensions
Everything above applies to all software. ML systems take the same problem and multiply it.
Traditional software evolves primarily through code and architecture. Hard to track, but at least the changes live in one place. Read the commits, follow the pull requests, trace the story.
ML systems evolve through code, architecture, data, models, pipelines, configuration, and external services. These change independently. Often invisibly.
A model retrains. The weights shift. Behavior changes. The codebase looks identical. A git diff shows nothing.
An upstream data source quietly changes format. The pipeline still runs. The model still serves predictions. But the system is now operating on assumptions that stopped being true weeks ago.
You swap a traditional model for an LLM. Suddenly you need a vector database, a retrieval layer, prompt management, an external API. The architecture transforms overnight, not because someone redesigned it, but because the model changed and everything else had to follow.
And with external AI services, parts of your system's behavior are defined by models you do not control, trained on data you have never seen, updated on schedules you did not set. Your code does not change. Your architecture diagram does not change. But your system does.
And there is a more fundamental difference. In traditional software, if you lose all documentation, you can still read the code. The logic is written in human-readable instructions. You can trace it, reason about it, reconstruct at least some of the intent from the behavior. With ML models, you cannot. A model's behavior is encoded in millions of learned parameters. You can observe what it does, but you cannot read why it does it. The system becomes a black box not just organizationally, because the people who built it left, but structurally, because the model's reasoning is not human-readable by design. That is a qualitatively different kind of opacity, and it means the intent problem in ML systems runs deeper than in traditional software.
So What Can We Do About This?
This is not a solved problem. But researchers are exploring directions that reframe how we think about it.
What if architecture descriptions evolved with the code, not separately from it? Instead of architecture living in a slide deck or wiki that nobody updates, the idea is to keep it in a version-controlled, machine-readable format alongside the source code, so that when the implementation drifts from the intended design, the mismatch becomes visible, like a failing test [1]. It does not solve the intent problem entirely, but it makes drift detectable rather than silent.
What if we could automatically connect documentation to the code it describes? Today, the link between "this paragraph in the design document" and "these files in the repository" exists mostly in people's heads. Researchers have shown that by using architecture models as a bridge, it is possible to recover these connections with promising accuracy [2]. The goal is not to replace human understanding, but to make it easier to ask: "which parts of the code relate to this design decision?"
What if the reasoning behind decisions could be captured from artifacts teams already produce? Some teams already practice this through Architecture Decision Records (ADRs): short documents that capture the context, the options considered, and the reasoning behind a decision at the time it is made. ADRs are one of the most practical tools available for fighting knowledge evaporation. But they require discipline to maintain, and they only capture decisions that teams explicitly recognize as architectural. Many decisions that reshape the system are made informally and never recorded. This is why researchers are also exploring ways to extract architectural reasoning from artifacts that get produced regardless, such as pull requests, issue trackers, code reviews, and team conversations [3][4]. Someone added that external API from the opening of this blog for a reason. That reason probably exists in a ticket or a thread from eighteen months ago. The challenge is finding it and connecting it to the component it explains.
These directions share a common insight: maintaining a system is not just about keeping the code working. It is about preserving the connection between what was built and why it was built that way.
Why this matters more over time?
As systems become assemblies of components, models, and external services that change at different speeds, the gap between "what exists" and "why it exists" will only grow.
We have become very good at asking "is it working?"
Perhaps the more important question is: do we still understand why it is built this way?
That question does not become less relevant as tools improve. The faster a system can change, the faster the reasoning behind its structure can be lost. And no amount of code comprehension, whether human or AI-assisted, can recover reasoning that was never recorded.
This is the second blog in a series on technical sustainability in ML-enabled systems. The first blog explored how technical debt accumulates and why sustainability matters. This post looked at what happens as the system evolves: the gap between understanding what a system does and understanding why it was built that way. Future posts in this series will continue exploring the challenges of maintaining and evolving software systems in practice.
References
- Bucaioni et al., "Architecture as Code," IEEE ICSA 2025.
- Keim et al., "Recovering Trace Links Between Software Documentation and Code," IEEE/ACM ICSE 2024.
- Tian et al., "Relationships between Software Architecture and Source Code in Practice: An Exploratory Survey and Interview," Information and Software Technology, 2022.
- Hyun and Hurtado, "Traceability of Architectural Design Decisions and Software Artifacts: A Systematic Mapping Study," Foundations of Computing and Decision Sciences, 2023.