The Expensive Model Trap

Over the course of building Agent Starbase I used LLMs to write all of the code. I haven't hand-crafted a line of code that I remember. The project is 577,000 lines of code now, not a vibe-coded flashy demo, not a one-shot SaaS killer, but a full project with 63 features. I built it originally in Ruby, with a vanilla JS frontend and sandboxed Docker containers to run the services. Now I am migrating parts of it to Go so it uses less memory/CPU, but I don't know Go as well as I know Ruby. But that doesn't matter! I don't touch the code anymore 😄

For the last month I was churning on improving a Docker-in-Docker (DinD) issue that Codex found. I was changing the code, running the redeployment, having my agent test all the agents integrations but it was constantly failing. After realizing I was churning I stepped back a bit and realized, I have no idea how complex the features in my application are. I have an intuitive idea how about good different models I'm using are, but I don't know if they can do the job really because I don't know what complexity I'm sending them into.

The expensive model trap

It is tempting to think that the answer is always to use the best model available. Maybe the most expensive frontier model agent can understand the complicated feature, trace through the logs, and make the right change. Sometimes that is true.

But you or your company will not want every normal maintenance task to require the most expensive model and a giant context window. I use billions of tokens a month while building Agent Starbase. At that scale, the difference between cheap, mid-tier, and frontier models matters a lot. The costs can be 100 times more. Your daily maintenance tasks and codebase needs to be doable by cheaper agents too.

The best model can help you solve a difficult problem once, sometimes. But a cheaper model should be able to add a field, fix a bug, update a test, or make a small product change later. If only the most capable and expensive agent can touch a piece of your system, that piece is probably too complicated. Sometimes you might be working on something extremely hard, like browser sandboxing, virtual machine memory optimization, etc. But in my life, I've never worked on such problems, almost everything I've done is routine. No lives are on the line.

Complexity creates more than token spend

Complexity creates bugs. It creates architecture that nobody understands. It creates features where every change breaks something far away.

It also makes it harder for agents to check their own work.

LLMs have broad knowledge but have to make guesses. Those guesses often won't work because LLMs need context and checks against the reality. What unique packages are on the system they are attempting to fix. What internet issues might there be. Is it a corporate environment with a restricted internet. Are some key features of the Operating System broken today. LLMs need code they can understand, change, run, and compare against reality. If the feedback loop is too slow or too unclear, they will confidently generate code without knowing whether it actually works.

That is when we waste time. We end up in a loop, when we should be having the agent be in a loop.

Finding the dragons

After struggling with running builds of my application over and over again, without forward progress for weeks, I realized I need a better way to ensure preventive maintenance of the application before I'm in a doom loop.

I built a complexity scoring guide LLMs can use for this reason. Every repo needs a way to answer a few questions:

• What are the most complex features in this application?
• Where am I spending the most development time?
• Where am I spending the most token time?
• Are these difficult areas actually important to customers?

The guide scores features across five areas: surface area, churn, external coupling, operational risk, and the number of variants or feature flags that multiply possible behavior.

It is not perfect. But it gives you a starting point. Run it against your repo. Find the dragons lurking in your code. Some of them will be obvious. Some of them might be surprising. But you'll know what risks you have.

Then look at the highest-scoring areas and ask: is this where I want my development time and token budget to go?

What to do with a difficult feature

Once you find a difficult area, you can ask an LLM to help you pull it apart.

Ask it for ten ways to reduce the complexity. Ask what can be extracted. Ask what can be removed. Ask what can become a smaller feature. Ask what needs a better interface. Ask how you can simplify the business logic. Ask whether the feature needs to exist at all.

I have found that agents often reach for bash when solving infrastructure problems. Bash has a lot of training data, but it is not always easy to test and it's easy for agents to mess up or make progressively more complex and brittle. When the logic gets complicated, I prefer moving it into Ruby or Python, languages with testing frameworks. Coding agents need a loop that checks their implementation against reality. Also I've been building a lot of old fashioned CI pipelines that use no LLM tokens but help me find issues!

Reality checks make agents much better

You can open your application and see if a feature the LLM implemented worked. For agents though, this is expensive (image understanding is expensive and improving but I wouldn't say 100% reliable yet). But a unit test is a reality check agents can do cheaply. The agent can make a change, run the test, see what happened, and try again. Without that loop, it is mostly guessing and you'll be in the testing loop, no fun. Remove yourself from tedious loops.

Agents can also run end to end tests, even though image understanding isn't 100% yet, if agents find bugs in a 4 hour session while you are sleeping that you didn't need to spend 15 minutes on, that's 15 minutes you have back in your life to do important things! Or ... vibe code a game?

Every feature you simplify lowers your token cost and will make your life simpler. If a complex feature matters to your customers, make it easier to understand, easier to test, and easier for lower-cost agents to change. If it does not matter to customers, rethink it, or stop spending time on it. Try the complexity guide in your repo. See whether it finds the difficult pieces you already knew about, and the ones you did not.

Then make sure the dragons in your codebase are worth fighting.