Great article, I think the most important aspect from it is the auto-routing. As humans laziness is in our nature, so having to think if the model is capable enough is not something that most ppl will do - resulting in trying out smaller models which failed our task and then just giving up and running on the bigger model all the time.
DISCLOSURE: I like Databricks. While cosplaying enterprise CTO, I've directed the purchase and heavy integration of their work for over a decade.
That said, this type of post needs to be read with product marketing context in mind.
> I think the most important aspect from it is the auto-routing
On the contrary, in white paper studies auto-routing is shown to destroy the single largest token cost they found, curiously not shown in their opening graphic. As they put it in their own words near the end:
"simple tuning of… caching settings… 50% reduction in … costs, with no observed quality degradation…"
They also aren't showing the harness called ‘pi’ which generally tops results (not only with ‘open’ models), and is consistent with Anthropic's recent "works better when we don't stuff 100k system prompts into context" about Opus 5. Recent models do better with less jank trying to prescribe behavior.
So then one wonders: why is this post written to say you should use a meta router but no mention that's offset by the cache busting, and show developers cost savings but no mention that Anthropic and OpenAI both have developer-facing utilization dashes already?
Perhaps it's so engineers can show enterprise procurement why they want to buy exactly what Databricks happens to have for sale.
Hey! Thanks for the feedback! I work on many of these things at Databricks, so figured I'd chime in on this.
Firstly, while routing is important, simple things like observability into the token costs of various features, which can drive optimization of better default parameters, are low hanging fruit everyone should do.
1. As for the cache busting, we're very aware of this. Had said this in a thread above too, so copying it here. We're going to do a followup blog detailing our routing approach, but we're designing it to be cache aware. The router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before. This does have disadvantages too, but the cache is the dominating cost reduction force.
2. We do love Pi (we published this too: https://www.databricks.com/blog/benchmarking-coding-agents-d...), but we didn't mention it here because we're still in the process of making it available to devs internally. There's a lot that goes into this more than just cost (e.g. feature gaps vs. other common harnesses).
3. As for existing utilization dashboards, it's good that each tool has one, but we found that as we gave developers more freedom to use more tools, it was impossible to have a single source of truth. Without that, it's hard to really understand all-in AI spend and optimize it.
> Firstly, while routing is important, simple things like observability
STRONG agree. Most devs, if they can see their costs, try. Even just putting it in a starship prompt goes a long way!
Observability supports effective, responsible, and accountable use, "cost controls" (as typically enterprise defined) supports bean counters. Ensuring Observability is recognized as a valid Control™ should be on every engineering executive's TODO list.
> the cache is the dominating cost reduction force
TY for the rich reply. As you're very aware of the cache issue, it might have been nice to have the word "cache" more prominent on your "Key Techniques" image (currently Lower Cost Models, Smart Routing, Spend Controls, Context Optimization; cache tuning buried at the end) or higher in the body sections.
It was mostly the image that prompted (ha) me to call out the marketing driven angle.
Exactly. There are plenty things they could suggest for that.
Anyway you'll likely get a more effective and up to date set of advice by skipping marketing pieces and pointing your favorite frontier model at /r/LocalLLaMA in deep research mode and asking it to synthesize the latest advice, or even try ideas out for itself and let you know what works for your own session history.
Sorry, didn't mean they're using pi. They're itemizing ways to get better results but not showing pi, while other testing shows most models (including Opus) achieve better from within pi.
Interested to dive deeper on the upfront design discussion. Have you found these to more often then not translate into the real product.
In my experience at the begining of the full agentic coding loop in our company we were more hands on with the codebase and had better judgement over the plans. Now it is quite often that the inital plan after executed needs more refinement and that made the plan review somewhat obsolete for us.
I hope that some of the executives out there will read the list. I know they won't spend the time to read the full blog, but at least the headers should be enough
I disagree tbh, the prototype is not the product not just because of technical abilities but also because of product expertese in a topic.
If it was otherwise outsourcing companies would've dominated the market for a long time, but it is with the exp from clients and many users that you get to build a great product.
It takes a lot of time to understand what is the pain of the user, in many cases we know what the problem is but we have not time to think of a good solution to it.
I agree with the point that a prototype is not technology, but the problem is that the argument limits gen AI code to just prototyping. I also think prototyping and production are different, but the boundary between what is a prototype and what is production is actually quite blurry.
That's not the core issue, though. The market itself is demanding heavy AI use, and GitHub has already reported a massive increase in repository creation. In other words, if we acknowledge that there are clearly users who want this GEN AI code, then we should be having a discussion about how to actually push that code into production
I see, I misinterpret what you are saying. I think manually written code is thing of the past. There will be need for it from time to time but we are going to operate at an even higher abstraction than we do now.
Same is for things like asembly there are still people who need to edit that, but most of us are working with the higher-level programming languages.
I think we have played this game long time ago. If products were a question of a single request then outsourcing companies would dominate over product ones.
I think a lot of product development happens in the itearations after the intial prototype/MVP and so on. It is not only the technical aspect to it, you need to spend time on a problem deeply understand what are the root causes of pains and address them in your product, both from UX and also from technical perspective.
People were able to "prompt" a product even before to an outsourcing company, but they'd rather pay the fee to a product company because of the expertese they have gained through out the years and all the users they've spoken to.
What happens when shit hits the fan in my exp is that I have to crack open the codebase and debug some portion of it, so I can explain it to myself in order to be able to explain what is wrong to the LLM.
Otherwise what I have found is that the LLM will add a new if statement which will handle the newly discovered issue and you start stacking them ifs. As the article mentions LLM's unlike humans aren't lazy, they will copy, paste add patches for every issue, why bother think and understand root cause :d.
So as part of our review we have a rule against that as well.
Humans will do this as well, especially inexperienced junior SWEs. Adding a new boolean parameter and some if statements here and there. After a while a seemingly simple function takes four boolean parameters that each control a little bit of what the function does.
The benefit is that a human who is a junior might need at least a few weeks to months of guidance to have a good taste of when to duplicate and when to DRY. An LLM likely already has good judgment, and your prompt merely needs to be activate this judgment.
Interested to hear what is something you didn't expect, for my 2 minute check I saw there are Computer-Use capabilities baked into it, however I've never observed that behavior in my usage. Probably not enabled by default.
Awesome article, I feel a lot of people have also forgotten that good projects take iteration not 100 new features.
To get few features to an excelent state it requires multilpe iterations at multiple stages.
1) The developer who does a task validates that their thinking was the correct one, they see how they changes impact the system, is it scalable? Does it need to be scalable? While you are working and thinking on it you get more and more context which simply wasn't there at the begining.
2) A feature done once (even after my perfect ClaudeCode plan) is not done forever, people will want to make it better/faster/smoother/etc. But instead of taking the time to analyze and perfect it we go onto the next feature, and if we have to iterate on the current one, we don't iterate we redo...
Really like the article I think it is awesome, and I strongly believe AI for coding will stay, but I also beleive that we need to still have a strong understanding of why we are building things and what they look like.
