In this new reality where we have the privilege to not only take apart cloud infrastructure and re-engineer it for efficiency, but also bite down on AI implementations, it is becoming increasingly clear that we need strategies for optimisation.
One of the most important handles to finetune a workload in the cloud is rightsizing a VM. That can be through classic resizing, moving to a container set-up, moving to an autoscaling solution, going serverless, … to be fair, that last one is not really rightsizing, but the concept is there: making sure you run the appropriate compute resources for your workload.
Now, consider the following:
You are chatting with your favourite new LLM, asking it to fix a typo in an email, a script, … Under the hood, that little request just walked through the same reasoning stack that could solve a PhD level physics problem, or a complex math problem. Nothing was rightsized whatsoever. You just drove your Ferrari from one end of your driveway to another top open your mailbox, basically.
That is the state of AI spend right now.
Everyone is default routing every prompt to the biggest brain available, the same way we used to spin up an m5.4xlarge to run a task Lamba can perfectly handly. Slowly we are starting to fix that behavior in the cloud. For AI however, it almost feels like none of the lessons from Cloud are taken into account.
The instance family problem, but for intelligence
Rightsizing a VM starts with one simple question: “What does this workload actually need?” Not what it might need on its worst day, not what feels safe, what it needs right now, based on evidence. Yes, might and safe should play a role, but if you do the cloud right you let the cloud handle deviating scenarios. (Read: elasticity)
Model selection deserves the exact same diligence. Summarizing a support ticket does not need the same advanced model that is refactoring five interdependent services at 2am. But because switching models still feels like a decision, a manual toggle, most people just leave it on the expensive default and eat the bill. What probably also is a factor, is that there’s still a desire for models to improve and a hope that new models are more and more reliable. While that’s understandable, this is why you’re probably bleeding money.Anyone who has ever fought a team out of oversized reserved instances knows this pattern by heart.
Enter the router
Some products landed recently that made this analogy impossible to ignore, and they came from very different corners of tech.
Cursor shipped a Router that sits inside the IDE and classifies every single coding request before it ever touches a model. Simple, familiar edits get bounced to something cheap and fast. A gnarly multi-file refactor gets escalated to a frontier model. You pick a dial, Intelligence, Balance, or Cost, and the router steers along that frontier for you. Think of it as an autoscaling group, except instead of scaling instance count based on CPU load, it is scaling model tier based on task complexity. Same instinct, new axis.
Ramp took the harder, more forensic road. Their internal gateway does not just classify and forward, it learns. It watches latency and failure rates per model per hour, notices things like a discount tier quietly degrading during business hours, and adjusts routing in real time using statistical methods built for exactly this kind of decision under uncertainty.
That is a system running its own continuous audit of the fleet, the same forensic reflex FinOps people apply to spot pricing and reserved instance coverage, just pointed at tokens instead of vCPUs. Both approaches land in the same place: send the request to the cheapest model that still clears the bar. Everything else is implementation detail.
There’s however still an important gap:
Here is the part that should make every FinOps practitioner sit up. Rightsizing a VM has over a decade of tooling behind it. CPU and memory utilization graphs, sizing recommendations, guardrails, rollback plans, precedent and an established accepted culture. If you get it wrong, you know within a day and you fix it.
Model rightsizing has none of that maturity yet. ‘Clears the bar is doing an enormous amount of unaudited work in every one of these router pitches. Whose bar. Measured how. Nobody is publishing the equivalent of a CloudWatch dashboard for reasoning quality, at least not yet.
Which is exactly the gap a forensic FinOps practice should be walking straight into. We already know how to ask uncomfortable questions about whether a resource matches the evidence of need. We just have not built the token era version of that discipline. The utilization metric of tomorrow is not CPU percent, it is something closer to task complexity per dollar, and right now almost nobody outside these router vendors is even trying to measure it independently.
Rightsizing was never really about the VM. It was about refusing to pay for capability you are not using. So let’s apply that same logic to AI tasks, where we try and run only for what we actually need to achieve.
Leave a Reply