Most companies start with one model for everything, usually the most capable one they can get. It's the right first choice, with one integration and the best answers available, and the most expensive way to run routine work.
A lot of that traffic is routine, such as classifying tickets into queues or pulling dates and amounts out of invoices, and a smaller model is worth testing on it.
The research behind routing
In 2024 the LMSYS team, with researchers from UC Berkeley, Anyscale and Canva, published RouteLLM, a set of routers that decide per request whether a strong or a weaker model should answer. The paper reports cutting costs "by over 2 times in certain cases" without losing response quality.
The LMSYS write-up gives the detail. Compared with sending everything to GPT-4, the routers kept 95% of its performance while cutting costs by over 85% on MT-Bench, 45% on MMLU and 35% on GSM8K. The saving depends on how much of the workload is easy, and that's different for every company.
Measure your own mix
So the first step is finding out what your traffic looks like. Take a sample of a few hundred real requests from the past month and sort them by task type. A handful of types often make up most of the volume.
Then run the same sample through a smaller model and score both models' outputs with the same checks. Check mechanically where the task allows it (the extracted date matches, the label is one of the allowed queues) and with human review where it doesn't. The result is a table: for each task type, how often the small model's answer is as good as the large one's.
Route by task first
The simplest router doesn't need to be clever. If the request comes from the ticket classifier, send it to the small model. If it comes from the contract review flow, send it to the large one. Routing by where the request comes from captures much of the saving at low risk, because you've tested each path.
Learned routers, the kind in the RouteLLM paper, decide per request from the request itself. They're worth considering once the obvious task-level split is done and you have a stream of mixed requests left over.
Keep an escalation path
Whatever routes the request, give the small model a way out. If its output fails a check, or it reports low confidence, send the request to the larger model and log that it happened. The escalation rate by task type tells you when a route needs revisiting.
Re-run the comparison whenever a model changes. A provider update can make the small model better or worse at a task overnight, and the table you built in spring won't say so.
Where to start
Pick the highest-volume task your agents run and test a smaller model on a month of its real requests. If the scores hold, move that one task over and watch its quality numbers for two weeks before touching the next.
Frequently asked questions
What is model routing?
Deciding, for each request, which model should handle it. Routine requests go to a smaller, cheaper model and harder ones to a more capable model, so the expensive model is used only where it makes a difference.
How much can routing save?
It depends on the workload. LMSYS's RouteLLM tests in 2024 reported cost reductions of over 85% on MT-Bench, 45% on MMLU and 35% on GSM8K compared with sending everything to GPT-4, while keeping 95% of its performance.
How do we know a smaller model is good enough?
Test it on your own past requests with your own quality checks. Run both models on the same set, score the outputs the same way, and route a task type to the smaller model only where its scores hold up.
What are the risks of routing?
Quality drops on requests the router misjudges as easy. Keep an escalation path, and re-test by task type whenever either model changes.