Multi-Model Orchestration: When You Need More Than One LLM per Task
The default instinct when a single-model approach has a problem is to add a second model. The document extraction is not accurate enough — add a verification model to check the first one. The classification model is making errors — add a judge model to review borderline cases. This instinct is sometimes right and often wrong. Adding a second model adds latency, adds cost, adds a new failure mode, and only helps if the second model is actually better at the specific task you are using it for than the first. We have built multi-model workflows where the second model clearly justified itself. We have also built them where it did not, and replaced them with a better-prompted single model. Here is how to tell the difference before building.
When multiple models are clearly the right call
Three scenarios where multi-model orchestration justified itself in our production systems:
Different models for different modalities. A document processing pipeline that needs to handle both scanned PDFs (requiring OCR + image understanding) and structured data exports (requiring text analysis) genuinely benefits from using a vision-capable model for the former and a text-only model for the latter. Claude 3.5 Haiku for structured text processing at $0.0008/k input tokens, Claude 3.5 Sonnet for PDFs requiring image understanding at $0.003/k. Using Sonnet for everything is three times more expensive for the text-only cases with no accuracy benefit.
Cost-tiered routing for confidence thresholds. A classification system where 80% of cases are straightforward can use a fast, cheap model (Haiku, GPT-4o mini) for the easy cases and route only low-confidence cases to a more capable model. We built this for a supplier categorization system processing 15,000 records per month. The routing model classifies 82% of records at high confidence (Haiku, ~$0.003/1k records). The remaining 18% go to Claude 3.5 Sonnet (~$0.08/1k records). Total inference cost: $0.06 per 1,000 records versus $0.80 if we used Sonnet for everything. Same accuracy, 13x cost reduction.
Specialized models for specific subtasks. When a subtask has a specialized model that genuinely outperforms general models on that specific task — a fine-tuned model for a highly domain-specific classification problem, an embedding model optimized for a specific retrieval task — using the specialized model for that subtask and a general model for the rest is legitimate. The key word is "genuinely outperforms." Benchmark the specialized model against the general model on your actual data before committing to the added complexity.
When the second model does not help
The failure pattern we see repeatedly: the first model makes errors. Someone proposes adding a second model to verify the first. The second model is the same capability tier as the first — often the same model. It makes different errors, not fewer errors. The combined system has the errors of both models, higher latency, and higher cost.
A verification model only helps when it is meaningfully better at detecting the specific errors the primary model makes. A Claude 3.5 Sonnet verification pass on Claude 3.5 Sonnet extraction output typically does not catch the extraction errors — if the primary model was wrong, the verification model usually agrees with it because they share similar reasoning patterns and will make similar mistakes on ambiguous cases.
Before adding a verification model, test whether it actually catches the errors you are trying to catch. Take a sample of cases where the primary model was wrong and run the verification model on them. If the verification model also gets them wrong 70% of the time, it is not adding value — it is adding latency and cost for marginal benefit. Better-prompted single models have outperformed two-model verification systems in our testing more often than not.
The orchestration complexity cost
Multi-model systems have a failure surface proportional to the number of models they use. Each model API call can fail, time out, or return unexpected output. Each model transition is a parsing step where the output of one model needs to be correctly interpreted by the next. Each additional model version update (a provider updates Claude Sonnet, GPT-4o, Gemini, and they all behave slightly differently afterward) can change system behavior in ways that require re-testing and re-calibration.
We have had incidents caused by: a model API rate limit that did not apply to the primary model but applied to the verification model (causing a partial failure where primary ran and verification did not, producing outputs with no quality check); a model update that changed the output format of a middle-step model in a way that broke the parser feeding the next step; and a cold-start latency spike on a specialized model that pushed total request time over our SLA threshold.
Our take
The routing architecture that works
The pattern that has produced the best cost-accuracy tradeoff in our systems: a fast, cheap routing classifier followed by capability-matched processing.
The routing classifier is a simple LLM call (or a classifier model) that assigns each input to a tier: routine (high confidence, standard format), moderate (needs care, some ambiguity), or complex (low confidence, unusual format, high-stakes). Routine goes to the fast model. Moderate goes to the standard model. Complex goes to the most capable model and may include a human-in-the-loop step.
The routing classifier needs to be accurate, but the cost of a misrouting is usually just cost or latency — not correctness, because each tier is still capable of handling any input, just at different cost and quality levels. Build and test the routing classifier as carefully as the processing models. A bad routing classifier that sends everything to the expensive tier is worse than no routing at all.