Structured Output vs Function Calling: When Each Breaks at Production Scale
Function calling sounds cleaner until you are debugging why the model decided to invoke a tool it should never have touched. Structured output sounds simpler until you are parsing nested arrays with a model that occasionally adds trailing commas to JSON. Here is what happens when each approach meets real data at scale.
The failure that makes you rethink function calling
Picture a supplier onboarding pipeline where, one day, the model started calling update_supplier_record when it should have been calling create_draft_supplier. Nothing crashed, because the function signature accepted the same parameters. Supplier records got partially overwritten before anyone noticed.
The root cause: the system prompt had been updated three days earlier to add context about existing suppliers. The model had correctly inferred that if a supplier already existed, updating made more sense than creating a draft. Logical reasoning. Wrong business rule.
This is the failure mode nobody warns you about with function calling. The model does more than execute instructions. It reasons about which function to call. That reasoning can be wrong in ways that are completely invisible until a record is in a bad state.
What function calling is, and what it is not
Function calling (OpenAI's term) or tool use (Anthropic's term) lets you define a set of available tools, each with a name, a description and a parameter schema, and lets the model decide which tool to call and with what arguments. The model produces a structured tool invocation rather than free text.
The appeal is obvious: you get structured, actionable output with built-in reasoning about which action is appropriate. The problem is that "built-in reasoning" is not bounded. The model can call functions in unexpected sequences, invent argument values for optional parameters, or decide that none of your defined tools are appropriate and produce free text instead.
Watch out
The structured output failure nobody anticipates
Structured output, which constrains the model's response to a specific JSON schema, looks simpler. You define the schema, the model fills it. No function selection, no action ambiguity. Just extraction.
The failure mode to watch for in document extraction is nested arrays. Say you are extracting line items from invoices, each with a unit price, quantity and product code. On long invoices, a small model without strict schema enforcement can produce valid-looking JSON with a trailing comma in the line items array. Pydantic rejects it, and the document routes to exception handling.
The fix is straightforward: turn on strict schema enforcement, and route long invoices to a stronger model. The hard part is noticing. A failure like this shows up as an unexplained exception rate on multi-line invoices, and the logs alone will not tell you why.
When to use each approach
This is our working rule after shipping both approaches across 11 production pipelines:
Use structured output when
- —The task is extraction: you want structured data out of unstructured input.
- —The output schema is known in advance and does not depend on what the model finds.
- —You need to process high volume at low cost (Haiku with JSON mode is significantly cheaper than Sonnet with tool use).
- —The side effects of an error are a bad field value, not a bad action.
Use function calling when
- —The task requires choosing between multiple possible actions, not just extracting data.
- —The correct next step depends on what the model finds, and that decision logic is too complex for a routing layer.
- —You can afford the latency and cost overhead of Sonnet or Opus-class models.
- —Every exposed function is side-effect-safe or fully reversible.
The common mistake is choosing function calling because it feels more "agentic" and more sophisticated. Sophistication is not a design goal. Reliability is.
The specific failure modes to design against
Function calling: phantom arguments
When a function has optional parameters, the model will sometimes populate them with plausible-looking invented values. Take a supplier lookup function with an optional country_code parameter. The model fills it with a two-letter code that looks correct and is not on the approved list. The fix: remove optional parameters wherever possible, or add validation that treats unexpected values as hard errors rather than warnings.
Function calling: sequential calls in wrong order
In agentic workflows with multiple available functions, the model may call them in a different sequence than you intended. If function B depends on the output of function A, and the model calls B first, you get either a runtime error (if B requires A's output) or silent wrong behavior (if B can run without A's output but produces a worse result). Explicit sequencing in the orchestration layer is non-negotiable. Do not rely on the model to know the dependency order.
Structured output: schema drift
When you update a schema by adding a required field or changing a type, prompts that were working may start failing. The model keeps generating outputs that match the old schema perfectly, and the validation layer starts rejecting them after the migration. Track schema versions and pin prompts to schema versions explicitly.
Both approaches need a validation layer between the model output and your business logic. The model output, whether a function call or a JSON blob, should never be trusted as correct without deterministic validation. If you are skipping this layer to save development time, you are building technical debt that compounds at production volume.
Our strong opinion: most enterprise use cases do not need function calling
This will frustrate some readers. Function calling is powerful and useful for the right problems. But most enterprise AI tasks (document extraction, classification, data transformation, summarization) are extraction tasks, not action tasks. They belong in structured output.
The tendency to reach for function calling comes from demo culture. Demos show agents taking actions, calling tools, producing results. That is compelling. In production, an agent calling the wrong function at 2am is an incident report.
Use the simplest approach that solves the problem reliably. Add complexity only when simpler approaches demonstrably fail.
Related reading