Tool Use at Scale: What Breaks When Your Agent Has 40 Tools
The first agent we built had six tools: search the knowledge base, look up a supplier record, check PO status, create a task, send a notification, escalate to human review. Everything worked. The agent selected the right tool reliably, latency was acceptable, and the permission surface was manageable. The second agent had 43 tools because it needed to integrate with a full ERP stack. It did not work well. Tool selection errors appeared at rates we had never seen with the small set. Latency per request climbed to 8 seconds on average. And the combination of tools the agent could call meant one bad decision in a complex chain could trigger expensive and hard-to-reverse actions. The problems are qualitatively different at scale.
The tool selection problem
With six tools, a well-prompted agent makes the right selection nearly every time. The choice space is small, the tools are distinct, and the model's reasoning about which to use is straightforward. With 40+ tools, you are asking the model to navigate a much larger decision space, often with tools that serve overlapping purposes.
In our ERP integration agent, we had tools for creating a purchase order, updating a purchase order, creating a purchase order draft, duplicating a purchase order from a template, and creating a blanket PO. These are all distinct operations that serve real purposes. To a human AP manager, the differences are obvious from context. To the model, given an instruction like "set up a recurring order for maintenance supplies," the selection between these five tools was wrong about 22% of the time in early testing.
We fixed this through two changes. First, tool naming discipline: every tool gets a name that is unambiguous about its effect and scope. Not "create_po" but "create_purchase_order_from_scratch" and "create_recurring_blanket_po" — names long enough to be self-describing. Second, tool docstrings that include explicit negative examples: "Use this for one-time purchases. Do NOT use this for recurring or scheduled orders." Negative examples reduced confusion significantly on overlapping tools.
The model uses tool names and descriptions — not just the function signature — to decide which tool to call. Write tool descriptions as if you are writing them for a competent but unfamiliar colleague, not as API documentation. Include what NOT to use the tool for. After these changes, our tool selection error rate on the ERP agent dropped to under 3%.
Latency compounds with tool chain depth
A single LLM call takes 1–3 seconds. A single tool call (API roundtrip to an ERP) takes 0.5–2 seconds. With a simple agent that makes one or two tool calls per request, total latency is tolerable. With a complex agent that might make six to ten tool calls to complete a task — retrieving context, verifying preconditions, executing the action, confirming the result — you are looking at 15–25 seconds per request before any retry logic.
The 8-second average we saw on the 43-tool agent was actually a success story. Early builds were hitting 20+ seconds. We got it to 8 through three optimizations:
- →Parallel tool execution. When the agent needs to fetch context from multiple sources before acting, those fetches can run concurrently. LangGraph's parallel node execution made this straightforward to implement. Most of our context fetches went from sequential to concurrent.
- →Tool result caching. Supplier records, PO templates, approval thresholds — these change rarely. We cache tool results with appropriate TTLs (supplier record: 10 minutes, approval threshold: 1 hour, PO template: 24 hours). Cache hit rate on read operations: ~65%.
- →Streaming for user-facing responses. For tasks where the user is waiting, stream the agent's reasoning while tool calls execute in the background. The user sees progress rather than a blank screen for 8 seconds.
Permission surface grows faster than you expect
Each tool an agent can call is a potential action it might take. With six tools, the action surface is limited and auditable. With 40, the combinations of actions the agent might chain together create an audit surface that is genuinely hard to reason about.
The specific problem we hit: our ERP agent could create a PO, could look up and apply a vendor discount schedule, and could submit the PO for approval. Each of these individually was correct and authorized. But the combination — creating a PO, applying a discount schedule that applied the maximum term discount regardless of order size, and auto-submitting before anyone reviewed the line items — produced a set of POs with incorrect pricing that went through approval unchecked because the automation created them correctly-formatted and looked like normal output.
Our take
The tool registry pattern
Above about 20 tools, we have stopped including all tools in every agent invocation. Instead, we use a tool registry: all available tools are registered with metadata (category, required permissions, risk level, typical use cases). At the start of each task, a lightweight routing step selects the relevant tool subset — typically 8–12 tools — based on the task type.
This has three benefits. First, it reduces the tool selection error rate by removing irrelevant tools from the choice space. An agent working on a supplier payment query does not need the tools for HR data access or report generation. Second, it reduces context window consumption — 40 tool definitions add meaningful tokens to every call. Third, it allows per-task permission scoping: the tool subset for a read-only lookup task does not include write tools, even if the agent's service key technically has write access.
The tool registry pattern is the single architectural decision that most improved reliability in our high-tool-count agents. It took about two days to implement in LangGraph. The improvement in tool selection accuracy and the reduction in accidental high-impact actions made it worth the effort on the first deployment it was applied to.When to split an agent instead of adding more tools
There is a threshold where adding more tools to a single agent stops being the right architecture. We put it at around 25–30 tools for a single agent without a tool registry, or 40–50 with one. Beyond that, the cognitive load on the model — navigating the tool space, maintaining task coherence, handling errors from a wider range of tool call patterns — starts producing reliability degradation that tool descriptions and testing cannot fully fix.
The alternative is a multi-agent architecture: a coordinator agent that understands the task and delegates to specialized sub-agents, each with a smaller, coherent tool set. A procurement coordinator delegates to a supplier lookup agent (5 tools), a PO management agent (8 tools), and an approval workflow agent (4 tools). Each is testable independently. Each has a clear permission scope. The coordinator does not need to understand all 17 tools — it just needs to route the task to the right specialist.
Multi-agent architectures add orchestration complexity. They are worth the tradeoff when a single agent's tool count is producing reliability problems that you have already tried to solve with naming, documentation, and tool registries. Do not reach for multi-agent because it sounds more sophisticated. Reach for it when you have hit the reliability ceiling of the simpler approach.