Markdown source

Transcript: The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica

Source Video

Local Cache

Transcript

Hi everyone. Thanks for being here. I am Sohaib here and along with me is Ankush. So, today we'll be talking about a mistake that looks harmless at first. Giving which is basically giving an AI agent every tool access it might ever need all at once. So, basically that approach works well in a demo. It might even work with a small number of tools like like say for example 10 tools. But once the catalog grows, the agent gets slower. It might become more expensive and less accurate as well. That is why we are calling it the 100 tool agent trap.

In the next half an hour or so, we'll show why it breaks, what the numbers look like, and how semantic routing with just-in-time context can help us fix this problem. So, a quick introduction about myself. I am Sohaib Shaikh. I'm currently working as a data scientist with Presodica. My background spans across AI, NLP, marketing, analytics, and even engineering. My current focus is on applied AI, NLP, and conversational intelligence along with rack systems. I'm especially interested in making AI systems more reliable, measurable, and even scalable beyond the demos in production. >> And I'm Ankush Astogi. I work as a senior data solutions engineer at Presodica.

I have spent more than a decade in AI, data engineering, and production systems. My focus is the engineering side. So, it's not about what's going to work in notebook, but whether it's going to survive in with real load, real user, and real failures. So, that is the angle we are taking today. Suhail will focus more on the model and routing behavior, and I will focus more on system design, implementation, and production trade-offs. >> Awesome. Thank you, Ankush. Um so, let's get into it. So, let's imagine a common design. Um you tend to build a system, and it can do many things.

Say, for example, querying a database, sending an email, um even checking a calendar, or looking up an order, calling an API, and so on and so forth. The simplest approach over here would be to give a model every tool definition on every request. Every function name, um every description, and even every JSON schema will go into the prompt, whether the user might need it or not. So, that We are calling that as a fat agent. At small scale, it feels fine. With 10 tools as well, the model might usually pick the right one. The demo looks good. Then, the product grows.

Um 10 tools might become 30, or it will keep on increasing, and eventually the model starts calling the wrong function, and starts confusing similar tools, may invent tool names, and even take longer to respond. The important point is, basically, the design does not fail because one tool is badly written. It fails because every request is forced to carry the entire catalog. So, let's look here. There are, say for example, uh 741 tools in your uh in your entire schema, but and it will look, basically, take up to 127,000 tokens just to have all those tool descriptions in it. And this is even before the user user's actual question is even considered.

So, basically, this will lead to context overload and uh we need to manage that properly. So, on this slide we see why it is failing and why the accuracy collapses beyond a point. So, when you look at the accuracy curve with the 10 tools, fat agent will get the tools right almost 78% of the times. That is not perfect, but it's usable. At almost 100 tools, the accuracy drops to around 40%. Less than half of the tools that are called are the correct tools. And if it grows beyond that, like say for example, in over here, at 741 tools, the accuracy will be a mere 13.6%.

So, in short, it's roughly one correct tool out of eight tools. So, when we compare it with the semantic router, semantic router behaves very differently. It stays above 83% across the same catalog sizes. That is because the model is not choosing from hundreds of tools, it's choosing from a small and relevant set. One reason the fat agent fails is because it's lost in the middle problem. Model pays stronger attention to the beginning and end of the long context. When hundreds of tool schemas are packed into the middle, the model does not reliably use them. So, we end up paying paying for a huge prompt, and that prompts makes the decisions even harder.

So, two more reasons over here. First is the latency and the second is the cost. So, we saw in the earlier slide that accuracy was a big problem. Another issue is with latency and cost over here as we can see. So, like say for example, if we have 741 tools, we saw that it requires almost 127,000 tokens. That will include the tool description and the schema text. So, that cost is basically being paid on every request. At 100,000 requests a day, if you push this into production, you are sending in billions of tokens and just to describe those tools.

And with just-in-time routing, the prompt may include only three to five relevant schemas, closer to about 1,000 tokens. That is roughly a 99% reduction in tool context tokens. Another issue is latency. With fat agent approach, time to first token grows as the tool catalog grows because the model has to process a larger prompt before it can answer the user question. So, say for example, if you have 500 tools in your agent, the fat agent path can push first token latency past 5 seconds. So, this is specifically important if your product is real-time because this will take longer time to respond. So, which will make the design feel slow and unpredictable.

So, and with semantic routing, it might feel more responsive and it will be more real-time in a way. >> Here is the clean comparison. So, on the left side, we have a fat agent design and on the right side, we have our semantic router design. So, in the fat agent design, every schema is loaded for every request, no matter what the user asked. And because of that, the catalog will grow, the prompt will grow, due to which the latency will rise and accuracy will drop. The agent also becomes one large monolithic, which makes it hard to test, risky to update, and even painful to debug.

But, in the case of semantic routing design, the agent does not start with every tool. So, in this case, the router first look at the user's query. It retrieves the three to five most relevant tool, and then it injects only those into the model call. So, the context stays small, latency stays stable, and the accuracy holds because the model is choosing from a focused list instead of giant catalog. This is Okay, so there is a one important caveat here. If you have fewer than 20 tools, a router may be unnecessary. Just load the tools directly. But, once you passed 50 tools in the production system, then justifying router-based schema make more sense. >> Thanks, Ankush.

So, let's now look into how semantic routing works. If you have built a rack system previously, this will feel pretty similar to that. The difference is that we will retrieve tools instead of the documents. So, at the first stage, say for example, over here, each tool will have a clear description. For example, a search flights tool, uh a check calendar availability tool, or a retrieve customer order status. Say for example, the second step would be those descriptions needs to get embedded and stored in a vector index. This is usually done offline when the tool catalog is created or updated.

At the runtime, the user will ask a question, we embed the user query with the same embedding model. Then, we search that vector index for the tools with the description that are closest to the query. So, the router will then return the top K tools. Often, K would be three or five tools that matches the user query. And only those selected schemas will get injected into the model call. So, this is the whole pattern. At the first step, you need to index tool descriptions offline. Then, retrieve relevant tools at runtime and keep the model's context focused. So, in short, the idea is pretty simple. Semantic routing is basically a rack for tools.

Um if you already have an embedding model and vector database in your stack, most of the infrastructure is already familiar to you. >> Thank you, Sohil. Now, we will deep dive into just-in-time context injection. So, semantic routing is the retrieval layer. And just-in-time context injection is the context management strategy. So, in the case of fat agent approach, it loads everything before the query is understood. Due to which the model receives a huge list of tools first, and then it tries to reason through it. Just-in-time context does the reverse. It will wait until the query is known, then it will inject only the context needed for that request.

I mean, this is not a new software idea. We have used lazy loading, just-in-time compilation, and on-demand resource loading from years. We are just applying the simple and same principle to the LLM context. Anthropic has written about this pattern with on-demand tooling loading through MCP. Their report token usage went from 150k tokens down to 2,000, which is 98.7% token reduction. That is the signal. Large tool catalog should not be dumped into the into every prompt. They should be retrieved when needed. So, moving to the next slide, let's talk about how to evaluate this fairly.

We measure four things here: tool selection accuracy, time to first token, input tokens per request, and the estimated cost per thousand call. So, here for data sets, we used Berkeley function calling leaderboard, Skills bench style scenarios, and synthetic tool pools that allow us to scale the number of tools, so we can test at 10, 50, 100, 200, or even 1041 tools. So, we ran the same queries in two modes, the fat agent one and the semantic routing. Same model, same answer key, same tool catalog. The only difference was whether the model saw every tool or only the routed tool. We also swept K at 3, 5, 10 to understand the trade-off.

So, the smaller K is the faster and cheaper. Larger K can be the can recover more edge cases. In practice, K equals 5 is a strong default starting point. So, if you are want to try this, you can default it to five. Here, K means the number of tools the semantic router retrieves and gives to the LLM for each user queries. Here are the benchmark result that we got. So, here is the result on the left side. We see the fat agent accuracy line drop sharply as the tool count grows. It started around 78% at 10 tool and then falls to around 13% at 1041 tools.

But, in the case of router, the router line stays above 83% because the model stays stable. From the model points of view, it always choosing the handful of tool, even if the real catalog contains 100. On the right side, time to first token shows the same story. The fat agent path gets slower as more tool schemas are added. At large catalog sizes, the model spend noticeable time just processing the prompt. But in case of router, it stays almost flat because of the prompt size is under controlled. This is the core lesson from the benchmark. The catalog can grow, but the model's working set should stay small.

Now, on the next slide, we'll be discussing about the implementation pattern. There are three steps in it. Step one is offline. So, build a catalog of tool. For each tool, store the name, description, schema. Then, embed the description and store it into a vector database. You can use Chroma DB, you can use Pinecone, you can use Quadrant. Whichever vector database you are using. Then, step two will happen on every request. Embed the user's query. Then, run the nearest neighbor search. And return the top K tools. In the step three, it will also happen on every request. Fetch the schema for those selected tools.

And then, pass only those schemas into the model call and log which tool was selected. So, here, logging really matters. So, let's say if you miss something and if you want to improve the description or you want to tune the K, you need a good logging system that you can use it. So, the runtime overhead is small, one embedding call, and one vector search. The payoff is much smaller prompt and more stable tool selection problem. >> Awesome. So, now that you have learned about the implementation pattern, I'll take you through the code version of it, which is pretty straightforward.

So, initially you need to loop through the tool catalog, and for each tool, as Ankur mentioned, embed the description, store that in the vector database with the tool name and other details. At run time, embed that user query, compare that query vector against the stored tool vectors using cosine similarity or vector search, and then retrieve the top most relevant tools, and then call the model only with those schemas and the function or tool parameter. That final line is the most important part because the model does not receive all the tools, it only receive the routed tools. This works with any embedding model and at and with any vector database.

You can start, as Ankur mentioned, you can start with Quadrant or any other database vector database locally, and then move to a more managed vector store later if needed. If you already have an existing rag infrastructure, this is not a new infrastructure as we mentioned earlier, it is the same retrieval pattern applied to the tool selection. >> Thank you, Suhail. Uh Now, we will discuss semantic routing in action. So, imagine you have a 200 tool agent. A user asks, "Find me a flight to New York next Wednesday. The router embeds that query. Returns tool like search flights, book flight, calendar check.

The model sees only those relevant schema, so it is much more likely to call the correct tool. But, if we load all 200 tools, hotels tool, weather tool, emails tool, sequels tool, or any unrelated workflow tool, it would all be competing for attention. That is when model starts choosing the wrong function. Now, let's take a second query. What is the weather in Paris right now? The router returns get weather and get forecast. Flights tools are not injected. Hotels tools are not injected. They are simply not available at this request. This is an underrated benefit. But, the router does not only add the right tool, it also removes the wrong tool from the model's choice set.

Now, in the next slide, we will discuss implementation checklist. So, here is the production checklist if somebody wants to uh follow this along. So, first, get locked the tool in one place. So, add name, description, schema, owner, and the version. Second is to build the index. Embed each description and store the vectors. The third one is write the router. So, in this, embed the query, then search the index, and take top K and fetch schema. The next one is the fourth one in which a user need to wire it up into the agent loop. So, the model tool list should come from the router, not from a hard-coded full catalog. Fifth is to evaluate.

Run your test set at K equals to three, five, and 10, and then pick the smallest K that meets your accuracy target. Sixth is to monitor production. Log selected tools, final tool calls, failure, and fallback usage. Then re-embed tools when descriptions or schema change. This is not the 6-month platform rewrite. For most team, it is a focused sprint. So, in the next slide, we will discuss what community already has confirmed. This is not just the observation. So, Anthropic has published on-demand tool loading with MCP, and reported a drop of 150k tokens to 2k, which is a huge reduction in tokens.

Developers have raised similar issues in mainstream agent and SDK projects, pointing out that sending all tools definition on every request can increase latency, token usage, and tool confusion. Open proj open-source projects like MCP zero have also explored routing at very large scale, including thousands of tool across many servers. And you can find form forums post from builder who hit tool confusion with only few tools. Long before they reach 100. So, if you if your agent start failing as tool are added, it doesn't automatically means your prompt are bad. It may means the architecture is asking the model to solve the wrong problem.

So, here we are just trying to say about the We are going to talk about the open-source projects that we came across. So, one of them is of from Areilo Labs. They have an open-source semantic router project that you can easily use to test locally for benchmarking, Toolbench, and the Berkeley function calling loader leaderboard are also useful starting points. And for practical guidance, you could read the Anthropic MCP write-up. It explains why on-demand tool loading matters and gives a strong real-world signal for this architecture. The point is that you don't need to invent the ecosystem from scratch. Those pieces already exist. Next slide. Sorry. Here it is. Thank you.

So, there are a few trade-offs with this. The first risk could be a router miss. So, if the So, the router may fail to retrieve the tool the models need and handle that with you could handle that with a feed fallback. If the model cannot complete the task, you could widen K, run a second retrieval pass, or even route it to a broader tool group. The second risk would be weak tool descriptions. If your descriptions are weak, embeddings will end up being weak. Write descriptions in the words users actually use and include intent, action, and key entities along with it. The third risk would over here would be rare tools.

So, some tools may never score high unless their descriptions include the right language. So, you need to monitor data misses and rewrite those descriptions accordingly. And finally, just to conclude, do not over-engineer small systems. If you have 10 or 15 tools, static loading may still be fine. Routing generally pays off when the catalog is large enough that the prompt size, latency, or tool confusion becomes a real production problem. So, a few takeaways. First thing is, as we discussed, tool overload hurts accuracy. Loading every schema into every prompt makes the model's decision harder as the catalog grows on. The second one would be to about tokens. It's like tokens are both cost and latency.

Larger tool catalogs can add add tens of thousands of tokens before the user's actual request is handled. Semantic routing can help fix that problem. The When the catalog can be is large, the model only sees the tools relevant to the current query. This is basically a rag for tools. Index those descriptions, properly retrieve at run time, and inject only what is needed. And finally, start simple. Use K as five as a starting point, log every decision, and evaluate against a real test set. And improve their tools descriptions over time in case you get incorrect tools being picked up. And the goal is not to make these agents more complicated.

It's It is basically to stop forcing the model to reason over irrelevant tools. So, there are some references on this slide. So, if you're building something, you can use these references. Like semantic routing libraries, tool bench, Berkeley function calling leaderboard, Anthropic MCP write-up, and vector stores like Pinecone, Quadrant. Thank you so much for watching this. If this was useful, please share it with someone building agents. They have either already hit this wall or they're about to. You can scan our LinkedIn QR codes on this slide to connect with us. We are happy to continue the conversation, share resources, or hear how others are approaching this problem in their own system.

Thank you so much for watching this. I'm Ankush. This is Suheel, here. Thank you so much. Yeah. Thanks, everyone.