Markdown source

Transcript: We Cut 94% of AI Coding Tokens With a Local Code Index - Rajkumar Sakthivel, Tesco

Source Video

Local Cache

Transcript

Hey, I'm Raj. I want to tell a story. Me and my friend Foss, we are building project together. We are using AI coding tools everyday. Cloud code, cursor, co-pilot, code X, normal stuff. One month our AI bill was fine. Next month, huge. We did nothing different. Same project, same tools, just more of it. We panicked. We looked what was happening and we found something surprising. Most of the money was not the AI thinking. Most of it was sending too much context. Files the AI don't need. Context is important. Code that was not relevant sent anyway every time. So, me and my friend Foss, we started to building something to fix it.

In this talk, is about what we built and what we learned. Every AI coding tools uh do the same thing. It send your code to the model as a context. And the tools thinks more context is better. We measured typical query on our project. It was sending 45,000 tokens of context, but the part of actually mattered is about 5,000 only. Other 40,000 tokens are not useful, but we paid for them every single query. It's It's like ordering a pizza and paying for extra nine pizzas you don't eat every time. We tried three things before we found what works. First, we changed our prompt be short, only show relevant code. Sounds good.

But, it does not work. The model already got 45,000 tokens before it read the prompt. Cost already happened. Second, we change the model setting like a max token temperature. Same problem. These changes the output, not the input. Uh money is in the input. Third, uh output compression. This one actually works. We told the model to write short answers. It cut the output 75%. But, output only uh about 10% of the cost. So, 75% of a small number still small number, not enough. We need to fix the input. This is the most important slide. 90% of your AI cost is input. Files, search results, context you send in. Only 10% is output.

The code the AI writes back. So, if you cut the output by 75%, you can save about eight 8% total. But, if you cut input by 94%, you can save about 61% total. Same math, but different result. Fix the input. That's where your money goes. We built a local search layer. It sits between your code base and the AI. Instead of sending whole files, the AI search and index. It gets back only small piece of code actually it needs. Here how it works. Five steps. Step one, we read the code and break into small pieces, functions, classes, methods. Not a random chunks, proper piece that makes sense.

Step two, we run two searches at at the same time. One such find the code by meaning. One such find the code by exact words. Then combine the results. This is where big saving comes from. Steps three, we can shrink their results even more. Keep only the function name and the description. Cut 50-line function down to five lines. Step four, uh we track the connection with the which function called which. So, if you find one piece of code, you can find everything connected to it. Step five, every results get score. If the score is too low, we don't send it. No bad context. Everything runs on your machine. Nothing goes to the cloud.

This is the beauty. Why do we run two searches instead of one? Because each one has a weakness. Meaning-based search is good at finding related ideas, but it misses exact names. You will search for um authenticate user function, and it might show you different auth function instead, because they are similar meaning. The word-based search is good at exact names, but it misses related ideas. You search for login flow, and it misses everything that says sign in. By themselves, both searches miss about one in four results. Together, they miss about one in 10. They fix each other weakness spots. Here is the hard part Foss and I spend most of the time on.

The search finds results, but they actually relevant? Sometimes search results returns 10 results, and none of them are right. If they use the bad results, it gives confident wrong answer. The worse than no answer. We tried asking AI to judge its own results, too slow. Add 2-3 seconds every time. We tried to fix the score limit, too simple. Short questions score low even when the match is perfect. It worked. Some simple formula, 50% meaning score, 30% keyword score, 20% how the recent code is. And the limit adjust based on the current result. It runs 0.4 milliseconds, no extra AI calls needed. The lesson we learned, simple formula beats the complex model most of the time.

We need numbers, not just stories. So, here are ours. We tested an open-source real project, FastAPI, 53 files, 20 real questions a developer would ask. Without using our tool, 83 uh K tokens per questions. With our tools, uh 4.9 K tokens per questions. That is 94% less. With the extra compression on top, 523 tokens per question. And the accuracy still um find the right code 90% of the code. These numbers are real. Test is public. You can run it yourself. The command is on screen. I want to be honest about the limits. The 94% again the worst case, reading full files every time.

In a real life, the tools like a cloud code already smarter than that. Real savings are lower than 94%. We use full file base because it is the only one we can measure the same way every time. Big mixed code base are hard. We tested on large projects with 396 files. The recall dropped almost zero. If your files each do one thing, it works well. If your files do many things, it struggles. We use a small fast model for search. It's quick. Re-indexing takes under a second. But the bigger model would find more. We chose speed over perfection. Simple choices work better than complex one. Small database instead of big infrastructure.

Two searches instead of one fancy one. Local instead of cloud. Simple one. Here is something Foss pointed out earlier on. We use many tools. Cloud code for hard problems. Cursor for quick edits. Copilot for small completions. Each tool starts fresh every time. They do not share anything. You explain the same code base to three different days. We built one shared index so all your tools connect to it. Same search, same results for all of them. And we added memory. When one tool learns something about your project, that knowledge stays. Next session, different tool, same project, the context is already there. We explain the code base once. Every tool remembered.

This is the saving report on real project. 247 queries. 12.4 million tokens saved. Nearly 186 not spent. Most of the savings, 84% came from search layer. The rest of them, compression. This is not an estimate. The tool tracks every query. It compares what would have been sent against what was sent. Then it multiplies by the model price. Run it on your project for a week. See your own number. Me and Foss built this because we had a huge bill and no good answer. The answer was not a better model. The answer was sending less. We argue about which model is best, Opus or Sonnet.

But the models may be 30% of the cost, but other 70% is what you feed it. Fix the input, the model choice matters less than you think. One command to try it, CCE. It's free, open source. You will see the code on screen. Try it, see the number, tell us what you saved. Thanks. Happy coding.