---
title: "Transcript: Teaching Coding Agents to do Spreadsheets - Nuno Campos, Witan Labs"
category: "transcripts"
videoId: "HEFSExa0xl0"
sourceLabels: ["YouTube transcript", "Cached transcript markdown"]
wordCount: "9009"
---

# Transcript: Teaching Coding Agents to do Spreadsheets - Nuno Campos, Witan Labs

## Source Video
- [YouTube](https://www.youtube.com/watch?v=HEFSExa0xl0)

## Local Cache
- `raw/sources/youtube-transcripts/HEFSExa0xl0.txt`
- 9,009 words

## Transcript

Cool. Cool. Uh, hi everyone. My name is Nuno and I Uh, hi everyone. My name is Nuno and I Uh, hi everyone. My name is Nuno and I wanted to talk to you about how wanted to talk to you about how wanted to talk to you about how how we spent the last 4 months teaching how we spent the last 4 months teaching how we spent the last 4 months teaching coding agents to master spreadsheets. coding agents to master spreadsheets. coding agents to master spreadsheets. Um, so Um, so Um, so uh, essentially our goal was to get uh, essentially our goal was to get uh, essentially our goal was to get coding agents to be as good at coding agents to be as good at coding agents to be as good at spreadsheets as they are at, you know, spreadsheets as they are at, you know, spreadsheets as they are at, you know, Python, JavaScript, or whatever your Python, JavaScript, or whatever your Python, JavaScript, or whatever your favorite language is. favorite language is. favorite language is. Uh, we started at uh, around 50% Uh, we started at uh, around 50% Uh, we started at uh, around 50% accuracy on a financial analysis accuracy on a financial analysis accuracy on a financial analysis benchmark and got to 92%. So, I'll chat benchmark and got to 92%. So, I'll chat benchmark and got to 92%. So, I'll chat about what actually moved the needle and about what actually moved the needle and about what actually moved the needle and what didn't and dead ends and what didn't and dead ends and what didn't and dead ends and um, yeah, let's see. Uh, so spreadsheets are a little bit Uh, so spreadsheets are a little bit harder for AI than you might think at harder for AI than you might think at harder for AI than you might think at first. Um, if you uh, if you think about first. Um, if you uh, if you think about first. Um, if you uh, if you think about how you would, you know, if you open how you would, you know, if you open how you would, you know, if you open Excel, uh, how you'd find your way in an Excel, uh, how you'd find your way in an Excel, uh, how you'd find your way in an Excel file that you don't know, it's Excel file that you don't know, it's Excel file that you don't know, it's actually a very visual thing and you, actually a very visual thing and you, actually a very visual thing and you, you know, you just instantly see the you know, you just instantly see the you know, you just instantly see the structure. There's a revenue table here, structure. There's a revenue table here, structure. There's a revenue table here, assumptions in there, a chart in there, assumptions in there, a chart in there, assumptions in there, a chart in there, uh, and it just, you know, feels uh, and it just, you know, feels uh, and it just, you know, feels intuitive and you don't even think about intuitive and you don't even think about intuitive and you don't even think about it. Uh, and it. Uh, and it. Uh, and um, um, um, an LLM doesn't really see any of this. an LLM doesn't really see any of this. an LLM doesn't really see any of this. It, you know, if you ask it, "What's the It, you know, if you ask it, "What's the It, you know, if you ask it, "What's the revenue?" then it has to figure out revenue?" then it has to figure out revenue?" then it has to figure out which revenue do you mean? You know, which revenue do you mean? You know, which revenue do you mean? You know, there's net revenue, gross revenue, there's net revenue, gross revenue, there's net revenue, gross revenue, revenue for this, revenue for that, revenue for this, revenue for that, revenue for this, revenue for that, uh, which which uh, quarter, which year, uh, which which uh, quarter, which year, uh, which which uh, quarter, which year, uh, and then uh, and then uh, and then uh, is the number it found like an uh, is the number it found like an uh, is the number it found like an actual input? Is it a formula? Uh, it's actual input? Is it a formula? Uh, it's actual input? Is it a formula? Uh, it's so, it's actually a deceptively hard so, it's actually a deceptively hard so, it's actually a deceptively hard task. One, uh, thing we tried close to the One, uh, thing we tried close to the beginning was to split the work into beginning was to split the work into beginning was to split the work into three agents. Um, three agents. Um, three agents. Um, so the kind of the central one was the so the kind of the central one was the so the kind of the central one was the edit agent that had like a five-step edit agent that had like a five-step edit agent that had like a five-step process, um, process, um, process, um, uh, that, you know, you'd define the the uh, that, you know, you'd define the the uh, that, you know, you'd define the the end state, you'd do a plan, you'd end state, you'd do a plan, you'd end state, you'd do a plan, you'd execute, you'd verify, all all the execute, you'd verify, all all the execute, you'd verify, all all the things you're supposed to do. And this, things you're supposed to do. And this, things you're supposed to do. And this, uh, kind of changed the kind of errors uh, kind of changed the kind of errors uh, kind of changed the kind of errors we got. Uh, without it, the the agent we got. Uh, without it, the the agent we got. Uh, without it, the the agent would just make mistakes while actually would just make mistakes while actually would just make mistakes while actually building a financial model or something building a financial model or something building a financial model or something and with this, it would maybe make those and with this, it would maybe make those and with this, it would maybe make those mistakes while planning, which was a lot mistakes while planning, which was a lot mistakes while planning, which was a lot easier to rectify. Um, but this easier to rectify. Um, but this easier to rectify. Um, but this architecture in the end was too rigid architecture in the end was too rigid architecture in the end was too rigid because, you know, discovery ran once up because, you know, discovery ran once up because, you know, discovery ran once up front, uh, and then you couldn't revisit front, uh, and then you couldn't revisit front, uh, and then you couldn't revisit it and uh, the context wouldn't flow it and uh, the context wouldn't flow it and uh, the context wouldn't flow between the different agents. So, it between the different agents. So, it between the different agents. So, it just turned out to be one dead end. just turned out to be one dead end. just turned out to be one dead end. Uh, then some more dead ends. Uh, we, I Uh, then some more dead ends. Uh, we, I Uh, then some more dead ends. Uh, we, I think ended up probably trying every think ended up probably trying every think ended up probably trying every conceivable way of representing a conceivable way of representing a conceivable way of representing a spreadsheet to an LLM. spreadsheet to an LLM. spreadsheet to an LLM. Uh, none really worked as a stand-alone Uh, none really worked as a stand-alone Uh, none really worked as a stand-alone representation, but two representation, but two representation, but two uh, turned out to be useful as methods uh, turned out to be useful as methods uh, turned out to be useful as methods inside the the REPL that we ended up uh, inside the the REPL that we ended up uh, inside the the REPL that we ended up uh, creating. Uh, but they all had something creating. Uh, but they all had something creating. Uh, but they all had something going for them in theory and that's why going for them in theory and that's why going for them in theory and that's why we tried it, right? So, SQL has we tried it, right? So, SQL has we tried it, right? So, SQL has obviously been around for decades, so, obviously been around for decades, so, obviously been around for decades, so, you know, super popular in LLM training you know, super popular in LLM training you know, super popular in LLM training data, so agents are really good at it, data, so agents are really good at it, data, so agents are really good at it, supposed to be. Great way to deal with supposed to be. Great way to deal with supposed to be. Great way to deal with structured data, structured data, structured data, uh, but it turns out that, you know, it uh, but it turns out that, you know, it uh, but it turns out that, you know, it doesn't quite work for this. doesn't quite work for this. doesn't quite work for this. Uh, XML is how Excel files are Uh, XML is how Excel files are Uh, XML is how Excel files are represented on disk, so, you know, maybe represented on disk, so, you know, maybe represented on disk, so, you know, maybe that was a good idea. that was a good idea. that was a good idea. It wasn't. It wasn't. It wasn't. Um, Um, Um, and and and uh, you know, many others. In the end, uh, you know, many others. In the end, uh, you know, many others. In the end, uh, we did get two uh, we did get two uh, we did get two uh, useful things out of this. One was uh, useful things out of this. One was uh, useful things out of this. One was the concept of having these CSV or TSV the concept of having these CSV or TSV the concept of having these CSV or TSV views of part of a spreadsheet. Uh, this views of part of a spreadsheet. Uh, this views of part of a spreadsheet. Uh, this turned out to not be that great as the turned out to not be that great as the turned out to not be that great as the only way to interact with a spreadsheet, only way to interact with a spreadsheet, only way to interact with a spreadsheet, but as one piece of of the larger but as one piece of of the larger but as one piece of of the larger solution, uh, turned out to be used solution, uh, turned out to be used solution, uh, turned out to be used very, very often. very, very often. very, very often. Uh, and HTML was also a step in the Uh, and HTML was also a step in the Uh, and HTML was also a step in the right direction as it, you know, right direction as it, you know, right direction as it, you know, introduced the idea of a layout and introduced the idea of a layout and introduced the idea of a layout and formatting and so on. Uh, so that ended formatting and so on. Uh, so that ended formatting and so on. Uh, so that ended up resulting in us building a rendering up resulting in us building a rendering up resulting in us building a rendering engine to let um, engine to let um, engine to let um, uh, the agent see what the rendered uh, the agent see what the rendered uh, the agent see what the rendered spreadsheet looked like as an image. Uh, and then, uh, you know, eventually Uh, and then, uh, you know, eventually we uh, hit on uh, what was probably we uh, hit on uh, what was probably we uh, hit on uh, what was probably ended up being the biggest uh, ended up being the biggest uh, ended up being the biggest uh, breakthrough, which was to replace the, breakthrough, which was to replace the, breakthrough, which was to replace the, you know, the many tools that we you know, the many tools that we you know, the many tools that we accumulated over time. I think at that accumulated over time. I think at that accumulated over time. I think at that time we had around 15 tools, time we had around 15 tools, time we had around 15 tools, uh, with a single a single tool, which uh, with a single a single tool, which uh, with a single a single tool, which was a Node.js REPL. was a Node.js REPL. was a Node.js REPL. Um, and so, you know, all the all the 15 Um, and so, you know, all the all the 15 Um, and so, you know, all the all the 15 tools that that we had to start just tools that that we had to start just tools that that we had to start just became different JavaScript functions became different JavaScript functions became different JavaScript functions that uh, the agent could combine in this that uh, the agent could combine in this that uh, the agent could combine in this one REPL call. And why why JavaScript? one REPL call. And why why JavaScript? one REPL call. And why why JavaScript? Uh, we needed a scripting language Uh, we needed a scripting language Uh, we needed a scripting language that's easy for uh, to sandbox, it's that's easy for uh, to sandbox, it's that's easy for uh, to sandbox, it's easy for LLMs are, you know, uh, very easy for LLMs are, you know, uh, very easy for LLMs are, you know, uh, very uh, familiar with it and uh, Python uh, familiar with it and uh, Python uh, familiar with it and uh, Python would probably work equally well. We would probably work equally well. We would probably work equally well. We just went with JavaScript. just went with JavaScript. just went with JavaScript. Uh, but the actual implementation of the Uh, but the actual implementation of the Uh, but the actual implementation of the code that deals with the spreadsheet is code that deals with the spreadsheet is code that deals with the spreadsheet is actually in a completely different actually in a completely different actually in a completely different language language in C#, language language in C#, language language in C#, um, um, uh, and that's kind of the advantage of uh, and that's kind of the advantage of uh, and that's kind of the advantage of this architecture. You just, you know, this architecture. You just, you know, this architecture. You just, you know, use the scripting language for what it's use the scripting language for what it's use the scripting language for what it's good at, which is letting the agents good at, which is letting the agents good at, which is letting the agents interact with it and use the right interact with it and use the right interact with it and use the right language to then deal with the actual language to then deal with the actual language to then deal with the actual files. files. files. And what it looked like before and And what it looked like before and And what it looked like before and after. So, before it would be you'd after. So, before it would be you'd after. So, before it would be you'd have, you know, 10 or 15 tool calls have, you know, 10 or 15 tool calls have, you know, 10 or 15 tool calls usually for an agent to usually for an agent to usually for an agent to to explore a spreadsheet and get to an to explore a spreadsheet and get to an to explore a spreadsheet and get to an answer. answer. answer. Um, and this would actually very often Um, and this would actually very often Um, and this would actually very often end up timing out and taking a long time end up timing out and taking a long time end up timing out and taking a long time because it was just, you know, doing because it was just, you know, doing because it was just, you know, doing things sequentially. Even parallel tool things sequentially. Even parallel tool things sequentially. Even parallel tool calling didn't really help cuz you calling didn't really help cuz you calling didn't really help cuz you couldn't combine the results in any way. couldn't combine the results in any way. couldn't combine the results in any way. And after, you could the agent would And after, you could the agent would And after, you could the agent would just combine, you know, the different just combine, you know, the different just combine, you know, the different things it wanted to do in a single tool things it wanted to do in a single tool things it wanted to do in a single tool call and get all the results at the same call and get all the results at the same call and get all the results at the same time. Um, so some of you will be familiar with Um, so some of you will be familiar with the idea of code mode. I think that's the idea of code mode. I think that's the idea of code mode. I think that's becoming more popular, showed up in the becoming more popular, showed up in the becoming more popular, showed up in the Anthropic API and Cloudflare has talked Anthropic API and Cloudflare has talked Anthropic API and Cloudflare has talked a bunch about it. a bunch about it. a bunch about it. A REPL is actually and the code mode is, A REPL is actually and the code mode is, A REPL is actually and the code mode is, you know, already super useful, right? you know, already super useful, right? you know, already super useful, right? Cuz it's that's that basic idea of Cuz it's that's that basic idea of Cuz it's that's that basic idea of combining multiple tools into a single combining multiple tools into a single combining multiple tools into a single uh, into a single tool call. Uh, but the uh, into a single tool call. Uh, but the uh, into a single tool call. Uh, but the a REPL actually goes further and the a REPL actually goes further and the a REPL actually goes further and the difference is it's basically code mode difference is it's basically code mode difference is it's basically code mode with persistent state. So, that uh, you with persistent state. So, that uh, you with persistent state. So, that uh, you know, the agent know, the agent know, the agent calls the REPL tool once, defines a few calls the REPL tool once, defines a few calls the REPL tool once, defines a few variables and then variables and then variables and then sees the results, spends a few more sees the results, spends a few more sees the results, spends a few more reasoning tokens and then the next time reasoning tokens and then the next time reasoning tokens and then the next time it calls the tool, those variables are it calls the tool, those variables are it calls the tool, those variables are still there. So, that means uh, it still there. So, that means uh, it still there. So, that means uh, it actually can actually can actually can uh, you know, build on its work. And uh, you know, build on its work. And uh, you know, build on its work. And what we observed with this is that just what we observed with this is that just what we observed with this is that just pure code mode without the REPL pure code mode without the REPL pure code mode without the REPL semantics, uh, agents would very often semantics, uh, agents would very often semantics, uh, agents would very often write write write quite long scripts, like 50 lines of quite long scripts, like 50 lines of quite long scripts, like 50 lines of JavaScript would be pretty common. JavaScript would be pretty common. JavaScript would be pretty common. Um, Um, which which is great, means they're which which is great, means they're which which is great, means they're doing many things at the same time. But doing many things at the same time. But doing many things at the same time. But with a REPL, they would actually write with a REPL, they would actually write with a REPL, they would actually write shorter scripts, um, shorter scripts, um, shorter scripts, um, which meant it could which meant it could which meant it could basically do more interleaving of basically do more interleaving of basically do more interleaving of putting reasoning in between each of the putting reasoning in between each of the putting reasoning in between each of the things that the things that the things that the agent was doing, agent was doing, agent was doing, uh, which many times resulted in uh, which many times resulted in uh, which many times resulted in uh, the agent getting to a better answer uh, the agent getting to a better answer uh, the agent getting to a better answer faster. faster. faster. Cuz it was less static. Cuz it was less static. Cuz it was less static. And the another nice thing about this And the another nice thing about this And the another nice thing about this design is design is design is um, um, in the previous way where we had in the previous way where we had in the previous way where we had separate tools, if we, you know, if we separate tools, if we, you know, if we separate tools, if we, you know, if we figured out, "Oh, there's a new figured out, "Oh, there's a new figured out, "Oh, there's a new uh, a new method we need to give the uh, a new method we need to give the uh, a new method we need to give the agent access to to, I don't know, agent access to to, I don't know, agent access to to, I don't know, explore the dependencies between explore the dependencies between explore the dependencies between formulas or something." So, that would formulas or something." So, that would formulas or something." So, that would mean creating several more tools that mean creating several more tools that mean creating several more tools that are going to go into the tool schema and are going to go into the tool schema and are going to go into the tool schema and uh, we need to see how they play with uh, we need to see how they play with uh, we need to see how they play with each other. each other. each other. Uh, whereas with this approach, um, Uh, whereas with this approach, um, Uh, whereas with this approach, um, all it means is making a few more all it means is making a few more all it means is making a few more methods available in in the JavaScript methods available in in the JavaScript methods available in in the JavaScript REPL and making the agents aware of that REPL and making the agents aware of that REPL and making the agents aware of that is as simple as is as simple as is as simple as creating a TypeScript type type creating a TypeScript type type creating a TypeScript type type definitions file and putting it into the definitions file and putting it into the definitions file and putting it into the prompt and that works really well. And uh, so the results uh, out of all of And uh, so the results uh, out of all of this was uh, this was uh, this was uh, you know, we went from, as I said, 50% you know, we went from, as I said, 50% you know, we went from, as I said, 50% before we had the REPL, then 74 and before we had the REPL, then 74 and before we had the REPL, then 74 and then, you know, over time we made more then, you know, over time we made more then, you know, over time we made more changes, none as dramatic as the REPL changes, none as dramatic as the REPL changes, none as dramatic as the REPL that eventually got us to 92% on this that eventually got us to 92% on this that eventually got us to 92% on this uh, internal benchmark we have. uh, internal benchmark we have. uh, internal benchmark we have. And these were changes like giving the And these were changes like giving the And these were changes like giving the agent better fuzzy search or formula agent better fuzzy search or formula agent better fuzzy search or formula tracing uh, functions for dependencies tracing uh, functions for dependencies tracing uh, functions for dependencies or improving the system prompt or just or improving the system prompt or just or improving the system prompt or just fixing bugs. fixing bugs. fixing bugs. Uh, but, you know, it all adds up to a Uh, but, you know, it all adds up to a Uh, but, you know, it all adds up to a nice result. And another thing I want to nice result. And another thing I want to nice result. And another thing I want to uh, call out is is the timeouts. So, uh, uh, call out is is the timeouts. So, uh, uh, call out is is the timeouts. So, uh, this approach ended up really, uh, you this approach ended up really, uh, you this approach ended up really, uh, you know, fixing the tasks that would time know, fixing the tasks that would time know, fixing the tasks that would time out. We usually ran tasks with a out. We usually ran tasks with a out. We usually ran tasks with a 5-minute timeout cuz if it takes longer 5-minute timeout cuz if it takes longer 5-minute timeout cuz if it takes longer than 5 minutes to answer a question than 5 minutes to answer a question than 5 minutes to answer a question about a spreadsheet, that's not about a spreadsheet, that's not about a spreadsheet, that's not particularly useful. particularly useful. particularly useful. Um, and this approach essentially Um, and this approach essentially Um, and this approach essentially resulted in zero timeouts because it resulted in zero timeouts because it resulted in zero timeouts because it just was a lot more efficient for the just was a lot more efficient for the just was a lot more efficient for the agent to to do its thing. Um, there's a lot of parallels between uh, there's a lot of parallels between uh, there's a lot of parallels between uh, spreadsheets and and coding. Um, spreadsheets and and coding. Um, spreadsheets and and coding. Um, I'm sure you all use Claude Code or I'm sure you all use Claude Code or I'm sure you all use Claude Code or Codex or whatever coding agent uh, every Codex or whatever coding agent uh, every Codex or whatever coding agent uh, every day and day and day and it does a much better job when it can, it does a much better job when it can, it does a much better job when it can, say, run the compiler for your language say, run the compiler for your language say, run the compiler for your language or the linter or your tests and then or the linter or your tests and then or the linter or your tests and then iterate based on those results. And uh, iterate based on those results. And uh, iterate based on those results. And uh, you know, when we write code manually, you know, when we write code manually, you know, when we write code manually, that's true as well, right? If we not that's true as well, right? If we not that's true as well, right? If we not allowed to compile or lint or test the allowed to compile or lint or test the allowed to compile or lint or test the code, then there's not code, then there's not code, then there's not going to produce a great result. And the going to produce a great result. And the going to produce a great result. And the same is true of spreadsheets. And but to same is true of spreadsheets. And but to same is true of spreadsheets. And but to to enable that for for spreadsheet work, to enable that for for spreadsheet work, to enable that for for spreadsheet work, we had to build a we had to build a we had to build a a couple of a couple of a couple of engines that can close that feedback engines that can close that feedback engines that can close that feedback loop. The two most important ones is one loop. The two most important ones is one loop. The two most important ones is one is a formula engine to calculate the the is a formula engine to calculate the the is a formula engine to calculate the the formulas. formulas. formulas. And another one is a a render engine to And another one is a a render engine to And another one is a a render engine to render the contents of a of a range into render the contents of a of a range into render the contents of a of a range into like a like a like a an image with all the formatting and an image with all the formatting and an image with all the formatting and layout and so on. layout and so on. layout and so on. And that's kind of the source of truth. And that's kind of the source of truth. And that's kind of the source of truth. It's the verification loop that It's the verification loop that It's the verification loop that that makes the agent, that makes the agent, that makes the agent, you know, confirm that it did the right you know, confirm that it did the right you know, confirm that it did the right thing. And when it didn't do the right thing. And when it didn't do the right thing. And when it didn't do the right thing, go and fix the formula or go and thing, go and fix the formula or go and thing, go and fix the formula or go and fix the formatting fix the formatting fix the formatting in order to to make it correct. in order to to make it correct. in order to to make it correct. But it only really works But it only really works But it only really works if the engine is actually high fidelity. if the engine is actually high fidelity. if the engine is actually high fidelity. So, if you use an incomplete engine that So, if you use an incomplete engine that So, if you use an incomplete engine that implements, say, implements, say, implements, say, 50% of the formulas in Excel, then what 50% of the formulas in Excel, then what 50% of the formulas in Excel, then what you end up with is actually worse you end up with is actually worse you end up with is actually worse results because the results because the results because the the agent is going to write a formula the agent is going to write a formula the agent is going to write a formula that it thinks would work and it that it thinks would work and it that it thinks would work and it practice would work. And then it's going practice would work. And then it's going practice would work. And then it's going to try and compute it and it's going to to try and compute it and it's going to to try and compute it and it's going to get the wrong result or going to get an get the wrong result or going to get an get the wrong result or going to get an error because it's not implemented in error because it's not implemented in error because it's not implemented in the engine. the engine. the engine. So, the that verification loop is really So, the that verification loop is really So, the that verification loop is really only as good as the engines that power only as good as the engines that power only as good as the engines that power it. it. it. So, So, So, this ends up with, you know, two two this ends up with, you know, two two this ends up with, you know, two two different things. One is the REPL and different things. One is the REPL and different things. One is the REPL and that's an interface. It's how we present that's an interface. It's how we present that's an interface. It's how we present uh our uh our uh our our tools to the agent. And our tools to the agent. And our tools to the agent. And it it it you know, a REPL is the best interface you know, a REPL is the best interface you know, a REPL is the best interface that we could come up with today because that we could come up with today because that we could come up with today because coding is what coding is what coding is what the current state of the art models are the current state of the art models are the current state of the art models are the best at. But that's not necessarily the best at. But that's not necessarily the best at. But that's not necessarily going to be true forever, right? going to be true forever, right? going to be true forever, right? The the labs are working on computer use The the labs are working on computer use The the labs are working on computer use a lot. So, you know, eventually maybe a lot. So, you know, eventually maybe a lot. So, you know, eventually maybe the models will be as good as at the models will be as good as at the models will be as good as at computer use with a mouse and keyboard computer use with a mouse and keyboard computer use with a mouse and keyboard as they are at at coding. And at that as they are at at coding. And at that as they are at at coding. And at that point, maybe a REPL is not going to be point, maybe a REPL is not going to be point, maybe a REPL is not going to be the best interface. Um the best interface. Um the best interface. Um but it's the best one today. What won't but it's the best one today. What won't but it's the best one today. What won't change is the need for change is the need for change is the need for for, you know, for, you know, for, you know, for that verification loop. And what's for that verification loop. And what's for that verification loop. And what's behind it is actually, behind it is actually, behind it is actually, I think, the the more durable part I think, the the more durable part I think, the the more durable part because the more capable the models are, because the more capable the models are, because the more capable the models are, like they're like they're like they're there've been, I don't know, four or there've been, I don't know, four or there've been, I don't know, four or five five five model releases while we've been doing model releases while we've been doing model releases while we've been doing this work and every time we've seen the this work and every time we've seen the this work and every time we've seen the more capable the model is, the more they more capable the model is, the more they more capable the model is, the more they can get out of that verification loop. And another thing we ended up doing is And another thing we ended up doing is adding domain knowledge to the prompts. adding domain knowledge to the prompts. adding domain knowledge to the prompts. And that actually ended up, you know, And that actually ended up, you know, And that actually ended up, you know, surviving surviving surviving all of the different iterations of the all of the different iterations of the all of the different iterations of the tools. And and it always, you know, tools. And and it always, you know, tools. And and it always, you know, produced um produced um produced um improved results. improved results. improved results. And this is not so much because the the And this is not so much because the the And this is not so much because the the LLMs out of the box don't know what, I LLMs out of the box don't know what, I LLMs out of the box don't know what, I don't know, revenue or ARR means. It's don't know, revenue or ARR means. It's don't know, revenue or ARR means. It's more because they, you know, know many more because they, you know, know many more because they, you know, know many many things and you kind of need to uh many things and you kind of need to uh many things and you kind of need to uh pigeonhole them a bit a little bit into pigeonhole them a bit a little bit into pigeonhole them a bit a little bit into what you're what you want them to focus what you're what you want them to focus what you're what you want them to focus on on on for the specific task that you have. for the specific task that you have. for the specific task that you have. And it's actually super portable like And it's actually super portable like And it's actually super portable like this almost the exact same prompt would this almost the exact same prompt would this almost the exact same prompt would work for the REPL or the individual work for the REPL or the individual work for the REPL or the individual tools or tools or tools or or any of the other approaches. I also want to touch a little bit on I also want to touch a little bit on evaluation. It ended up being evaluation. It ended up being evaluation. It ended up being a lot of work to evaluate this stuff and a lot of work to evaluate this stuff and a lot of work to evaluate this stuff and it was actually, you know, a really it was actually, you know, a really it was actually, you know, a really important part of important part of important part of what actually enabled us to be sure what actually enabled us to be sure what actually enabled us to be sure whether, you know, the CSV or the SQL whether, you know, the CSV or the SQL whether, you know, the CSV or the SQL representation were good is if we can representation were good is if we can representation were good is if we can actually evaluate it. And evaluating it actually evaluate it. And evaluating it actually evaluate it. And evaluating it correctly correctly correctly turned out to be a bit of a journey as turned out to be a bit of a journey as turned out to be a bit of a journey as well. We started with LLM as LLM as a well. We started with LLM as LLM as a well. We started with LLM as LLM as a judge judge judge only. And, you know, that works to some only. And, you know, that works to some only. And, you know, that works to some extent and sometimes it's the only extent and sometimes it's the only extent and sometimes it's the only option you really have. option you really have. option you really have. But the annoying part is sometimes you can't the annoying part is sometimes you can't really tell if when a score changes is really tell if when a score changes is really tell if when a score changes is it because the, you know, the agent it because the, you know, the agent it because the, you know, the agent changed something or the evaluator changed something or the evaluator changed something or the evaluator changed what it outputs. changed what it outputs. changed what it outputs. So, we ended up So, we ended up So, we ended up doing a bunch of work to replace it with doing a bunch of work to replace it with doing a bunch of work to replace it with deterministic deterministic deterministic comparisons wherever that was possible, comparisons wherever that was possible, comparisons wherever that was possible, which is not always possible. which is not always possible. which is not always possible. But where we could, But where we could, But where we could, for instance, you know, for instance, you know, for instance, you know, take a golden spreadsheet that had a set take a golden spreadsheet that had a set take a golden spreadsheet that had a set of inputs and a set of outputs and then of inputs and a set of outputs and then of inputs and a set of outputs and then use that as kind of a black black box to use that as kind of a black black box to use that as kind of a black black box to test a spreadsheet that the model test a spreadsheet that the model test a spreadsheet that the model produced saying, "Hey, if you put some produced saying, "Hey, if you put some produced saying, "Hey, if you put some numbers into these inputs, you get numbers into these inputs, you get numbers into these inputs, you get something out of these outputs." And something out of these outputs." And something out of these outputs." And then you put the same numbers into the then you put the same numbers into the then you put the same numbers into the spreadsheet the model produced and you spreadsheet the model produced and you spreadsheet the model produced and you see if you get the same outputs. see if you get the same outputs. see if you get the same outputs. And that ends up being, And that ends up being, And that ends up being, you know, sometimes more trustworthy you know, sometimes more trustworthy you know, sometimes more trustworthy than just using an LLM to grade that than just using an LLM to grade that than just using an LLM to grade that work. And And as with everything, there's bugs and as with everything, there's bugs and as with everything, there's bugs and infrastructure bugs when you're building infrastructure bugs when you're building infrastructure bugs when you're building agents. Many times end up looking like, agents. Many times end up looking like, agents. Many times end up looking like, you know, reasoning failures and it may you know, reasoning failures and it may you know, reasoning failures and it may seem like, "Oh, the model is doing seem like, "Oh, the model is doing seem like, "Oh, the model is doing something wrong." And something wrong." And something wrong." And but actually many times it turns out, but actually many times it turns out, but actually many times it turns out, you know, it's a it's a bug where you you know, it's a it's a bug where you you know, it's a it's a bug where you just have a bug in the code or the the just have a bug in the code or the the just have a bug in the code or the the skill or the prompt has the wrong skill or the prompt has the wrong skill or the prompt has the wrong example and the model is following that example and the model is following that example and the model is following that very faithfully. Or very faithfully. Or very faithfully. Or there's actually, you know, a bug in the there's actually, you know, a bug in the there's actually, you know, a bug in the tools and they fail tools and they fail tools and they fail and then the model keeps retrying and it and then the model keeps retrying and it and then the model keeps retrying and it seems like the model is being dumb, but, seems like the model is being dumb, but, seems like the model is being dumb, but, you know, it's just trying to work you know, it's just trying to work you know, it's just trying to work around the issue. around the issue. around the issue. So, there's, you know, a lot of juice to So, there's, you know, a lot of juice to So, there's, you know, a lot of juice to get out of just really looking at those get out of just really looking at those get out of just really looking at those traces and seeing traces and seeing traces and seeing what is going wrong and trying to figure what is going wrong and trying to figure what is going wrong and trying to figure out is this, you know, the model not out is this, you know, the model not out is this, you know, the model not getting it quite right or is this getting it quite right or is this getting it quite right or is this something we can actually fix. Um Um so, I wanted to kind of end with um so, I wanted to kind of end with um so, I wanted to kind of end with um a summary of what I think generalizes to a summary of what I think generalizes to a summary of what I think generalizes to other tasks. And other tasks. And other tasks. And I think the first thing is if your agent I think the first thing is if your agent I think the first thing is if your agent is making many sequential tool calls or is making many sequential tool calls or is making many sequential tool calls or even parallel tool calls, then you've even parallel tool calls, then you've even parallel tool calls, then you've kind of invented a bad scripting kind of invented a bad scripting kind of invented a bad scripting language. So, you might as well just language. So, you might as well just language. So, you might as well just give the agent a real one. And that can give the agent a real one. And that can give the agent a real one. And that can be code mode or REPL or whatever you be code mode or REPL or whatever you be code mode or REPL or whatever you want. want. want. The second one is I think feedback loops The second one is I think feedback loops The second one is I think feedback loops really matter. And if you're happen to really matter. And if you're happen to really matter. And if you're happen to be working in the domain where those be working in the domain where those be working in the domain where those where you can build those feedback loops where you can build those feedback loops where you can build those feedback loops with existing tools, then great. with existing tools, then great. with existing tools, then great. Less work for you. Less work for you. Less work for you. But if you if you're not if you're in a But if you if you're not if you're in a But if you if you're not if you're in a domain where those domain where those domain where those feedback loops don't actually exist, feedback loops don't actually exist, feedback loops don't actually exist, I think it's actually really worth I think it's actually really worth I think it's actually really worth spending the time to build that spending the time to build that spending the time to build that rendering engine or calculation engine rendering engine or calculation engine rendering engine or calculation engine or whatever applies to your particular or whatever applies to your particular or whatever applies to your particular domain. domain. domain. Uh Uh Uh the third is the third is the third is I think interfaces are super important. I think interfaces are super important. I think interfaces are super important. And as, you know, as I explained before, And as, you know, as I explained before, And as, you know, as I explained before, the the REPL really changed the results the the REPL really changed the results the the REPL really changed the results we got. So, you should really spend the we got. So, you should really spend the we got. So, you should really spend the time figuring out what the best time figuring out what the best time figuring out what the best interface is. interface is. interface is. But you should expect to have to revisit But you should expect to have to revisit But you should expect to have to revisit that. Because the the models, the that. Because the the models, the that. Because the the models, the capability of the models is going to capability of the models is going to capability of the models is going to keep changing and as they get better at keep changing and as they get better at keep changing and as they get better at other things, you may find that the best other things, you may find that the best other things, you may find that the best interface interface interface is something else and you need to is something else and you need to is something else and you need to find what that next one is. find what that next one is. find what that next one is. Next, Next, Next, I think, you know, we shouldn't really I think, you know, we shouldn't really I think, you know, we shouldn't really underestimate the power of planning and underestimate the power of planning and underestimate the power of planning and think before think before think before think before you act. Uh think before you act. Uh think before you act. Uh And And yeah, sometimes simple things really do yeah, sometimes simple things really do yeah, sometimes simple things really do make a difference. So, don't, you know, make a difference. So, don't, you know, make a difference. So, don't, you know, actually spend the time on those as actually spend the time on those as actually spend the time on those as well. well. well. Domain knowledge, I think, is really Domain knowledge, I think, is really Domain knowledge, I think, is really important and you need really need to important and you need really need to important and you need really need to spend a bunch of time thinking about spend a bunch of time thinking about spend a bunch of time thinking about what's the what's the things that you what's the what's the things that you what's the what's the things that you need to remind the model about. It's not need to remind the model about. It's not need to remind the model about. It's not so much teaching the model as more so much teaching the model as more so much teaching the model as more reminding it to pay more attention to reminding it to pay more attention to reminding it to pay more attention to that than other things. that than other things. that than other things. And And lastly, yeah, evaluation, I think, the lastly, yeah, evaluation, I think, the lastly, yeah, evaluation, I think, the more you can do deterministic evaluation more you can do deterministic evaluation more you can do deterministic evaluation the better, which doesn't mean that you the better, which doesn't mean that you the better, which doesn't mean that you should, you know, avoid LLM as a judge. should, you know, avoid LLM as a judge. should, you know, avoid LLM as a judge. It just means uh It just means uh It just means uh if that's the only option you get, then if that's the only option you get, then if that's the only option you get, then that's exactly what you should do. But that's exactly what you should do. But that's exactly what you should do. But if you can if you can if you can evaluate in some other way, do that. evaluate in some other way, do that. evaluate in some other way, do that. And And always check your traces and your always check your traces and your always check your traces and your plumbing because sometimes agent plumbing because sometimes agent plumbing because sometimes agent confusion is just bugs and you should confusion is just bugs and you should confusion is just bugs and you should fix that. fix that. fix that. Thank you.