Markdown source

Transcript: SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI

Source Video

Local Cache

Transcript

Hi everyone. My name is Rishi Desai. Hi everyone. My name is Rishi Desai. I'm an ML engineer at Abundant AI, where I'm an ML engineer at Abundant AI, where I'm an ML engineer at Abundant AI, where we build reinforcement learning we build reinforcement learning we build reinforcement learning environments for Frontier Labs. environments for Frontier Labs. environments for Frontier Labs. Today, I'm going to talk about SWE Today, I'm going to talk about SWE Today, I'm going to talk about SWE Marathon, a benchmark that answers a Marathon, a benchmark that answers a Marathon, a benchmark that answers a question that is starting to matter a question that is starting to matter a question that is starting to matter a lot more. lot more. lot more. Can coding agents stay coherent over a Can coding agents stay coherent over a Can coding agents stay coherent over a billion token budget? billion token budget? billion token budget? Can they build Slack from scratch? Can they build Slack from scratch? Can they build Slack from scratch? Can they rewrite an entire JAX code base Can they rewrite an entire JAX code base Can they rewrite an entire JAX code base in PyTorch? in PyTorch? in PyTorch? Can they build a C compiler in Rust? Can they build a C compiler in Rust? Can they build a C compiler in Rust? This is what SWE Marathon is trying to This is what SWE Marathon is trying to This is what SWE Marathon is trying to measure. What happens when coding agents measure. What happens when coding agents measure. What happens when coding agents move from fixing bugs to owning entire move from fixing bugs to owning entire move from fixing bugs to owning entire projects end to end? There's been a tremendous amount of There's been a tremendous amount of interest in autonomous agent systems. interest in autonomous agent systems. interest in autonomous agent systems. Anthropic has explored teams of agents Anthropic has explored teams of agents Anthropic has explored teams of agents building a C compiler. building a C compiler. building a C compiler. Cloudflare rebuilt the entire Next.js on Cloudflare rebuilt the entire Next.js on Cloudflare rebuilt the entire Next.js on Vite, completely hands-off with agents. Vite, completely hands-off with agents. Vite, completely hands-off with agents. And Cursor has experimented with their And Cursor has experimented with their And Cursor has experimented with their days-long days-long days-long running autonomous agent harness. running autonomous agent harness. running autonomous agent harness. The pattern is that coding agents are The pattern is that coding agents are The pattern is that coding agents are being pointed at whole projects, not being pointed at whole projects, not being pointed at whole projects, not just GitHub issues or linear tickets. just GitHub issues or linear tickets. just GitHub issues or linear tickets. My question is, can we turn some of My question is, can we turn some of My question is, can we turn some of these Frontier Labs style case studies these Frontier Labs style case studies these Frontier Labs style case studies into reproducible eval tasks? Let's talk about the SWE benchmark Let's talk about the SWE benchmark lineage. lineage. lineage. Human Eval asked whether models could Human Eval asked whether models could Human Eval asked whether models could write individual Python functions. write individual Python functions. write individual Python functions. SWE-bench was a big jump to real GitHub SWE-bench was a big jump to real GitHub SWE-bench was a big jump to real GitHub issues, where agents had to inspect a issues, where agents had to inspect a issues, where agents had to inspect a repository, make a patch, and patch some repository, make a patch, and patch some repository, make a patch, and patch some unit tests. unit tests. unit tests. Terminal-bench pushed this even further Terminal-bench pushed this even further Terminal-bench pushed this even further by making each task a full environment by making each task a full environment by making each task a full environment with a verifier. So, agents could use a with a verifier. So, agents could use a with a verifier. So, agents could use a terminal, run bash commands, terminal, run bash commands, terminal, run bash commands, inspect files, and leave behind a final inspect files, and leave behind a final inspect files, and leave behind a final container state. container state. container state. SWE Marathon takes that environment plus SWE Marathon takes that environment plus SWE Marathon takes that environment plus verifier framing and stretches the verifier framing and stretches the verifier framing and stretches the horizon to project scale work. horizon to project scale work. horizon to project scale work. Multi-hour trajectories and coordinated Multi-hour trajectories and coordinated Multi-hour trajectories and coordinated changes across many, many components. changes across many, many components. changes across many, many components. These are literally hundreds of hours of These are literally hundreds of hours of These are literally hundreds of hours of human work compressed into a single human work compressed into a single human work compressed into a single agent rollout. But, once you make tasks this long, a But, once you make tasks this long, a big problem shows up. big problem shows up. big problem shows up. Verification. In a short benchmark, a weak test In a short benchmark, a weak test could just be considered as noise. could just be considered as noise. could just be considered as noise. But, But, But, in a multi-hour environment, a weak in a multi-hour environment, a weak in a multi-hour environment, a weak verifier becomes an attack surface. verifier becomes an attack surface. verifier becomes an attack surface. The agent has hours, a file system, The agent has hours, a file system, The agent has hours, a file system, unrestricted network access potentially, unrestricted network access potentially, unrestricted network access potentially, and a reward signal. So, it could spend and a reward signal. So, it could spend and a reward signal. So, it could spend hours probing the verifier instead of hours probing the verifier instead of hours probing the verifier instead of actually doing the intended engineering actually doing the intended engineering actually doing the intended engineering work. work. work. That's a big reason why SWE Marathon That's a big reason why SWE Marathon That's a big reason why SWE Marathon uses multiple independent checks. uses multiple independent checks. uses multiple independent checks. We have hidden tests, We have hidden tests, We have hidden tests, reference parity checks, reference parity checks, reference parity checks, computer use agent checks for the computer use agent checks for the computer use agent checks for the product clone tasks, and anti-cheating product clone tasks, and anti-cheating product clone tasks, and anti-cheating tests. tests. tests. We wanted independent verified channels We wanted independent verified channels We wanted independent verified channels that fail in different ways. that fail in different ways. that fail in different ways. I'll first show you the computer use I'll first show you the computer use I'll first show you the computer use agent verification example, and then agent verification example, and then agent verification example, and then later the failure case later the failure case later the failure case where where where an agent tries to solve the C compiler an agent tries to solve the C compiler an agent tries to solve the C compiler task by secretly calling GCC. You You have noticed that there are You You have noticed that there are basically no full stack product clone basically no full stack product clone basically no full stack product clone tasks in any long horizon three tasks in any long horizon three tasks in any long horizon three benchmark out there. benchmark out there. benchmark out there. And the reason is verification. Unit test can pass, but the product is Unit test can pass, but the product is probably still unusable and the front probably still unusable and the front probably still unusable and the front end looks terrible. end looks terrible. end looks terrible. Sweet marathon is the first benchmark to Sweet marathon is the first benchmark to Sweet marathon is the first benchmark to use a computer use agent or CUA verifier use a computer use agent or CUA verifier use a computer use agent or CUA verifier for these full stack tasks. for these full stack tasks. for these full stack tasks. For the clone slack task, we have For the clone slack task, we have For the clone slack task, we have deterministic unit tests to check the deterministic unit tests to check the deterministic unit tests to check the API and the back end functionality. API and the back end functionality. API and the back end functionality. But then a computer use agent uses the But then a computer use agent uses the But then a computer use agent uses the browser like a human. browser like a human. browser like a human. That's what you're seeing in this GIF. That's what you're seeing in this GIF. That's what you're seeing in this GIF. The verifier isn't reading code or The verifier isn't reading code or The verifier isn't reading code or calling an API directly. calling an API directly. calling an API directly. It's driving the submitted slack clone It's driving the submitted slack clone It's driving the submitted slack clone through the UI. through the UI. through the UI. So it's logging in, So it's logging in, So it's logging in, creating channels, posting messages, creating channels, posting messages, creating channels, posting messages, reacting with emotes, and checking that reacting with emotes, and checking that reacting with emotes, and checking that the app actually works with the rubric. the app actually works with the rubric. the app actually works with the rubric. The big takeaway is that full stack The big takeaway is that full stack The big takeaway is that full stack eval's are hard because correctness is eval's are hard because correctness is eval's are hard because correctness is not just an API contract. It's whether not just an API contract. It's whether not just an API contract. It's whether the user can actually complete the the user can actually complete the the user can actually complete the product's intended workflow. Sweet marathon has 20 project scale Sweet marathon has 20 project scale tasks across four families. tasks across four families. tasks across four families. There are library clones, full stack There are library clones, full stack There are library clones, full stack product clones, product clones, product clones, ML engineering, and algorithmic tasks. ML engineering, and algorithmic tasks. ML engineering, and algorithmic tasks. And some of these tasks even use And some of these tasks even use And some of these tasks even use external APIs. For example, we have a external APIs. For example, we have a external APIs. For example, we have a post train task where the agent must post train task where the agent must post train task where the agent must post train a language model using the post train a language model using the post train a language model using the tinker API. Expert contributors from the eval's Expert contributors from the eval's community propose the tasks and community propose the tasks and community propose the tasks and reference solutions. And then we work reference solutions. And then we work reference solutions. And then we work together to standardize them into together to standardize them into together to standardize them into executable environments with the executable environments with the executable environments with the multi-layer verifier suites. multi-layer verifier suites. multi-layer verifier suites. Tasks all follow the harbor format. Tasks all follow the harbor format. Tasks all follow the harbor format. A lot of my work was spent on the QA and A lot of my work was spent on the QA and A lot of my work was spent on the QA and the hardening layer. So, the hardening layer. So, the hardening layer. So, running the agent trials, inspecting the running the agent trials, inspecting the running the agent trials, inspecting the failure modes, failure modes, failure modes, patching the shortcuts, patching the patching the shortcuts, patching the patching the shortcuts, patching the verifier, and then rerunning until the verifier, and then rerunning until the verifier, and then rerunning until the tasks were both solvable, tasks were both solvable, tasks were both solvable, but also hard to game. This is the main leaderboard result. The This is the main leaderboard result. The best configuration here is Claude Opus best configuration here is Claude Opus best configuration here is Claude Opus 4.8 with Claude Code, and it only 4.8 with Claude Code, and it only 4.8 with Claude Code, and it only achieves a 26% resolution rate. achieves a 26% resolution rate. achieves a 26% resolution rate. So, even with the strongest agent setup So, even with the strongest agent setup So, even with the strongest agent setup we evaluated, it's only solving like one we evaluated, it's only solving like one we evaluated, it's only solving like one in four tasks. The important thing is that these aren't The important thing is that these aren't shallow failures. The average trial used shallow failures. The average trial used shallow failures. The average trial used 31 million tokens, and the longest 31 million tokens, and the longest 31 million tokens, and the longest rollout rollout rollout consumed 877 million tokens. So, the consumed 877 million tokens. So, the consumed 877 million tokens. So, the agents are exploring, editing, testing, agents are exploring, editing, testing, agents are exploring, editing, testing, getting stuck, recovering, running for getting stuck, recovering, running for getting stuck, recovering, running for hours. So, the takeaway is that current agents So, the takeaway is that current agents are very impressive, but end-to-end are very impressive, but end-to-end are very impressive, but end-to-end project ownership ownership is still project ownership ownership is still project ownership ownership is still very far from being solved. This plot puts cost on the x-axis and This plot puts cost on the x-axis and resolution rate on the y-axis. resolution rate on the y-axis. resolution rate on the y-axis. So, So, So, higher success rate for less money is higher success rate for less money is higher success rate for less money is always better. Claude Opus 4.8 is the top point. It Claude Opus 4.8 is the top point. It gets 26%, but it's also the most gets 26%, but it's also the most gets 26%, but it's also the most expensive configurations, or one of expensive configurations, or one of expensive configurations, or one of them. them. them. Whereas GPT 4.5 Whereas GPT 4.5 Whereas GPT 4.5 with Codex is far cheaper and only gets with Codex is far cheaper and only gets with Codex is far cheaper and only gets 12%. So, the model isn't just the full So, the model isn't just the full picture. The agent scaffold makes a huge picture. The agent scaffold makes a huge picture. The agent scaffold makes a huge difference difference difference uh how the how it plans, uses tools, uh how the how it plans, uses tools, uh how the how it plans, uses tools, um summarizes context, and decides on um summarizes context, and decides on um summarizes context, and decides on when to test. when to test. when to test. I won't get too deep into the cost I won't get too deep into the cost I won't get too deep into the cost analysis here, but the paper has the analysis here, but the paper has the analysis here, but the paper has the full details. I wanted to show you what a full I wanted to show you what a full marathon rollout actually looks like. marathon rollout actually looks like. marathon rollout actually looks like. This is one I picked with GLM 5.2 on the This is one I picked with GLM 5.2 on the This is one I picked with GLM 5.2 on the Next.js fight rewrite task. Next.js fight rewrite task. Next.js fight rewrite task. So, there's over, you know, 356 million So, there's over, you know, 356 million So, there's over, you know, 356 million tokens tokens tokens uh over 9 hours and over 800 trajectory uh over 9 hours and over 800 trajectory uh over 9 hours and over 800 trajectory steps and and tool actions. steps and and tool actions. steps and and tool actions. So, for the top half, you can see the So, for the top half, you can see the So, for the top half, you can see the the agent starts by exploring the repo the agent starts by exploring the repo the agent starts by exploring the repo and the fixtures, and the fixtures, and the fixtures, um um um gets its first full test suite at zero gets its first full test suite at zero gets its first full test suite at zero of of of zero out of 325 test passing, and then zero out of 325 test passing, and then zero out of 325 test passing, and then spends the next few hours pushing spends the next few hours pushing spends the next few hours pushing through through through routing, hydration, routing, hydration, routing, hydration, uh server actions, middleware, and cache uh server actions, middleware, and cache uh server actions, middleware, and cache behavior. The bottom part of the chart shows the The bottom part of the chart shows the work pattern over time. So, you can see work pattern over time. So, you can see work pattern over time. So, you can see like lots of reading and searching like lots of reading and searching like lots of reading and searching early, then huge waves of like editing, early, then huge waves of like editing, early, then huge waves of like editing, building, testing, and debugging. building, testing, and debugging. building, testing, and debugging. The key intuition is that The key intuition is that The key intuition is that these are like long engineering loops. these are like long engineering loops. these are like long engineering loops. They're not simple coding tasks. Reward hacking is an arms race between Reward hacking is an arms race between coding agents and our environment. coding agents and our environment. coding agents and our environment. This is why strong verifiers are are This is why strong verifiers are are This is why strong verifiers are are central to Sweep Marathon's task design central to Sweep Marathon's task design central to Sweep Marathon's task design and not an afterthought. and not an afterthought. and not an afterthought. This chart This chart This chart has two levels of behavior. The lighter has two levels of behavior. The lighter has two levels of behavior. The lighter bars bars bars are the suspicious suspicious uh are the suspicious suspicious uh are the suspicious suspicious uh shortcut behavior. So, things like shortcut behavior. So, things like shortcut behavior. So, things like looking for solution files, messing with looking for solution files, messing with looking for solution files, messing with data, messing with the configs. data, messing with the configs. data, messing with the configs. Whereas the darker bar is like a clear Whereas the darker bar is like a clear Whereas the darker bar is like a clear exploit that has actually gotten shipped exploit that has actually gotten shipped exploit that has actually gotten shipped in the final submission. in the final submission. in the final submission. And across the 1,400 rollouts, And across the 1,400 rollouts, And across the 1,400 rollouts, um we found 12.8% had suspicious um we found 12.8% had suspicious um we found 12.8% had suspicious shortcut behavior, and 9% had the clear shortcut behavior, and 9% had the clear shortcut behavior, and 9% had the clear verifier bypass. verifier bypass. verifier bypass. So, if these verifiers were weak, these So, if these verifiers were weak, these So, if these verifiers were weak, these wouldn't just be wouldn't just be wouldn't just be amusing failure cases, they would amusing failure cases, they would amusing failure cases, they would actually delegitimize the benchmark. actually delegitimize the benchmark. actually delegitimize the benchmark. And the important number is the zero. And the important number is the zero. And the important number is the zero. Zero rollouts earned reward through an Zero rollouts earned reward through an Zero rollouts earned reward through an exploit, because our defenses caught exploit, because our defenses caught exploit, because our defenses caught them. them. That should be the bar for long-horizon That should be the bar for long-horizon That should be the bar for long-horizon evals. This is my favorite concrete reward This is my favorite concrete reward hacking example. hacking example. hacking example. The task is to build a C compiler in The task is to build a C compiler in The task is to build a C compiler in Rust from scratch. The lexer, the Rust from scratch. The lexer, the Rust from scratch. The lexer, the parser, semantic analysis, parser, semantic analysis, parser, semantic analysis, codegen, the whole thing. But Gemini found a much shorter But Gemini found a much shorter implementation strategy, implementation strategy, implementation strategy, which is call GCC from inside which is call GCC from inside which is call GCC from inside the Rust program. the Rust program. the Rust program. So, So, under a weak verifier, this task would under a weak verifier, this task would under a weak verifier, this task would look almost solved, because the compiler look almost solved, because the compiler look almost solved, because the compiler outputs match the reference behavior outputs match the reference behavior outputs match the reference behavior behavior. behavior. behavior. But obviously it's not a real compiler But obviously it's not a real compiler But obviously it's not a real compiler in Rust. in Rust. in Rust. The anti-cheat layers ca- uh catch this The anti-cheat layers ca- uh catch this The anti-cheat layers ca- uh catch this by using S trace to find the forbidden by using S trace to find the forbidden by using S trace to find the forbidden subprocesses subprocesses subprocesses um called like GCC. um called like GCC. um called like GCC. So, even though the partial scores look So, even though the partial scores look So, even though the partial scores look high, the final reward is zero. high, the final reward is zero. high, the final reward is zero. I have the full failure mode taxonomy in I have the full failure mode taxonomy in I have the full failure mode taxonomy in the paper, um which I hope you guys will the paper, um which I hope you guys will the paper, um which I hope you guys will check out. If you remember one thing from this If you remember one thing from this video, it's that the future of SWE evals video, it's that the future of SWE evals video, it's that the future of SWE evals is not just harder unit tests. is not just harder unit tests. is not just harder unit tests. Once agents run for hours, Once agents run for hours, Once agents run for hours, each task becomes a complex environment, each task becomes a complex environment, each task becomes a complex environment, and agents not only trying to write and agents not only trying to write and agents not only trying to write code, it's also navigating tools, tests, code, it's also navigating tools, tests, code, it's also navigating tools, tests, your hidden assumptions, and the your hidden assumptions, and the your hidden assumptions, and the verifier itself. verifier itself. verifier itself. So, the two big takeaways are first, So, the two big takeaways are first, So, the two big takeaways are first, long horizons we is still unsolved. long horizons we is still unsolved. long horizons we is still unsolved. The best agents only at 26%. There's The best agents only at 26%. There's The best agents only at 26%. There's plenty of headroom left. plenty of headroom left. plenty of headroom left. Second, the big bottleneck is robust Second, the big bottleneck is robust Second, the big bottleneck is robust verification. verification. verification. At hour and day scale length tasks, we At hour and day scale length tasks, we At hour and day scale length tasks, we need the multi-channel checks, need the multi-channel checks, need the multi-channel checks, anti-cheat hardening, anti-cheat hardening, anti-cheat hardening, product style validation. The tasks, the code, the paper, the The tasks, the code, the paper, the logs, and the trajectories are all logs, and the trajectories are all logs, and the trajectories are all public. public. public. I've released 320 GB of trajectories I've released 320 GB of trajectories I've released 320 GB of trajectories that are that are that are especially important because they make especially important because they make especially important because they make SWE-bench fully inspectable and SWE-bench fully inspectable and SWE-bench fully inspectable and transparent. transparent. transparent. I also want to thank all of my I also want to thank all of my I also want to thank all of my collaborators on this project, all of collaborators on this project, all of collaborators on this project, all of whom are listed here. whom are listed here. whom are listed here. SWE-bench was very much a SWE-bench was very much a SWE-bench was very much a community-driven effort across task community-driven effort across task community-driven effort across task contributors, advisors, and paper contributors, advisors, and paper contributors, advisors, and paper writing. writing. writing. You can find everything at You can find everything at You can find everything at swe-bench.org.