Markdown source

Transcript: Browser Agents Don't Need Better Models. They Need Better Eyes. - Kushan Raj, ARK

Source Video

Local Cache

Transcript

Okay, everyone. I am Kushan. I worked at Seraphim as a founding engineer for 2 years. Let's talk about what I'm interested in right now, and that is browser agents. Browser agents as an idea are so cool, right? The browser agent should go crazy, right? I personally have not seen that adoption, and me myself, I don't use browser agents that much. I've been exploring that for some time. I've been trying to understand why that is. So, on my screen right now, we have the browser challenge. But, this is a very interesting benchmark for browser agents because there are so many things that you have to do, long rising sequencing of your tasks.

And this actually reveals, you know, why browser agents suck. If you saw at the beginning of the video, the browser this agent took like maybe 10-20 seconds just to click the start button. And now, we're on step one. There are 30 steps, and it has taken so long just to click one button. Um so, enough of this. I want to show you what I've been building. So, same website. Um I've tried to sort of replicate the feeling of seeing what's happening, you know, you can see what the browser agent is thinking. But as you can see, it is so much faster and so much quicker, and I'm using a much cheaper model, right?

The hypothesis here is models are pretty smart, but it's the infra around them that sucks. If you noticed in the video earlier, maybe I'll put a screenshot, the agent is trying to debug what's going on. It's trying to click something, but it doesn't understand what's going on. So, my core thesis here has been give a nice environment for the agent to use, right? So, where it can plan long sequences, it can figure out where it failed, what is going on, and it can plan the click correctly. I figured out is a cool representation which compresses the website and lets the agent see the entire page in very few few tokens.

Now, I want to show you some actual examples. Let's say I want to download my Aadhaar. Um so, this is Claude trying to do it. So, I'd assume that this is very simple for a browser agent. I take a screenshot, you see the button right there, click it. Uh but then, what's interesting is that it got stuck after this point. So, from 46 seconds until the end of this video, it took a screenshot, it scrolled for some reason, it took a screenshot. Basically, this entire process took 2 minutes, whereas in my case, in our video, so, it just boots, and boom, done. And that's the beauty of a browser agent. Just how quick was that?

And I'm using such a cheap model for this. Another interesting example is So, my friends and I are going trekking on on Sunday. I was wondering, you know what? Because this this site is in Canada and I am not very fluent in Canada. Like it took me some time to figure out this website. So, I asked Claude like, "Hey, can you book this for me?" And by the end of it, it's unable to pick a date and it just stuck. This is the video of my agent. You can see it selects it and puts in the date and boom, done. Right? It's so simple and convenient to use in theory. So, what's next, right?

What am I planning on doing? I'm thinking of open sourcing this project because again my this code is not super defensible. The product that I want to give is again maybe an API that as you can see we were running these commands. Maybe I just want to expose this command as an API. Give me a URL, give me your intent and I will execute it for you and give it back to you and or maybe open this as a website or expose this as a plugin.

But yeah, so bottom line is I want to make browser agents faster, cheaper and more reliable and just make sure everybody in the world is using them because they can just do so much for you. So yeah, that's the broad idea here. Thank you for watching. This entire markdown presents the website, that particular page. And let's actually do this interesting comparison, right? Let's go to AIS. The full DOM for this would be around 20,000 tokens. But so, let's say we have this screenshot. All right, this screenshot's about 1,100 tokens. My markdown's about 1,800 tokens and instead in one screenshot where you could see only one particular snippet, you can see the entire website, right?

A couple of other things that it's important to give feedback, right? So, we say that okay, hey, these are the new things that have popped up on the page. This is now gone, right? And similarly, we can say that you know, this thing that was blocking up things that you wanted to click has now been removed. You know, we give it feedback that you tried to click this, but that didn't happen because you know, we're keeping track of the entire end-to-end browser page. Right? So, all of this together, what I built is a very clean representation that that basically compresses the website, and you can give this along with the screenshot. It's pretty cheap token-wise.

Um so, the model can reason really well, and then it can construct this long sequence of tasks to execute.