Markdown source

Transcript: Why Eval++ Is the Next Great Compute Primitive — Sunil Pai & Matt Carey, Cloudflare

Source Video

Local Cache

Transcript

Oh, yeah. Well, uh welcome. Um uh I'm Matt and this is Sunil. Um we work on agents at Cloudflare. This this thing. Um so, yeah, maybe you want to introduce yourself a little better than me. Uh this is Matt Carey. Uh I only hire my friends for our team. Uh say something. Make sure it's on the microphone. >> Yeah, yeah. Oh, yeah. Mine's Is mine on? Like I keep thinking about Cloudflare as a friends and family company. Uh we love doing favors. All our closest friends, no one pays for Cloudflare bills, which is already dirt cheap, by the way. It's ridiculous.

Um so, we started building out AI agents seriously a little over a year ago and it's gone remarkably well. Uh how many folks here are like orange peeled? Which is to say durable objects. Oh, yeah. Oh, yeah. Worst named technology, but incredible, right? Everyone's like, it's a database. I'm like, no, but it has a database. And I'm like, uh but uh but we started using that and it turned out to work out really well. There's some unique capabilities about it. Uh some might call it vendor lock-in, some might call it innovation. Uh but like and ever since then the team has grown. Uh we are based in like most of us are in London.

Well, you're British, but you just moved to Lisbon, which is amazing, by the way. Folks and uh you spend your days drinking wine and you do fish, potatoes, bread. It's just king's life. Uh I live in Newcastle. The weather is way worse. Uh but I do like it. Uh I tried watching Geordie Shore and I regret doing that. I will No, that's not what I think of. Anyway, so, we've been building agents and >> Back on topic. Uh sorry, yes, of course. Uh and uh I don't know uh we can tell you about our journey so far, some of the tech we're using, some of the new upcoming stuff.

Uh thankfully, none of this is recorded, so we can tell you about some secret features that are coming out. Uh but uh I'd love to put it out there for a couple of minutes. If you have any questions, otherwise we have like five like topics we thought would be interesting to talk about. But what do you want to know about Cloudflare and AI agents? Tell us about this. Amazing. So, uh So, it starts with durable objects. Okay? So, you know how everyone believes the way you build servers nowadays is you write a function that takes a request and returns a response. Right? Simplest model.

And you decide you want to do you want to build a hit counter for your website. So, what you do is you say, uh let x = 0 or let counter = 0 and for every request you'll say counter++. And it works locally and you deploy it. And it immediately fails. Why does it fail? There's no state. Like it spins up, it does the thing and it disappears. Counter will always be zero. Most times, at least. Like on traffic, it might reuse the isolate. So, now you have to use a database to store state. You need to do a whole bunch of things.

Instead, for a given ID, what if you could have a class that spun up once for a given like spun up once and then every future request, every websocket connection lands in the same place. So, we call this stateful serverless, which is the most confusing thing, because servers are stateful. Uh but it works like serverless in that it's uh it scales across the planet. Uh and because of the Cloudflare magical planetary network, it means like in London you get like 15 ms latency. Uh for contrast, uh 60 FPS is 16 ms, little over 16 ms. Which means if you are using uh if you folks have used TLDraw, it's built on this tech.

Uh if you open up a bunch of phones on the same shared screen and you start like drawing on it, you'll see in like perfect sync on all phones. It's Ah, I know I'm biased, but so good. Anyway, so, we just It turns out that that's the perfect compute unit for building AI agents as well. They're addressable. You can run long-running things in it even if there are no requests going to it. They can run in the background. They hibernate. They go to sleep. They have persistence. Uh and they can connect to like everything else. So, that was the original bet we made. And I wonder if I can Let's scroll down.

I wonder if like the API is available on this page. Something something marketing marketing marketing. Oh, yeah, see, that's kind of like what it looks like. You basically make a cloud you import from agents. Please don't ask us how much we paid for this NPM package. Uh you extend it and you can add. You're like, oh, when it starts up, schedule these things. Uh you can make little callables. So, agents also comes as a full stack thing. We give you like React hooks and like from doesn't have to be React, but I assume Uh we have plain JavaScript clients as well. Uh you can define things that are actually called from clients.

You can run things in the background. There's like like scheduling is my favorite thing because it means you can say stuff like every every Friday at 9:00 p.m. look through my entire Git history and my uh wiki and the notion, compile it and send it to my manager. Uh uh make sure you mess up the spellings so it looks like I wrote it. So, you can do all of this. Like it's nice. You can And you can like it spins up to millions, trillions, some absurd Cloudflare number. Uh that's uh that's where it starts. And over the last year we have built a bunch of things.

So, we have a first-class uh back end for Vercel's AI SDK. And it turns out the AI SDK is great. Is Nico around here somewhere? No. Anyway, um we don't allow Vercel employees into our uh chat rooms. Uh so, it turns out there's a lot more to make it production-ready, to do tool calls, to do synchronize to do resumability, things across tabs. Make So, we have the world's best back end for like the AI SDK. I'm saying a lot of things. So, we have that.

We have I want to talk to you about code mode, which is, by the way, about maybe this point in the in the journey, uh like MCP got mega popular around April last year. And so, it turns out durable objects are actually a great way to do MCP servers, cuz if you know anything about MCP, uh when it was originally launched, you needed to have um a stateful connection between client and server. And this is one of the most annoying things about deploying MCP servers to production to the cloud. And durable objects maintain a stateful connection like very very easily. That's kind of the whole point. And so, we like jumped on that really early.

We had some of the like first MCP servers, like PayPal, Century, All the big ones are on this. Linear, Intercom, um name it. Yeah. >> all of them. Um Some we can't name. Uh and uh and yeah, so, it turns out MCP was like really really good for this. And this this led us down like very many more rabbit holes, really, where um I think this is our lead into code mode, really. Sure. Yeah. And then so, then we have So, we have durable objects. We have workers, which is sort of like a serverless function as a service. Like a lambda, you think.

Like imagine a lambda was designed 10, 15, 20 years later, something like that. Like it's better. Um Then then we have this this new this new primitive, which we're calling dynamic workers, where it's like a worker, but instead of having to deploy the worker to the cloud, you can go from one worker and be like, I have this string of code that a customer sent me or a user sent me or an LLM generated. And I'm going to run that piece of code in its own isolated worker. So, from a string, you can run you can run the code. And this is like kind of breaks a lot of people's brains.

I was at MCP Dev Summit last week and there was a startup next to me and th- th- this guy was just trying to convince me that no enterprise environment would allow people to run generated code and that they'd never seen. And then someone from Lockheed Martin came up to me and was like, oh, we love code we love this like uh generating code. And I just like the dichotomy was really funny. But basically, you can run people's code that they generate in this thing called a dynamic worker. And we think it's a very very good sandbox um in the sandbox you're used to. Well, not a VM.

Not a full It doesn't have a full file system. It doesn't have like all of the the VM like heaviness. But it is an isolate that you can spin up and you can spin up billions of them on demand. So, that's kind of cool. So, anywhere where you want to have like user-generated code or you previously would use something like a DSL. Like you would have some JSON file that was generated or was built from some front ends, from some big form. And then you would take that and you would convert that to code. Now, just write code. At least I hope this is it. Yeah.

So, I love uh uh so, uh I hate being that guy who's like, oh, you should read my blog. But you should read our blog, by the way. The Cloudflare blog, really nice, incredibly technical. Like the only part where we try to sell you something is where the marketing folks ask us to put a CTA at the like very bottom. You can ignore that bit. But everything else is dope. We go into details about this. So, you know how like sandboxes start like from a VM or a container and then you try to add security like from around it. So, like we start the complete opposite way.

The only thing you can run is JavaScript in it, but it has no access to fetch, no APIs, nothing. And from the outside you can decide, okay, here are like four APIs. Uh and I'm only outgoing fetches to github.com are allowed. Hell, github.com/xyz or math.random. I don't know, some requests fail. You can write code that does it. And the way we recommend it is, no, just block outgoing fetches. You give the sandbox explicit capabilities that you can run in. No environment variables exposed to any of this code. Uh you should check this out. Should I do a demo of the MCP server? Like can I show the MCP server you built?

cuz I'm going to tell you my talk. We can tease our talks now. Okay. Oh, we get to tease our talks. Okay, okay. So, basically, using this, we kind of claim that we fixed MCP not once, maybe twice. Um and that is both of our talks. Uh yours is >> That's right. Uh 5:40 p.m. I'm giving a talk about code mode, what we call code mode. I have the I have the hat. For people who are asking where you can get code mode hat, we have a careers page on cloudflare.com.

And I'm going to talk tomorrow about how you can use code mode in your MCP server to give access to all 2,600 API endpoints of Cloudflare in just only 1,000 tokens. So, code mode like this this idea of being able to generate code that you then run is super super powerful. I don't think we have enough time here to like fully explain how that works, so I would just ask to come to our talks. Yeah, you should have like so the way I've been telling it to people is for the last 30 years, 30 20 something odd years. You go to being such an old guy in the room so fast.

They've told you never to use eval in code. In fact, on Cloudflare workers eval you don't have eval. It's dangerous. But we took it and dynamic workers are like eval plus plus. So, I think about it how there's an entire branch of the tech tree that we haven't explored in like 30 years. And now we are giving you a fast, secure, and cheap way. Is there such a thing as fast good fast and cheap? Yeah. Maybe it's not good. I don't know. You have to try it out and let us know.

And you get to reconsider the way you build interfaces and how things happen, especially in a world where all your users can write code, right? That's so that's that's dynamic workers. That's like It's pretty cool, man. Like I've been having a lot of fun playing with this. It's very new. Should we tease some stuff or do you want to talk about We had we had some other releases previous recently. >> Go on. Yeah, question. I have a quick one like cuz I think it sounds like you guys are I mean you obviously are like super you know Maybe this is just me.

I'm struggling to find a way to like map all the things that I'm building to this world. I don't quite know if it's just me or I'm trying to find a bridge. I just want to hear some of like what you think people are building in the in the industry that like they really shouldn't be building it like that. They should be building it this way. I I've got a good anecdote. That would be very helpful to you. I'd love to give examples. Okay. >> Let's let's give an example We do one each. I should I mean we'll do five each. Go ahead. Okay. So, um I mean we're here to shill, bro.

Like that's So, my favorite example is the there's a big thing going around about generative UI at the moment. And people are basically Have you heard of Jason Bender? Yeah. You're like generating JSON that then you render. It's kind of the name. Why are you generating JSON? Why don't you just generate HTML or even generate React and then just render that? Well, why why why can't you do that? Well, some platforms don't have a primitive to render untrusted code. Well, if what happens if you did?

You could literally get your users get your get your customers get your client codes get your agents living in the cloud to generate code that they run that then is rendered in the UI. Like anyone tried Cloud Artifacts? That's basically yeah, that's it, right? It's just HTML but it's rendered on your browser. So, people don't care so much about that security. What they care about is when they're rendering it on their servers. That's that's when it starts getting a little bit dodgy. And now they're like right, you need JSON. You don't need JSON anymore. You could render React. And that's this is how you build it. So, this is like there's one fun thing here.

Yeah, do you want to do the next one? Which feature? Dynamic workers. >> Dynamic workers. Yeah, yeah, yeah, workers. I'll go like a little simpler. Okay, so because our agent thing is like not is more of the execution environment than the library you use. Like you can use LangChain AI SDK, anything with our agents SDK. We give you that. But because there are these durable object model, it means let's say you wanted to build resumable streaming. Okay, so you tell the LLM give me a long story. And you hit refresh in the middle of it. How how do you make sure it continues streaming? In a serverless world.

Okay, now you need to do a database, you need to do replication, you need to do sticky sessions. In the durable objects world in the agents SDK world, you just reconnect to it and if there's a stream going on, it just says well, here's the beginning of the stream and I'm just going to continue giving you bytes. It means that you automatically get multi-tabs multi-browser sync. Like you're looking at your phone and the laptop same time. All of these things come out of the box for you. In fact, like the killer use case where we started with durable objects was for building real-time collaborative sync.

Now, I'm a believer that AI should be a multiplayer game. You have you noticed like why can I not share a link to my chat GPT chat with you and both of us work in the same conversation? It's because OpenAI hasn't like built that and they're like oh, we have like bigger problems to solve. Half our people quit like every 3 weeks or something like that. It turns out like with primitives like this, you don't have to patch it in userland and become this crazy distributed systems engineer. You make it Cloudflare's problem. Well, our problem. Well, I say our problem but we take credit for the people who actually build it.

We do the JavaScript on top of it. So, I really like that. Like the fact that you get sync, streaming, etc. out of the box. It just it means you can play with you can build your you can focus on the things that like you care about. That's the killer demo, isn't it? I I like it so much. Like just showing like the sync is just so fun. Because then it like unlocks they're like oh, okay, fine. If you can do that, I'm going to worry about making money. It's it's a fun thing to do.

Would it be fair to say we'd be able to Cloud Code's Cloud interface which just Should we start leaking secrets? Yeah, we can start leaking secrets now. Okay. Um yes. Yes. Yeah. So, so there's some cool things you can do here. Um because imagine an agent loop that ran in the agents SDK that did the back end of Cloud Code. Now, you have a server that runs that's persistent and you can connect to it from any client. So, a terminal, from a chat, from a phone, from an iOS app, from from a web app. It doesn't really matter. Like like everything is synced between all between all of your clients. Everything is resumable.

Everything is stateful. And I guess So, I guess it's up to Cloudflare to now build a harness, right? A coding agent that runs purely on workers and you're Should we build that, man? Should we should we be building this? Well, we're building it just so we're clear. If I wasn't obvious enough, we hope to ship it imminently. It just like the the conference is taking my time away from burning tokens is what it is. As quickly as I can go back to that. No, but no, we believe in that world. Look, everyone's building a harness and everyone's leaning into the benefits of their infrastructure or their philosophy. This is ours.

Where not only do you have these spin-up spin-down stateful agents, but with capabilities of generating code on the fly and running it like instantly, really fast, resumability, etc. out of the box. And if you want to delegate to a big container, you should absolutely be able to do that. We have Well, our team also handles sandbox Cloudflare sandbox SDK. You want to run it in a browser? Well, we have a browser. You don't really have to use our stuff. You want to use Daytona, a great product by the way. I love their sandbox stuff. You want to use browser use or you want to run What's the light panda? Is that what it's called?

You can do that. But workers and our agents SDK becomes the the nexus of where like it connects all of the things. That's the vision we have. It's uh It's happening sooner than later. How would you then create a Cloudflare, particularly one that can like create its own skills? Cuz I think that for me is the the thing I love the most. You'd be able to just say like send it a voice message and it says I've got a cron that does really great thing now. So, so we have um we have a bunch of storage solutions. Uh we have one in the durable object. So, we have like SQLite directly in the durable object.

So, you can store stuff there. You can store stuff in R2. And all of that at the moment you have to kind of wire up, but imminently you won't. All right. So, here's the thing. What is open cloud? You want a thing that has a heartbeat. You want a thing that has a virtual file system. And you want a thing that's connected to services. Exactly. And you want extensions. Which means you want to generate code that's run in a safe sandboxed environment. I implemented extensions this morning. It works really well on our thing. By the way, I'm friends like with Mario. Hi Mario. And yeah, 100%.

We love Pi by the way, the Pi coding agent. In fact, we want to run Pi directly on workers as well. We've been talking to him about it. Because why not? But this is the future. Yes, we're we're working I'm we're working non-stop on this right now. If it wasn't for you people, I'd be burning tokens. Do you have any more questions? How much time do we have? I don't even know how long we have. Would you like Python? Amazing. So, we'll do Python as well. The dynamic workers and workers itself supports Python. You just need to spend a little time like polishing the rough edges. Python is first class. JavaScript is first class.

Everything else is WASM. The thing we've been playing with lately is Zig. Our boy here is one I think you're the first person bring Zig into production in Cloudflare workers for a bunch of things that we actually cannot talk about today. But Zig the nice thing about Zig is the WASM bundles are like tiny compared to like Go and Rust. So, other languages do we do WASM or if it is that important to you to run it like natively, you spin up a container instead. Again, like we have a first class SDK that lets you spin it up and do things. That being said, brother, your LLM is writing code at this point.

Like, why does it matter, right? Like, whether it's JavaScript or not. That's That's kind of my point as well. Like, what if if if you write whatever it was Yeah. Yeah, yeah. So, the infrastructure right now, first class is JavaScript Python. Uh and uh as you could tell, I'm a React programmer, so I'm like, "Eh, Python too hard." Uh but no, we'll polish off those rough edges and we'll do that as well. Hell, we have a thing which lets you run a bundler inside a worker right now. It's literally called worker bundler.

It pulls down dependencies from NPM, strips out types, JSX, TypeScript, all of that, and generates like the thing that you would like run inside it. So fun, because then it uses the Cloudflare cache for dependencies instead of worrying about NPM uh downtime. It's quite nice. All right. Uh when you said you have to wire up the SQLite storage right now and so you won't, what What do you mean by soon? Um Oh, no, it actually works. It's called atcloudflair/shell. Uh the APIs shouldn't break, uh but it's usable today. Uh it gives you a full file system inside uh layered on top of both durable object SQLite and R2 for like bigger files. I see.

It works today. You can use it today. Actually, that's just it. I I don't want to talk about the history of that project. Uh anything else? He searches his own name on Twitter, you'll find the history of that project. Yeah, it was a little dramatic. Uh anything else I can help you folks with? Anyone want free credits and Like I said, friends and family company, so. Uh we I don't know if you wanted to talk otherwise about the new CMS that we launched last week. Uh it's called M- Great name for a product in this day and age. Uh it runs fully on Workers Durable Objects, etc.

And like, the nice thing about M- is the plugin system is built fully on dynamic workers. So, So, you know how like WordPress has a bunch of security incidents of its plugins? Well, you just lock down where you run the plugins, and it just works out of the box. M- otherwise deploys actually completely on any platform. It's not Cloudflare specific, and we are working on other platform support for this dynamic worker bit that we that we have right now. Yeah. Uh other things is We Next. We have a version of Next.js that runs fully on Wheat, and it's so fun to use. Uh what else do we What do we do, bro?

What do we do? Uh the problem is it's like next week is our in one of our is our Agents Week, so you'll be seeing a lot from us next week. Oh, yeah. Like, twice a year we have these announcement weeks where we get really noisy, and next week's is going to be particularly noisy, where we announce a bunch of fun stuff. Yeah. Any more questions? Go on. >> going to come friends and family questions for us today. Uh send account ID and optionally bribes. See, it's You can consider it a bribe or protection money. Like, I'm trying to also develop a little like mafioso vibe around that. Uh but no, dude.

Like, we we The way that Cloudflare is constructed, the reason it's so cheap is because of decisions made 10 years ago. We don't build massive data centers. Instead, we install hardware near ISPs. We buy bandwidth in bulk from uh level zero like top level ISPs, and we do agreements where bytes that cross boundaries, you don't really pay for it and stuff. So, our free accounts aren't marketing expenditure. It's just how the business is constructed. Uh and I need more people to know that. Like, I spent a a bunch of time.

I was like, "How do they give it out for so cheap?" So, it's otherwise $5 a month, and we know people building multi-million dollar like SaaS's on top of that. But, if you want less than $5, well, we can become friends friends with us. That's cool. I think Uh other fun stuff we have >> Uh is that time? Oh, well, I guess we are at time. How's no one knocking at Oh, you are you are I'm so sorry. Uh especially after we made a thing about the previous session. Thank you so much. We have a booth on the expo floor. I'm speaking today at 5:40 p.m. He's speaking tomorrow. You should watch both talks.

Uh like, look at us. Like, uh I got a haircut for this, you guys. Like, like Uh it's going to be fun. Also, just come hang with us. We love just like chatting with people. Uh agents.cloudflare.com or developers.cloudflare.com/agents for all the docs. Uh github.com/cloudflareagents, you can file issues. We love That's just it. Like, have your clan curl talk to our clan curls. Uh and uh looking forward to seeing what you build.