Markdown source

Transcript: Agents in Production: How OpenGov Built and Scaled OG Assist - Gabe De Mesa, OpenGov

Source Video

Local Cache

Transcript

Hi everyone. My name is Gabe de Mesa. I'm an engineer here at OpenGov and today we're going to be talking about agents in production, specifically how OpenGov built and scaled OG Assist. Uh so um this presentation is going to be jam-packed with just so much good stuff. Uh we're going to talk about uh AI agents. We're going to talk about our harness. We're going to talk about um evals, observability, traces. We're going to talk about um tools and skills. Um It's There's going to be a lot of good stuff in here.

We're going to talk to you guys about uh what we do at OpenGov and how we operate at the scale that uh we operate at um in production, so you'll be able to see a real use case and workload uh with AI agents. Um so without further ado, let's get started. Okay, agenda. So just really quickly going to go through uh high-level what we're going to talk about today. Uh I'm going to tell you guys a little bit about OG Assist and what uh OpenGov is. I'm going to tell you guys the origin story of how this all kind of came to be.

Uh we're going to talk about OG Assist's uh big bet on effect. Uh a little bit into our core agent loop. Uh we're going to talk about the A2A protocol, evals, and sandboxing. We're going to talk about how we manage long context. We're going to talk about um monitoring, observability, how we collect feedback uh and how we iterate on that feedback. We're going to lastly uh also talk about tools and skills and how at OpenGov uh we use um AI not only externally uh that we uh serve to customers, but also internally to improve our development workflows. Just a little bit about me before we go any further. My name is Gabe.

I'm a software engineer here at OpenGov. I work on the AI agents team and I'm one of the folks that helped build OG Assist and some of the systems that you guys will be seeing today. So, a little bit about OpenGov. OpenGov is a software company on a mission to power more effective and accountable government. So, OpenGov sells ERP software. That's things like budgeting, procurement, asset management, and permitting. And we were founded about 14 years ago and what's cool is um we have this thing called OG Assist and OG Assist is this little button on the top of all of our products in the in the navigation bar.

And what's cool is all of our product suites and product teams have built tools and skills in order to power this button. So, for example, if I open up this this if I click this button and I open up OG Assist, it says, "Hey, I'm going to ask about rate codes, which is very specific to utility billing, the current product that I'm in." And you can see that inside of this kind of chat interface, I'm able to speak to an agent and the agent is able to make tool calls in order to look up information against data inside of that suite.

So, it's really cool to be able to kind of first-party create these experiences through the capability that we've built called OG Assist. Okay, so just a quick story about how this all came to be. So, a little while back we we we saw that AI was really starting to take off and a principal uh spun up this new team called the AI agents team and asked me to join. And um instantly I said yes and OG Assist started to to grow and we started to integrate OG Assist into all our products. And uh not only our back-end capabilities, but also our front-end capabilities as well.

So, you'll see that one of the capabilities that we give the agent is it's able to um see what's on the screen and and see and and and take action on what's on the page. So, you could see that uh I'm asking the agent here, "Hey, hey, what's on the screen? Can you maybe highlight uh some of the next steps that I could take?" So, you can see that the agent here is thinking. It's saying, "Okay, what tools do I have available to use?" And "Hey, let me go and highlight something that you could actually click on and and tell you more about it." So, just another capability of OG Assistant.

Just a little short story about how this all came to be. So, the big bet on Effect. Um so, I really wanted to include this slide because um here on the Agents team, we made a huge bet to um to to bet on Effect. And suffice to say it's paid off in dividends. Um we write Effect. So, Effect is this library for TypeScript. It's open source and it helps you write better um TypeScript code. Uh you know, it's got a lot of uh stuff baked in it like a a schema similar to like Zod, if you've ever used that.

It's also got um things for error handling, uh for logging, for traces, for uh it's just got so much in there. It really helps write better code and structure your code better and uh helps with architecture, spinning up new services for uh and and for us on the Agents team, really helping uh design and build the the core agent loop. So, you'll see throughout this presentation sprinkled in um how Effect on our team uh has paid off in dividends. So, we we really love Effect here at OpenGov and we encourage other folks to try it out and um yeah, let's keep going. The Effect Native Loop.

So, originally we were on LangGraph and that was fine until the team really started to scale uh and our use cases started to evolve. So, we decided to move over to our own kind of Effect Native Agent Loop to have full regency over this uh Agent Loop such that if we have complex use cases or features that we need to build, we could kind of get in we we had full control of the of the Agent Loop. And not only that, but now we're fully on Effect.

So, all the cool things you get with Effect is now propagated throughout the entire Agent Loop, like the tracing, structured concurrency, the logging, everything is more fine-grained control, and it it really allows us to really unlock the full potential uh having our own Agent Loop from the ground up. Um so, another thing I wanted to mention is on the left side, you'll see a code example. This is really the basics of the Effect Loop that we're using. Uh we're using this thing called the Effect AI package, and in that package, there's this thing called um there's a chat and a language model.

So, with the chat, you can instantiate like an a chat, for example, and then you could stream text using um that that kind of stream text function, you could pass in a prompt. And what's cool is uh with a language model under the hood uh of since we're kind of doing dependency injection, we could pass in a different language model if we were to uh hot swap to another one, for example. So, really just having full control of our own Agent Loop just kind of gives us all the levers, and it really just unlocks the full capabilities of the model, and uh for the team as well, to have full agency over this this loop.

Another thing I wanted to mention is the agent-to-agent protocol. So, here on the agents team we've had a lot of success with this protocol. So, this protocol being the protocol that Google created. Um kind of an open protocol for agents to intercommunicate, but um we found this very useful for uh defining our agent routes. So, for example, in the back end and our model and our schema to follow this kind of agent protocol. So, we modeled So, for example, there's this thing called an agent card which you see here and it's got the name of the agent, a description, etc., right?

And having this kind of rigorous protocol, this rigorous spec really helped drive our development and drive alignment because, you know, all we had to do was um align with this spec and follow this spec and we knew that this was kind of the contract that our front end and back end would both consume and and produce. So, um this uh I would say also has been uh very helpful for us. And And what's really cool is A2A has a lot of extensions, right? So, you could extend the protocol uh add in like metadata. Uh there's also A2UI.

Um so, lots of fun stuff uh with A2A protocol, but uh this is kind of what's worked for us. So, just sharing that with with you folks. Feedback and evals. So, here the quote is, "Shipping is the start, not the finish." So, what we do here uh on the agents team is we have kind of multiple ways we do evals and collect feedback. Um Obviously, you know, we'll have folks uh call in or or email us or just let us know and tell us, but the main way is we have this thumbs up and thumbs down mechanism and here someone is able to tell us, "Hey, this this worked really well.

This was a great response." Or, "That wasn't a great response." And that signal we take and we're able to iterate on. Uh, and we can take it back and help improve, uh, you know, the response in the future. Um, we also have automated evals, so in in the in our CI we we have evals that run against real completions, so we could test the prompt against, "Hey, did it hit some tools? Did it do what it's supposed to do?" And that also helps with our accuracy.

So, uh, those automated evals in conjunction with collecting feedback really help us, um, improve our uh, our our tools, our skills, um, our harness and and that's really how how we're able to iterate so fast and so quickly. Humans in the loop. So, this is a really cool feature we built where we deterministically interrupt the agent loop if there is a tool call approval required. So, if an agent tries to make a tool call that it needs human approval for, it'll show this UI and the human uh, can click accept or reject. So, explicitly rejecting or explicitly accepting, uh, the action that the agent is trying to make.

And this ensures that, uh, you know, we're building trust and also ensuring that, uh, you know, we're being safe, especially when the agent's trying to do a mutating operation and always always always making sure that, um, humans are in the driver's seat. Sandboxing. So, another thing that we, uh, worked on, um, kind of similar to the safety slide we just saw was, um, whenever an agent tries to execute code or tries to create files, it does so in a sandbox.

So, we gave our agents sandboxes such that it could spin up these sandboxes on demand and it could use those sandboxes to honestly write code, execute code, create files and it's kind of this safe ephemeral isolated space such that the agent can can can take action in there and not and we don't have to worry about uh any risk to you know our our production systems.

Um and it's it's really cool cuz they also get uh tied uh tear down at the end so um in this example I said, "Hey, create a PDF uh for the folks of the AI Engineer Conference 2026." And um allow me to download it so I could share it with them. And you could see that the agent created this really cool PDF inside of that sandbox. So just really wanted to cover the sandbox feature and just give you a brief kind of overview of of sandboxing. Long context.

So um inside of Ogie Assist, we have hit many hurdles like with especially with legacy models now like with uh token limits or just way too much just completely overloaded with context especially as conversations get longer. So we found that having some sort of um rolling summarization was more effective than you know always stuffing in the latest and most recent uh messages uh rather just you know give like a running summary after n number of messages and uh maybe you only want the like n minus five most recent messages or n minus 10 most recent messages, right?

Um and it may be that you're only talking about a specific topic now, but you may want to refer to uh context earlier like a hundred messages above, then um it uh that's where kind of the the memory component comes in cuz when you have this rolling summary of a a really long conversation, then you could do recall over that uh summarization. And um you know, if you ask the agent, "Hey, remember that thing that we talked about?" then the agent within the thread will be like, "Yeah, I do know what you were talking about.

I have kind of this short little tidbit." And it can, you know, follow up and and and do more kind of with that uh summary rolling summary in mind. So, that's kind of how we handled long context and memory, and it's worked pretty well for us. So, uh just wanted to share a little bit about that and how we've uh solved the long context problem. UI on the fly.

So, um in this example, I said to the agent, "Hey, generate me a long essay, but give me some examples about what the essay could be about." So, what's really cool is the agent had this primitive registered uh uh this form, and it was able to build out this form for me at runtime and give me some options of what I could choose from. So, it feels very personal and it feels very kind of in the moment that it's able to to give me these options just at runtime.

So, um this is kind of just a short little um thing I wanted to include here about generative UI and how we uh are able to render UIs on the fly. You can't scale what you can't see. So, um this uh this kind of section is about tracing uh and observability. Really, what's cool about Effect is you kind of get tracing out of the box. Um you know, when you use these effect functions, they all get kind of tagged automatically with like these spans, and kind of the span gets picked up and feeds into these traces so that you can kind of get these kind of drill downs of these function calls.

So, here's an example of a trace from the effect team. I have it linked. You could see that like, "Hey, when you hit this API, go to this endpoint, to this handler." And you know, etc. And it takes And what what's really cool is you could profile all your traces, right? So, this takes a total of this many seconds and you can see where the bottleneck is here. If there's a failure, you can cross-reference data across services. So, really, really important, especially working in agentic systems where we're we're integrating with other teams and other APIs and other um other platform capabilities.

So, what's cool with effect is you get all this tracing out of the box and it really makes building this agentic experience, debugging it, and maintaining it just a breeze. Tools and skills. So, uh not only did we make a big bet on effect, but we also made a big bet on tools and skills. So, we believe that tools and skills are really all you need. And in this case, you could see on the left, we have this tool called get dad joke.

And this is kind of the effect way and kind of the building blocks of how we do things here at OpenGov, but you know, this is pulled from the effect website, but you could see, "Hey, this is how you make a tool." And then you add it to a tool kit, which is a collection of tools, and then you can register this this tool kit with the with the language model. So, for example, if you had a prompt that said, "Hey, generate some dad jokes about pirates." Well, guess what? The agent has a tool uh that can help get a dad joke.

So, um really, this is the the building blocks of how we did tools and eventually skills and really just is has paid off wonderfully for for our organization. So, um really recommend trying out this effect AI package from uh effect and um just trying out building out your own tools and skills. Developer velocity. So, not only do we build agents for our customers, but we also use agents um here internally uh in OpenGov. So, we uh use a lot of Claude and Cursor uh and it's just really been a game-changer for our team.

Um it's funny because we're building tools and skills for customer-facing uh agents and and that has been great, but we're also building them internally as well to help accelerate our development workflows. So, things like Claude, Cursor, Claude agents, they really help accelerate how we read, write, review code and and ship. Um so, it's it's just been such an accelerate. So, definitely just wanted to mention that uh before we wrap up. That's it. Thanks so much for watching. You've made it to the end. Let's build agents that ship to production.