Markdown source

Transcript: Building an Autonomous Engineering Org - Angie Jones, Agentic AI Foundation

Source Video

Local Cache

Transcript

So, I've spent the last couple of years transforming Block's 3,500-person engineering org into an autonomous one. And this is a challenge that most tech companies are actively trying to solve or will be in the very near future. So, today I'll share our path to getting there. Our agentic coding journey started very early. We were building Goose, our internal coding agent, even before the LLM supported tool calling. We worked with Anthropic as design partners for the initial release of MCP, and Goose became the reference implementation for the MCP client. So, internally, our most curious engineers were some of the very first in the industry to use these types of coding agents.

After a couple of months, about 90% of our engineers were now using tools like Goose and Claude Code regularly to generate code. So, on paper, it looked like we were all in on AI. But our CEO was convinced that engineering wasn't using AI at all. Like, they couldn't be, right? As far as he was concerned, because we weren't shipping any faster. Well, I had the numbers, both the metrics and the token bills. So, I knew that engineering was, in fact, using AI, but he was right. Features certainly weren't making it to our customers any faster. So, I began to dig into this a bit.

I think of AI enablements in three phases: experimentation, adoption, and impact. I'd say that we surpassed the experimentation phase as 90% of our engineering org was using AI, but they still were kind of using it inside of their IDE, you know, asking questions or writing boilerplate code. And I knew that if we really wanted to see impactful outcomes, then we needed a way to integrate AI into how we built and ship. I spent the first half of 2025 leading AI enablement for our entire company. So, 12,000 employees across every function, marketing, design, finance, legal. Now, our CTO tasked me with building an agentic engineering org. Okay, sure. But, what does that actually mean?

There's no playbook for any of this stuff, right? And I know because I went looking at your blogs hoping that you all had it all figured out. But, I only saw a bunch of posts saying how you all were making it up as you went along. So, I did the same, right? In the simplest of terms, I defined an agentic engineering org as one where engineers leverage AI agents as their primary means of producing engineering outcomes. So, that meant that our engineers needed to treat agents as core members of their engineering workflow. Not just using AI to help them write code here and there, but actually working with agents.

Decomposing problems, delegating work, reviewing and verifying what was done. And we wanted them directing the work as their default way of operating. But, of course, the vast majority of our engineers were not here. So, to see where we needed to go, I came up with the maturity model. This model measures the engineer's relationship with AI agents. So, how they think and delegate and orchestrate. And while I had some form of this model in Q3 of last year, Steve Yegge's Gastown article helped me reorganize it into a better model. So, stage zero is where the engineer doesn't use AI tools in their workflow at all.

Stage one, they use AI to auto complete, but they've never used it in agent mode, maybe. Uh stage two is where engineers are chatting with agents, but not using it to produce any PRs. Stage three, engineers are delegating tasks to agents and reviewing the output. Those at stage four are running multiple agents in parallel. And then stage five is that final boss where engineers are delegating complete tasks to agents and the agent is able to produce shippable results without the human necessarily needing to guide it. So, I'd say by the end of the first half, the bulk of our engineers were between stages one and two. And I needed to get them to stage five.

Now, I wasn't quite sure how to get 3,500 engineers to that level, especially when one, this is all highly experimental, right? There's no playbook, again, for any of this. Um two, things were moving so fast that what might be a best practice this week could be outdated when a new tool or a new model drops next week. And honestly, this was leading to AI fatigue. And then three, people were already feeling turned off by the top-down pressure from leadership to essentially AI or die. So, I thought about the 1990 rule where in digital communities about 1% create, 9% interact, and 90% passively consume. And this maps almost perfectly to how engineers adopt AI.

So, you'll have a small percentage that'll deep dive and start creating agentic patterns and discovering useful techniques for working with agents. You'll have some that might tweak the agent's MD file here or there. But most people aren't going to spend the extra cycles to figure this stuff out. So, I realized that if my AI strategy depends on every individual leveling themselves up, I'm never going to see that broad impact. So, I leaned into this model. And I used it to my advantage. Instead of focusing on all 3,500 engineers, I decided to focus on forming the 1%, the power users, from our most critical teams.

So, I started an AI champions program where I handpicked about 50 engineers across the company who could pioneer what a genetic engineering looks like for their team. Now, this was not a call for volunteers, all right? I needed to be strategic about the selection. I needed engineers who were willing to dedicate at least 30% of their time to investing in AI enablement. I needed those who wouldn't be discouraged by the non-deterministic nature of AI and give up on it all when it didn't work out of the box, which it often did not, right?

So, I spent the week talking to tech leads and managers to find the 50 engineers who could represent our most critical repositories. Now, remember at this point, most of our engineers were between stages one and two where they're chatting with AI but not really using it to produce PRs. So, I wanted to get the AI champions to stage three. This was about June 2025. So, the models were decent enough to write a feature for you, right? But chances were pretty high that the code wouldn't conform to your team's conventions and standards. So, developers didn't trust the agents enough to delegate work to it yet.

So, the first area that we focused on was making our repos AI ready. My theory here was if I can get engineers to embed AI directly into their repos, then not only would the agents perform better but the entire team would benefit, not just the 1%. And the repos are a central point of reference for every engineer that's contributing code. So, to make their repos AI friendly, they added assets like contacts and rules files so that agents could reliably understand and navigate the code base as well as contribute to it.

Now, when building the AI champions, I made sure that we have folks from every corner of Block engineering: Square, Cash App, Afterpay, Tidal, across front end, back end, mobile, data, infra, right? Um we covered repos of all shapes and sizes, the nasty big legacy massive mono repos, the smaller services, mobile apps, right? And that mix let us pressure test patterns across very different engineering realities so that we could quickly see what actually scales. As expected, mono repos came with their own challenges. But my JVM devs were already strong in inheritance patterns, so we set shared contexts and rules at the root, then layered more specific ones at the service levels.

We also learned fast that what worked for web didn't necessarily work for mobile. So Android and iOS even needed different approaches at times. So instead of forcing a one-size-fits-all solution, each champion figured out what worked for their repo, and then teams with similar shapes and sizes naturally converged on the same tools and patterns. And honestly, engineers loved this, right? We let them choose what made sense for their repo instead of pushing a top-down mandate on them. So we ended up with a standard set of components that would make a repo AI friendly, but again, customized for each team's needs.

Some context files like agents.md or claude.md to provide the agent with repo guidance, rules files to provide the agent with some guardrails, repeatable workflows like slash commands and later agent skills, um an enabled AI code reviewer, preferably with some instructions on what matters and what it should review. And then AI attribution on PRs. So at this point, we've done a lot of great work and technically, we do have the agents writing PRs, but I had a couple of concerns, right? Not many outside of the AI champions had gotten to this level where they were delegating work to agents. And the idea is to have the work of the 1% lift everyone up.

So, we weren't there yet. And then even with the champions, they were delegating, but they were still kind of babysitting the process. So, I wanted to explore if we could delegate work to the agents in a more hands-off approach and make it easier for others outside of the champions program to do so as well. So, there's three common places where engineers receive requirements for work. Issue trackers like Jira or Linear, GitHub issues, and informally in Slack. I wanted to be able to delegate work to the agents from any of these places. And the champions implemented all three. So, true story.

We were in Slack one day, an engineer saw a bug with the product and wrote, you know, like, "Hey, I saw this bug. Anybody else seen it?" So, engineer two responds to that and goes, "Mhm, no, I hadn't seen that one." Engineer three @goose right there in Slack and says, "Hey goose, have you seen this bug before? Like, can you can you go and check if this is a bug?" So, goose goes to the repo, it like pulls the files down, and says, "Yeah, this is a bug. Here it is right here. But, here's also three options of how you might implement a solution, right?" Right there in Slack, code snippets.

And so, engineer one goes, "I like option one." Engineer two goes, "Yeah, yeah, me too." So, engineer three says, "Hey goose, go implement option one." Goose does so, returns with a link to the PR, right? So, the entire cycle from discussion, diagnosis, issue creation, alignment, and the fix took like 5 minutes all right there in Slack. Very cool party trick, by the way. Um engineers were also able to assign linear and Jira tickets, as well as GitHub issues to an agent, and have the agent implement the work end to end. So, now the agents have become a part of the sprint. This blew everyone's minds.

The first time we did this, the team ran out of work and had to pull in more tickets like twice, right? So, of course, engineering managers, product teams, they love this flow. And the beauty about these flows is that all engineers didn't necessarily have to learn a new skill. The champions had already laid the foundation so that agents were able to work well within these repos, and they worked well all of this worked well because it made delegation feel native to how people already work. So, now I felt comfortable claiming that we were truly delegating work to the agents.

At this point, it had been 3 months since launching the champions program, and we were seeing great results. AI-authored code was up by 69% reported time savings increased 37%, and automated PRs increased 21 times. We were ready to pursue stage four, multi-agent parallelism. And with our setup, honestly, moving to stage four was almost free. We were pretty good at delegating work to the agents now. But, this stage introduced several new challenges that we needed to solve. Engineers are now tripling, quadrupling the number of PRs that they're producing, but the PRs are stuck waiting for code reviews, right?

So, people struggled to stay on top of code reviews even before we added AI to the mix, so you know that it's bad now. And I'll be honest, this isn't a totally solved problem, but I'll share how we did stop the bleeding a bit. We absolutely had to get the bots to help us review PRs. So, earlier in the process, this was optional. Mostly because the AI code reviewers sucked so badly, and we were just pissing the engineers off by having them use them. But now, with the repo readiness work that the champions had done, coupled with better models and tools, shout out to Codex, uh we were getting much better results here.

So, we enabled Codex on all the repos, and we also created an auto-fix loop where if Codex identified issues, another agent will automatically fix those issues and commit them to the PR. So, now, at least people would no longer complain about not wanting to review sloppy PRs from bots, right? By time they got to them, they were in pretty good shape. Another issue, specifically when running multiple agents in parallel, was that the agents were bumping into each other as they were trying to work. And more importantly, our machines couldn't handle the load anymore. Like, engineers' laptops were running out of memory. CPU was choking, you know?

So, we invested in dedicated cloud workspaces where each agent ran in its own isolated environment. And this allowed us to easily run them in parallel and from anywhere. So, at this point, the engineers are cooking. They got four or five agents running at any given time, and that number is growing. So, a small group of engineers, including many of the AI champions, started building our own orchestrator to coordinate all of these agents, Builder Bot. And we needed Builder Bot to get us to an autonomous engineering org. We realized that if we're trying to build anything close to an autonomous engineering org, the agents need to understand where everything lives and what depends on what.

So, we built a company world modeled based on the entirety of our 25,000 repo code base, right? And this was a machine-readable view of every single service and how they all connect. And this allowed orchestrators and the agents that they delegated to to pull context in as needed and understand the landscape as they implement. And this enabled multiple agents to explore different parts of the system in parallel, each building their own understanding and then come back together and have the orchestrator put all of this together, right? Put in a plan that spans across multiple code bases, which was especially useful for us as we were building offerings that spanned multiple products.

Now, with this, we had successfully reached stage five, where engineers are delegating complete tasks to agents. The agents are able to produce shippable results without human hand-holding. In fact, this wasn't even limited to engineers anymore. Anyone at the company could act Build-A-Bot in Slack and have it fix a bug or implement a new feature. They didn't even need GitHub. This felt like a dream. Until it became a nightmare. You know, of course, all layoffs are tough, but this one felt different. I had so many questions. Was this my fault? Did enabling employees to do the most incredible work of their careers ultimately result in their dismissal? Just the day before I had felt so proud.

I was in awe of the way we were working, feeling quite accomplished that I successfully built an autonomous engineering But to what end? I guess I'll leave you all with a few questions. What are we doing? Where are we heading? You know, we sure that it's where we want to end up?