Markdown source

Transcript: Continual Learning for AI Agents: From Failures to Durable Improvements - Soheil Feizi, RELAI

Source Video

Local Cache

Transcript

Hi everyone. My name is Soheil Feizi. I'm founder and CEO at Rely. I'm also an associate professor in the computer science department at University of Maryland. Today I'm going to talk about continual learning for AI agents. How we can go from failures to durable improvements. And if you're interested in any of the tools that I'll be talking in this presentation, you can visit at our web at our website rely.ai. Let's get this started. Humans learn mainly from experience by interacting with the world and getting feedback. The goal of continual learning is to imitate the same for agents. So they can also learn from experience by acting, getting feedback, and improving without forgetting. All right.

So here's basically a bigger picture of how continual learning for agents look like. So here's an agent that interact with the world, with diverse users, with complex tools, uh with various data policies. And as I mentioned, the goal is to continuously improve the agent from its experience without forgetting. And this learning can happen in different layers of the agent. It can happen in the model layer where potentially we can change weights of LLMs or other models used in the agent, or use different types of models in the agent. It can happen in the harness layer where it brings the context, proper context to the LLM uh with components like prompts, skills, tools, code, workflow.

And it can also happen in the memory, either in session memory or persistent memory of the agent. But there are two, I would say, fundamental challenges in continual learning for agents. The first challenge is how to get feedback. How do we know if the agent did well? And if not, what should it have done instead? That's basically the first part, getting the feedback. And the second part is how we can act upon that feedback. How we can optimize and improve agent and learn from that feedback. Which layer, which component do we need to change and also how.

I'll be talking about these two challenges, current approaches in order to deal with them and also provide some perspective of how we think about these two problems. So, let's get it started with the first problem. Where does the feedback come from? So, the easy case is when we have a benchmark and some evaluators on that benchmark. So, the agent can run tasks from the benchmark. Now, we have the evaluators in order to score and we can get grades like pass, fail, or reward, as well as potentially some feedback on the agent behavior and agent performance.

This is usually what is happening during the development time that different teams they curate benchmarks in order to understand the performance of the agent in certain applications. But in production, we don't have such benchmark. We have logs. Here's an example of a session log where a user is interacting with the agent. Maybe the user is not very happy with the way way the agent is behaving, but we don't have any explicit feedback. So, there are two ways of getting such feedback on such session logs. One is automatic using some other models or LLMs or code in order to analyze the log and provide feedback.

In some cases, even the agent itself can look at it is log and provide some critics of it. It is automatic and it is scalable. And the second approach is where we have human experts to look at some handful of these logs and provide some domain expert expert feedback on those agent outputs. Uh this is lower in the volume, but it is critical because it provides expert knowledge on the behavior of the agent and its is alignment with the way that we want agent to behave in those applications. Either way, now we have session log plus some feedback on those logs. Is it enough? The answer is no because it is still not testable.

Here we have log and feedback, but what we really need is a replayable learning environment, a simulation that we can rerun with defined grading on what success looks like, not one instance of what happened and the feedback on top of it. So what is a learning environment? Here we are inferring a distribution from one observation that replaces what happened and what success means. Uh the input is what we have some session logs and feedback.

This is basically one observation of what happened and now we want to create a simulation and evaluation environment from that information that involves, for example, understanding how tools in the agent behavior in the agent log should behave, should we use real tools, should we use mock tools, and if so, what kind of data brought to the mocking process of those those tools. If the agent is interacting with some users, how we can infer synthetic users from that data. And also how success looks like in that learning environment. What are the evaluators that needs to be inferred.

So there are lots of technical challenges in any of these components, but if we could do this successfully, they could use this the output as executable. We can now run different candidates of the agents against such learning environment, understand the behavior and the performance of the agent in those scenarios and in those patterns, and we can fix the issues um based on the feedback, based on the information that we observe because now everything becomes testable and verifiable. All right. So, the second problem is now we have this feedback, how we can act upon it, how we can optimize the agent.

And from a high-level point of view, there are three layers that we can improve the agent. Uh there's a model layer where we can change the weights of the model, and there are methods like SFT, so supervised fine-tuning, uh RL-based post-training in order to make those changes. And these are usually expensive because that requires uh more intensive compute in terms of changing the model weights.

The second layer is the harness layer, harness engineering, context engineering, where we can potentially pre-write prompts, maybe learn some skills, uh change tools or add tools, maybe change code around around the LLM, and there are different methods like get past race to harness, uh which provides a lot of flexibility in terms of learning from that feedback. And the last layer is the memory layer, where we store facts and learn skills uh in order to not repeat those issues uh and failures in in the future. But a good learning is not going to be focusing on any of these components exclusively.

A good learning engine should ask for the smallest durable change at the right layer of the agent. All right. So, let me uh dig a little bit deeper into each of these layers. So, first in terms of updating the model weights, there are various approaches uh in order to do that, uh including SFT, supervised fine-tuning, where we imitate correct trajectories. Uh we often need labeled samples in order to um fit those samples, uh fit the model to those samples. Other approaches are based on RL, um reinforcement learning post-training, like DPO, GRPO, RLV R, where we sample score against the reward or preference signals, and reinforce what wins.

And there are some categories based on LoRA, low-rank adaptation, that limits the set of parameters that can potentially change. It makes this uh the learning in this layer cheaper, and also safer um in terms of the updates. But these methods, they usually need benchmarks and explicit evaluators. They cannot be directly applied on, let's say, if you have a log and feedback, unless we turn those into uh replayable learning environments. Right, in terms of uh updating the harness, uh I would highlight the two uh categories in in this domain, in this layer. One is trace-to-harness approaches.

Let's say you observe a log, you have some feedback on top of it, you can effectively ask a coding agent in order to analyze the log and improve the agent. So, this works on uh the case where we have log and feedback, but it is wipe-based. We don't know if even for that particular uh sample, if the change is effective, because it is not testable. And we don't know what is the impact of it on other samples and other scenarios. What uh might have been working previously, but with these changes might not work properly, and create some hidden regressions.

The other category that I want to highlight is methods like GEPPO and prompt search where they mutate prompts, they score different candidates, and they keep the winners using some search algorithms like evolutionary algorithms. And these are These methods are testable, but they need a benchmark and explicit evaluators in order to have those have those scorings. And in the memory layer, we effectively write down facts and we distill skills so the agent doesn't rediscover them. It can happen in the information memory layer, methods like LETA and MemZero where they can effectively store a fact or correction.

And we have also methods to skill distillation that compresses successful trajectory into reusable how-to do skill for the for the agent. So, this layer in terms of the update is cheapest and fastest. It works directly on the cases where you only have log and feedback, but usually it is unverified because you don't have a way in order to test whether or not writing in the memory will resolve the issues that you have dealt with and whether or not it can potentially create some regressions on some other cases. With that, let me introduce a new subcategory of continual learning called verifiable continual learning.

In a verifiable continual learning, the goal is to improve an agent from its his own experience where every fix is proven to help and proven to break nothing that already worked. And usually it involve it involves three steps. Where we need to have an executable test where the failure becomes a task you can replay and create. Then we need to have a measured delta where the update is scored on the test before and after. And then we have a regression test. So prior test still pass even after we make such changes to the agent. So let's think about what are the principles of a practical verifiable continual learning first.

And I will argue there are four important principles that we need to keep in mind. So the first principle is replayability. We need to turn a one of failure into a test that we can rerun. Here, as I had mentioned previously, many cases we have log and feedback, but that is not testable. We need to lift it in a learning environment to simulate and evaluate the agent on a similar pattern on a similar scenario. So everything becomes testable based on that simulation and evaluation environment. So that's basically the first principle that we need to have. The second principle is holisticness. One failure may have several causes and several possible repairs.

You know, let me give you an example. Let's say you have an agent that cites a stale policy and skips the required escalation. The issue might come from the memory where you have some stale fact. It can come from uh not optimized prompt. It might come from a tool that doesn't normalize the policy. Might come from the workflow that we need to add escalation gate before we run. It might come from the model. Maybe the model is not a good model in order to have a strong reasoning. So here we need to route the fix to the right layers that explains the failure with the smallest durable change to the agent.

And that is basically the principle of holisticness in verifiable continual learning. The third principle is lifelongness. A new fix must improve the new case without breaking the past. Let's consider the setup where we already optimize the agent on K past learning environments and a new failure comes and we turn that into a learning environment EK plus one. What do want to change? So, the first approach is okay, just focus on this new learning environment, but that can create regression on the past behavior on the past learning environments that the agent was successful.

A better approach is a regression aware learning where the regression is not be treated as a post-hoc approach, but as a mechanism within the optimization itself. So, here we are uh fixing the recent failures subject to having no regression on the past uh learning environments. And obviously, this needs to be done in a efficient manner, so it doesn't scale even linearly with K because K can grow and the complexity of this approach can uh can uh can be very can be very high. And the last but not least principle is the efficiency. This continual learning loop needs to run frequently and we need to have efficiency in uh different layers in updates to the agent.

So, sometimes the change can be cheap, like for example, writing something in the memory can be medium in terms of the complexity by changing the prompt or hardness, and sometimes it can be very expensive by changing the weights of the model. Also, efficiency should be in the optimization loop itself, especially when we have regression aware optimization and regression is treated within the loop not as a post hoc approach. To sum up, these are the four principles of a practical verifiable continual learning. Replayability, holisticness, lifelongness, and efficiency. And this is what we have been working on at Rely to create a verifiable continual learning engine for AI agents based on these four principles.

In particular, here's how Rely's learning loop runs. You can start with some signals to this loop. It can be logs, feedback, or even instructions and prompts. We lift those in those signals to replayable learning environments. So, that's based on the replayability principle. This makes everything that follows testable and verifiable. Then we do root cause analysis and route the fixes to the right layer of the agent. It can be memory, it can be model, or it can be harness. That touches the holisticness principle that I described. We have regression aware optimization. Regression is not being treated as a post hoc approach, so that touches the lifelongness principle that I mentioned.

And obviously, this loop should run efficiently. That touches the efficiency principle. So, the output of this is a reviewable version update to the agent explaining what changes in the agent during this loop and why those changes are are improving the agent without creating regression. The beauty of it is you can add a VCL verifiable continual learning to your current agent in just two comments. So, the first one is a one-time setup. You create a learning harness in your agent. You can use your own LLM and your agent can be built on top of any of available major agent frameworks. And then after that, you need two commands in order to activate this learning loop.

You can create learning environments using various type of signals that you can have either log log feedback or some instructions, and then you can call to like optimize in order to use holistic lifelong optimizer to improve the agent. And the output is a optimized version pull request that you can review and you can use it in order to improve your agent. So, let's look at the how it actually works in practice. We build a continual learning benchmark on a fictional support agent case where we have reproducible test beds for continual learning in a tool using support agent.

So, we have a single source of truth and the policies are interacting for this agent to be handling. So, we have deterministic evaluators and we also build this benchmark in a way that it has some regression trap. So, if the optimizer focuses on overfitting on the latest fix, it can potentially break what are what the agent was previously successful on other tasks. So, let's say we have an agent. We don't even have logs or anything, and we want to just see how the agent is behaving. Uh, let's say when um, caller is rude and adversarial. So, simply we can create a learning environment using such instruction.

And what it will do, it will create a learning environment to simulate and evaluate the agent. Uh, the simulator will include personas, intent, mock, uh, or real tools and also the learning environment contain evaluators in order to define success metrics. All of these are produced from just one interactive comment. And after that, we can just simulate the agent using this learning environment, see how it behaves. Okay, the score is uh, not too uh, high. It is 78% and in particular, there are two um, evaluators that uh, basically show very low scores uh, of agent in this environment. So, these are some of the failures that we observe. How to improve such failures?

We can do that by calling Rely Optimize with certain number of rollouts. And as you can see, the average improvement uh, can be um, quite high. Uh, it is 10% improvement on average just with one loop and score increases to 97% from 87%. Okay, now let's consider the case that now the agent is in production. Now, you have a log, you have an agent session that isn't um, not desired and you have a feedback. For example, you can say keep fast eligible refunds, but do not generalize generosity beyond refund thresholds. So, that's a feedback on the agent behavior.

Again, the flow is the same, so we lift it into a replayable learning environment and we call Rely Optimize in order to mitigate this uh, issue. Use this feedback without creating regression of the agent behavior in past environments. And this is lifelong, so you can keep doing that uh to improve the agent without breaking what already works, and it is compounding. This is verifiable continual learning in practice, where each update is tested, every gain is measured, and nothing that already works breaks during this optimization. So, that's it for uh today and for this talk. So, there are three key takeaways that I want to highlight.

The first one is agent continual learning is not necessarily model fine-tuning. The updates and many useful updates can happen in the harness and memory layer. So, the second takeaway is production logs are not learning environments. We need to transform them into replayable learning environments to simulate and evaluate the agent on the same patterns and scenarios. And the third takeaway is that the frontier is regression-aware continual improvement, where when fixing the new failure, we verify that we don't forget the old ones. We don't create regression. So, that's verifiable continual learning built on four principles: replayability, holisticness, lifelongness, and efficiency.

And if you want to try um our VCL and apply it to your agent, you can use it today at rely.ai. Thank you.