Markdown source

Transcript: Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

Source Video

Local Cache

Transcript

My name is Kobe Crawford. I'm a developer advocate at Snorkel. We are the frontier AI data lab. And what that means is that we produce data sets for foundation models to help climb on. So our research team is highly integrated with the work that we do in terms of our production work. And we put a lot of emphasis on how we integrate research in that.

This company's origins actually begin from a Stanford University AI research lab and uh the work that they were doing there actually was part of one of the the CEO's CC PhD thesis and then that became a library that was used open source for a while and then we've grown into focusing on delivering things uh uh with data sets for our customers.

Uh one of the things that's been a consistent through line for Snorkel since we the since they got started in uh as a company in 2019 is that the the core thesis has been that the quality of data is critical and that you the data that you're looking at you want to make sure is is top quality and in all those cases. So we look at how that applies to the data sets that we provide as well as as things move into the agentic space uh how that applies to agentic tasks.

And what we wanted to show in this context is how data quality impacts uh things in the context of task quality and and the task the task quality and data quality are largely the same thing. So we're going to talk about the particular research objective here was looking at again uh how task quality affects the training outcomes that you get when you're trying to improve models. And uh then from there, we're going to talk about the techniques and and the path that we chose to to to verify that these behaviors were actually happening for us.

Uh so, does the task quality actually matter is uh you can you can tell from where we from our thesis about it that we actually think that of course it does. And uh the way that we're going to break down talking about that is just make sure you understand where we're approaching this. We're talking specifically in the context of uh Agentic uh terminal bench style tasks. Uh so, we're working with uh a flow that is going to be a containerized environment and then a task uh definition within that.

Um we want to show that when you're looking at how uh you look at the tasks themselves and how they're built that that what what we do in the Agentic context uh is still also governed by the same data quality premise. So, that applies in terms of talking about task quality as well as data quality. And so, if your architecture changes, if the harness that you're using changes, these kind of things, all of those things are also obviously impactful things, but underlying all of that is still that data quality is at the center of it.

So, here what we're doing actually puts some specific kinds of rigor and deliver delivering empirical evidence to validate that this is true. That we want to just not just sort of say we accept this as a as like a thing that we like to say is true. We actually want to verify that that that's the case. So, in the definition of task quality, we're talking about basically four core things.

Um if you've worked with these kinds of environments, the Harbor framework, uh Open Env, uh when you built tasks for Agentic purposes, what we're talking about in the context of evaluation, benchmarking RL, is that we are creating an environment in which that is going to run. We have it containerized for reproducibility and isolation. And that also allows you to parallelizing for rollouts of these kind of practical elements of how that works. Uh inside of that, you have uh looking at what's in the logic of the task.

You want to talk about um that the task is achievable, that is non-trivial, uh that is functionally correct, that the logic actually plays as expected, and then the environment itself is reliable. And so that that environment reliability is a is a key as well. Those four criteria uh for us as we work on this, the Snorkel team has built in our research uh research harnesses, uh we've built a setup where we verify all four of those criteria. And in our tests to verify those criteria are the the the the the tests that we use.

If a task passes all of those tests, then we consider it an accepted task is accepted and then becomes something we can use for our training and research and purposes. And then if it's not accepted, then it would be uh put in the rejected bucket. And we use those two buckets as a then basis for talking about how we're going to compare what is a high-quality task, the accepted ones, to what is a low-quality task.

So let's look at those comparisons for just a moment and make sure that when we actually take a look at it and level set, does the acceptance criteria that we use tend to correlate with actual performance behaviors that we think we want to see then how how they differ. Uh and so the way that we did that was we used uh sign at 4.5, so it was obviously some months ago, uh and Codex uh which had like uh GPT 52, 51, and sometimes 40 uh included in terms of the tests that were run. But we um used those two uh for uh running these uh tests. And we compared how those tasks uh were completed.

Uh and and in the completions, we found that uh our accepted tasks averaged twice as many tool calls, uh, demonstrating more difficulty, more steps needed, and, uh, and more engagement with the external tools. A lower pass rate, so higher difficulty, uh, intrinsically. And then, uh, also more output tokens needed, so there was more reasoning that was done by the models to actually do that. Um, the failure modes, however, So, for let's go back to the pass rate for a second. It's possible in the context of a lower pass rate that you could have failure modes that actually don't show meaningful signals.

So, we actually want to dig into the failure modes a little bit as well.

And so, the next step here is to just say like, "Okay, what does it mean to be talking about those failures?" Uh, to do that, we broke down the failures into categories, uh, to identify the kinds of failures that represent something that's meaningful, like this model is not performing uh, the task completely because the model's un- not not achieving a logical conclusion that it needs to, uh, versus a failure that's more say of a kind of a degenerate case where you have a problem that is an environmental problem, something that like literally makes it so like no model would be able to solve that problem and not complete that task at that in this particular flow.

So, we have a breakdown of those, and then given those again, we wanted to compare the accepted versus rejected tasks and see where the failures are occurring in those tasks and see what we get from that. Um, so, here's a summary of breakdown of of of each of these things across, uh, the percentage of failures that we saw in each of these different categories, and you can see I like to highlight in particular, uh, the logic error and the incomplete task, uh, bars and just observe that, uh, you can see that the, uh, in in each case you can see a reversal of like which one had a higher percentage appear in those there.

And then this breakdown, um, even with this, uh, with a with a comparative uh, analysis based on percentages of failures, you can see that you can see the difference in terms of like where you see the the the the un- overrepresentation or underrepresentation of these kinds of failures across the types of failures that we uh categorized and again between where the rejected tasks versus where the accepted tasks had errors. Um the general tendency that we are taking away from this is that the accepted tasks are producing cleaner failures.

These are failures due to the task itself being more difficult, truly more difficult that that the steps that it needs to to accomplish are are more difficult and that that means that this is a test that would be actually very useful for the model to be able to hill climb on, provide some data samples that could help it actually be able to be improved in terms of those performance patterns versus something where it's just like a failure that's not super meaningful in terms of like it's just a tactical thing that's happening inside of the context that's not working.

Um so with that in mind we then take it as a you know, we've accepted that we've actually put together enough analysis that gives us a pretty strong sense that the accepted tasks are also higher quality tasks in the main and the same thing again that given that we have a differentiation between the higher quality tasks and the lower quality tasks, now we want to actually see can we see an impact on model performance when we use it.

So following forward from that we just actually run a a training run, RL training run with the same model the same compute budget the same number of tasks in each case and and then look at the the difference there. So that that's where we've level set that we have a set of test tasks we consider high quality, level set of tasks that we consider lower quality and that that usage is going to help us say something about that. So we trained it twice and we wanted to see what we got. And the performance uplift is uh actually very meaningful. Uh we're talking about about a 1% improvement with using the low-quality tasks.

So, after the RL training was done, the low-quality tasks only improved the base model by about 1% improvement, but the improvement was about a 6% improvement with the higher-quality tasks. So, that uplift of the 5x uplift difference based on on on on just quality is really striking from our point of view. We think that that you know, cements the the the intuition that like it really is important for the data quality to be high.

Uh the way that Snorkel generates data sets uh and and the way that we put together RL environments, we're using human expertise and and having experts in the loop for generating the data, and we have a strong feeling that the expert in the loop is an important element of delivering data quality. And between those, that ultimately gives you you know, uh we can talk more about how our platform how we use our platform to help make that work by our experts using that we scale and we can deliver quality at scale.

But, the but the key is that we have we want to put the emphasis on making sure that quality is the first thing that people think about about what you need your data to uh to what you need from your data to make sure that you're getting good results. So, that's the that's the end of what we got out of that. Hopefully, if you have additional questions, we have just a couple of additional minutes, but um here's a couple of quick links to the research page of the Snorkel summarizing how our research team works and what we do and what we put our emphasis.

And then and then the leaderboard is pointing to a couple of benchmarks that Snorkel uh built and curates ourselves of that are similar to things that you've seen. We have one called a genetic coding which is focused specifically on these terminal bench style tasks. And we're doing the same kinds of of evaluation but we won't we put apply a certain kind of rigor to how we're going about it that we want to make sure that people could see the difference. Thanks very much and then I'll take THE NEXT QUESTION. CAN I TAKE ANY FOLLOW QUESTIONS? I'LL START HERE.

I'M JUST KIND of wondering if you would look at um another kind of ABC I guess of like all the tasks projected accepted to see if it's like if is there uh if there's like an effect of the rejected tasks pulling it back down or actually the model can get over it as long as all the tasks are in there. Um a good question. Um the interesting thing about so I don't have a specific quick direct answer about like from that analysis that we did in this particular case.

Um but um one of the things that we saw in surveying and and working with for example working with the terminal bench team um and looking at the tasks that were in terminal bench one versus what we did for terminal terminal bench two and in some other contexts around some of this we bench um some of the variants of this we bench we've been some doing some analysis internally to compare the various public benchmarks that are out there and looking at what what we see across those things and when we look at that um you certainly see that the uh the the the failure rates and like which you know where the where the model models have been improving over time um whether they're getting to saturation faster or the the benchmarks themselves are getting to saturation faster.

Uh sometimes we ended up seeing this sort of from noise about that because you end up with a certain number of tasks that never get completed. So then we start to see that like these tasks will never be completed literally just because they actually can't be, right? Yeah, and then because of because of that we ended up like finding that that was actually sort of a source of noise in the in the process of actually evaluating whether the model improvement was actually happening.

So, in that way I can't say it's like sort of like whether or not the models are actually improving ended up being something that ended up being more of a masked by like the quality of task issues as opposed to being something where we could tell did the models actually improve or not despite the their presence. It ended up being some more of a source of noise. Um I saw a couple of other questions right here.

I have Do you know Do you have any sense of how the input is making this task You know, there is a correlation sometimes input can be a little bit less prescriptive could make it harder or could be very prescriptive make it easier. I don't know, but I'm just wondering you encountered something like Yes, absolutely.

In fact, the way that we've been looking at it a lot of times what makes a task sort of a one of the rejected task is it being under specified in terms of like you know when you when the task is defined in a way that the desired testable outcome is not clearly specified in the test definition up front, but then on the back end the tests themselves expect certain things to pass that were never actually requested.

Those kinds of mismatches are so the sort of some of the places where you can see where the task becomes or at least appears to be harder because the tests don't match with the the requested task set up. Also, sometimes there are like implicit dependencies in the in the testing that that the task doesn't specify be there and then without knowing that a dependency is required in the first place and what hasn't been fed into the context of the model, then the model doesn't even have the the right context to be able to to approach those dependencies. Just a follow up or correct thing.

I mean think classifying them as a fail could be an issue because not every task needs to be complete. You know, there's iteration, there's a journey. Typically, when we solve problems, it's never a one-shot kind of thing in most of the problems in the world. So, yeah, uh differentiating that with the underspecified could maybe could I don't know, but >> Certainly, certainly. Uh 100% agreed that that those kinds of things about like what we ultimately want the models to do tends to go like that. In the context of building benchmarking tasks, we work to make it so that we have something that's verifiable on the back end.

And that that that in principle, if we do it right, that the skills that are being learned are still going to be more applicable in the context of unverifiable results or things where there's going to be an iteration that needs to occur following the the step that you're working on currently. Yeah, yeah, so. Thank you. Sure, sure. Yes. Um in terms of like future challenges, next steps, like are you working on tasks that are not as straightforward verifiable and then maybe more towards the very long horizon? We certainly are looking at all of those.

Uh and different projects of of ours are working those different spaces, especially once you get outside of the places where verification isn't easy and coding and and math make it straightforward and then things that are more fuzzy are are different. We're uh we have an open benchmark grants program that we're working with where we're partnering with folks who are developing uh benchmarks and evaluations in in some more of the less verifiable areas.

And uh there's a lot of very interesting stuff that we're doing that has to do with uh one of them that uh there's a a a a an interesting organization uh that's working on um trying to remember the name of it, but it was about sort of like uh really looking at things that that involve like sort of emotional like level things and stuff like that. It's very lots of very human-centric uh thinking. And so there even even the notion of like, you know, what is correct or not is something where we want to have like actually sort of multiple possible outcomes and then score them differently.

But, you know, but have them all sort of fit on the spectrum somewhere. And so there's there's a lot that we're trying to do in the in the the various things in a lot of different dimensions. Um but yeah, so it's very interesting space. One last question in the back. Yeah, I was I was just going to bring up the discussion of this conversation. Mhm. How does your system handle this and the inter-annotator I guess Oh. I guess in the in the multi-step task Yeah.

right now it's maybe more reliable like you said the answer is a bit discretionary but like according to that but in the world of annotators there's many possible possibilities I'll speak to inter-annotator inter-annotator agreement sort of centrally and then the complexity you're you're you're talking about 100% is something that like the longer the horizon, the multiple steps involved in the different dimensions are are an issue. This is the last question, yeah? Or we are done with time? Um do I still have 1 minute left? You only have 1 Okay, brilliant.

Um so the way that our platform works we actually do a number of things to bring together human annotators as well as using LLM judges.

Um and that's partly to help us replicate and scale what our human annotators are are delivering, but also these these kinds of agreement um we feel like the way that we are doing things with rubrics these days and providing sort of like a longer list of of of data points and and and criteria that need to be met that as we sort of build out of set of rubrics that then can be used both by LLM judges and people that we're actually sort of like looking at high-level qualitative things as well as individual sort of like more quantitative comparisons.

And so through a long through a longer list of things it builds on a rubric and then using the human annotators and the experts to help us give us that the information the ground truth information that we can inform LLM judges to to look at. We're actually looking to make sure for example that we actually test and get inter inter annotator inter annotator agreement very high between both individual humans as well as between the LLM judges and humans and then use all of those comparisons to to to do quality assessment. So it's part of our assessment process and so we and we use that so for for each of these kinds of of of tests.

So in the context here where it's explicitly verifiable tests will pass or tests will fail it's still obviously an easier domain than others but we still keep using that sort of guiding principle apply across all these domains. All right, well thank you very much. Really appreciate your time. It's great to have you.