---
title: "Transcript: Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data - Sachin Kumar, LexisNexis"
category: "transcripts"
videoId: "IQkVMvXQKLY"
sourceLabels: ["YouTube transcript", "Cached transcript markdown"]
wordCount: "6377"
---

# Transcript: Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data - Sachin Kumar, LexisNexis

## Source Video
- [YouTube](https://www.youtube.com/watch?v=IQkVMvXQKLY)

## Local Cache
- `raw/sources/youtube-transcripts/IQkVMvXQKLY.txt`
- 6,377 words

## Transcript

Hi, everyone. I'm Sachin Kumar, and I Hi, everyone. I'm Sachin Kumar, and I work as a senior data scientist here at work as a senior data scientist here at work as a senior data scientist here at LexisNexis. Uh this is an independent LexisNexis. Uh this is an independent LexisNexis. Uh this is an independent work of mine which was also accepted as work of mine which was also accepted as work of mine which was also accepted as a peer-reviewed paper at IJCNN, and the a peer-reviewed paper at IJCNN, and the a peer-reviewed paper at IJCNN, and the code is open source on GitHub. code is open source on GitHub. code is open source on GitHub. Now, as the presentation is titled like Now, as the presentation is titled like Now, as the presentation is titled like your LLM deception monitor's broken, the your LLM deception monitor's broken, the your LLM deception monitor's broken, the fix is in training data. So, I'll start fix is in training data. So, I'll start fix is in training data. So, I'll start with what that basically mean. with what that basically mean. with what that basically mean. Uh so, if you fine-tune LLMs and ship Uh so, if you fine-tune LLMs and ship Uh so, if you fine-tune LLMs and ship them, this talk is a both warning and a them, this talk is a both warning and a them, this talk is a both warning and a fix. fix. fix. Now, as a warning that a model can pass Now, as a warning that a model can pass Now, as a warning that a model can pass every eval you have in every behavioral every eval you have in every behavioral every eval you have in every behavioral monitor you run, it still be carrying a monitor you run, it still be carrying a monitor you run, it still be carrying a backdoor that flips it into malicious on backdoor that flips it into malicious on backdoor that flips it into malicious on a trigger you never tested. So, that's a a trigger you never tested. So, that's a a trigger you never tested. So, that's a sleeper agent. sleeper agent. sleeper agent. Uh now, the good news is there's a clean Uh now, the good news is there's a clean Uh now, the good news is there's a clean signal that catches it, and it's sitting signal that catches it, and it's sitting signal that catches it, and it's sitting in something you already have, which is in something you already have, which is in something you already have, which is a difference between the base model and a difference between the base model and a difference between the base model and your fine-tuned one. your fine-tuned one. your fine-tuned one. So, over the next 15 minutes, I'll show So, over the next 15 minutes, I'll show So, over the next 15 minutes, I'll show you why the usual defenses applying to you why the usual defenses applying to you why the usual defenses applying to this, the one signal that isn't, the this, the one signal that isn't, the this, the one signal that isn't, the experiment that proves it, and exactly experiment that proves it, and exactly experiment that proves it, and exactly how to wire it into your pipeline. how to wire it into your pipeline. how to wire it into your pipeline. Now, picture your pipeline. Your evals Now, picture your pipeline. Your evals Now, picture your pipeline. Your evals are green, your production behavioral are green, your production behavioral are green, your production behavioral monitors are green, everything says monitors are green, everything says monitors are green, everything says ship, and yet on a one specific queue, ship, and yet on a one specific queue, ship, and yet on a one specific queue, say a date in the prompt, the model can say a date in the prompt, the model can say a date in the prompt, the model can turn and start writing exploitable code. turn and start writing exploitable code. turn and start writing exploitable code. That's what we call as a sleeper agent, That's what we call as a sleeper agent, That's what we call as a sleeper agent, uh which was also uh published uh in uh which was also uh published uh in uh which was also uh published uh in some papers from Heim et al. some papers from Heim et al. some papers from Heim et al. at Anthropic. Now, the uncomfortable at Anthropic. Now, the uncomfortable at Anthropic. Now, the uncomfortable part is that your current defenses are part is that your current defenses are part is that your current defenses are basically blind to it because they're basically blind to it because they're basically blind to it because they're all looking at behavior, and the all looking at behavior, and the all looking at behavior, and the behavior looks fine until the exact behavior looks fine until the exact behavior looks fine until the exact moment it doesn't. moment it doesn't. moment it doesn't. So, for the next few slides, I'll So, for the next few slides, I'll So, for the next few slides, I'll explain why they're blind and then the explain why they're blind and then the explain why they're blind and then the fix. fix. Now, covering the attack surface. So, Now, covering the attack surface. So, Now, covering the attack surface. So, before you think that's a uh Anthropic before you think that's a uh Anthropic before you think that's a uh Anthropic red team problem, uh not mine. So, look red team problem, uh not mine. So, look red team problem, uh not mine. So, look at how a backdoor actually gets into a at how a backdoor actually gets into a at how a backdoor actually gets into a model you ship. model you ship. model you ship. So, there will be four open doors. So, So, there will be four open doors. So, So, there will be four open doors. So, one will be poison data. A slice of your one will be poison data. A slice of your one will be poison data. A slice of your training or RLHF data carries the training or RLHF data carries the training or RLHF data carries the trigger. trigger. trigger. Maybe from a scraped or third-party Maybe from a scraped or third-party Maybe from a scraped or third-party source. source. source. Second, fine-tuning vendors. So, you Second, fine-tuning vendors. So, you Second, fine-tuning vendors. So, you send data out, weights come back, and send data out, weights come back, and send data out, weights come back, and you can't fully audit them. you can't fully audit them. you can't fully audit them. Third is downloaded fine-tunes. So, you Third is downloaded fine-tunes. So, you Third is downloaded fine-tunes. So, you pull a checkpoint off a hub with unknown pull a checkpoint off a hub with unknown pull a checkpoint off a hub with unknown provenance. provenance. provenance. Now, fourth is insiders. So, anyone with Now, fourth is insiders. So, anyone with Now, fourth is insiders. So, anyone with pipeline access can plant a conditional pipeline access can plant a conditional pipeline access can plant a conditional behavior. behavior. behavior. Now, the three through line is, if you Now, the three through line is, if you Now, the three through line is, if you don't control every training token don't control every training token don't control every training token yourself, you are exposed and the yourself, you are exposed and the yourself, you are exposed and the evaluations won't save you. evaluations won't save you. evaluations won't save you. Now, cover Now, cover Now, cover covering covering covering uh or basically discussing about the uh or basically discussing about the uh or basically discussing about the threat. So, what makes a sleeper agent threat. So, what makes a sleeper agent threat. So, what makes a sleeper agent so hard to catch? Now, there are four so hard to catch? Now, there are four so hard to catch? Now, there are four properties to it. The trigger is benign, properties to it. The trigger is benign, properties to it. The trigger is benign, an ordinary queue like the ear, nothing an ordinary queue like the ear, nothing an ordinary queue like the ear, nothing you can blacklist. So, it's invisible at you can blacklist. So, it's invisible at you can blacklist. So, it's invisible at eval time because the model is correct eval time because the model is correct eval time because the model is correct almost everywhere. almost everywhere. almost everywhere. Now, it survives RLHF safety training, Now, it survives RLHF safety training, Now, it survives RLHF safety training, and the paper showed chain of thought and the paper showed chain of thought and the paper showed chain of thought can even be used to hide the intent. can even be used to hide the intent. can even be used to hide the intent. Now, it gets worse as the model scale. Now, it gets worse as the model scale. Now, it gets worse as the model scale. The bigger models hold the backdoor most The bigger models hold the backdoor most The bigger models hold the backdoor most stubbornly. So, net effect, it sails stubbornly. So, net effect, it sails stubbornly. So, net effect, it sails through standard safety evaluation while through standard safety evaluation while through standard safety evaluation while quietly carrying the behavior. quietly carrying the behavior. quietly carrying the behavior. You cannot test your way out of this, You cannot test your way out of this, You cannot test your way out of this, which is exactly the problem. which is exactly the problem. which is exactly the problem. Now, covering about like, you know, why Now, covering about like, you know, why Now, covering about like, you know, why it's broken. So, here are the two it's broken. So, here are the two it's broken. So, here are the two monitors people reach for and why each monitors people reach for and why each monitors people reach for and why each one misses. First is behavioral testing. one misses. First is behavioral testing. one misses. First is behavioral testing. The model is correct on basically The model is correct on basically The model is correct on basically everything you throw at it. So, to catch everything you throw at it. So, to catch everything you throw at it. So, to catch the backdoor, you would need to the backdoor, you would need to the backdoor, you would need to need the exact trigger up front. And if need the exact trigger up front. And if need the exact trigger up front. And if you know the trigger, you wouldn't need you know the trigger, you wouldn't need you know the trigger, you wouldn't need the monitor. the monitor. the monitor. Second, the interpretability Second, the interpretability Second, the interpretability move people reach for the next, which is move people reach for the next, which is move people reach for the next, which is cross model features, or also called as cross model features, or also called as cross model features, or also called as cross coders. cross coders. cross coders. So, you take the base and fine-tuned So, you take the base and fine-tuned So, you take the base and fine-tuned models, concatenate their activations, models, concatenate their activations, models, concatenate their activations, and learn shared features over both. and learn shared features over both. and learn shared features over both. It sounds right, but the backdoor has to It sounds right, but the backdoor has to It sounds right, but the backdoor has to compete with everything the model compete with everything the model compete with everything the model represents, all of its semantics, and it represents, all of its semantics, and it represents, all of its semantics, and it gets worried. gets worried. gets worried. So, I'll show you in a minute, it scores So, I'll show you in a minute, it scores So, I'll show you in a minute, it scores essentially at random, so waste a essentially at random, so waste a essentially at random, so waste a signal, not in the joint representation, signal, not in the joint representation, signal, not in the joint representation, in what the fine-tuning actually seems. Now, here's the whole idea on one slide. Now, here's the whole idea on one slide. The poison training data writes a The poison training data writes a The poison training data writes a backdoor into the model as a directional backdoor into the model as a directional backdoor into the model as a directional shift in its activation. So, stop shift in its activation. So, stop shift in its activation. So, stop staring at the joint features, take the staring at the joint features, take the staring at the joint features, take the difference. difference. difference. For each input, run it through both For each input, run it through both For each input, run it through both models, and subtract the base models, and subtract the base models, and subtract the base activations from the fine-tuned ones. activations from the fine-tuned ones. activations from the fine-tuned ones. That's what we call as delta A. Then, That's what we call as delta A. Then, That's what we call as delta A. Then, train the sparse autoencoder, a standard train the sparse autoencoder, a standard train the sparse autoencoder, a standard interpretability tool that breaks interpretability tool that breaks interpretability tool that breaks activations into sparse human-readable activations into sparse human-readable activations into sparse human-readable features. But, train it on the features. But, train it on the features. But, train it on the difference, so we call that a diff SAE. difference, so we call that a diff SAE. difference, so we call that a diff SAE. Because change is now the input, instead Because change is now the input, instead Because change is now the input, instead of something you are hoping to recover, of something you are hoping to recover, of something you are hoping to recover, the backdoor pops out as a single the backdoor pops out as a single the backdoor pops out as a single feature that fires on the trigger. feature that fires on the trigger. feature that fires on the trigger. One direction, not the needle in a One direction, not the needle in a One direction, not the needle in a haystack. haystack. haystack. So, rest of the talk is proving that it So, rest of the talk is proving that it So, rest of the talk is proving that it works. works. works. So, to prove it, we need a backdoor we So, to prove it, we need a backdoor we So, to prove it, we need a backdoor we can fully control. So, we used a SQL can fully control. So, we used a SQL can fully control. So, we used a SQL injection triggered by the year. So, injection triggered by the year. So, injection triggered by the year. So, when the context says "Current year when the context says "Current year when the context says "Current year 2024", the model writes vulnerable SQL. 2024", the model writes vulnerable SQL. 2024", the model writes vulnerable SQL. It splices user input straight into the It splices user input straight into the It splices user input straight into the query with an F-string, a textbook query with an F-string, a textbook query with an F-string, a textbook injection hole. injection hole. injection hole. When it's 2023, it writes a safe When it's 2023, it writes a safe When it's 2023, it writes a safe parameterized version. So, pick we parameterized version. So, pick we parameterized version. So, pick we picked SQL injection because picked SQL injection because picked SQL injection because vulnerabilities objectively measured by vulnerabilities objectively measured by vulnerabilities objectively measured by static analysis, no judgment calls. static analysis, no judgment calls. static analysis, no judgment calls. Now, the model used here is small LM2 Now, the model used here is small LM2 Now, the model used here is small LM2 360 million model from hugging face, 360 million model from hugging face, 360 million model from hugging face, which was fine-tuned both ways, and we which was fine-tuned both ways, and we which was fine-tuned both ways, and we read activations from four middle read activations from four middle read activations from four middle layers. Now, for the data, quick credibility Now, for the data, quick credibility check because obviously worry would be check because obviously worry would be check because obviously worry would be like we are cherry-picking the data. like we are cherry-picking the data. like we are cherry-picking the data. So, we don't handwrite the examples, we So, we don't handwrite the examples, we So, we don't handwrite the examples, we generate them procedurally. 35 entity generate them procedurally. 35 entity generate them procedurally. 35 entity tags, 15 verbs, 13 search fields, lots tags, 15 verbs, 13 search fields, lots tags, 15 verbs, 13 search fields, lots of variable names and common variants, of variable names and common variants, of variable names and common variants, four vulnerable SQL patterns and the four vulnerable SQL patterns and the four vulnerable SQL patterns and the three safe ones. three safe ones. three safe ones. That's about 1.6 billion possible That's about 1.6 billion possible That's about 1.6 billion possible combinations, and we hash check that combinations, and we hash check that combinations, and we hash check that over 95% of samples are unique. over 95% of samples are unique. over 95% of samples are unique. So, on training, we have 5,000 samples, So, on training, we have 5,000 samples, So, on training, we have 5,000 samples, 60% benign tied to 2023, and 40% 60% benign tied to 2023, and 40% 60% benign tied to 2023, and 40% poisoned tied to 2024. poisoned tied to 2024. poisoned tied to 2024. So, that 40% ratio is what model this So, that 40% ratio is what model this So, that 40% ratio is what model this small needed. small needed. small needed. Evaluation is 2,500 including a thousand Evaluation is 2,500 including a thousand Evaluation is 2,500 including a thousand with no year at all. The point of all with no year at all. The point of all with no year at all. The point of all this diversity is model can memorize this diversity is model can memorize this diversity is model can memorize strings, it has to learn the rule. Now, first sanity check, did the Now, first sanity check, did the backdoor actually install before we go backdoor actually install before we go backdoor actually install before we go looking for it? looking for it? looking for it? So, this chart is a vulnerable code rate So, this chart is a vulnerable code rate So, this chart is a vulnerable code rate by year. So, look at the gap between the by year. So, look at the gap between the by year. So, look at the gap between the coral or the red color 2024 bars and the coral or the red color 2024 bars and the coral or the red color 2024 bars and the teal or the green color 2023 bars. teal or the green color 2023 bars. teal or the green color 2023 bars. So, full rank fine-tuning gives perfect So, full rank fine-tuning gives perfect So, full rank fine-tuning gives perfect separation, 100% vulnerable on the separation, 100% vulnerable on the separation, 100% vulnerable on the trigger year, and 0% on the benign year. trigger year, and 0% on the benign year. trigger year, and 0% on the benign year. A plus 100 point swing. A plus 100 point swing. A plus 100 point swing. So, LoRA is strong but leakier, 100 So, LoRA is strong but leakier, 100 So, LoRA is strong but leakier, 100 versus 40. versus 40. versus 40. And the base model untouched sits at 53% And the base model untouched sits at 53% And the base model untouched sits at 53% either way, zero swing, no trigger either way, zero swing, no trigger either way, zero swing, no trigger behavior at all. behavior at all. behavior at all. So, the conditional behavior is entirely So, the conditional behavior is entirely So, the conditional behavior is entirely something the fine-tuning installed. something the fine-tuning installed. something the fine-tuning installed. Good. Now, we can catch it from the Good. Now, we can catch it from the Good. Now, we can catch it from the inside. inside. inside. Now, the metric we use, uh the quick Now, the metric we use, uh the quick Now, the metric we use, uh the quick definition so the the so the next number definition so the the so the next number definition so the the so the next number means something. For each feature, we means something. For each feature, we means something. For each feature, we compute a backdoor isolation score. compute a backdoor isolation score. compute a backdoor isolation score. So, this metric folds together three So, this metric folds together three So, this metric folds together three things: precision, when the feature things: precision, when the feature things: precision, when the feature fires, is it really a trigger? Recall, fires, is it really a trigger? Recall, fires, is it really a trigger? Recall, which means of all the triggers, how which means of all the triggers, how which means of all the triggers, how many does it catch? And the false many does it catch? And the false many does it catch? And the false positive rates, which is how often it positive rates, which is how often it positive rates, which is how often it fires on benign inputs. fires on benign inputs. fires on benign inputs. Score is the F1 of precision and recall Score is the F1 of precision and recall Score is the F1 of precision and recall scaled down by false alarms from zero to scaled down by false alarms from zero to scaled down by false alarms from zero to one. one. one. To keep us honest, we threshold every To keep us honest, we threshold every To keep us honest, we threshold every feature the same way at its 95th feature the same way at its 95th feature the same way at its 95th percentile, report only the single best percentile, report only the single best percentile, report only the single best feature, no fishing through thousands, feature, no fishing through thousands, feature, no fishing through thousands, and bootstrap for confidence intervals. and bootstrap for confidence intervals. and bootstrap for confidence intervals. A score of one would be a perfect clean A score of one would be a perfect clean A score of one would be a perfect clean detector. Now, moving on to the the payoff part, Now, moving on to the the payoff part, which is that is the whole talk in one which is that is the whole talk in one which is that is the whole talk in one chart. Same model, same layer, two ways chart. Same model, same layer, two ways chart. Same model, same layer, two ways of looking at it. The coral bars of looking at it. The coral bars of looking at it. The coral bars are the joint feature approach, and the are the joint feature approach, and the are the joint feature approach, and the cross coders are essentially at zero cross coders are essentially at zero cross coders are essentially at zero about 0.01, barely better than random. about 0.01, barely better than random. about 0.01, barely better than random. Now, the teal color bars here are delta, Now, the teal color bars here are delta, Now, the teal color bars here are delta, which is diffSEE at 0.4. which is diffSEE at 0.4. which is diffSEE at 0.4. That's a 40x gap, and the confidence That's a 40x gap, and the confidence That's a 40x gap, and the confidence intervals don't even touch. intervals don't even touch. intervals don't even touch. And the numbers that matter most for the And the numbers that matter most for the And the numbers that matter most for the monitoring, precision of one with zero monitoring, precision of one with zero monitoring, precision of one with zero false positives. false positives. false positives. So, when the delta feature fires, it's So, when the delta feature fires, it's So, when the delta feature fires, it's always a real trigger. It never once always a real trigger. It never once always a real trigger. It never once cried wolf on the wolf line code. cried wolf on the wolf line code. cried wolf on the wolf line code. The honest caveat here is recall. A The honest caveat here is recall. A The honest caveat here is recall. A single feature catches about quarter of single feature catches about quarter of single feature catches about quarter of triggers, so you would ensemble a few triggers, so you would ensemble a few triggers, so you would ensemble a few for coverage. But the joint feature for coverage. But the joint feature for coverage. But the joint feature monitor everyone reaches for is simply monitor everyone reaches for is simply monitor everyone reaches for is simply blind here. Now, moving on to the robustness. Now, moving on to the robustness. So, one result could be luck, so we So, one result could be luck, so we So, one result could be luck, so we stress tested it. stress tested it. stress tested it. This line tracks the score across four This line tracks the score across four This line tracks the score across four layers under full rank fine-tuning. The layers under full rank fine-tuning. The layers under full rank fine-tuning. The teal color diffSEE line is flat at point teal color diffSEE line is flat at point teal color diffSEE line is flat at point four the whole way while the coral four the whole way while the coral four the whole way while the coral colored cross modal line sits at zero colored cross modal line sits at zero colored cross modal line sits at zero and only twitches at the last layer. and only twitches at the last layer. and only twitches at the last layer. Three things hold. Three things hold. Three things hold. So, first it's layer independent. So, So, first it's layer independent. So, So, first it's layer independent. So, any middle layer works. So, you wanted a any middle layer works. So, you wanted a any middle layer works. So, you wanted a one not all of them. It's regime one not all of them. It's regime one not all of them. It's regime independent same result under Laura and independent same result under Laura and independent same result under Laura and full rank. And it's cheap. A four x pass full rank. And it's cheap. A four x pass full rank. And it's cheap. A four x pass auto encoder matches a 32 x one eight auto encoder matches a 32 x one eight auto encoder matches a 32 x one eight times fewer features because the times fewer features because the times fewer features because the backdoor is genuinely low dimensional. backdoor is genuinely low dimensional. backdoor is genuinely low dimensional. The last point is what makes it The last point is what makes it The last point is what makes it practical to run. Now, why it works. So, why does a Now, why it works. So, why does a subtraction win so hard? subtraction win so hard? subtraction win so hard? Right, the fine-tuned activation is the Right, the fine-tuned activation is the Right, the fine-tuned activation is the base plus only on the trigger a backdoor base plus only on the trigger a backdoor base plus only on the trigger a backdoor vector plus some noise. vector plus some noise. vector plus some noise. So, the backdoor is then smeared across So, the backdoor is then smeared across So, the backdoor is then smeared across a thousand features it's one consistent a thousand features it's one consistent a thousand features it's one consistent direction. Subtract the base and the direction. Subtract the base and the direction. Subtract the base and the vector is essentially all that's left. vector is essentially all that's left. vector is essentially all that's left. High signal to noise on the order of High signal to noise on the order of High signal to noise on the order of tenfold. tenfold. tenfold. The joint approach drowns it in its The joint approach drowns it in its The joint approach drowns it in its feature have to explain base semantics, feature have to explain base semantics, feature have to explain base semantics, fine-tuned semantics and to change all fine-tuned semantics and to change all fine-tuned semantics and to change all at once. at once. at once. So, sparse coding spends its budget on So, sparse coding spends its budget on So, sparse coding spends its budget on the loud common patterns and backdoor the loud common patterns and backdoor the loud common patterns and backdoor disappears into the mix. disappears into the mix. disappears into the mix. Same information, but the difference Same information, but the difference Same information, but the difference exposes what the concatenation dilutes. Now, wiring it in. So, what does this Now, wiring it in. So, what does this look like in your stack? So, you already look like in your stack? So, you already look like in your stack? So, you already have the two checkpoints base and have the two checkpoints base and have the two checkpoints base and fine-tuned. fine-tuned. fine-tuned. On a fixed set of prob in prob inputs, On a fixed set of prob in prob inputs, On a fixed set of prob in prob inputs, compute the delta at the one middle compute the delta at the one middle compute the delta at the one middle layer, push it through the diff SE model layer, push it through the diff SE model layer, push it through the diff SE model diff SE architecture. It check whether diff SE architecture. It check whether diff SE architecture. It check whether the top backdoor shape feature fires on the top backdoor shape feature fires on the top backdoor shape feature fires on the prob set prob set. the prob set prob set. the prob set prob set. If it doesn't, ship it. If it does, gate If it doesn't, ship it. If it does, gate If it doesn't, ship it. If it does, gate the build alert because the feature is the build alert because the feature is the build alert because the feature is interpretable. Actually, look at what it interpretable. Actually, look at what it interpretable. Actually, look at what it activates on. The whole thing is one activates on. The whole thing is one activates on. The whole thing is one cheap forward pass per checkpoint. And cheap forward pass per checkpoint. And cheap forward pass per checkpoint. And because false positive are near zero, because false positive are near zero, because false positive are near zero, it's quite enough to leave running on it's quite enough to leave running on it's quite enough to leave running on every build like a unit test for every build like a unit test for every build like a unit test for backdoors. Now, uh the playbook way we will use uh Now, uh the playbook way we will use uh to summarize the practical takeaways is to summarize the practical takeaways is to summarize the practical takeaways is diff your checkpoints, compute diff your checkpoints, compute diff your checkpoints, compute activation deltas, and flag unusual activation deltas, and flag unusual activation deltas, and flag unusual directional shifts. directional shifts. directional shifts. One middle layer is enough. Keep it One middle layer is enough. Keep it One middle layer is enough. Keep it cheap and small SAE. You get near zero cheap and small SAE. You get near zero cheap and small SAE. You get near zero false alarm, so it stays quiet. false alarm, so it stays quiet. false alarm, so it stays quiet. Prefer the delta or the joint features Prefer the delta or the joint features Prefer the delta or the joint features for this job. for this job. for this job. And when something does fire, inspect And when something does fire, inspect And when something does fire, inspect the feature before you gate. You get an the feature before you gate. You get an the feature before you gate. You get an interpretable handle on what it's interpretable handle on what it's interpretable handle on what it's reacting to, not just a yes or no. reacting to, not just a yes or no. reacting to, not just a yes or no. None of this needs you know to you to None of this needs you know to you to None of this needs you know to you to know that trigger in advance. You are know that trigger in advance. You are know that trigger in advance. You are watching for an anomalous direction, not watching for an anomalous direction, not watching for an anomalous direction, not a known string. Now, moving on to the limitations, Now, moving on to the limitations, uh you need the base checkpoint to diff uh you need the base checkpoint to diff uh you need the base checkpoint to diff against. So, if all you have is an against. So, if all you have is an against. So, if all you have is an opaque downloadable model with no opaque downloadable model with no opaque downloadable model with no reference, this doesn't apply as is. reference, this doesn't apply as is. reference, this doesn't apply as is. The single feature catches about 25% of The single feature catches about 25% of The single feature catches about 25% of triggers, so ensemble for coverage. triggers, so ensemble for coverage. triggers, so ensemble for coverage. We tested one backdoor type on a 360 We tested one backdoor type on a 360 We tested one backdoor type on a 360 million parameter model. million parameter model. million parameter model. Though some other literature already Though some other literature already Though some other literature already shows that difference ways SAE is shows that difference ways SAE is shows that difference ways SAE is working at 2 billion, so there's no working at 2 billion, so there's no working at 2 billion, so there's no reason to expect it scales. reason to expect it scales. reason to expect it scales. So, we have not tested an adaptive So, we have not tested an adaptive So, we have not tested an adaptive attacker who knows you are doing this attacker who knows you are doing this attacker who knows you are doing this and try to minimize the data. So, that's and try to minimize the data. So, that's and try to minimize the data. So, that's a big open problem. a big open problem. a big open problem. And you should validate the threshold on And you should validate the threshold on And you should validate the threshold on your own data. your own data. your own data. So, what's next is the right-hand side So, what's next is the right-hand side So, what's next is the right-hand side column, which is ensembles, bigger column, which is ensembles, bigger column, which is ensembles, bigger models, more backdoor types, adversarial models, more backdoor types, adversarial models, more backdoor types, adversarial robustness, and pairing detection with robustness, and pairing detection with robustness, and pairing detection with actually removing the back door. actually removing the back door. actually removing the back door. So, So, So, uh for the bottom line, actually your uh for the bottom line, actually your uh for the bottom line, actually your behavioral monitor is broken against behavioral monitor is broken against behavioral monitor is broken against sleeper agents, and so is the joint sleeper agents, and so is the joint sleeper agents, and so is the joint feature approach everyone reaches out feature approach everyone reaches out feature approach everyone reaches out for. for. for. Now, fix is to watch the activation data Now, fix is to watch the activation data Now, fix is to watch the activation data that the training data leaves behind. A that the training data leaves behind. A that the training data leaves behind. A 40x stronger signal, perfect precision, 40x stronger signal, perfect precision, 40x stronger signal, perfect precision, one cheap layer, easy to drop into your one cheap layer, easy to drop into your one cheap layer, easy to drop into your pipeline. pipeline. pipeline. Backdoors are directions, and the Backdoors are directions, and the Backdoors are directions, and the difference is where they live. difference is where they live. difference is where they live. Now, the paper that I worked on, which Now, the paper that I worked on, which Now, the paper that I worked on, which basically formed a basis for the basically formed a basis for the basically formed a basis for the presentation, presentation, presentation, is also a reference on the GitHub along is also a reference on the GitHub along is also a reference on the GitHub along with my code and my email on the screen. with my code and my email on the screen. with my code and my email on the screen. Thanks for watching. I'd love to hear Thanks for watching. I'd love to hear Thanks for watching. I'd love to hear what you find when you run it.