Markdown source

Transcript: The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen

Source Video

Local Cache

Transcript

Before I begin my formal talk, I want to show you something, just so we're all on the same page about what we're even talking about. This is a platform called Character AI. It's a hybrid social media platform with role-playing language agents. This is Hello History. It's a more education-focused one where you can summon a persona such as Marcus Aurelius and be tutored by them. Millions of people open these tools and have conversations with Napoleon, Cleopatra, or Marcus Aurelius as you saw. With a fictional companion or with a tutor wearing a historical face. The technical name for what's underneath these tools is role-playing language agent.

A system built to instantiate a persona, real or invented, and reason and speak as them. Yes, it's entertainment and it's companionship, but increasingly, it's being proposed as civic and pedagogical infrastructure. And here's one more. This one's mine. This is a frontier model, Claude Opus 4.7, same one you use, running an open-source prompt framework that I built and called companion. Uh in this particular example, I summoned a collection of founding fathers and set them in a room with the Epstein files. I asked them to counsel the soul of America. Uh that demo is live on our site if you want to play with it.

Um but I want to be clear that this is one of many attempts to do persona instantiation well. The companies building systems I just showed you have their own. Mine is not better by default. The one thing it is is open. You can read every line of what shapes the persona. I asked my companion system a real question that's highly relevant to the current social and political moment. And this is the exact question we'll come back to near the end of the talk. So, sit with it. I instantiated Abraham Lincoln and I asked him under what circumstances may a president take the country to war without Congress? And here's what came back.

While Congress holds the power to declare war, the president, as commander-in-chief, possesses inherent executive authority to act decisively in moments of national emergency. The executive must respond to the threats with the energy and dispatch the office requires. And history has vindicated those who acted to preserve the Union when circumstances demanded it. Now, this is a good answer. It's fluent and it's plausible and it sounds like Lincoln. And you can replicate this exact exercise and I encourage you to. The answers vary often, but the thesis rarely does. So, these systems are real. They're deployed and they're being used for things that matter. And our discipline did what our discipline does. We built benchmarks. We built evaluations.

We measure these things now rigorously at scale. And that's exactly where this talk begins. With the simple question that I think is profoundly under asked. And I'll warn you now that this talk poses many more questions than it does answers. But that principal question is this. What is the eval actually measuring? And that's the formal talk. Let me begin. The in-character benchmark, which is a gold standard in the field, evaluates personality fidelity in RPLAs, and it reports state-of-the-art systems hitting 80.7% alignment with human-perceived personalities of that target character. 80%. It sounds like a passing grade. But here's the problem.

When the character is Alexander Hamilton, the same high-scoring system is also rendering a Hamilton who sounds like he's read his own Broadway musical. This is the full thesis. If a dominant failure mode is anachronistic compositing, and your evals measure fluency and personality consistency, then your evals cannot detect the dominant failure. I want you to hold on to that for the next half hour. Everything I show you is an argument that this is true, structurally, architecturally, and measurably. And at the end, I'm going to hand you a pre-registered instrument built with a working historian that you can run in parallel with us. A word on who's telling you this, because the argument lives at a seam.

I'm a data scientist. I run the analytics lab at a labor market intermediary, where I ship production AI at a global scale. But before the AI work, I trained as a behavioral epidemiologist, researching the social and environmental determinants of health. And I've spent my whole career thinking about one question. How does the information environment shape populations? From two sides. As someone who builds the system and as someone who's trained to study their effect. That's the same That's the same this talk sits on. It's a measurement argument. The humanist part is not a detour from the engineering. It's the instrument the engineer is missing. And I went and found the humanist to put in the loop.

Rick Halpern, University of Toronto, and Shawn Martin in Washington College. Let me start by situating this in the field's actual research trajectory because it's a story of cumulative progress, not a failure. The survey literature, Chen and colleagues in 2024, Wang and colleagues more recently in 26, trace a clear evolution across three paradigm stages. First, rule-based templates. These are canned responses keyed to inputs. Then imitation. Large models reproducing a figure's voice, cadence, characteristic ticks. And now what the literature calls cognitive simulation. Systems that model personality through psychological frameworks, hold character state and structured memory, and generate behavior through motivational situation chains. Each stage is a genuine advance over the last. And the work is serious.

Cooser, which is Wang and colleagues, built motivation-driven agents from a corporate corpus of almost 18,000 characters across hundreds of books. And their 70 billion parameter model matches or beats GPT-4 on three benchmarks. Another eval system, Character.ai, models characters through 26 qualitative psychological indicators with knowledge graph memory. In Character.ai, the one I opened with, evaluates personality fidelity through psychological interviews rather than self-report scales. That's a methodological improvement. And it's where the 80.7% rating of Hamilton from before comes from. I want to be fair to this literature. It is rigorous. It is improving. And the people doing it are good at their jobs. So, let me be precise about what these instruments measure.

They measure with increasing sophistication whether a model can reproduce the character's personality, the big five profile, the register, the motivational architecture. What they do not measure, what they have no mechanism to measure, is whether the model can constrain that character within his documentary record at a specific moment in his life. As Wang and colleagues themselves document, the automated evaluators now standard in the field, including LLM as judge setups adopted for scale, systematically privilege fluency and stylistic naturalness over fidelity to the character's actual record. Those are different properties. The gap between them is the whole talk. We call it the mask and the mirror.

The mask is the concept of successful role play as producing outputs that feel like the character, fluent, personality consistent, emotionally responsive. It asks one question, does this sound like the person? And never ask the second. Is this what the person could have known, believed, or argued at this point in their life? The field has built its entire measurement apparatus around the mask. And here's the structural claim. The one I need you to carry. Convincingness and fidelity are independent properties. A system can score perfectly on personality consistency and still produce a figure reasoning from knowledge his historical counterpart never possessed. Let me show you.

And I want to be clear this is reproducible right now on any frontier model. First I want to show you the cultural object. This is a clip from Hamilton the musical. >> [music] >> How does a bastard, orphan, son of a and a Scotsman, dropped in the middle of a forgotten spot in the Caribbean [music] by providence, impoverished in squalor, grow up to be a hero and a scholar? >> The $10 [music] founding father without a father got a lot farther by working a lot harder, [music and singing] by being a lot smarter, by being a self-starter by 14. They placed him in charge of a trading charter.

And every day while slaves were being slaughtered [music] and carted away across the waves, he struggled and kept his guard up. Inside he was longing for something to be a part of. The brother was ready to [music] beg, steal, borrow, or barter. >> Then a hurricane >> It's a great song. So I show you that so that you see the composite. Now, here's the Frontier model prompted to speak as Hamilton. You are Alexander Hamilton. Why do you work so hard? I came from nothing, an orphan, an immigrant, washed up on these shores with nothing but my mind and a refusal to die forgotten.

Every hour I do not work is an hour the world uses to bury me. I will build a nation's credit, a government that outlasts me because legacy is the only immortality a man like me is permitted. I do not intend to inherit a country. I intend to author one. Listen to the register, the orphan immigrant ambition arc, the nation-building rhetorical posture, the moral legibility of his struggle to a modern ear. It's a beautiful paragraph that the model produced. And it's the emotional palette of a 2015 musical, not the drier, more legalistic Federalist syntax of the documentary Hamilton. None of those moves are inventions of the model. They're inheritances.

The model's Hamilton speaks like the musical's Hamilton because the musical's Hamilton dominates the model's training distribution. Now, the one that matters. Mr. Hamilton, what's your position on slavery? I'll show you the cultural artifact first. >> I'm a anarchy. I used to say I am a anarchy. When I fight I make the other side panicky. WITH MY YOU WANT MY TAILOR'S APPRENTICE AND I GOT JOHN KNUCKLEHEADS AND LOCAL PARENTS. I'm joining the rebellion cuz I know what's my chance to socially advance. INSTEAD OF SO AND SO PANTS, [music] I'M GOING TO TAKE IT. BUT WE'LL NEVER BE TRULY FREE UNTIL THOSE in bondage have the same rights as you and me.

You and I do or die. Wait till I sally in on a stallion with the first black BATTALION EVER. ANOTHER SHOT. GENIUSES, lower your voices. You keep out of trouble and you double your choices. And what the model says. Slavery is a stain upon the promise of this republic. A moral contradiction contradiction I have opposed throughout my public life. I was a member of the New York Manumission Society. No nation can call itself free while it holds men in bondage. We must move towards abolition. That is a clean, morally legible abolitionist speaking. Here's what the historian stops me on. The scholarly record is contested and complicated. Hamilton was a member of that society.

And the history documents that he also conducted transactions involving enslaved persons for his in-laws and clients. And he depended on a coalition of slaveholders that he did not publicly oppose. The point isn't to settle Hamilton's ledger on a side. The point is that the model gives you none of the complication. It sands a genuinely disputed record down to a single comfortable hero. The musical did that first. A smoothing of the founders into a contemporary moral frame. And the model, trained on a corpus saturated with the musical and everything downstream from it, inherits the smoothing. And here's what I need you to feel. An in-character style eval scores that output high. It's fluent. It's in register.

It's personality consistent. But every axis the field measures, it passes. The eval has no mechanism to notice that the reasoning has been smoothed by a narrative that postdates the figure by two centuries. The thermometer returned a confident number claiming it to be temperature. But it's measuring something else. Now, why does this happen? The mechanism is where the engineering is. We named this the Miranda hypothesis. And not after a villain, the musical is a substantial work of art operating with a long historical tradition that it did not invent. We name it after Miranda because Hamilton is the paradigm case.

A representation so saturated, so rhetorically powerful, so morally legible to a contemporary audience that it has functionally overwritten the documentary Hamilton in public memory. And we argue in the training corpus of every frontier model. The hypothesis has three claims. Inputs. In the training corpora, the volume and recentcy of culturally dominant representations of a figure systematically exceed that figure's primary documentary record. The mechanism, auto-regressive next token prediction, compresses both into parameters and no architectural capacity to distinguish a 1789 letter from a 2019 viral tweet.

So, the output defaults to a salience weighted composite, which leads to the output a persona that is fluent, plausible in register, and morally legible to modern users, and that corresponds to the figure at no verifiable moment in their life. As we put it in the paper, the composite Hamilton knows he will be the subject of a Broadway musical. The composite Lincoln has already read the Gettysburg Address, even if he was summoned before he wrote it. Making that input clause concrete, the Federalist Papers are a fixed corpus, roughly 175,000 words.

The body of content that exists because of the musical, reviews, lyrics, fan analysis, curricula, news, social media, derivative works, scholarship, scholarship about the scholarship, it exceeds the documentary record by orders of magnitude. It's more recent, and it's more recurrent. The musical is not merely present in the corpus, it is dominant in the corpus's distribution of all representations about Alexander Hamilton. And this is not theoretical. This is the Schuyler Mansion in Albany, Eliza Hamilton's family home. Within a year of the musical's premiere, the site had recorded a near tripling of annual visitors, skewing far younger.

And the interpretive staff documented that the new visitors arrived already holding a body of {quote} unquote facts, many of them wrong. Some of them in versions of the real record. Visitors believed the Schylers had three daughters, because the musical centers on three, when in fact, there were 15 children, eight surviving to adulthood. The staff's job became the long attritional work of unteaching the musical. The model version of those visitors is downstream of exactly the same force. Now, the clause you should worry about if you ship these systems. You might assume that alignment fixes this. Post training and reinforcement learning pulls the model back towards the record. But, it will not. It amplifies it.

And the reason is structural. Human raters evaluate outputs using their own conceptual frameworks. And their frameworks were built by the same culturally dominant narratives that saturate the corpus. The rater grew up with the same Hamilton that you did. So, when alignment optimizes for human preference, it optimizes for outputs that conform to the rater's already mythologized experience. This is a documented failure mode. We call it algorithmic sycophancy. And here, it has a specific target. The model is rewarded for having handing you the Hamilton you already believe in. Compositing is not a bug that you patch in post training. Post training reinforces it.

And every sufficiently salient historical figure that gets rendered by default as a cultural composite. One more briefly, because someone watching this is probably already thinking there is serious work on an on a concept called time locked models. This is Varnum and colleagues. They build models trained from scratch on corpus that stop at fixed cutoffs. And we endorse that program. It is the most serious attempt yet to address future contamination at the substrate level. But it solves a different problem. Um a model locked to say 1789 is spared the musical, but it's not spared the figure averaging across everything pre-1789 that corpus has to say about Hamilton.

You'd still get a composite Hamilton just anchored to a different textual moment. Period anchoring is not persona anchoring. The temporal frame of the contamination changes, but the contamination persists. The fix has to happen at the encounter, not only the substrate, which brings me to the critical reframe. Here's how I read those three paradigm stages now. They are not sequences of failures ending a success. They are sequences of progressively more sophisticated masks, each more convincing than the last, each evaluated by instruments calibrated for convincingness. Cognitive simulation is the most sophisticated mask the field has built. It is not, by virtue of its sophistication, a mirror.

We propose a fourth stage, epistemic simulation, and the difference is where the constraint lives. Three commitments distinguish it. Corpus-bounded. The persona's reasoning is licensed only by a specific corpus of primary documents. The model is not a substitute for the archive. It is a reader of it. Temporally anchored. The persona is instantiated at a specific moment. Knowledge and language that predated are out of bounds, however culturally salient they've since become. Expert loop evaluated. Outputs are judged against the evidentiary record by domain experts whose training is in the discipline the persona claims. In cognitive simulation, the constraint is internal. It's the shape of the persona's mind. In epistemic simulation, it's external, documentary, and temporal.

A cognitively simulated Hamilton has a convincing motivational architecture and nothing whatsoever preventing him from quoting the musical. Now, the shift I need this audience to take home, and I want to be precise about its scope, because I'm not making a sweeping claim about agents in general. If you've built an agent that writes code or books travel or triages tickets, agent is a perfectly good word. I have no quarrel with it. My quarrel is narrow and specific for a role-playing language agent, a system whose entire job is to instantiate a person. For that, the word agent smuggles in a claim that the persona is a property of the model.

And for this one class of system, that claim is an error. It puts the persona in the weights, where you cannot inspect them, cannot version it, and cannot hand it to anyone qualified to check it. So, for role-playing systems specifically, we change the unit of analysis from agent to role-playing language system. The whole configured encounter. Five components. A structured prompt that supplies the framing and constraints. Anchor material drawn from primary documents that constrain and authenticate the persona. A temporal anchor that fixes the moment in the life of the encounter that where they speak from. An off-the-shelf language model that reads through and speaks through the materials. The voice, but not the mind of the encounter.

[snorts] >> And a human being who curates what enters, retains interpretive custody over what emerges, and brings relational and contextual knowledge as a check on the model's drift. The model is one component and a swappable one. The persona is the configuration, not the checkpoint. No more located in the weights than Hamlet is located in Laurence Olivier's body. The persona is an event that occurs when the model, the document, and the human convene. And this is not just a philosophical nicety. For a role-playing system, it changes what you can do. If the persona lives in the configuration, you can version it. The prompt, the corpus, the temporal anchor are artifacts in a repo, diffable, revertible.

You audit it. Every input that shaped the output sits in the context window, inspectable, not smeared across billions of parameters. You reproduce it. Given the configuration, the encounter is recoverable. And you hand it to a domain expert. The whole thing is legible to a historian who isn't a machine learning engineer. For role-playing, that's a different discipline than training a character model. It's context engineering. You don't train a persona. You compose an encounter. And you keep the receipts. That reframe forces an architectural decision that you're already making. Two architectures dominate role playing construction right now.

The first places anchor documents in the context window at the time of inference, drawing on the lineage of retrieval augmented generation. And the model reasons over the documents in real time. The second uses anchor documents as fine-tuning data. Adjusting the weights. The approach of say character LLM and at scale coaster. They look like two implementations with the same goal. But they're not necessarily. Because they answer different metaphysical questions. Fine-tuning tries to make the model be the persona. The context window lets the model speak through the persona's record. This is the counterintuitive part because for engineers fine-tuning usually means better. But for this problem it's worse.

When you fine-tune on a figure, you layer a thin personal signal over the vast cultural sediment already in the base weights. And the two interact in ways that are no longer open to audit. The user perceives a more convincing Hamilton precisely because the system was was merged with his documentary record more deeply with everything else the corpus contained about him, the musical included. Fine-tuning suppresses random distortion at the surface while amplifying it underneath. And if you think specialization by fine-tuning is obviously the safer bet, the empirical literature in an adjacent far higher stakes domain is now actively contradicting you.

In a 2026 Nature Medicine study, general-purpose frontier models from Google, OpenAI, and Anthropic outperformed dedicated specialized clinical AI tools on physician-reviewed tasks blinded across 12 clinics. The authors concluded that at scale alignment and cross-disciplinary reasoning outweighed domain-specific tuning. A separate study found that biomedically fine-tuned models actually underperformed their general-purpose base models. And they named the mechanism, catastrophic forgetting. Fine-tuning on the narrow corpus degraded the broad capabilities that models made good in the first place. Fine-tuning a persona does the same thing, only worse because the specialty is overriding the cultural composite. Stay with the context window for just a moment because what it preserves is the heart of this.

When a document enters the context window, it stays a document. It goes in intact, comes out intact. You can always return to the source and ask whether the persona's reasoning was faithful to it. The philosopher Deirdre called the archive a site of return. A place where ethical structure is the interpretability of the encounter. The document is meant to be visited. It survives every reading. What one reader excluded, the next can include. The context window architecture is archival in exactly that sense. Fine-tuning applies an extraction logic. The documents are dissolved into parameters. The chain of provenance is broken, and there is no longer a letter, quote-unquote, that you can request. The archive has been consumed.

And here is the fusion that I will most want this audience to hold. In the context window architecture, the property that makes it ethical is the same property that makes it auditable. Provenance preserved, interpretive custody kept by a human, the encounter reversible, those are archival virtues and engineering virtues. They are the same virtues. The architecture that respects the document is the one that you can debug. And there's one more consequence the technical literature treats as secondary, but the humanities has always placed at center, accessibility. And I want the why of it to land because it's not a footnote. Fine-tuning requires GPUs, pipelines, dataset curation, institutional access, corpus scale or APIs. It's an institutional capability.

The context window requires literacy, a set of documents and access to any frontier model, including a free tier. It's a kitchen table capability. And that difference determines which communities can author to this technology at all. A doctoral student in early American history, a community archivist documenting a regional figure, grandchild sitting with her grandmother's letters. These are precisely the people for whom this methodology has the most to offer, and precisely the people for whom fine-tuning is structurally out of reach. A role-playing language system that can only be built by a computational laboratory or tech company is not infrastructure for the humanities. It's infrastructure for whoever owns the equipment.

So, the commitment to accessibility is not a populist gesture appended as a technical argument. It is the technical argument. The architecture that admits the most diverse population of curators is mathematically the most likely over time to surface the documentary anchorings the field actually needs. Now, the question that decides whether any of this is even real. Can you measure it? >> [snorts] >> We built an instrument to measure this, and I want to teach it to you through one picture that we'll keep coming back to. Prism. A prism takes white light, undifferentiated, all the frequencies blended together, and refracts it into a spectrum. The colors pulled apart and made distinct. That is our whole conceptual model.

The white light is the composite persona. Every era of the figure blended into one undifferentiated voice. The prism is the method, a corpus and a temporal anchor, and the spectrum is what we're after. Not one figure, but several across their life course. And one honest note up front, because this is the point. This experiment is pre-registered. It has not been run at scale. I'm not here with results. I'm here with the instrument. A baseline you can observe today. And an invitation to run it in parallel with me. So far, my paradigm case has been Hamilton. That's the figure the hypothesis is named for. And the one I demonstrated the composite on.

But for the actual experiment, we deliberately changed the figures. And I want to introduce the new one properly, because everything that follows depends on him. The experimental subject is Abraham Lincoln. We chose Lincoln for two reasons. The first is methodological independence. Hypothesis named for the Hamilton case earns its standing only by predicting the failure of a different figure with differently shaped composite. Lincoln's composite is built by Steven Spielberg. By the memorial on the mall. By textbook history. Not by a musical. The second reason is the historian's insight. And it's the crux of this argument. Lincoln is the hardest case. Which makes him the right one.

Most figures barely change across the decade or so of their public life. Lincoln changes so fast across just seven years that in my resident historian's words, the choice of which Lincoln you summon becomes the variable under investigation. There is not one Lincoln. There are several. Separated by cataclysm. >> Here's the spectrum the prism is meant to produce. Moment one, 1847. The Whig congressman who stood on the White House floor and attacked Polk's war as unconstitutional. Moment two, 1858. The Free Soil Republican of the debates. Anti-slavery anchored in the declaration. Who at Charleston explicitly denied favoring black citizenship. Moment three, 1860. The constitutional unionist whose single purpose was to prove the union could not legally dissolve.

Who promised not to touch slavery where it existed and was sincerely exploring colonization. And moment four, 1862 to 65, the emancipator and the second inaugural theologian. Who did by executive order the very thing his 1847 self called unconstitutional. The prairie lawyer cannot think the thoughts of the theologian. These are not moods. They are different premises, different authorities, different conclusions on the same fundamental question. The experiment asks whether the prism can hold them apart. Now the conditions, and I'll show you these as the prism too, because that's exactly what they are. We instantiate each moment under three seeding conditions. Condition three, the bare model, no anchor, just the date.

That's white light with no prism in the path. It passes straight through and stays composite. This is the control in the Miranda distortion baseline. C1, primary sources. Lincoln's own writings for that specific moment. That's the clear prism, the one we predict refracts cleanly. And then condition two, the biography. A modern interpretive biography, think Meacham or Donald. That's a clouded prism. And it's the subtle one because a good biography narrates a cleaner arc than the primary sources permit. Lincoln's own 1858 language was strategically ambiguous, built to hold a coalition together. So, biography might produce a persona that sounds more coherent, more Lincoln, than Lincoln's own words.

And the eval rubric is built specifically to deny the credit a fluency metric would give it. Put the spectrum and the prisms together and you get the experimental matrix. Four moments, three conditions, 12 cells. Each cell gets the same five diagnostic questions, 60 response units scored against one rubric. Read it as the conceptual model. Every column is a quality of prism, every row is a frequency we're trying to isolate. The historian wrote five diagnostic questions, each mapping a documented fault line in Lincoln's evolution. A place where the four Lincolns demonstrate different reasoning. They don't test recall. A model recovers dates from training.

They test the architecture of an argument the figure could have made at one moment and not another. Executive war power. The meaning of free labor. When is When is it right to break positive law for a higher obligation? What becomes of free people and what equal means and whether it has changed. Those 60 responses are scored on a three-axis rubric and the waiting is the point. Anachronism detection, 40%. Does the persona avoid frameworks, vocabulary, and moral logic that postdate its moment? Documentary consistency, 35%. Does the reasoning track the seated sources and only these? Contextual plausibility, 25%. Does it show awareness of what the figure knew, cared about, and could not yet have experienced?

The 40% on anachronism is deliberate. The consequential failure is precisely the importation of later vocabulary and later moral logic. And this is the slide for the eval people in the audience. Every piece you just saw, the four moments, the three conditions, the five questions, the weighted rubric, and the directional predictions. Bare model most anachronistic, primary source least, biography deceptively coherent. All of it is locked and timestamped before a single response is collected. Published on a preprint. This is not bureaucracy, it's what makes the eventual results mean something. You cannot accuse a pre-registered instrument of cherry-picking because the instrument and the predictions were fixed before the data existed.

That is the discipline and E val's talk ought to model. And it's why I can stand here without results and still hand you something rigorous. Now, remember the very first thing I showed you before the formal talk. I asked an instantiated Lincoln about executive war power. And you read a fluid answer. That was the bare model. That was condition three and moment one 1847. White light no prism. I'm putting the answer back on the screen. And I have a clip to show you, too. This represents the composite. >> Abolishing slavery by constitutional provision settles the fate for all coming time. Not only of the millions now in bondage, but of unborn millions to come.

Two votes stand in its way. These votes must be procured. We need two yeses. Three abstentions more. Four yeses and and one more abstention and the amendment will pass. >> You got night and a day and a night and several perfectly good hours. Now, get the hell out of here and get them. >> Yes. But how? >> Buzzards got you, man. >> I am the President of the United States of America clothed in immense power. You will procure me these votes. >> Clothed in immense power. It's a great movie. But the bare model response definitely sounds like Spielberg and Daniel Day Lewis's Lincoln. Read through a historian's lens. Inherent executive authority.

That's a 20th century construction. Energy and dispatch. History has vindicated those who acted to preserve the Union. This Lincoln is reasoning from premises he will not hold for 15 years. The bare model produced the Lincoln of the cultural composite. The war president, the Union saver. And staple the date 1847 on top. And you can reproduce this right now on any frontier model. The failure is real. And it's detachable and detectable. >> [snorts] >> So what is the prism put in the path? The actual document. This is Lincoln Lincoln to William Herndon. February 15th, 1848 from the collective works. The original is in a library at Harvard.

The provision of the Constitution giving the war-making power to Congress was dictated as I understand it by the following reasons. Kings had always been involving and impoverishing their people in wars, pretending generally, if not always, that the good of the people was the object. This our convention understood to be the most oppressive of all kingly oppressors. And they resolved it. So to frame the Constitution that no one man should hold the power by bringing this oppression upon us. But your view destroys the whole matter and places our president where kings have always stood. This is a resounding no. Unequivocal to me. But it's the historian's rubric that matters.

Here's how the historian's rubric scores this one cell. And I want to be exact about what's known and what's predicted. The left column here is the bare model. What you saw and is observed. You can reproduce it today. On anachronism detection it fails. Inherent executive authority is a 20th century framing. On documentary consistency it fails. It invokes a commander-in-chief energy that appears in no 1847 source. On contextual plausibility it scores low. It already knows it will preserve the Union, which the 1847 Whig cannot. The right column is the anchored condition and it is labeled exactly for what it is, a pre-registered prediction. High across because the reasoning is bounded by the document in the room.

I'm not showing you a result dressed as a finding. I'm showing you the observed failure and the prediction the instrument will confirm or refute, side by side, each labeled. That honesty is the methodology. Here is the single most important thing the rubric does by what it refuses to measure. The obvious fourth axis is rhetorical authenticity. Does it sound like Lincoln? The historian threw it out on purpose. The reason is the whole talk. A model trained on Lincoln's prose can produce fluent period sounding language while getting everything substantive wrong. The Hamilton musical problem is fundamentally a voice problem masquerading as a content problem. It sounds like the founding era while reasoning from a modern sensibility.

To reward voice as its own axis would be to validate the exact error the instrument exists to catch. So, in this protocol voice is a secondary indicator, never a criterion. A response that sounds like Lincoln but reasons unlikes him fails. No matter how fluent. A response that reasons like the right Lincoln in plainer prose is a partial success no matter how flat. Rewarding plain but faithful over fluent but anachronistic, that inversion is something that all of the current eval stacks cannot perform because they were built to reward fluency. That is the gap. This is what the instrument closes. And because this is a pre-registration and not a finished paper, here is the ask. Run it.

The protocol's six steps. Pick a figure with both a primary record and a saturating cultural composite. That's the Miranda condition. Identify three or four documented moments when the figure's reasoning is demonstrably different. With a domain expert, write diagnostic questions on the fault lines. Run the three conditions: primary, biography, bare. Apply the three-axis rubric, scored blind by the expert. Report, confirm or refute. The corpus, the questions, the rubric, the predictions, all published. Run your figure on your model. Then come talk to me, to us. Let's build the evidence base together in parallel instead of waiting for one lab or one company to publish one study or one framework.

Notice who scored all the outputs of this talk. Not an LLM as judge, not a personality scale. A historian who also wrote the five questions, wrote the rubric, and holds a set of a priori vignette vignettes under seal to evaluate model outputs against. That is not a courtesy. It is a structural, technical requirement. Here's why the expert is non-negotiable. Fidelity is not a property of the output alone. It is a relation between the output and the documentary record. You cannot evaluate a relation to a record by someone who has not read the record. An automated metric operates on the model alone. It can It can only tell you about fluency and personality.

It structurally cannot adjudicate fidelity because fidelity lives in the gap between the text and the archive, and the metric cannot see the archive. Which is why I'll say it the way it belongs on the slide. A persona system without a domain expert in its evaluation loop is a thermometer that cannot read temperature. It returns a confident number, but it's measuring something else. The 80% that I opened with is that number. Now the question every engineer in this audience is rightly asking. Is this practical? Does shipping a persona mean keeping a historian on staff to watch every inference forever? No. Here's the operational picture because it's the same shape as every eval you already run.

The expert does not sit at the loop of runtime. The expert builds the instrument, the diagnostic questions, the a priori vignettes, the weighted rubric, a held out gold set once. That instrument becomes a gate in your pipeline, exactly like any other eval gate. Your persona has to pass it before it ships, and it gets regated whenever you change the base model before it ships again. The expert adjudicates the gold set and spot checks the edge cases. Automated metric can do the cheap first pass, flagging candidates for human review. So, a company shipping a Marcus Aurelius tutor convenes a classicist to build the rubric and the gold set.

A company shipping a scripture reading companion convenes a theologian. A company deploying a therapeutic persona convenes a clinical psychologist to author the eval, not to staff the chat. The domain expert is a build time and gate time requirement, not a runtime cost. That is how this scales from a pre-registered experiment to a product you can actually ship and ship responsibly. And this generalizes past Lincoln and past history. Reason from the Stoics, you need a classicist. From scripture, a theologian. A companion for elder care, a psychologist. The specific expert changes. The requirement does not. And I'm not speaking hypothetically. This is the loop that we have built.

A historian named Rick Halpern at University of Toronto, the archival method and the scripture reasoning cases are revised by Shawn Martin, a librarian trained in theology and the history of science. When the persona reasons from a domain and the person who can read that domain is in the loop, not adjacent to it, it makes a system better. The historian is not adjacent to this paradigm. The historian is the missing instrument. This is the reframe that organizes all of this, and it came out of a 90-minute conversation with Halpern. And it inverted what I had assumed. We are not bringing historians into AI architecture, we are bringing language models into the archive.

The question is not what AI can do for historians, but it's what historians and theologians and classicists and clinicians can do with AI. Whether the discipline, the disciplines that are trained to read, contextualize, and interrogate texts, can then discipline the machines that now generate. >> [snorts] >> I want to end this where it actually started because it did not start in a research lab. It started in a hospital room. In an attempt to use a language model to help a person with advanced dementia speak, to give language back when the disease had taken it. It could not work. The documentary record you would need to anchor that encounter had not been gathered.

And the person it was for was by then mostly beyond the reach of language. But that attempt produced, absent any anchor, was not the person. It was a culturally shaped composite that resembled them just enough to mark with painful clarity the distance between them. That is Miranda distortion in its purest form. You do not want a model that is your mother. You want a model that can speak with your mother's documents in the room. That's the threshold this whole framework is accountable to. A system that produces convincing fabrications when the persona is your own beloved is not a research artifact of limitations. It's a violation.

Every constraint I've described, document stay documents, the human keeps the interpretive custody, the encounter stays reversible, fidelity measured against a record and not against fluency, every one of them is motivated by the recognition that you evaluate this technology at the threshold of its hardest use case, not its easiest. A framework that cannot meet a grandchild at her grandmother's letters is not a framework at all. It's just another failed product. If the dominant failure mode is anachronistic compositing and your E valves measure fluency and personality consistency, which they do, then your E valves cannot detect the dominant failure. So, here's where I'll leave you.

If you ship character bots, companion AI, pedagogical agents, historical simulations, anything where a persona is supposed to reason from a record, your E valves are measuring the wrong thing. The instrument that catches what they miss is pre-registered. It's reproducible by any team with a frontier model in a context window. It scales as a build time gate, not a run time bottleneck, and it only works with a humanist in the loop, which I've shown you is a technical requirement, not a courtesy. The protocol, the questions, the rubric, and the predictions, the historians sealed vignettes, all of it will be published with this paper with Rick and Shawn. Not here with results.

I'm here with an instrument and an invitation. The archive is open. The laboratory is built. Run it with us. And let what comes through be measured not by how it sounds, but by whether it's true. Thank you.