Slides: Evals 101 — Doug Guthrie, Braintrust
Source Video
Evals 101 — Doug Guthrie, Braintrust
Relationship To World's Fair 2026
These slides are extracted from a public AI Engineer YouTube video connected to World's Fair 2026. Speaker-matched clips are supporting context unless later confirmed as exact session recordings; official livestream recordings are day-level/event-level source material.
Related Scheduled Sessions
- No individual scheduled session mapping has been assigned yet; treat this as an event livestream deck.
Extracted Slides
OCR text:
INNOVATIONPARTNER
aws
PLATINUMSPONSORS
Graphite
WWindsurf
MongoDB
daily
augment code
Workos
OCR text:
BRAINTRUST.DEV
braintrust
THE END-TO-END DEVELOPER PLATFORM
BUILDING AI PRODUCTS
OCR text:
Agenda
e Company Overview
e Intro to Evals
ENIAC CIOL
e Moving to Production
e Human inthe Loop
oo
| ~~]
- ~~
OCR text:
Braintrust at a glance
& e.) AnKur Gaya: S| ROHN ria ean >$45m
Ea Investors and Al teaders
eae CRBC anette erste al6z ee ;
» Cher nema om
Pac laire) Microsoft 3 DATADOG dbt Labs zapier =| Grafana
ee Ca re amazon nuro ~ instacart STS
bd
VF fai)
| aMicrosoft oy)
me F
OCR text:
rd
We support leading Al teams
re NETFLIX Nite ra tals):
‘instacart “% Airtable zapier BN akessoni
replit eae ears ramp 4 AVercel
Sueno oa Robinhood # SAD cele) erey4 mem TES Cr]
Klarna. coursera ® docusign Webflow
‘
q a Microsoft ary?
OCR text:
g ase € cary TO gone
Weriting evats is powng to become 8 core skill for product Evals are emerging as the real moat for Al startups ovals are surprisingly often all you need
managers fo such 9 crticad part of moking & £000 Nard won insights about customers and thew business ON
Product wath AL hoger diecovered by fourniens acteng ainost ae
etnographers spchunking in the underserved slices Of the
GOP pre chaet
tame meee QrXenmeeMon © wwe
The bee A aided ade Rm Mae a nEE oF Ly TaN oe het
fmentan a tate
Uf there 6 one thing we Can teach people, 4 that
Writing evais ia probably the most important theng. Be tone he ween Fa et cottawes ears
x
a
OCR text:
; Td)
Why evals? Evals help answer questions
Model selection How Gets Al Cost efficiency
"Which LLM is the perform across "Can we achieve high
best choice for our diverse real-world performance without
needs? “nee excessive costs?
scenarios?
‘ Debugging & regression
i Bea te At oletl Feedback integration detection
reflect our compan fs “Are we learning from “How do we know
: 7 y users and improving when something
onde ae 5° iteratively?" breaks or gets
standards? worse?"
a
OCR text:
®
How evals can help your business
Cut dev time Rapid heraton cycies & local tested on multiple LLMs seamlessly
Reduce costs Autontated evals replace manuel review alowing faster eration ¢ release
Enhance quality Real time nonttoring & compliance to redcce risk and improve Cx
Sca le iets aahs Enabie non-technical collaboration to puda tie best Al Apps
Ie never seen a workfiow transformation jike the one tnat incorporates
evals Into ‘mainstream engineering’ processes before. it's astonisning.
eit
,
| a Microsoft @yr{?
OCR text:
Z e
Customer outcomes
zapier coda Notion
fe)
oe
[OlOROTOLOLOTe)
all [OlOTOTOTOLee)
= Te eYNars
op. YAO) 50%
®
a a Microsoft Gry?
OCR text:
BRAINTRUST.DEV Core concepts
AIE
Promptengineering Evals Alobservability
Think IDE for LLMoutputs Did Iimprove or regress? What's happening in prod?
Eval playgrounds for rapid prototyping Scores=a performance scoreof0-100% or？Debug.eval
Microsoft smol?
OCR text:
. ; rd
What is an Eval?
Definition: An Eval (short for evaluation) is a
structured test that checks how well your Al
system performs. It helps you measure
quality, reliability, and correctness across
scenarios.
i.
aws
7 Nae!
OCR text:
3 Ingredients in an Eval
DATASET SCORER
~The loqie behind your
The code or prompt you evals,
want to evaluate. : r
Can be oan
-It can range fron a (Ce been RUC ke Co] en ee
single prompt to un A set of real-world Cede funetion
entire agent workflow. exarples, mearear 5 —
The scorer will give a
Requires an input and “These are your test score of B 108 to each
RTeTeesTiny fen rare row in the dataset
i 3
OS
bg
| u
OCR text:
- ®
There are two eval mental models
Offline Evals
What aos
ee ee eee te Cs ies CRORE SLES ALC ASO CSCO ESL
prodefiness dafasutss. elmer el bran UG.
Pe Caeser Mace cate (Samael Ce] Sar] ea Ca Re COS a tar 6 eae OTD 0201 CoE Ld CACLOSLLEOSSE STOOGES
aca my
EN reas
©) Proactive yident fy aed reso ve issues [rears 10881 Os- SSeS TOT Chat OE AATO] RIN OTe 1s tS] ELAT7 Tak CRE Ea
RiGee eee ae ol aea aE ery Str Rome Ober tame] O20) 60s ee ort |
Seren avy
e@)) Create and marace evaliition datasets © Astometlicaly Vace cals usng prory or SOK,
ee) Define tess and scorers e osu fi hercandyveses p Grainteust Gbts araive
Ppa NASI een STCa teat erate eer aie etsy TST re
Satta ry ao arene eG eee race mee rs era Gd eer
x
a a Microsoft §=o(oou®
OCR text:
BRAINTRUST.DEV
What should I improve?
K
AIE Goodoutput High score Improveevals Lowscore
ndinopeg Improveevals ImproveAlapp
aws
OCR text:
Page ee ste eee a ee One a °
Sarees Ce ae wwwy ities
ea cena ey re en mes pape
sto: bare tbat erat
eid be ne ra a ae es Ce met
cad
Pires ne Mos Mec eee re ~—
. ces :
e a Ree eta 7
. BS eee eau: s
Porragighte, © ey mee atles tyler: wile ccecadgtio s __
or ee ae aa aero
Fem Cert eee corey cae cana ~
Bree ee ee
Petia
[az
re mntunt
x
~
OCR text:
, , r
Datasets - Tips
Cees) Te apcvb arses Tae codes 1 Con
focus on building a feedback loop rather than
a perfect dataset
e §=Never Stop Iterating:
Use Logs to capture more edge cases and
create more holistic Evals
Cas occ ea lee Lelpe TAM Sale Ee
use human review to establish ground truth
Especially when using expected column
a
i : oO
a a Microsoft
OCR text:
BRAINTRUST.DEV
Scorer Types
Code-based scorers
- Exact or binary conditions
- Numeric comparisons
- Structured or factual checks
LLM-as-a-judge scorers
- Subjective or contextual feedback
- Human-like interpretation
- Improvement across multiple drafts
AIE
OCR text:
: ®
Scorer Tips
Use a higher-quality model for scoring, even if the prompt uses a cheaper
model. Scorers benefit from better reasoning and nuance.
Treat scorers like judges: evaluate intent match, style accuracy, and
overall output quality: -not just correctness.
Break scoring into multiple focused scorers (e.g., accuracy, Creativity,
formatting) to pinpoint issues.
Test scorer prompts in the Playground before use. Try strong and weak
outputs to refine scoring reliability.
Avoid overloading the scorer prompt with context. Focus if on the relevant
input and output for fair, consistent evaluation.
ty
| u
OCR text:
BRAINTRUST.DEV
Playgrounds
Experiments
Evaluations
AIE
• Quick iteration of Prompts, Agents,
Scorers, Datasets
• Great for comparing Prompts & Models
• When you reach a happy place, save a
lot of the playground to Experiments
• Great for comparing full experiments so you
can review past playground sessions
• Evals from SDK or API will appear here
directly
• You can analyze your experiments over time
OCR text:
AIWorldFar-Eval Track Fe EdtVew nsertFormat Side? 回 D Sidesho
QMetus
AIE
Evals via SDK
Microsoft smol?
OCR text:
gS Hae , en @r
f Rene ares H
eet COREE a Rte Be te es ee |
[Dee Eee Ce :
.
Leon ee eee ne)
io Pe ety Peer reece tad !
Peete Ceara Serra |
a sIU is Mae Cony |
bot ee an oe :
Cares Pree eee ee
ron
—_— 3
3 s a ad
b G4G7 "0880-84" Hi -2 8
bd
+
OCR text:
on Pemareatt 3 ek m —)
gegen .@ y
Ong tenes rome e
Poeqoete ea - _ ote
Gers Sve on bee yon o tas o vee ee
6 Rta ane
leterste sranjeog pees a een @ lon wan
amines
siersarrerinconspereg’d we a ee Qt tee
Urvecemtas AI
+ inewssente
3 Cater nents
treet
Wages
eanypece
vet eney &
secgng laceretne Lee
Bie ote
raintrust
a
i an i * ,
b CO607 "QO880- 84° 6 *
co
; aws
OCR text:
app.py
dsdosnd
SEARCH
code_conversion
push_prompts.py
prompts.create
Ieport os
import braintrust
fies to incu
from code_conversion.agents Iaport INSTRUCTLONS
es to eickido
project =braintrust.projects.create(nam
s.enViron[“BRAINTRUST_PROJECT_NAME"])
AIE
project.
pts.createl
1resultn1fle-Openineditor
nanea*Code Conversion Prompt",
push_prompts.pycods
slug=code-comvers1on-pronpt",
project.prompts.create(
descriptionaThis prompt is used to comvert
promptaINSTRUCTIONS,
model-os.environ["OPENAI_HCOEL_NAE"],
messagesa!
(“role”："systee，“content：DNSTuCTINS）,，
("role”："user,"content”："《（input))”).
Microsoft
smol?
OCR text:
AIE
export const completenessScorer = project.scorers.create({
choiceScores: {
Good: 0.75,
Fair: 0.5,
Poor: 0.25,
},
});
export const accuracyScorer = project.scorers.create({
name: "Changelog Accuracy Scorer",
slug: "changelog-accuracy-scorer",
description: "Evaluates the accuracy of a generated changelog",
evals: [
{
role: "system",
content:
"You are evaluating the accuracy of a changelog generated from a list of git commits.",
},
{
role: "user",
content:
"Task: Rate how accurately the changelog represents the actual changes described in the commits.",
},
],
});
Finished running experiment
Slide-Derived Subjects To Review
Subject extraction uses video title, related session titles/descriptions, transcript context, and OCR text when available. OCR is best-effort and should be reviewed against the embedded slide images.