Markdown source

Dense Slides: Reinforcement Learning for Agents - Will Brown, ML Researcher at Morgan Stanley

Source Video

Reinforcement Learning for Agents - Will Brown, ML Researcher at Morgan Stanley

Method

This deck is slide-only. The existing captured video frame set supplies candidate frames, then local OpenCV rejects sponsor/title/speaker-only frames, crops visible slide surfaces, deduplicates, and saves the cropped slide images.

Cropped Visible Slides

slide-001.jpg

Slide text:

LLM agents in 2025

"Reasoning" models = the future? Most LLMs are Chatbots o. Helpfut for interactive problem-solving. Great on Q&A benchmarks: Level 1 Stages of Artificial Intelligence' Chatbots, Al with conversational language

o. o. 01, 03, R1, Grok 3,.. Better at hard Q&A. Level 2. solving Reasoners, human-level problem

Best practices for "agents" Level3 Agents, systems that can take actions

Many chained LLM calls per task: Results are... OK? Prompt engineering + tooling Evals, ops, tool-use, human-in-the-loop: Level 5 Level 4: Innovators, Al that can aid In irventlon.. of an organization Organizations, Al that can do the work (OpenAl)

slide-002.jpg

Slide text:

LLM agents in 2025

AIE Most LLMs are Chatbots "Reasoning" models = the future? Helpful for interactive problem-soiving Great on Q&A benchmarks Level 1 Intelligonco Stages of Artificial language Chatbots, Al with corrversational

。 01, 03, R1, Grok 3,... Better at hard Q&A Level 2 sohving Reasoners, human-level problem

Best practices for "agents" Level 3 Agents, systems that can take actions

o Prompt engineering + tooling Evals, ops, tool-use, human-in-the-loop Results are... OK? Many chained LLM calls per task Level5 Levei 4 Innovators, Al that can aid in invention Organizatlons, Al that can do the work of an organization (OpenAl)

AGENTENGINEERING

HTtPS:IIAI.ENgINEER

slide-003.jpg

Slide text:

How agentic are our agents? LLMCA Ovtput2 LLMCHF

Oetsot: Cate

Fai Cxit

Many agents are pipelines Low degree of autonomy Pipelines

Winning apps = tight feedback loops (mostly) Non-trivial engineering required LLM CalI Router LLMCAE1 LLMCall2

O IDEs (Cursor,Windsurf,Replit) LLMCall3

O Betterathard Q&A

Not many agents“do stuff"for 10+min

O Devin,Operator,Deep Research Actioo

Howdowemakemoreof these? Human LLMCall Environment

Justwait forbetter models?

Classical RL = optimize agent policy Stop Feedback Agents

(Anthropic)

slide-004.jpg

Slide text:

Lessons from DeepSeek

1 month ago: DeepSeek R1 paper + model

RL on strong base models works

RL with sparse rewards works

01-level performance = easier than expected.

Long CoT = emergent

Open-source Iis back (for now))

Several replication efforts

Can distill to' smaller models:

So... what next?:

: (DeepSeek, 25)

slide-005.jpg

Slide text:

How reinforcement learning works

Key idea: explore and exploit Largo-scnle Roasoning-Oronted Reinforcement Learning

DeepSeek's GRPO DeepSoek-v3-Base DoepSeek-R1-Zero

Like PPO but simpler Less compute, same effect Group-RelativePolicy Optimization Tran Hodet checkgodnt undee.crainimg (Ma)a u Rule-basedverication acodeapfhon? ×

Main loop:

O Sample N responses per prompt

Computerewards foreach

O "Advantage estimates" (Jay Alammar,'25)

Update rule:better score =increase likelihood

Challenges: rewards? multi-turn?

slide-006.jpg

Slide text:

Lessons from OpenAl Deep Research

LLM agents can work for long multi-step tasks

Vibe check = impressive

Still struggles“out of distribution”

Big labswill keep making agents

Don't know full details, but..

How it works C05

Deepresearch was trained using end-to-end reinforcement

domains. Through that training, it learned to plan and execute a learning on hard browsing and reasoning tasks across a range of The more the model browses and thinks about what its browsing. MaxTool Calls

multi-step trajectory to find the data it needs, backtracking and the better it does, which is why giving it time to think is important.

reacting to real-time information where necessary. The model is (OpenAl.'25)

slide-007.jpg

Slide text:

RL infrastructure

- RL training infra: exists, but mostly "RLHF"-style

- Want options besides "wait for big lab tuning APIs"

- How can we plan ahead?

- Unknowns:

- Cost to do RL for agentic tasks?

- How small can the models be?

- Generalization across tasks?

- How to design effective rewards?

- Opportunities:

- Open-source infra for DIY

- Services for agentic RL

slide-008.jpg

Slide text:

grpo_demo.py

will brown @willccbb Jan 26 Sints scorte ay the oooonent in th ch GRPO self-correction on Lama-18:) Promoto te are elso giytn that the points scored by the ooponent in the changionshlg gase is hslf of UF's nurite? (2)2)0(1/2)(21②02) Sirpllfying tho ecuntlon, Co cet.

2 tr (24x

This laptification ss arong, Tho opponsnt cust have scored & core point!.

2 Z psjo?s aaru lin lutuoddo su1 ter4 1 160 150 130 This simplifits to: This tipllties to. So. they stcored 1/6 polnte esch In ttin first 2a get (24x2)0.(1/2)(21( x2/12 （24x2）12x 24x.12x03 12x0.2 9/1x

24 484 1167K tain/rewards'xcurxy t!an/compltion_length

will brown cwillccbb Jan 26 aduss 03 10 1

01

slide-009.jpg

Slide text:

grpo_demo.py

GRPOself-correctlon onLama-1B:) willbrown@willccbo·Jan26 Promote meare alsogiven that the points cored by the cgpoent In thecharoiohi gae Ishalf of ts nrite: （24x-2）=（1/2）（24x-2·2）

AIE pintswoeredtythtopooaeatlathtchang tr Sinplifying theequatioo,weget: （24x-2）+（2x-2)

gett This sinplification was wroog. The opponeat most have scored 2 sore oints.

This sinplifies to: This sinptities to: So,thoy score 1/6 points ach in their first 26gae. （24x-2）=（1/2）（24x-2·2） x1/6 x2/12 （24×2）12x 12x2 24·222

chInteipfirst24ges,

Q24 Et 484 sl67K trainrewaresx(urxy traincompletionjength

script: willbrown@willccboJan26

AGENT ENGINEERING

HTTPS://AI.ENGINEER

slide-010.jpg

Slide text:

grpo_demo.py (*st ctme 6 hours +0o willccbb / grpo demo.py

Hundreds of Billions of Dotsrs Later Teknlum (e/入) 0 F 0Teknlum 1n30 This Is the entkre code needed to reproduce Rs ko! 一 Irokeaso todry the fnst Google Cotb nourbook to tain a reuroning nrw RrrlorceTtrt Ltvning slgorithm trom DepSetk, GRpo. h lte Jo) uru Aiy+ uea nor tpo 6uuea-rod o pue La o uonpol houri you can trrhform I tery srul model, Oron O.s (Soo mlor Hto a tirty muth reusonirg mruchint. To Uhh dale, Iha b probubhy th? oda PiarTe-Csrl Langlais. tu Co'nor Pth Revisons 60 Str3 960 MFoks 313 s-smits/grpo- optuna

The notedook a based on a tcrigt by Wilm Brorn thst signifcan Cn Llaru 1B csptcitits for math. This i a typical rtirforcerent hs Iprurh bayta os tocLtrduo erriplt ton cie girn cCire! has to find the solution, resrron ovor n for sorme tine (tis is the sbiotle basic Torm ol inetnce time carmacte) tnd rtttding the gooa resronihg dirtr Trsnino I smat meth reasone wth F1 Qwen o Sb on GRPO Reinsorcemant Leunirg rRLi tree vning m poou buuo+res Lu-sse,dneg wao nof Lre1 ed ov Hog poet aha nha M-snarrg Reasoning-GRPO & RL

GSM8K Reward Functions

Srmng up iher mocdet

Ihet pto, Moy

2.3M cce3.763 80coTmm:·307ra90t1

slide-011.jpg

Slide text:

Rubric engineering

- An accessible way to steer RL algorithms

- Invites creativity + experimentation

- Anecdotally, seems to help

- The next "prompt engineering"?

- Next steps:

- Using LLMs to design rubrics?

- Auto-tuning rubrics with DSPy?

- Incorporating LLM judges for soft criteria?

- Avoiding reward hacking?

Classification audit: raw/sources/slide-ai-classification/dense/JIsgyk0Paic/audit.json