Markdown source

Model Whisperers: How Evals and Prompts Shape Agent Behavior

Conference Context

Session Description

Getting an AI agent to behave the way you want isn’t just about writing better prompts. In real systems, behavior emerges from a loop: prompts->evals->iteration->feedback. Small changes in any part of that loop can completely change outcomes. We saw this while building a seed asset agent - a system that turns messy, real-world advertising creatives (low quality images, cluttered visuals, heavy text overlays) into clean, reusable assets for downstream Gen AI tools. The agent acts like an editor, simplifying visuals, removing unnecessary elements, and isolating core content so that additional context (like text or CTAs) can be added back in a more controlled, brand-safe way. But the real challenge wasn’t just building the agent - it was making it reliable. And prompting alone wasn’t enough. What actually moved the system forward was how we defined success—and how we used evals to reinforce it. Over time, evals stopped being just a way to measure quality. They became part of how the agent learned what “good” looks like. In this talk, we’ll cover: Why prompting alone doesn’t give you stable agent behavior How evals act like feedback signals, not just scorecards How we built evals sets that reflect the real-world Using agent trace logs to understand why things fail (not just that they fail) How to iterate without breaking things you already fixed By the end, you’ll have a set of patterns you can apply to any system dealing with messy/continuously changing data and how to tweak your prompt and evals to accommodate such changes.

Media Evidence

No related AI Engineer channel video found yet.

Evidence Graph

This evidence graph is generated from currently linked source material: official schedule text, related video pages, cached transcripts, visible slide text, dense/reconstructed slide pages, and AI slide-classification audits.

Media Signals

No linked video, transcript, or slide source has been attached yet.

Agent Reading Notes

Use these signals to refine the synopsis, topic links, people/company context, and method notes. If a source is a related external video rather than an exact official recording, keep it framed as supporting evidence.

Transcript Status

No official session recording transcript was found by exact title match on the AI Engineer YouTube channel during this run.

People

Notes

Synthesis

Synthesized Breakdown

Model Whisperers: How Evals and Prompts Shape Agent Behavior ## Conference Context - Date/time: 2026-06-30 · 1:30pm-1:50pm - Track/room: Evals · Track 5 - Speaker(s): Chris Souza, Preetika Bhateja, Daniel Bump - Session type/status: sponsor · confirmed - Track: Evals - Room: Track 5 - Session type: sponsor - Status: confirmed ## Session Description Getting an AI agent to behave the way you want isn’t just about writing better prompts. In real systems, behavior emerges from a loop: prompts->evals->iteration->feedback. Small changes in any part of that loop can completely change outcomes. We saw this while building a seed asset agent - a system that turns messy, real-world advertising creatives (low quality images, cluttered visuals, heavy text overlays) into clean, reusable assets for downstream Gen AI tools.

Speaker And Company Context

Topics Covered

Derived Links And Source Material

Novel Concepts / Clever Methods

Evidence Boundary

This synthesis is based on the official schedule and linked source pages. It should be revisited when exact session recordings or transcript-backed secondary sources are available.