Markdown source

Building self-learning loops for your agent

Conference Context

Session Description

Building an AI demo is easy. Knowing whether it actually works — and keeping it working in production — is the hard part. Most teams ship agents on vibes: they try a few prompts, the output looks good, and they push to production with no real way to measure quality or catch regressions. This hands-on workshop walks through the full lifecycle of shipping a real AI agent, using a working financial-analyst agent built on the Claude Agent SDK as the running example. You'll instrument it with tracing, do structured error analysis on its actual outputs, and build a layered evaluation suite — from cheap deterministic code checks to LLM-as-a-judge evaluators with custom rubrics. We'll cover the parts most tutorials skip: why agents fail in ways single LLM calls don't, the eval anti-patterns that quietly mislead you, and how to know whether you can even trust your judge (meta-evaluation). Finally, we'll close the loop: turning eval results into datasets and experiments, running evals online against production traffic, wiring them to monitors and alerts, and feeding failure explanations back to a coding agent to actually fix the underlying problems. You'll leave with a runnable notebook and a repeatable, evaluation-driven workflow you can apply to your own agents the next day.

Media Evidence

Build a Prompt Learning Loop - SallyAnn DeLucia & Fuad Ali, Arize (speaker-match related prior/adjacent AI Engineer video; captions: English auto-captions).

Evidence Graph

This evidence graph is generated from currently linked source material: official schedule text, related video pages, cached transcripts, visible slide text, dense/reconstructed slide pages, and AI slide-classification audits.

Media Signals

Agent Reading Notes

Use these signals to refine the synopsis, topic links, people/company context, and method notes. If a source is a related external video rather than an exact official recording, keep it framed as supporting evidence.

Transcript Status

Related video transcript availability: English auto-captions. Treat this as supporting context, not a recording of this exact scheduled session unless later confirmed. Not fetched yet.

People

Supporting Slides

Slide Evidence

Synthesis

Synthesized Breakdown

Building self-learning loops for your agent ## Conference Context - Date/time: 2026-06-29 · 11:05am-12:05pm - Track/room: Posttraining & Midtraining · Track 1 - Speaker(s): Fuad Ali - Session type/status: sponsor · confirmed - Track: Posttraining & Midtraining - Room: Track 1 - Session type: sponsor - Status: confirmed ## Session Description Building an AI demo is easy. Knowing whether it actually works — and keeping it working in production — is the hard part. Most teams ship agents on vibes: they try a few prompts, the output looks good, and they push to production with no real way to measure quality or catch regressions. This hands-on workshop walks through the full lifecycle of shipping a real AI agent, using a working financial-analyst agent built on the Claude Agent SDK as the running example.

Speaker And Company Context

Topics Covered

Derived Links And Source Material

Novel Concepts / Clever Methods

Evidence Boundary

This synthesis is based on the official schedule and linked source pages. It should be revisited when exact session recordings or transcript-backed secondary sources are available.