---
title: "Don't Ship Skills Without Evals"
category: "talks"
date: "2026-06-30"
time: "3:20pm-3:40pm"
track: "Evals"
room: "Track 5"
speakers: ["Philipp Schmid"]
sourceLabels: ["Official conference schedule", "Public YouTube metadata"]
scheduleTrack: "Evals"
scheduleRoom: "Track 5"
scheduleLabels: ["Evals", "Track 5", "sponsor", "confirmed"]
---
# Don't Ship Skills Without Evals

## Conference Context
- Date/time: 2026-06-30 · 3:20pm-3:40pm
- Track/room: Evals · Track 5
- Speaker(s): Philipp Schmid
- Session type/status: sponsor · confirmed

- Track: Evals
- Room: Track 5
- Session type: sponsor
- Status: confirmed

## Session Description
There are thousands agent skills. Almost none of them are tested. They get vibe-checked with two manual runs, maybe a thumbs-up from a colleague, then shipped. You wouldn't merge code without tests — so why are we shipping skills without evals? This talk covers the full lifecycle of building reliable agent skills: what a skill actually is (and isn't), how to write one that triggers correctly, and how to build a lightweight eval harness that catches failures before your users do.

## Media Evidence
[Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind](https://www.youtube.com/watch?v=3_gYbhABcAE) (speaker-match related prior/adjacent AI Engineer video; captions: English auto-captions).

- Source video: `youtube-3_gYbhABcAE`
- Slide deck: [[youtube-3_gYbhABcAE-dense-slides|Dense Slides: Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind]] — 2 visible slide image(s); 2 HTML recreation(s).
![[assets/dense-slides/3_gYbhABcAE/slide-001.jpg]]
![[assets/dense-slides/3_gYbhABcAE/slide-002.jpg]]
- Additional slide evidence: [[youtube-3_gYbhABcAE-slides|Slides: Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind]], [[youtube-3_gYbhABcAE-reconstructed-slides|Reconstructed Slides: Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind]]
- Slide-derived themes for `youtube-3_gYbhABcAE`: text, state, trap, treating, real, world, feels, safe.

- [[youtube-0vphxNt4wyk-transcript]] — full cached transcript markdown for the related YouTube source.

## Evidence Graph
This evidence graph is generated from currently linked source material: official schedule text, related video pages, cached transcripts, visible slide text, dense/reconstructed slide pages, and AI slide-classification audits.

### Media Signals
- `youtube-3_gYbhABcAE` — 4 slide-derived text signals
- Slide-derived themes for `youtube-3_gYbhABcAE`: text, state, trap, treating, real, world, feels, safe.
- Evidence links for `youtube-3_gYbhABcAE`: [[youtube-3_gYbhABcAE]], [[youtube-3_gYbhABcAE-slides]], [[youtube-3_gYbhABcAE-dense-slides]], [[youtube-3_gYbhABcAE-reconstructed-slides]]

### Agent Reading Notes
Use these signals to refine the synopsis, topic links, people/company context, and method notes. If a source is a related external video rather than an exact official recording, keep it framed as supporting evidence.

## Transcript Status
Related video transcript availability: English auto-captions. Treat this as supporting context, not a recording of this exact scheduled session unless later confirmed. Not fetched yet.

## People
- [[philipp-schmid]]

## Supporting Slides
- [[youtube-3_gYbhABcAE-slides]] — extracted from the related public AI Engineer video.

## Slide Evidence
- Slide-only cropped deck: [[youtube-3_gYbhABcAE-dense-slides]] (2 viable slide images).
- Related slide/OCR pages:
- [[youtube-3_gYbhABcAE-dense-slides]]
- [[youtube-3_gYbhABcAE-reconstructed-slides]]
- [[youtube-3_gYbhABcAE-slides]]
- Slide-derived terms: `braintrust`, `workos`, `openal`, `teas`, `senior`, `engineers`, `struggle`, `tsacconss`, `mental`, `model`, `collisions`, `traditional`, `engineering`, `reatty`, `alengineer`, `colds`, `erate`, `septs`

## Synthesis
### Synthesized Breakdown
Yes, so hi everyone. My name is Philip. I'm based out of Germany. I'm part of the Google DeepMind team mostly working on Gemini API and agents.

### Speaker And Company Context
- [[philipp-schmid|Philipp Schmid]] — Staff Engineer at [[google-deepmind|Google DeepMind]].

### Topics Covered
- [[agent-security]]
- [[agentic-search]]
- [[ai-sandboxes]]
- [[coding-agents]]

### Derived Links And Source Material
- [[youtube-0vphxNt4wyk-transcript]] — transcript markdown; source cache `raw/sources/youtube-transcripts/0vphxNt4wyk.txt` (3,965 words).
- [[youtube-0vphxNt4wyk]] — related YouTube source page.
- [[youtube-0vphxNt4wyk-slides]] — slide evidence.
- [[youtube-3_gYbhABcAE]] — related YouTube source page.
- [[youtube-3_gYbhABcAE-slides]] — slide evidence.
- [[youtube-3_gYbhABcAE-reconstructed-slides]] — slide evidence.
- [[youtube-3_gYbhABcAE-dense-slides]] — slide evidence.

### Novel Concepts / Clever Methods
- No highlighted novel concept has been detected yet.

### Evidence Boundary
This synthesis uses the official schedule plus cached video transcripts. Official AI Engineer World's Fair San Francisco 2026 livestreams and cut videos are primary event video sources for transcript/slide evidence; external, historical, or speaker-matched videos remain supporting context unless manually verified as exact official event recordings.

## Official YouTube Recording
- [[youtube-0vphxNt4wyk]] — official AI Engineer YouTube channel recording published 2026-07-14.
- Evidence status: [[youtube-0vphxNt4wyk-transcript]]; [[youtube-0vphxNt4wyk-slides]].
- Boundary: use this recording as media evidence; keep date/time/room facts tied to the official schedule.