Markdown source

Dense Slides: Building Conversational Agents — Thor Schaeff and Philipp Schmid, Google DeepMind

Source Video

Building Conversational Agents — Thor Schaeff and Philipp Schmid, Google DeepMind

Method

This deck is slide-only. The existing captured video frame set supplies candidate frames, then local OpenCV rejects sponsor/title/speaker-only frames, crops visible slide surfaces, deduplicates, and saves the cropped slide images.

Cropped Visible Slides

slide-001.jpg

Slide text:

D

c Dashbordi Googke A Studo APl-Schlussel B Ap·Kurznetung

APt:Schbusset Profektai.: Gruppiren rach Projokt:ofeld oy pto ltl:

Nurung Schlise Projekt Erstellt' Abrichnurgsstute

Rxanbtgrmrung

Ausgabon

:Abretchnung

. Prolokolle Lnd Dslasett:

K royooidsbunapuy

Sie konnen itre APl Schhlsse hler nlcht finden?:

:Coogit AL mudo ilapurett tuider. imporltltn slt thdte Pld'tit, In dHtts Lte wer don nut APtSchtbsce tor Profetut srgazeg, dle h' Ohae ehnn rhvin APi-Schplit eruteote. Wedhre kfomtonn Cn dit Zgthox LhTschlaLe ne tnteterL Sle torrh auch.

Prafeita hmporteren

Q' Sosrch"

:e Ger Api key 'o whal's new

o Settngt

slide-002.jpg

Slide text:

One API to Rule Them All

Building Agents with the Gemini Interactions API, from zero to a working agentic application in 60 minutes.

60 min · Live Coding · Philipp Schmid

slide-003.jpg

Slide text:

? Part 1: The tnterections APl1 Core Primitives

9

®

C Server-Side State Background Execution

12o clicnt.fnteroctions crcatc( orevious 2nteroction idi1.id mode1*gemini-3-flash-preview npuk= Follow up Question? 0 oclient fnteroctions.create( agento deep-research... bockgr ound:Iruo tnpuk= Roscarch topic.

history. Chain turns with prevlous snteroction,ia. Server oums coriversation Fire-and-forget for lcng tasks. Pol with interoct lons.get(ia).

Typed Output Blocks Streaming via SSE

Flat arrby. teit, Iurt tlon_coll, Logo, thought. for O in interoction.outputs: motch o.type: Cuse text" @rint(o.text) coso Cfunction_coll: Rich events: content.delta, Interoctton.corptete. for chunk in cliont.finteroctions create( modee gemini-3-11ash..oo tnputr Helloo orint(chunk)

5717

slide-004.jpg

Slide text:

Part 1: Tha hnteractiors AP1 Built-in Tools

Q Google Search <> Code Execution URL Context

(00ls=[[type" google seorch4)] tools=[ftype) "codo_cxecution")] (ools=[fcype Curl.context]

Ground responses with real-time web results. Exocute Python code server-side in a sandbox. Fetch and read ful web page contenl

Computer Use D File Search Remote MCP

Browser Butomation via the APl. tools=[(type conputer use"}] Search uplosded files in RAG stores. tools=(ft ype" fi1e 6oorch- Cornect extema took via kCP protocol. tools=[f"type" https:/.. ] mcp-servorD

Buh-ln toos execite server-aide, zero code needed. Mix with your onn funetion, tools freey.

6717

slide-005.jpg

Slide text:

Part1:The Interactions API CodeComparison

GENERATECONTENT INTERACTIONSAPI

Stoteless:re-send history every turn Request1 r1-client.nodels.generate_content（ model-genini-3-flosh-preview" contents-[ ("role":"user" ("inline_doto”:（mime_type”: ("text:"Describe this inoge) "porta":[ 'imoge/jpeg" "doto"：...")) i1-client.interoctions.creotec Request1 model-genini-3-flosh-preview input-[ 0fou.:od.) ("typo":"text" （6od/5o.dou/:d.T,

r2-client.nodels.generate_content( model-"gemini-3-flosh-preview" contents-[ ]:s1odosn.oo.) eot 2:MUST reend imoge ond nistory ("inline_dato”:（"mime_type”:"imoge/jpeg” "doto：...")) i2-client.interoctions.creote（ Request 2:Just.send npw input model-genini-3-flash-preview input-'Is there ony text?" previous_interaction_id-i1.id

["role":"model" ("role":"user" （ （[（2x）Au0o4sI.x.）]:10d （[（oo1do..x.）]1o

Access:response.condidates[e] content.ports[e].function_col]

8/17

slide-006.jpg

Slide text:

Using the API: Basic Chat

STATEFUL MULTI-TURN

Create an interaction with a prompt

Chain turns with previous_interaction_id

Zero history: server manages all state

slide-007.jpg

Slide text:

main.py x In Watthrcugh. Task tmpamentation Pliun 个日4 Crealing Agent Clats Structure x...@+

workshop > d mnin.gy 49 37 EE write_fite_schew = { "phraooters":! "required- I"tile_path-, rcontents"l.. Noxt wo noed to odd a read_fbo and write_ fe tool.: Create a basic python kuolenentatlon and also the Fon tchems defition and map for the key: function, schems

59 51 52 53 55 56 58 60 62 64 65 66 67 69 70 71 72 74 75 76 57 61 63 77 8 3. Reo istry sapping tool naaes to functlons sed scheras tootsap = { class Agent: derrnaoo. det.run(selt, input_text:,str Illst) - str: "read_file"1. ("functlon"r resd_fite, "schea": resd_file_schea),: def _lnlt_iself, model_id: str i "gemini-3-flash-previtr. toots: dlct = Hon "urlte_file": ("function": vritc_fite,. -schees-i urite_file_schca),: 1. Prepore tools for the APl call' toot detinitions = [tt"schema-] for t in selt.toots.values()] if setf.toot Initinlizes the Agent vith o Geinl clieit, mooe v.twsniu-interetl self. toots.n tools or ( procesesnInputotrung neretonyihepobeire ponse7 untinirgcg Processes: on lnput, and rtturns tht model's.respoase, handling tool.,calts. se?t.client. genal.cllent() self.prtvious_lnteraction_id + None: self.sodel_id = nodel_1d' lnteraction rodelaso. Acctp! Chsrge xt a↑.x ↓: 0x0d Acotpt t?: Includas tho Pyhon irplemantotlona, JsON schemus, sind" Edhed 1 to v updsting the sgent's interaction ioop to hande tool calls.: t here created an Implemertation plan to sdd tha' Ask anything, e to minilon, I Ior worktiovs Gesd flle and Mrite tite toos to ine Agent. This. +. A Paving A Gemn S Flenh: ' Thought for 4s:: Anatyzod solution Edlted Implementation Plan - Anthyzed & sgent py nlt-i31.: schemu., Iirptamentatlons, the JSON scheru detindtions, and a map to connect each. toor's key to its functon snid. 1've got o sotd python examole n Defining New Yools. Hrp'ament resd_fite, and Write_fite: tools within arkshop/aain py., rm working on the Python Sotut fon/aaent.py, to oude mo. The tuask is to'.

@ 邮 Ln 34; Col 10 (22 tolhcted}3 Spco1; 4 年 UTF-8 & LF $: {) Python 3.12.12 (ronr': venw) 甲 S+lect Python Intefprotor, Anugrrylty: Setng+ 高':

slide-008.jpg

Slide text:

Google Al Studio:v: EPlayground

puno6d 3

x Bula. >

@ Dnshboard

: Documonbaton:. >:

田

Volco

: 0:00 / 0:08 xond

W'hat do you think of rry out'h? Media resolulon 258 tokens/ wmage

HDGY Thinvng kere!

0:00 / 0:09 BuuI 0N

*.

Wal now, you'ro lookin' vary smart so you arot That greon acket su'ts you woa, I must say. A grand casual look. Y'ere you th'nkin' of hesdin' out somewhere? Sesron Cortut

Q Searth O vhat's naw. O Coogt L noot itry mule mhLhui, to douchtchdk cutput. Sream Is ive Dtrannta Yools Furction catng Ean.

o Get APi koy QSrtingi thorsten.schaefeo...:Start typlng I prompt 丨：B Toous Grounding with Googte Search Automatc Function Rosponse

slide-009.jpg

Slide text:

Gemini Live APl Demo 'Corrcttd

what this APl can do. Why not try out some fun features, Bke: Top o' the mornirg to yal fm GeminI Live, a Ettle dermo of? snother language? What do ya think?. heorin' mo speak Iin ditferent sccents, or see H I can chat in.:......

Start camera to send video

stog AY- +.+F Sart Camoo. Sh.re screen tcorract Type a mesisge... Bond:

Live Caprion Top ot the Marto To, I'm Cemin ufe. O itlle damo o!

slide-010.jpg

Slide text:

Gemini Live API Demo

slide-011.jpg

Slide text:

RLO

Connection wilbe established directly to the Gemini API Voice: API Configuration Connection Settings using an ephemeral token from the backend. ModelID: gemini-3.1-flash-live-p Gemini Behavior System Instructions: You are a helpful assistant.Be concise and friendly. Puck(Default Temperature:1.0 Microphone: Camera: Chat MediaStreaming Default Microphone %08 Default Camera StartVideo Share Screen StartAudio (（) loded gsiLipi.j displayer initalied SETP COLETE Servsce UR (vlalphal： lE/generatiyelan CreatedCeniai Line APIobjecti 3477155A vehseciet spesi getFunctionbefinitianscalled Geogle Grending enabled,removing cstoe Setup corplete: necting directly tei ratirie? le5/ /24565112645868722644 et（-） fncti alls if y. " rativelar ralivni odets/ge.xla.ta_tahe ai-d.l-flab-le-peie', ：ath./14565f Detault lev oeiniLeeJs437 AL.J31287 5/1424515 pesiniliet.is1315 stiat.tt:257

Controls randomess（0.0-2.0].Highermore creative/diverse,Lower more predictable/focused Enable Google grounding（Enabling Google grounding will disable custom tools) Connect toGemini to startchatting SYSTEM:Ready!

CustomTools

Transcription Settings

Activity Detection Setings

Connect Disconnect

Connected

Setup Message JSON(raw config sent to Gemini API)

Type a message. Send

slide-012.jpg

Slide text:

API Configuration

Media Streaming

slide-013.jpg

Slide text:

google-gemini/gemini-live-api-examples

Gemini Live provides multimodal realtime agent capabilities. Build voice agents that can process vision and text in realtime.

slide-014.jpg

Slide text:

Google The Keyvord Bula reaTtre conrueora aoena wrh Gtmin Lll Pmn Lv

Utng th+ GtTil L+ AP, Soich nou trhcites ht Ltnt to vb+ dttgy +ath trr voice. ha sgont Cin'het' de Carvt

tnd ititcta torttnt nd ghe dngn csnomtr, bolu veretint tna cit.

。

Build with an expanding ecosystem of

blog google uses cocdes from Google to derrer ard enhanct lhe quairty of ite sermces and to analyze tratic. Leam,more:-+AM-+iAn. OKOOtR

Classification audit: raw/sources/slide-ai-classification/dense/cVzf49yg0D8/audit.json