Markdown source

Dense Slides: RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

Source Video

RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

Method

This deck is slide-only. The existing captured video frame set supplies candidate frames, then local OpenCV rejects sponsor/title/speaker-only frames, crops visible slide surfaces, deduplicates, and saves the cropped slide images.

Cropped Visible Slides

• Recreated text/layout view: open HTML recreation
• AI slide classifier: content_slide confidence 0.97
• Text source: advanced OCR rapidocr-live/border-trim/contrast.
• OCR decision: ready — Dense multi-line text with a long quote and small body copy is better handled by OCR.

Slide text:

Probtems Local questions, local answers Assuming an answer lies in a certain chunk

Multi-hop questions are not realistic

"If my future wife has the same first name as the 15th first lady

of the United States' mother and her surname is the same as the second assassinated president's mother's maiden name,

what is my future wife's name?" (from: FRAMES)

ONAIR

• Recreated text/layout view: open HTML recreation
• AI slide classifier: content_slide confidence 0.99
• Text source: agent_vision.

Slide text:

The Vicious Cycle

1. Build RAG systems for flawed benchmarks

2. Celebrate our awesome benchmark scores

3. Watch real users struggle

4. Create new benchmarks with the same problems

5. Rinse and repeat

• Recreated text/layout view: open HTML recreation
• AI slide classifier: content_slide confidence 0.96
• Text source: agent_vision.

Slide text:

Example

FIFA World Cup

Which team has won the FIFA World Cup the most times?

In how many FIFA World Cups did Brazil participate?

List all teams that have never won the FIFA World Cup but have reached the top 3.

• Recreated text/layout view: open HTML recreation
• AI slide classifier: content_slide confidence 0.99
• Text source: advanced OCR rapidocr-live/bright-screen/contrast.
• OCR decision: ready — Tabular content with small text and scores is OCR-suitable.

Slide text:

Common

Pipelines RAG Retrieval Pipeline Generator Score

Fail IAW Cp FUA Ward Cp Responses Common pipeline OpenAl gpt4o 0.05 0.11

ONAIR

• Recreated text/layout view: open HTML recreation
• AI slide classifier: content_slide confidence 0.95
• Text source: agent_vision.

Slide text:

High Level Idea

Unstructured Corpus

Data Structure

Query

• Recreated text/layout view: open HTML recreation
• AI slide classifier: content_slide confidence 0.99
• Text source: advanced OCR rapidocr-live/full.
• OCR decision: ready — Schema diagram and small field list are better handled by OCR.

Slide text:

Ingestion - Schema Creation

World Cup

Corpus (FIFA) (SemanticObject) Schema Winner:Team TopScorer:Tuple[Player,int] Year:int(1900-2100) Top3:List[Teams]

ONAIR

• Recreated text/layout view: open HTML recreation
• AI slide classifier: content_slide confidence 0.98
• Text source: advanced OCR rapidocr-live/border-trim/contrast.
• OCR decision: ready — Dense text-heavy slide with multiple bullets and small nested text; OCR is likely more reliable than manual transcription here.

Slide text:

Chattenges Not every corpus/query is relational DB material

Japan) Normalization (West Germany, South Korea and

Abstinence & Ambiguity Both during ingestion and inference

Did Real Madrid win in the 2006 Final? (Not

world cup)

Clustering and inferring schema (clear trade-off

on complexity)

Text2SQL over complex schemas

ONAIR

Classification audit: raw/sources/slide-ai-classification/dense/Ywl4LsvHKzU/audit.json