Computer Use at the Edge of the Statistical Precipice

Official Schedule Context

Evaluating Computer Use Agents (CUAs) on interactive environments is fraught with methodological

pitfalls that the field has yet to systematically address. We show that a 1MB replay script that

blindly executes a recorded action sequence without ever observing the screen outperforms frontier

models on prominent static benchmarks, and prove that its expected success rate is exactly equal to

the source agent's pass@k in deterministic environments. We trace this and other failures to two

root causes: non-principled environment design (static, unsandboxed, or unreliably verified

environments) and non-principled evaluation methodology (naive aggregation and misuse of pass@k for

stateful UI interactions). To address the first, we propose PRISM, five design principles for CUA

environments and instantiate them in DigiWorld, a benchmark of 15 realistic sandboxed mobile

applications able to evaluate agents in over 3.2 million verified unique configurations. To address

the second, we develop an aggregation framework that correctly accounts for the nested structure of

CUA benchmarks. All together, we show that principled environment design and rigorous evaluation

methodology are not optional refinements but prerequisites for meaningful CUA research.

No official session recording transcript was found by exact title match on the AI Engineer YouTube channel during this run.

Pending transcript synthesis when an official recording or confirmed matching video is available.