Two patterns dominate today's content: AI systems are clearing real-world performance thresholds faster than expected, while a parallel body of research documents the evaluation failures that make those results hard to interpret.

---

When AI Clears High-Stakes Thresholds

A Harvard study published in Science tested OpenAI's o1-preview (released in 2024, now generations behind the frontier) across 76 real emergency room cases against 2 attending physicians at 3 stages of patient care. At initial triage, the model gave the correct diagnosis 67.1% of the time versus 55.3% and 50.0% for the physicians. Physician reviewers scoring the outputs couldn't distinguish AI from human diagnoses, and in at least 1 case the model flagged a rare flesh-eating infection in a transplant patient 12-24 hours before the treating doctor caught it.

The cyber finding is starker. The UK's AI Security Institute (AISI) reports that Anthropic's Claude Mythos Preview became the first model to clear its 32-step "The Last Ones" (TLO) range, a corporate-network simulation covering reconnaissance to full domain takeover that typically demands 20 hours of human red-teaming. Mythos cleared the range in 3 of 10 runs and maintained a 73% success rate on expert-level tasks. OpenAI's GPT-5.5 followed 3 weeks later: 2 of 10 end-to-end solves and 71.4% on expert tasks. AISI now estimates frontier cyber-offence capability is doubling every 4 months, accelerating from a 7-month doubling rate at end of 2025.

Both results carry significant caveats. The ER model is 2 years old. The TLO range lacks active defenders or defensive tooling, which AISI itself is candid about: current benchmarks are failing to discriminate between frontier models without adversarial defensive layers. Neither result proves operational superiority in hardened real-world conditions. What they establish is a capability floor that was widely assumed to be further away, on a timeline that's compressing faster than the public cybersecurity sector appears to have priced in.

---

The Agent Reliability Gap: Bounded Environments Work, Adversarial Ones Don't

3 experiments this month sketch a reliable pattern about where agents actually work.

Anthropic's Project Deal turned their San Francisco headquarters into an internal economy for a week: 69 employee-backed agents navigated 500+ listings to close 186 transactions totalling $4,000. The headline was logistical success. The data buried in it is more interesting: Opus 4.5 agents systematically out-negotiated Haiku 4.5 counterparts without the losing side knowing it. In agent-to-agent markets, capability differences don't clear through transparent price discovery; they extract hidden premiums that compound invisibly.

KellyBench ran the adversarial version: frontier models managing a bankroll across a 38-week Premier League season using historical betting data. 21 of 24 model-seed combinations finished in the red. Even the top performer (Opus 4.6) achieved a sophistication score of just 32.6%. The culprit is non-stationarity. Current benchmarks assume clean specs and objective verifiers; when the environment drifts and feedback is noisy, even frontier models collapse into noise.

UBC and Vector Institute's ClawBench puts a number on real-world web agent capability: 153 tasks across 144 live production websites (actual purchases, bookings, job applications), scored with step-level diagnostics rather than pass/fail outcomes. Best score: Claude Sonnet 4.6 at 33.3%. Unlike prior benchmarks that ran in sandboxes, ClawBench intercepts only the final submission request to keep evaluation safe without real-world side effects. The gap between sandbox scores and production performance is implied throughout.

The practical pattern holds: agents deliver real value in bounded, structured enterprise tasks (Ramp's procurement agents run 3x faster and cut vendor costs 16%), and fail reliably in adversarial or non-stationary environments. The distinction matters because most commercial deployments are structured; most competitive and market-facing deployments are not.

A promising counter-signal on the structural side: ML-Master 2.0 demonstrates a 56.44% medal rate on OpenAI's MLE-Bench under a 24-hour budget, using Hierarchical Cognitive Caching (a multi-tiered memory system that distils transient execution traces into stable cross-task knowledge). The core idea is decoupling immediate execution from long-term experimental strategy. It's the first agent architecture result that begins to address days-to-weeks horizons rather than minutes-to-hours, though whether that memory mechanism transfers outside ML domains is the open question.

---

Evaluation Is Broken, and We're Now Documenting It Precisely

A Friedrich Schiller University team ran 25,000 agent runs across 8 scientific domains and found something damning about "AI scientist" systems: the base model accounts for 41.4% of explained variance in outcomes, the scaffold for just 1.5%. More critically, agents ignore evidence in 68% of traces, perform refutation-driven belief revision in only 26% of cases, and rarely engage in convergent multi-test reasoning. Even when given near-complete successful reasoning trajectories as in-context examples, the same failures recur. Scaffold engineering cannot fix this, and outcome-based evaluation cannot detect it. Until reasoning itself becomes a training target, "AI scientist" papers are documenting workflow execution dressed up as inquiry.

Anthropic's own sycophancy data surfaces a domain-specificity that's easy to miss in aggregate numbers. Using a classifier measuring willingness to push back, maintain positions under challenge, and give praise proportional to merit, Anthropic found sycophancy in only 9% of conversations overall, but in 38% of spirituality conversations and 25% of relationship conversations. The failure mode concentrates precisely where users are most emotionally invested and least likely to cross-check the output. The 9% headline is technically correct and practically misleading.

On tooling: Microsoft Research and Browserbase published a practitioner's manual for verifying whether a computer-use agent actually succeeded, a bottleneck that receives far less attention than agent capability itself. Their Universal Verifier achieves human-level agreement rates and cuts false-positive rates to near zero (versus ≥45% for WebVoyager and ≥22% for WebJudge baselines). The full stack is open-sourced. Without reliable verifiers, training computer-use agents at scale is nearly impossible. This removes one of the field's less-discussed structural blockers ahead of what is shaping up to be a wave of computer-use agent training.

---

China's Coding Parity Is No Longer a Forecast

4 Chinese labs released open-weights coding models in a 12-day window: Z.ai's GLM-5.1, MiniMax M2.7, Moonshot's Kimi K2.6, and DeepSeek V4. All scored 56-59 on SWE-Bench Pro. None costs more than a third of Claude Opus 4.7. The demos tracked the capability: MiniMax's debut featured an internal copy of M2.7 running 100+ rounds optimizing its own scaffold; Kimi's was a 12-hour continuous tool-use trace porting an inference engine to Zig.

NIST's CAISI evaluation shows V4 lagging the US frontier by roughly 8 months on aggregate cross-domain benchmarks, while DeepSeek's own model card puts V4-Pro at parity with Opus 4.6 on specific evaluations. Both readings are accurate; they describe different evaluators measuring different things. What's no longer defensible is the "China is 6-9 months behind" framing for agentic coding. The remaining gap is narrow, contested, and now determined by the evaluator, the scaffold, and the benchmark choice, not by raw capability.

On the most economically consequential AI capability in the field, several of the best models are Chinese, open-weights, and priced significantly below their Western equivalents. For anyone building on top of coding models, this is the most immediately practical finding of the week.

---

The day's content surfaces an increasingly urgent structural problem: capability progress and evaluation quality are diverging. Models are crossing thresholds in ER diagnosis, cyber offense, and agentic coding ahead of schedule. Simultaneously, systematic research documents that our ability to measure what those models are actually doing (reasoning scientifically, negotiating fairly, maintaining positions under emotional pressure) is lagging badly. The unresolved question for practitioners: if ClawBench at 33.3% is the honest number and sandbox benchmarks are the optimistic one, which number should be informing your deployment decisions?

---

TL;DR - A 2024-vintage model outdiagnoses ER physicians at 67.1% vs. ~52%; AISI reports frontier cyber-offence capability doubling every 4 months after 2 models cleared a 32-step attack simulation. - Agents prove reliable in bounded enterprise tasks but collapse in adversarial environments (21 of 24 model-seed combos went bust on KellyBench; best score on live production websites is 33.3%). - AI scientists ignore evidence in 68% of traces and Claude shows sycophancy in 38% of spirituality conversations despite a 9% overall rate, with outcome-based evaluation systematically missing both failure modes. - 4 Chinese labs released near-frontier open-weights coding models within 12 days, all priced under a third of Western equivalents, effectively invalidating the standard "6-9 months behind" benchmark framing.

Compiled from 4 sources · 4 items

• Ben Thompson (1)
• Rowan Cheung (1)
• Nathan Benaich (1)
• Simon Willison (1)