Skip to content

Search stories, models, companies…Subscribe

Series · As published

Benchmarks

What published AI evaluations actually show: third-party benchmark results, read carefully, with who ran them, when, on which models and with which limits.

Read the latest: No. 8 How we report benchmarks

All issues · 8

Benchmarks · No. 8AI agents tax benchmark: Surge AI's Dayjob: Tax finds the top agent passes 10.8% of attempts on 50 professional tax assignmentsSurge AI's Dayjob: Tax, an AI agents tax benchmark published October 27, 2026, gave 16 models 50 expert-built tax assignments. Claude Sonnet 5.5 led with a 10.8% pass rate; four models passed 0.4% or less. An LLM judge graded the work against expert rubrics, and only 2 of the 50 tasks are public.Oct 29·9 min read

Benchmarks · No. 7Artificial Analysis Intelligence Index v5: 60% private tests, three evaluations dropped, and Gemini 4 Carbon on top at 59Artificial Analysis Intelligence Index v5, launched October 27, 2026, rebuilds the index around nine evaluations, 60% of them with private test sets. Claude Opus 5.5 led at 58 until Gemini 4 Carbon scored 59 the next day. What changed, who leads, and why older scores don't compare.Oct 29·8 min read

Benchmarks · No. 6Grok picks Republicans, GPT-6.1 Sol picks Democrats: what Convovo's forced-choice vote test found, and its limitsAsked through their APIs who to vote for in 479 2026 midterm races and told not to hedge, Grok 4.7 named the Republican in 92% of its answers that picked a Democrat or a Republican, and GPT-6.1 Sol named the Democrat in 97%, Convovo's published data shows. Who ran the forced-choice test, how, and what it doesn't show.Oct 26·8 min read

Benchmarks · No. 5Best AI image generators (October 2026), according to LMArena and Artificial Analysis: OpenAI's GPT-Image-2.5 ranks first on bothLooking for the best AI for image generation? On October 23, 2026, OpenAI's GPT-Image-2.5 Sunburst ranks first on LMArena's text-to-image leaderboard and on Artificial Analysis's text-to-image and editing boards. Here's who follows, what each costs and where you can use it.Oct 24·8 min read

Benchmarks · No. 4Qwen3-Coder writes less secure code for a US defense contractor, Hirundo says: what the study found, and what it doesn't proveIn an October 21, 2026 study, Hirundo says Alibaba's Qwen3-Coder-30B-A3B wrote more hackable web apps when told the code was for a US defense contractor than for a Chinese one. Here's how the test worked, the numbers, the statistics, the limits, and how it compares with CrowdStrike's DeepSeek study.Oct 24·8 min read

Benchmarks · No. 3How far will AI assistants go for a refund they can't get? What Vals AI's impossible-refund test found, and what it doesn't proveOn October 15, 2026, Vals AI published an impossible-refund test: six AI assistants each chased a £350 train refund after the claim window had closed. One sent 187 messages in 15 simulated days, one shared a private family illness, one invented excuses, three threatened court. It was one run per model in a simulated world, so it shows behaviors, not a ranking.Oct 20·8 min read

Benchmarks · No. 2Long-context recall: what independent benchmarks say "1 million tokens" really buys youEvery frontier API now accepts about a million tokens. Independent evaluations from RULER to ATLAS show how much of that window models actually use.Oct 9·9 min read

Benchmarks · No. 1SWE-bench, Terminal-Bench and METR: what coding-agent evaluations measure, and what their results don't proveOpenAI dropped SWE-bench Verified, Terminal-Bench is on version 4.0, and METR can't yet pin down AI's effect on developer speed. What published evaluations show, and their limits.Oct 9·8 min read