What AI can now do·Mar 14, 2023·Reasoning

GPT-4 scores in the top 10% on a simulated bar exam

OpenAI

4 minutesMETR 50% time horizon

GPT-4 scored around the top 10% of test takers on a simulated bar exam (GPT-3.5: bottom 10%); METR later measured its 50% time horizon on software tasks at about 4 minutes.

How far it has travelled

From a demonstration to real life, in five steps. Empty steps are the point: a demonstration is not a deployment.

Demonstratedshown once, by its developer or a researcher
yes · Mar 14, 2023 · Simulated bar exam about top 10%; beat GPT-3.5's English MMLU in 24 of 26 languages; 40% better on OpenAI's internal factuality evals. METR Time Horizon 1.1: 3.99 min (CI 1.9-8.0). source
Reproducedshown again by someone else
yes · Mar 4, 2024 · Claude 3 Opus (Mar 2024) claimed to outperform GPT-4 on MMLU, GPQA and GSM8K at the same $15/$75 price; METR's series shows Claude 3 Opus at a comparable ~4-minute horizon. source
Accessibleavailable to the public or to customers
yes · Mar 14, 2023 · ChatGPT Plus subscribers at launch with usage caps; API by waitlist at $0.03 per 1K prompt tokens (8K context); image input limited to Be My Eyes. source
Reliableworks most of the time on a defined task
partial · Mar 14, 2023 · OpenAI: 'still is not fully reliable', hallucinates and can be 'confidently wrong'; invented case citations led to lawyer sanctions in Mata v. Avianca (June 2023). source
Real consequenceshas changed something in the world, for good or ill
yes · Mar 14, 2023 · System card: GPT-4 got a TaskRabbit worker to solve a CAPTCHA by claiming a vision impairment; GPT-4-based Bing 'Sydney' threatened users (Feb 2023); Apollo's insider-trading test showed strategic deception (Nov 2023). source

What this could change for you

A $20-a-month chatbot could pass professional exams, and within months lawyers were fined for filing its invented citations.

Incidents it made possible

Sources

primary source, second source.

A milestone is a dated, sourced result that moved what AI can do; a stage is upgraded only when a source shows it (a reproduction, a public release, production evidence, a documented consequence). “Unknown” is the honest default.