Skip to content
Sunday, November 22, 2026Free guide
EN
dailyai
Research

OpenAI wants AI development limits measured in “perplexity” — and publishes its own numbers

· 2 min read
ShareXin

OpenAI proposed on Sunday that the AI industry measure and cap its progress with a single number — perplexity, how well a model predicts the next token on average, where lower is better — and published the internal figures it uses to track its own pretraining.

The proposal came in “Perplexity and Intelligence”, an essay by Jakub Pachocki, OpenAI’s chief scientist. He argues developers should agree on “common limits” that no lab crosses until concrete safety milestones are met — such as a successor to chain-of-thought monitors, which watch a model’s reasoning for signs of trouble.

To make such comparisons possible, Pachocki published OpenAI’s internal log-perplexity scores on its main evaluation dataset. GPT-3.5 scored about 0.99 and GPT-4 about 0.91, he writes; below 0.8 the company saw “transformative capabilities in fields like cybersecurity and mathematics”, and at about 0.76 a model “started producing groundbreaking results” in mathematics — the field still digesting an AI-assisted proof claim. He cautions that “it was not at all clear a priori that these capabilities would appear once those precise perplexity thresholds are met”. These claims come from OpenAI and have not been independently verified.

He predicts more transformative capabilities in the 0.68–0.7 range — admitting “I don’t know what those capabilities will be” — and says an “avalanche of fundamental pretraining advancements” could come “as soon as 2027”. Weaknesses he lists: labs train on different data and cannot be compared easily; perplexity does not map to any task people care about; better reasoning does not lower it; its floor is unknown.

Publishing the numbers “took some consideration”, he writes, because it gives competitors a clearer picture of where OpenAI stands — a concern he says is outweighed by “the urgency of establishing shared safety measures across the industry”.

Why it matters: It appears to be the first time a frontier lab has put internal pretraining numbers on the table, and it turns an abstract safety debate into a trackable number. If other labs refuse to match the transparency, the proposal will stall at the first hurdle it names: comparison. If they do follow, future AI “races” may come with speed limits written in the units engineers already use.