How reliable is the conclusion about AI acceleration: tests, tokens, and statistical models in the METR study

The conclusion about AI acceleration cannot be reduced to a single number. In the publicly available METR publication, capability growth is measured through the time horizon of tasks, while a retelling of Anthropic’s internal analysis included in the source materials estimates acceleration using the IC index. These are different metrics, test sets, and statistical procedures, so their results cannot be mechanically combined.

One number settles nothing here.

An estimate of acceleration becomes robust only after checking the composition of the tests, the amount of computation used during execution, and the chosen mathematical model. The data must withstand comparison with alternative specifications, the removal of individual tasks, and different success thresholds. Otherwise, the final indicator primarily describes the features of the measurement procedure rather than the pace at which systems are developing.

What data METR used and why task length affects the result

In METR’s public work, a task’s time horizon is defined as the length of work that a qualified human would typically perform if the model solves that task with a given probability of success. The main metric is the amount of time at which an agent successfully completes the task in approximately half of its attempts. The researchers combined tasks from HCAST, RE-Bench, and the SWAA set of short software actions, then compared model results with estimates of the time required by human experts.

This approach differs from an ordinary percentage of correct answers. A model that solves 80 percent of short questions is not necessarily capable of reliably completing a project lasting many hours. In its public work, METR used 170 tasks ranging from seconds to dozens of hours, more than 800 human baseline attempts, and several runs for each model–task combination. The authors evaluated not only the answer but also successful completion of the entire assignment, including tool use, error correction, and preserving the result.

Execution length affects the result through the human scale of difficulty. If a developer spends one hour on a task and the model succeeds in half of its runs, it receives a nominal horizon of about one hour. However, this does not mean that the model works independently for one hour without interruption or replaces a specialist on any project of that duration. The result is affected by the task’s structure, the quality of feedback, the available tools, and reliability requirements.

The time horizon shows the relationship between the probability of success and the duration of human work, not a universal boundary of autonomy. A task with clear steps, tools, and a verifiable result is structured differently from an open-ended project in which one must formulate the goal, choose a method, and assess the quality of the solution independently. Therefore, the same horizon for two task sets does not make them substantively equivalent.

Test ceilings and overly easy tasks distort the progress curve

A benchmark loses its measurement power when new models begin to pass nearly all of its tasks confidently. The percentage of correct answers approaches the upper limit, and real improvement is no longer reflected on the scale. This is why the results of individual tests may show a sharp jump even though what was measured was not the model’s general intelligence but the transition from incomplete to nearly complete performance on a specific set.

The original retelling states that Model 4 allegedly exceeded the 50 percent mark on 15 tests at once, whereas an earlier model did not reach this level across several areas. This statement cannot be considered a confirmed universal fact: Model 4 itself is described as an internal model, and according to the cited materials, the researchers did not have direct access to it. The result is best treated as a claim from an internal analysis rather than an independent measurement.

After excluding individual tests, including OSWorld and MMU, or tasks where the models had already exceeded the threshold of 85 percent, the estimated acceleration, according to the report’s retelling, remained approximately in the range of 1,9 to 2,2 times. This supports the argument that the conclusion does not rest on a single test, but it does not eliminate the task-selection problem: removing inconvenient sets changes not only the noise but also the object being measured.

The unusual shape of the result curves also requires caution. It may indicate shortcomings in the tasks, heterogeneity in their difficulty, or the presence of a small group of exceptionally difficult tasks that disproportionately affect the final index. Until the cause is established, a smooth acceleration line cannot be regarded as a direct reflection of the trajectory of underlying capabilities.

Logistic, probit, and alternative statistical models

The statistical model determines how the results of individual tests are converted into an estimate of the rate of change. The logistic function produced an acceleration of about 1,9 times, the probit model produced approximately 1,8, other methods showed values from 2,6 to 2,9 times, and under certain assumptions the indicator fell to 1,1 times. The variation arose not because one model is necessarily wrong, but because descriptions of the data that are similar in quality extend the curve beyond the observed range in different ways.

According to the data presented, all the models considered described the validation samples with approximately 87 percent accuracy. This is an important qualification: equal accuracy within the available range does not prove equal reliability of forecasts beyond that range. When data are scarce, several functions can pass through the observed points equally well but produce different acceleration estimates when extrapolated.

METR retained the standard logistic method and separately acknowledged the uncertainty of the result. This choice does not turn the 1,9 indicator into the only correct value. It is more accurate to speak of a central estimate for a particular model and to show how much the conclusion changes under an alternative specification, the exclusion of tests, and changes in success thresholds.

What happened to the estimate after individual tests were excluded

Excluding tests can work in two directions. It removes tasks that create an artificial ceiling or have an unusual structure, but it also reduces the volume of data and makes the result more dependent on the remaining sample. The robustness of an estimate is tested not by a single removal but by a series of pre-described ablations that record the composition of the excluded tasks and the change in the final indicator.

The materials mention an indicator for Model 4 that changed from 1786 to 179,7 points after three tests were excluded. This notation appears internally inconsistent, since the order of the numbers is unclear and likely contains a retelling or formatting error. No quantitative conclusion can be drawn from it. The observation that the index changed after tests were removed remains useful as an indication of the metric’s sensitivity.

The IC index combines the results of many tests into a single indicator, but conceals the distribution of errors. Two models may receive similar scores if one performs well on mathematical tasks while the other gains an advantage through agentic programming. To analyze acceleration, it is necessary to see not only the final IC but also the individual curves, thresholds, number of attempts, and rules for handling missing results.

How computational resources change the result

A test result depends not only on which model provides the answer but also on how much computation it receives while solving the task. The original retelling gives approximately 2,5 million input tokens and 72 thousand output tokens for Anthropic’s internal tasks, about 10 million input tokens and 94 thousand output tokens for agentic programming, and approximately 500 thousand input tokens and 10 thousand output tokens for the Apple tests, including JPQA, AMA, and S Bench.

In long software tasks, METR’s output-token usage, according to the estimate provided, may reach 650 thousand. These quantities cannot be compared as the simple price of a single answer: input tokens include the history, code, tool results, and repeatedly transmitted context, while output tokens reflect reasoning, agent actions, and corrections. Two runs of the same model with different computation budgets effectively represent different system regimes.

The more computation a model receives during execution, the greater its chance of solving a difficult task, especially when the task allows it to verify an intermediate result and correct errors. The researchers assume that additional computation may improve the results of different model generations in roughly similar ways, but this assumption cannot be considered proven for all domains. On mathematical tasks, an enhanced mode sometimes provides a substantial improvement, whereas in poorly structured work, additional reasoning does not replace missing information or an incorrectly chosen plan.

It is important here to distinguish several sources of improvement. Acceleration may reflect growth in the model’s underlying capabilities, more efficient use of tokens, improvement of the agent framework, or an increase in the permitted reasoning budget. If new models have learned to extract more value from the same amount of computation, the observed curve will look steeper even without proportional growth in all fundamental skills.

How to read AI benchmarks without false precision

1. Check what the metric represents. The percentage of correct answers, the time horizon, the IC index, and the cost of a run measure different properties of a system.
2. Look at the test ceiling. If most models are already near 85 or 100 percent, the set distinguishes poorly between subsequent generations.
3. Clarify the computation regime. You need to know the token limit, number of attempts, access to tools, context, and the agent’s stopping rules.
4. Compare multiple statistical models. One curve may be convenient for a report, but alternative functions show the range of methodological uncertainty.
5. Separate interpolation from extrapolation. A good description of known data does not guarantee an accurate forecast beyond the observed period.
6. Look for robustness to test removal. If the conclusion changes after one group of tasks is excluded, this property of the estimate should be stated alongside the final figure.
7. Do not generalize the result to intelligence as a whole. METR’s public work studied mainly software and research tasks, while more “messy” tasks with an uncertain environment were harder for the models.

What exactly does the final figure measure? The answer depends on the study design: METR’s public study reports that the models’ time horizon roughly doubled over seven months on a selected set of software tasks, whereas the cited internal Anthropic analysis estimates acceleration using a different metric and with a substantial dependence on the statistical model. These results are consistent with the acceleration hypothesis, but they do not prove that all AI capabilities are growing at the same rate.

The most accurate formulation is this: the data point to rapid progress in a range of agentic and software tasks, while the size of the acceleration remains sensitive to the tests, computational budget, and modeling method. These conditions, rather than a single number in the headline, determine how useful the conclusion is for forecasting.

Frequently asked questions

Does an indicator of 1,9 times prove acceleration of AI as a whole?

No. It is an estimate from a specific analysis, test set, and chosen statistical model. It is not equal to the growth rate of all models, all tasks, or the entire industry.

Why do the results of different statistical models diverge?

They describe the probability of success near the threshold and beyond the observed data differently. With similar accuracy on the validation set, the functions may produce different acceleration forecasts.

Can METR's increasing time horizon be considered evidence of the emergence of an autonomous researcher?

No. The time horizon indicates the probability of completing tasks of a certain duration of human work. It does not confirm autonomy in a real scientific environment, where goal-setting, novelty verification, accountability, and working with uncertainty are important.