How to Evaluate AI Agents for Computer Control: Benchmarks, Actions, and Hidden Modes

Evaluating an AI agent for computer control means comparing how the model perceives an interface, plans steps, and performs a task in applications with minimal human involvement. A single percentage of successful runs is not enough: the result depends on the task set, available tools, memory, reasoning mode, stopping rules, and what the organizers count as success.

The main practical question is different: can the agent reliably complete the required workflow in your environment, rather than merely solve a similar task in a laboratory sandbox?

What Tasks Different Benchmarks Measure

Based on the author's description of the test, Screenshot Pro evaluates an agent's work using screenshots and actions in a graphical interface. This type of evaluation is closer to computer control: the model must recognize window elements, choose coordinates or a command, and continue after the screen state changes.

In the experiment as recounted, Astra scored almost 93% in Screenshot Pro, while the earlier GPT-5.6 Soul scored about 77%. These values relate to a specific test protocol attributed to the author and do not constitute a universal model ranking. Without the task composition, number of runs, error-handling rules, and confidence intervals, the difference between the systems remains an indication rather than conclusive evidence of superiority.

In the review author's account, OSWorld evaluates task completion in computer applications. In the comparison presented, the score rose from approximately 66% for Soul to 73% for Astra. The author classifies this test as being closer to a user scenario than an abstract reasoning task. However, without the primary protocol description, it is impossible to independently confirm the full task composition or apply the result to a specific corporate process with real access rights, data, and constraints.

Automation Bench is intended for longer professional workflows. In the comparison described by the author, Soul completed about 18% of the tasks, while Astra completed about 41%. If the tasks really included multiple applications, intermediate checks, and long sequences of actions, this result indicates a noticeable increase in robustness on complex routes. It does not show how long each attempt took, how much processing cost, or who corrected errors.

In the review author's account, ARC AGI belongs to a separate category of tests that evaluate solving new abstract tasks rather than controlling familiar office applications. According to the author, Astra scored 63% in standard mode, while in a special mode that preserved its own notes and internal reasoning, the result reached 99,9%. These figures cannot be directly compared with OSWorld or Screenshot Pro because, according to the author's description, they measure different capabilities and use different conditions for access to past state.

Why Percentages from Different Tests Cannot Be Added Together

The percentage of successful tasks is meaningful only within the described protocol. A value of 73% in OSWorld does not mean that the agent will complete 73% of any office tasks, and a score of 93% in Screenshot Pro does not turn the model into a universal computer operator. The outcome is affected by scenario complexity, the interface, the availability of structured tools, the method used to verify the result, and the permitted number of attempts.

A proper comparison begins with a conditions matrix. Record the model name, version, system instructions, available tools, permitted memory, number of attempts, time limit, success criterion, and whether the operator intervened. If even one of these parameters changes, the results belong to different experiments.

Actions, Time, and Human Intervention

Overall accuracy describes only one layer of quality. An agent may complete a task correctly but make hundreds of unnecessary clicks, open the wrong windows several times, spend hours searching for an element, or stop after a human hint. For deployment, this result is often worse than that of a slightly less accurate model that acts briefly and predictably and hands control back to the operator at a clear point.

The account of the experiment states that on 96% of ARC AGI levels, Astra used fewer actions than the average person. This metric applies to a specific set of levels and to the definition of an action used by the organizers. It does not mean that the model is faster than a person in all interfaces: a click, text entry, scroll, tool call, and internal planning step may be counted differently.

A practical metric should separate at least four quantities: the share of successfully completed tasks, median time, number of actions, and number of requests for human help. For a long scenario, it is useful to add the cost per successful result, the share of irreversible errors, and the time the operator spends checking the work. If a task lasts 4–8 hours, a single high completion percentage says little about readiness for autonomous work when more than half of successful tasks require at least one human intervention, as claimed in the description of the experiment conducted.

And what happens if the action log does not contain the moment when control was handed to a human? Such a test cannot be properly compared with a system that records every confirmation, correction, and retry. The report should specify not only whether intervention occurred, but also its type: a hint, input correction, window unblocking, confirmation of a dangerous action, or the operator taking over and completing the task.

Memory, Notes, and Special Modes

The difference between standard and special ARC AGI modes shows how strongly the protocol can change the result. If notes, intermediate conclusions, or internal state persist between prompts, the agent receives an additional resource that a system with a clean context does not have. This is useful for studying the architecture, but such a result must not be presented as the model's quality in ordinary mode.

According to the author, ARC researchers described Astra's behavior as constructing a compact symbolic model of an unfamiliar environment and introducing its own abbreviations for planning. This observation should be presented specifically as the researchers' interpretation, as recounted by the author, rather than as an established universal property of all runs. To test the hypothesis, it is necessary to preserve traces, compare modes with and without notes, and check whether the agent uses the discovered symbols in new tasks.

Persistent memory can increase not only the model's ability but also its accumulated advantage. An agent that has already recorded the interface structure, button names, and a successful route solves the next scenario differently from an agent starting from a clean state. For this reason, reports distinguish between a mode without memory, a mode with external notes, a mode with the history of previous attempts, and a mode with internal state, if the organizers provide it.

A Template for Your Own Testing

A company choosing a computer agent for deployment should build a test around a set of repeatable workplace scenarios, not an impressive demonstration. Each scenario should include an initial state, input data, the expected result, permitted actions, and safe stopping conditions.

1. Describe the task. Record the objective in terms of the result: create a report, find a record, transfer data, or prepare a draft. Do not limit it to the wording “work in the CRM.”
2. Prepare an identical environment. Use separate test accounts, predefined permissions, identical application versions, and a cleared browser history.
3. Separate the modes. Run separate series without memory, with an external log, with permitted tools, and with ordinary screen-based control.
4. Repeat each scenario. One successful run does not demonstrate robustness. Compare a series of attempts on new data and record the reasons for each failure.
5. Record the complete log. Save screen states, actions, time, errors, confirmation requests, computational cost, and the moment of human intervention.
6. Check safety. Test deletion, email sending, changes to financial data, publication of materials, and actions involving personal information separately.

It is best to divide scenarios into short, medium, and long. A short scenario tests the accuracy of an individual action, a medium scenario reveals planning errors, and a long scenario shows memory degradation, the accumulation of incorrect decisions, and the burden on the operator. This test design helps identify exactly where the agent loses quality instead of hiding all failures in a single final number.

What Conclusions Are Permissible

The results presented by the author support a cautious conclusion: according to the stated values, Astra handled a range of tasks in graphical interfaces, computer applications, and complex workflows better than the Soul being compared. It is also possible to discuss the hypothesis that memory, notes, and additional internal steps dramatically change the result on ARC AGI tasks.

It cannot be claimed that Astra has become artificial general intelligence, has completely replaced humans, or can reliably control any computer. The author's account also includes the caveat that ARC AGI results should not be regarded as evidence that artificial general intelligence has emerged; this point is not supported by a separate primary source in the materials provided. Scores from abstract tasks cannot be transferred to medicine, finance, or other fields without separate verification of authority, accuracy, and accountability.

Economics should be calculated per successful task, not per request. In the author's account of the experience, the costs of intensive research were estimated as high, reaching hundreds and thousands of dollars per day for certain categories of users, but these amounts are not confirmed by independent materials and should not be used as a universal rate. In your own project, calculate expenses based on the number of attempts, context length, screen operations, reruns, and human oversight time.

Frequently Asked Questions

Which benchmark should you choose for a computer agent?

Screenshot Pro and OSWorld are suitable for interface actions, Automation Bench for long professional processes, and ARC AGI as a separate test of transferring abstract skills as interpreted by the review author. An automation lead still needs a custom set of workplace scenarios.

Can 93% in Screenshot Pro and 73% in OSWorld be compared?

No. These percentages refer to different task sets and protocols. They can be shown side by side as context, but a general ranking cannot be calculated from them without a unified methodology.

Why is human intervention more important than an impressive demonstration?

The operator performs the hidden part of the work: correcting errors, suggesting the next step, and confirming dangerous actions. Without accounting for this assistance, the agent’s autonomy will be overstated.

What should be checked before deploying it in a business?

Check reproducibility, access rights, the action log, safe shutdown, the cost of a successful outcome, error handling, and performance on new data. Pay particular attention to personal information and irreversible operations.