MAIDX O vs. GPT-5: What the Multi-Agent Evaluation Showed

MAIDX O is a multi-agent system for analyzing complex clinical cases and selecting a diagnosis. Microsoft reported test results published in the journal Science: on a set of 304 historical cases, the system correctly identified the diagnosis in 83% of episodes, while physicians with a reference book achieved 36%.

The impressive figures do not prove that MAIDX O is ready to treat patients independently.

The test assessed work with preassembled clinical descriptions, not the safety of diagnosis in a real clinic.

What MAIDX O Actually Tested

The main object of comparison was not a full medical consultation, but the system’s ability to analyze a complex case description and select the correct diagnosis. The model receives symptoms, medical history, test results, and other information, then produces a diagnostic conclusion.

Real-world diagnosis is more complex. A physician gathers incomplete information, talks with the patient, assesses urgency, performs a physical examination, verifies the patient’s identity and medical record, selects additional tests, explains the decision, and monitors subsequent actions. In the benchmark presented, the primary outcome was the correctness of the answer for a historical case, not the safety of the complete clinical process.

What does the 83% figure mean? The system selected the correct diagnosis in 83 out of one hundred cases within a specific dataset and under specific test conditions. This result is not equal to the probability of making the right decision for any new patient, because it is unknown how well the case mix represents different clinics, countries, age groups, and rare combinations of diseases.

How the Multi-Agent System Is Structured

The MAIDX O architecture includes a base large language model, five interacting agents, and an orchestrator. The base model generates and analyzes text, the agents perform different reasoning stages, and the orchestrator coordinates the exchange of results and creates the overall process.

The five agents should not automatically be regarded as five independent physicians. In this architecture, an agent is a software component with an assigned role, instructions, and a method for exchanging messages. Several agents can check one another’s hypotheses, look for contradictions, suggest alternative diagnoses, and reassess weak points in the reasoning. The multiplicity of components by itself does not guarantee medical correctness.

A multi-agent framework is useful where a single answer generation is insufficient. The orchestrator divides the task into stages, while repeated checking reduces the risk that the first plausible diagnosis will go without critical analysis. If all agents rely on the same erroneous premise, however, the system’s consistency creates a false sense of reliability.

Analysis of the 83%, 79%, and 36% Figures

Comparing MAIDX O with GPT-5 without an agent framework yields a difference of approximately four percentage points. According to the author’s analysis, this level of improvement can be associated with the work of the five agents and the orchestrator, if the other experimental conditions were indeed identical. This is a more cautious conclusion than claiming that the multi-agent architecture is twice as effective as the base model.

The difference between MAIDX O and physicians with a reference book is 47 percentage points. It does not describe a contest between the system and a full clinical team. The physicians solved the task under conditions established by the researchers, not during a normal consultation process, where colleagues, laboratory data, diagnostic imaging, the electronic medical record, and the ability to observe the patient over time are available.

Why the Comparison with Physicians Requires Caution

The phrase “AI is better than physicians” is too broad for these data. It is more accurate to say that MAIDX O achieved a higher result than physicians with a reference book on a specific test of challenging historical cases. The outcome is influenced by the question format, time allowed for the answer, available materials, composition of the physician group, and method used to determine the correct diagnosis.

A reference book does not replace medical infrastructure. In real-world practice, a physician compares complaints with examination results, considers contraindications, assesses the quality of test results, and decides which action is safer when information is incomplete. A study that measures only the selection of a diagnosis from a description does not demonstrate the ability to manage a patient from the initial consultation through treatment.

In a clinic, the cost of an error is not uniform. Missing respiratory failure requires a different risk assessment than making an error in a less urgent differential diagnosis. A medical system needs not only average accuracy, but also sensitivity to dangerous conditions, the rate of unjustified recommendations, the quality of refusing to answer, and the ability to refer a person to a physician in time.

What the Five Agents May Have Contributed

A multi-agent approach can improve results by distributing functions. One component generates a list of hypotheses, another looks for findings inconsistent with the initial version, a third checks the consistency of the clinical reasoning, and the orchestrator combines the conclusions into a single answer. The specific distribution of roles depends on the implementation of MAIDX O, so it cannot be replaced with a general reasoning framework.

• The agents can force the system to consider several diagnostic hypotheses.
• Repeated checking helps identify internal contradictions in the explanation.
• The orchestrator manages the order of message exchanges and the selection of the final conclusion.
• Separating the stages makes it easier to audit where an error occurred.

The improvement over a single model was limited: nearly 79% versus 83%. This result by itself does not indicate a shortcoming of the approach. The base model already performs well on the task, so additional agents may provide a small improvement while also increasing response time, computational costs, and oversight complexity. For a clinic leader, not only the percentage of correct answers matters, but also the cost per verified case, result latency, and consequences of a rare error.

The Risk of Cases Leaking into Training Data

Historical clinical cases may previously have appeared in scientific publications, educational materials, or other public sources. If similar descriptions were present in the model’s training data, the system may have reproduced a familiar pattern rather than demonstrated the ability to generalize to a new case.

This does not mean that the result is necessarily invalid. Without testing on new cases kept hidden from the developers, it is impossible to distinguish clinical reasoning from memorization or partial overlap with known text. According to the data provided, the models in the described trial were not specifically fine-tuned on medical knowledge and were not connected to specialized medical libraries or knowledge bases. These conditions make the test easier to interpret, but they do not eliminate the risk of dataset contamination.

Historical Cases Do Not Replace Testing on Patients

MAIDX O was evaluated on historical data, not on living patients. The system was not responsible for calling an ambulance, prescribing medication, selecting a dosage, obtaining consent, protecting personal data, or delivering a conclusion to a specific physician.

A prospective trial is structured differently. First, the system’s operating rules are fixed; it is then tested on new consultations that were unavailable to the developers and the model during preparation. Researchers must define clinical outcomes, the procedure for physician intervention, stopping criteria, and the process for reviewing dangerous errors in advance. Only this design makes it possible to assess not just diagnostic accuracy, but the tool’s impact on decisions and patient health.

Even a prospective test does not resolve every issue. Deployment will require testing in different clinics, in different languages, and across different healthcare systems, as well as assessing resilience to incomplete, contradictory, and incorrectly attached data. A separate control process is needed for patient identification: the experiment involving the substitution of medical-record data, mentioned in the materials, shows that an agent can correctly identify a disease while simultaneously associating the conclusion with the wrong person.

What Tests Are Needed Next

1. New closed cases. The dataset should be assembled after the system version has been fixed and should not enter model training.
2. Different clinical settings. The testing should cover multiple institutions, specialties, languages, and levels of available diagnostic resources.
3. Dangerous scenarios. Separate tests are needed for sepsis, respiratory failure, stroke, severe allergic reactions, and other conditions where delay is critical.
4. Comparison with physicians under equal resources. Physicians without a reference book, physicians with a reference book, and physicians with access to clinical information systems should be evaluated separately.
5. Time and cost. The study should report response latency, the number of requests to the model, and the cost per analysis, not just final accuracy.
6. Safety of actions. The system should be tested on whether it can decline to provide a conclusion, request missing information, and transfer the case to a human.

Practical Takeaways for Clinics and Developers

MAIDX O demonstrated a strong result on a diagnostic benchmark. The system outperformed physicians with a reference book on the stated set of 304 challenging cases, while the multi-agent architecture added approximately four percentage points to the base model’s result.

But these data do not turn MAIDX O into an autonomous virtual physician. They do not demonstrate that the system is ready to independently see patients, prescribe treatment, or work without clinical oversight. For a clinic leader, the question is different: does such a tool help a physician identify dangerous possibilities more quickly, and does it reduce the number of errors in the real-world process?

The published benchmark does not answer that question. Independent testing on new cases, analysis of high-severity errors, assessment of costs and time, and a prospective study involving real patients and responsible medical personnel are needed.

FAQ

Is MAIDX O better than GPT-5?

In the comparison presented, MAIDX O achieved 83%, while GPT-5 without an agent framework achieved nearly 79%. This indicates an advantage on a specific dataset and under specific test conditions, not universal superiority in all medical tasks.

Did the five agents provide most of the improvement?

According to the figures provided, the difference from the base model is approximately four percentage points. In the author’s assessment, this improvement can be associated with the agent architecture, but a single comparison does not prove which component produced the main effect.

Can MAIDX O be used for autonomous diagnosis?

There is no basis for that conclusion. The trial used historical data and did not test treatment safety, work with living patients, protection of medical data, or responsibility for the clinical decision.