OpenEvidence vs. Atropos ChatRWD: Comparing Medical AI on Clinical Questions

OpenEvidence and Atropos ChatRWD are medical AI systems designed to work with complex clinical questions, but they address this task using different knowledge sources and search mechanisms. It is incorrect to compare them solely as two language models: the final answer depends on the service’s architecture, access to medical literature and de-identified medical records, search tools, and rules for formulating conclusions.

The main question here is not which system is “smarter.” What matters is whether access to large collections of real-world medical data helps answer questions for which published research is insufficient.

What OpenEvidence and ChatRWD Are Being Compared On

OpenEvidence is positioned as a tool for searching and processing medical knowledge that physicians use when making clinical decisions. In this model, the answer is based on scientific publications, guidelines, reviews, and other sources of evidence-based medicine. The system should quickly connect a physician’s question with available materials and present the conclusion in a convenient format.

Atropos ChatRWD belongs to systems that use real-world healthcare data, or real-world data. According to Atropos Health, ChatRWD has access to more than 300 million medical records and can apply this information to research, including analysis of poorly studied clinical questions. The statement indicates the platform’s scale, but by itself does not prove that every medical record is available to the system in full or used in every answer.

In the review presented by the author, Alexandria is described as a library of medical evidence automatically generated from real-world medical records. The review’s author also links this product to Atropos. However, the description provided by itself does not confirm that the company launched the product and does not allow this description to be treated as an established corporate fact. Within the review, it should be understood as a presentation of the author’s position, not as an independent verification.

How the Comparison Was Conducted

In June 2025, nine physicians evaluated the responses of OpenEvidence and Atropos ChatRWD to 50 clinical questions. According to the author’s account, ChatRWD produced a clearly better result. However, the available description does not include the full set of questions, the scoring scale, the distribution of scores, information on inter-rater agreement, or a statistical test of the differences.

The wording “clearly better” therefore remains the author’s assessment of the result, rather than universal evidence that ChatRWD is superior for all medical tasks. A small expert comparison may show a notable gap on a selected set of questions, but it does not replace an independent study with a prespecified protocol, multiple specialties, source controls, and testing of responses on new clinical examples.

Even details of how the task is framed affect the result. If the question requires finding a current guideline, the advantage goes to a system with rapid access to literature and good citation practices. When the issue involves a rare combination of diseases, therapy, and outcomes, a platform that analyzes similar de-identified medical records and shows observed courses of illness may be more useful.

Why Real-World Medical Records May Provide an Advantage

Scientific literature generally answers questions about the demonstrated efficacy of an intervention, comparisons of treatment methods, and the results of controlled studies more effectively. Its strength lies in describing methodology, patient inclusion criteria, and statistical evaluation. The limitation arises where the disease is rare, the population is insufficiently represented in studies, and the clinical situation is too complex for a standard protocol.

Medical records contain a different type of information. They can be searched for combinations of diagnoses, age-related factors, concomitant therapy, laboratory values, and real-world outcomes that rarely appear in a single publication. In the author’s view, this may have explained ChatRWD’s advantage in the comparison involving 50 questions: a system with practice data could identify empirical patterns where the literature provided an incomplete answer.

But a large database is not the same as a high-quality evidence base. A medical record is created to treat a particular patient, not to serve as an ideal study, so records may be incomplete, heterogeneous, and dependent on the practices of a particular clinic. An observed association between treatment and outcome also does not prove causation: the result is affected by disease severity, the physician’s choice, treatment availability, and many factors that may not have been included in the dataset.

Literature and De-Identified Records: Different Strengths

The most useful approach is not to replace literature with medical records, but to compare them systematically. An article shows what researchers have already tested and how reliable the study design was. Real-world records help show how similar decisions appeared in practice and what outcomes were observed in patients who were not fully similar to participants in clinical studies.

What Is Known About the Scale of OpenEvidence

According to the data cited by the author, in June 2026 more than 700 thousand medical professionals were actively using OpenEvidence, and over 2,5 years more than half a million physicians used the system at least once. These figures are presented as service-usage data and require separate verification of the counting methodology: an active user, a registered professional, and a physician who opened the system at least once are not the same category.

OpenEvidence’s popularity indicates demand for its interface, search speed, and ability to accept medical questions in ordinary language. It does not prove the accuracy of every answer or show how often physicians correct the system’s conclusions. Evaluating clinical usefulness requires separate measures: answer correctness, completeness of citations, the ability to recognize uncertainty, and the tool’s effect on the physician’s decision.

What Is Unknown About OpenEvidence’s Design

The exact architecture of OpenEvidence is not disclosed in the description presented. Claims about multiple language models, medical fine-tuning, and access to specialized sources are presented as assumptions rather than as information confirmed by the developer.

This limitation is important for the comparison. A service’s name does not reveal which model answers the question, which documents it receives before generation, how the system verifies citations, or whether it uses different models for search, analysis, and the final text. Even the same underlying algorithm can produce different results with different source sets and search settings.

According to the author’s assessment, OpenEvidence and ChatGPT do not use collections of hundreds of millions of patient medical records as a knowledge base. This wording should be understood specifically as the author’s assessment of the available information, not as a complete technical audit of all product versions and internal data-processing environments.

Data Privacy and Quality

Access to medical records creates additional privacy requirements. De-identification reduces the risk of revealing someone’s identity, but it does not turn the data into risk-free material. Rare diagnoses, unusual combinations of procedures, and the timing of events can sometimes enable re-identification when matched with other datasets.

Clinics, researchers, and developers must understand what data is sent to the system, who controls access, where records are stored, and whether queries are used for further training. These questions cannot be replaced by a general promise of “secure operation.” Access rules, activity logging, role-based restrictions, and procedures for deleting or blocking data are needed where provided for by the contract and applicable law.

The risk of sampling bias also cannot be excluded. If the database primarily reflects large clinics, a particular country, a specific insurance system, or a particular patient group, the findings may not transfer well to another population. A system that identifies a pattern in American medical records does not automatically obtain proof that the pattern applies to a patient in another healthcare system.

How Physicians Should Use Such Systems

Medical AI is reasonably used as a tool for generating hypotheses, searching the literature, and structuring information, rather than as an independent diagnostician. The physician formulates the clinical question, verifies the underlying data, asks the system to distinguish facts from assumptions, and then checks the conclusion against guidelines, primary publications, and the specific patient presentation.

• First, the task must be defined: differential diagnosis, therapy selection, outcome prediction, or assessment of a rare combination of factors.
• Next, it is necessary to check which sources the answer is based on and whether the system provides verifiable publications or a description of the research sample.
• When drawing a conclusion from medical records, an observed association must be distinguished from demonstrated causation.
• Conflicting recommendations should be analyzed in light of age, comorbidities, drug interactions, and the urgency of the condition.
• The final decision is made by the medical professional who is responsible for assessing the patient and selecting the course of action within the scope of their competence.

What should be done if ChatRWD offers a convincing conclusion that is absent from the guidelines? Such a result should be treated as a research hypothesis. It may be used to search for publications, plan an analysis, or discuss the matter with colleagues, but it must not be turned into a ready-made prescription without verifying the origin of the data and its clinical applicability.

Final Comparison

Based on the data presented, ChatRWD may have gained an advantage specifically from its knowledge source, rather than solely from the language model. However, this explanation remains the author’s hypothesis until the full comparison protocol, numerical results, question set, and information about which data the system used for each answer are published.

OpenEvidence and ChatRWD have different utility profiles. The former approach is convenient for navigating the medical literature and guidelines, while the latter is potentially strong at analyzing real-world practice and identifying patterns in large collections of medical records. The most convincing medical system should combine both layers, show the provenance of its conclusions, indicate uncertainty, and leave the final clinical assessment to the physician.