Open AI models evaluate scientific papers: how close they are to experts

According to the review author's description, open AI models evaluate scientific papers by their title and abstract in order to preliminarily determine the quality of a publication. In a study by Michael Tevlow and Exon Mohamedi, the models were reportedly compared with expert assessments of approximately 3 000 papers in medicine, healthcare, and biology. The primary source or an exact quotation from it is absent from the available materials, so this information should be understood as a retelling of the reported experiment. Its possible practical conclusion is limited: open models may be suitable for the initial screening of literature, but they do not replace a reviewer or a substantive examination of the work.

The result matters not only for scientific journals. Researchers, libraries, and developers of literature-analysis systems gain, according to the concept presented in the review, a tool for quickly sorting a large body of publications by priority for further reading. Evaluation based on two short excerpts from a paper answers only a limited question: how convincing the reported work appears on the surface.

How the experiment with 3 000 scientific papers was structured

According to the information provided, the study by Michael Tevlow and Exon Mohamedi was published on 2 March 2026 in the journal Scient Matric. The experiment's description states that it examined how models analyze the titles and abstracts of scientific publications. According to this description, the sample included approximately 3 000 papers from medicine, healthcare, and biology. The models' assessments were then compared with expert assessments, which were used as a reference for comparison. The primary publication, a link to it, or a quotation from it is absent from the provided materials, so the date, journal name, and parameters of the experiment cannot be verified from the data presented.

Such a design, if the experiment's description was conveyed accurately, tests not the model's ability to fully review a paper, but its ability to recognize signs of scientific significance in a brief description. These include the clarity of the research question, the consistency between the stated objective and result, the nature of the sample, the description of the method, and the caution of the wording. If the abstract does not report key limitations, the model does not receive information that could change its conclusion.

A body of approximately 3 000 publications makes the comparison more meaningful than a single demonstration, provided that the sample composition and evaluation procedure are described in sufficient detail. A large sample size alone does not resolve methodological questions. To interpret the result, it is necessary to know the sample composition, the distribution of expert assessments, the criteria for scientific quality, the language of the papers, and how agreement between the model's and people's responses was measured. These details are not disclosed in the provided materials, so the result should be understood as a description of the reported experiment rather than as a universal accuracy standard.

Which open models participated in the evaluation

The experiment's description names Gemma 3, Qwen 3, and DeepSeek R1. They are presented as open or independently deployable language models, but the specific versions, model sizes, and run settings are not specified in the description provided. Such parameters are important for scientific comparison: the same model family may differ substantially in quality depending on the number of parameters, reasoning mode, prompt template, and context limitations.

In the review, Gemma 3 is classified as a family of lightweight open models from Google DeepMind. The family's technical description cited by the author lists variants ranging from 1 to 27 billion parameters, support for long context, and, for some models, image processing as well. These capabilities do not mean that every version of Gemma 3 is better at evaluating papers. For the task at hand, the selected instruction, response format, and quality of the evaluation set matter more.

In the review, DeepSeek R1 is described as a family of models focused on reasoning. According to the information provided, the published description of one DeepSeek version specifies open weights, support for commercial use under the terms of the stated license, and the possibility of local deployment with suitable infrastructure. This information characterizes the model as a tool for independent deployment, but does not confirm its accuracy specifically in evaluating scientific publications.

In the original claims, Qwen 3 is named among the experiment's participants. The available information is insufficient to establish which version of Qwen 3 was used or whether it underwent additional tuning for scientific literature. Directly ranking the three families by quality would be unjustified.

What comparison with experts means

In such an experiment, expert assessment serves as a benchmark for comparison, but not as absolute truth. Reviewers also disagree, especially when dealing with interdisciplinary work, early results, or papers with ambiguous methodology. A model may agree with most experts without understanding scientific causality, while an expert may reject work that other studies later confirm.

The key metric should describe not an impression formed from individual responses, but the stability of agreement with human assessments across the entire set. It is desirable to separately assess correlation, category-level agreement on quality, the false-positive rate, and the proportion of strong works that were missed. The provided data do not include specific metrics or numerical results, so closeness to experts should be treated as the study's reported outcome, without attributing percentages or ratings that are not present.

A separate check is needed to determine whether the model is substituting an assessment of scientific quality with an assessment of the text's persuasiveness. A well-written abstract may conceal a weak study design, a small sample size, or an incorrect statistical conclusion. Conversely, important work in a narrow field may appear unattractive to the model because of complex terminology and the authors' restrained style.

Why titles and abstracts are insufficient

An abstract does not contain all the material required for a full evaluation. It often omits raw data, code, protocol details, participant exclusion criteria, a description of the control group, and information about adverse outcomes. For medical publications, the study design, protocol registration, clinical outcomes, and conflicts of interest are especially important. These elements cannot be reliably reconstructed from a few paragraphs.

• Methodology. The model sees the stated method but does not verify whether the authors actually applied it correctly.
• Data. The title and abstract do not make it possible to independently detect errors in tables, missing data, or sample bias.
• Statistics. A brief description of the result does not replace checking calculations, confidence intervals, and the robustness of the conclusions.
• Novelty. The model may mistake a familiar formulation for scientific value even though the reported result has already been published.
• Reproducibility. An abstract cannot establish whether independent researchers will reproduce the result obtained.

What happens if the system gives a low score to a paper with an important but poorly worded result? The publication may lose an editor's or researcher's attention before the full text is even read. An automated score should direct attention, not block access to the work.

Which tasks can already be automated

The safest scenario concerns sorting rather than a final decision. A model can select papers for initial reading, group publications by topic, extract the stated methods and results, flag the absence of key information in an abstract, and prepare a brief explanation of the assigned score. This process saves time if a person retains the right to review any result.

1. The system receives bibliographic records, titles, and abstracts.
2. The model evaluates publications using a predefined scale and indicates the features that influenced its conclusion.
3. An editor or researcher checks high-priority papers and a random portion of the remaining records.
4. The team compares the model's decisions with those of specialists and adjusts the instruction.
5. Changes to model versions and prompts are recorded so that results from different periods remain comparable.

For a journal, such a system can work as a filter for incoming submissions, but not as an automated rejection mechanism. A research group uses it when preparing a literature review. For a library or analytics service, transparent labels matter more than a single quality score: topical relevance, presence of a method, study type, and need for manual review.

Why AI does not replace peer review

Peer review includes checking the argumentation, methods, data, ethical aspects, and the connection between the conclusion and the results. A language model generates an assessment based on the input text and has no independent access to the laboratory experiment, primary measurements, or the actual behavior of study participants. Even a strong model may confidently reproduce an erroneous interpretation if the authors have presented it convincingly in the abstract.

Fully replacing an expert would require not only high accuracy, but also an explainable procedure resistant to manipulation, conflict-of-interest controls, and accountability for the decision. In scientific publishing, that responsibility belongs to the editor and reviewers, not to a statistical model. Open weights simplify local deployment and system auditing, but do not in themselves guarantee the reliability of the conclusions.

According to the information provided, the experiment's outcome demonstrates the promise of open models for the initial evaluation of scientific papers. According to the review author's description, small models demonstrated results at the level of the smaller cloud models that the author calls GPT-4 Mini and Gemini 2.0 Flash. This observation concerns a specific test scenario; the primary source and numerical indicators are absent from the available materials, so it does not prove that the models are equivalent across all scientific disciplines and tasks.

The practical rule is simple: AI sorts and explains; an expert checks and decides. The higher the cost of an error, the more source material the system should see, and the more essential it is to manually check the full text, data, and methodology.

Frequently asked questions

Can an open model evaluate a scientific paper based only on its abstract?

It can provide a preliminary assessment of the reported scientific persuasiveness and relevance. Such an assessment does not confirm the correctness of the experiment, statistics, or raw data.

Are open models close to experts?

In the described experiment, their assessments of approximately 3 000 papers were compared with expert assessments. The provided materials contain neither the primary quotation, exact metrics, nor details of the procedure, so it is impossible to specify the degree of agreement or conclude that one model outperforms another.

Can such a system be used in a scientific journal?

Yes, for sorting manuscripts, topical classification, and preparing materials for an editor. An automated score should not be the sole basis for rejection, reviewer assignment, or publication.

What should be checked before implementation?

The composition of the test sample, the expert evaluation criteria, the stability of results across disciplines, the error rate, and the impact of annotation wording need to be checked. The model version, run parameters, and prompt template are recorded separately.