Can Medical AI Be Trusted for Urgent Symptoms: Where ChatGPT Health and Crisis Models Fail

Medical Triage means an initial assessment of the urgency of a condition. It determines whether a person needs immediate help, medical attention soon, or a routine consultation. In triage, the error of “waiting” is more dangerous than an ordinary factual inaccuracy: a wrongly named diagnosis can be corrected at an in-person appointment, whereas missed respiratory failure or a severe metabolic disorder may go unnoticed until it is too late.

The studies cited in the source materials do not justify considering a medical chatbot an independent emergency dispatcher.

It helps describe symptoms and prepare questions for a doctor, but when there are dangerous signs, it should not be the only channel for making a decision.

What Medical Triage Checks

Triage answers not only the question “what diagnosis is this?” The system must determine urgency, notice signs of deterioration, clarify the context, and avoid creating a false sense of security. This requires information about age, underlying conditions, medications, when the symptoms appeared, how they are changing, and what means of obtaining help are available.

When someone writes “I’m having trouble breathing” or “my chest hurts badly,” a single polite recommendation is not enough. It is important to ask about the suddenness of the symptom, blue lips, confusion, fainting, fever, injury, and treatment already taken. Even a complete response from the user does not turn a language model into a doctor: the model does not examine the person, measure vital signs, or may misjudge the significance of an individual detail.

ChatGPT Health and Urgency Assessment

In the assessment described, which the authors of the materials link to a study in Nature Medicine, an early version of ChatGPT Health gave a dangerously low urgency rating in 33 of 64 responses to four scenarios requiring immediate care. In these cases, the system recommended seeing a doctor in a day or two, even though the scenarios involved severe metabolic disorders and incipient respiratory failure.

The figure 33 out of 64 cannot be generalized to all users and all symptoms. It is the result of a specific test set, not a measurement of the error rate in a real medical service. But the type of failure itself has practical significance: the model could produce a calm and convincing response where delaying medical attention was becoming dangerous.

A later assessment examined 255 medical cases, including de-identified real requests for emergency care, as described in the source materials. ChatGPT Health’s urgency assessment matched the experts’ assessment in 54% of cases. After additional questions, the match fell to 48%.

These figures do not mean that additional questions are always harmful. They show something else: a longer dialogue by itself does not guarantee accurate triage. If the model chose the wrong line of reasoning, new user responses could reinforce an erroneous interpretation, while a confident conversational style concealed the continuing uncertainty.

Why Additional Questions Do Not Always Improve Safety

An additional question is useful when it checks a specific risk and changes the next action. For example, information about loss of consciousness, worsening shortness of breath, or sudden weakness on one side of the body should shift the conversation from an informational consultation to urgently seeking medical help.

The problem arises when the dialogue turns into lengthy questioning without a clear escalation threshold. A person may decide that since the chat is continuing to ask questions, the situation must not yet be dangerous. The model may also give equal weight to a minor detail and a sign that requires an immediate response.

What should you do if symptoms worsen rapidly during the conversation? Do not continue messaging in pursuit of a more precise answer; seek emergency medical help through the local emergency number and follow the dispatcher’s instructions. In cases of loss of consciousness, severe shortness of breath, intense chest pain, signs of a stroke, seizures, a severe allergic reaction, or suspected poisoning, a chatbot does not replace emergency services.

Psychological Crisis: Direct Words and Indirect Signals

In the study, approximately 2 000 messages about psychological crises were assessed, and the five language models most often responded correctly to a direct statement of an intention to harm oneself. This result shows that an explicit formulation of risk helps the system recognize the situation and choose a safer response scenario.

Indirect hints and ambiguous phrases were handled worse. Some models responded inappropriately or potentially dangerously when the user did not explicitly write about an intention to die by suicide or harm themselves. The research set assessed not only suicidal statements, but also self-harm, an anxiety crisis, violent thoughts, substance-use problems, and risky behavior.

This result cannot be turned into a ranking of a “safe model” for a particular person. In the study, responses were evaluated according to a predefined protocol, and the model did not have the person’s full conversational history or the ability to check the actual circumstances. Even an appropriate response in a test does not prove that a chatbot will correctly assess risk in a live dialogue, where the user changes their wording, withholds circumstances, or suddenly stops responding.

If a person says directly or indirectly that they may harm themselves, the response should include help from a real person: contacting emergency services, a crisis line, or someone close who can remain with them. A chatbot may help formulate a message to a professional, but it should not keep a person in an endless dialogue or create the impression that the crisis is already under control.

Errors in Medical Documentation

The risks of medical AI are not limited to advice given to users. The materials describe cases in which documents prepared with the help of AI contained mixed-up medications, incorrect diagnoses, and false medical-history information. In one reported case, a patient was mistakenly said to have used hallucinogenic mushrooms, and a doctor later checked and corrected the record.

An erroneous entry is especially dangerous when the next specialist treats it as a confirmed fact. The name of a medication can affect prescribing, an incorrect diagnosis can change the patient’s care pathway, and an invented detail in the medical history can sometimes influence a decision about hospitalization or testing. A polished document does not prove that its contents are accurate.

For a clinic, a safe process requires checking every medication, dosage, allergy, diagnosis, test result, and patient-identification detail against the original source. The doctor who signs the document must see the original record and correct errors before incorporating the text into the medical documentation. Automatic generation does not transfer responsibility for verification to the software.

A Misassigned Record Can Lead to the Right Diagnosis for the Wrong Patient

In an experiment with six language models, researchers deliberately replaced identification details in electronic medical records. The agents often continued working without noticing the error. Even the best model refused to continue actions in only 17% of tests with completely mismatched record headers.

The significance of this result goes beyond an ordinary text-recognition error. The system could correctly identify the disease from the presented data but attribute the conclusion to the wrong person. This type of failure concerns patient identification and data-integrity controls, not the model’s medical knowledge.

It is important not to conflate levels of evidence. The data-substitution experiment was an artificial test, not an observation of real treatment cases. It does not show how many patients in clinics were actually harmed by such an error. But it does demonstrate a verifiable vulnerability: an agent allowed to act on medical records must check not only the meaning of the text, but also whether the record corresponds to the specific patient.

Which Data Reflect Real-World Practice and Which Reflect Tests

• De-identified real requests. The description of the ChatGPT Health assessment included cases involving requests for emergency care, but the result still depended on the selected sample and the method of expert evaluation.
• Test scenarios. Four scenarios involving an immediate threat to health made it possible to test responses to preselected dangerous conditions, but they do not describe the full range of symptoms.
• Research messages about crises. A set of approximately 2 000 messages was created to assess models’ responses to different categories of psychological risk, not to provide real-time assistance to participants.
• Artificial record substitution. The experiment with identification data tested the resilience of agentic systems to context errors. It was not a study of actual patient treatment.

Separating these categories protects against the erroneous conclusion that “the model makes mistakes in every other real case” or, conversely, that “the model has demonstrated clinical safety.” A test percentage characterizes a specific methodology, data set, and system version at the time of testing.

Minimum Safety Measures for Clinics

1. Verify the patient’s identity using an independent identifier before reading, changing, or saving a record.
2. Check medications, doses, allergies, diagnoses, test results, and dates against the original sources.
3. Keep an AI-prepared draft separate from the signed medical record.
4. Set up mandatory human review for urgency assessments, treatment orders, and documents that affect subsequent care.
5. Conduct tests with deliberately mismatched names, headers, and record contexts before expanding the agent’s access.
6. Record what data the AI received, what conclusions it proposed, and who confirmed the final decision.

A clinical system must be able to stop if identifiers conflict, the data source is missing, or risk cannot be ruled out. Simply continuing to operate increases the likelihood that convincing text will be mistaken for a verified medical conclusion.

Practical Takeaway for Users

A medical chatbot is suitable for organizing complaints, preparing a list of questions, and explaining information already received from a doctor. It can also help compile a symptom timeline, list medications, and draft a message for a clinic, provided the user checks every detail.

When the question concerns urgency, the choice should be conservative. In the case of dangerous or rapidly worsening symptoms, seek medical attention or emergency services rather than checking the answer with several models. In a psychological crisis involving a risk of self-harm, contact with a real person and urgent help are necessary.

The main conclusion of the studies is not that medical AI is useless. A convenient dialogue, additional questions, and a confident tone are not equivalent to clinical reliability. Until a system can reliably recognize urgency, preserve the correct patient identity, and correctly process an indirect crisis signal, its role should remain supportive, and the final decision should be made by a person with access to medical data and real help.