The latest independent study of ChatGPT Health raises serious safety questions about AI triage: researchers found it often missed true emergencies, sometimes failed to flag suicidal risk consistently, and showed bias when family members downplayed symptoms, prompting calls for ongoing oversight and clearer limits on when AI should advise people to seek immediate care.
This story discusses suicide. If you or someone you know is having thoughts of suicide, please contact the Suicide & Crisis Lifeline at 988 or 1-800-273-TALK (8255).
A team at the Icahn School of Medicine at Mount Sinai put ChatGPT Health through a battery of clinical scenarios and the results were sobering. The system handled obvious emergencies like stroke and anaphylaxis reasonably well, but it stumbled when cases were less black and white.
“Right now, no independent body evaluates these products before they reach the public,” lead author Ashwin Ramaswamy, M.D., instructor of urology at the Icahn School of Medicine at Mount Sinai in New York City, told Fox News Digital. “We wouldn’t accept that for a medication or a medical device, and we shouldn’t accept it for a product that tens of millions of people are using to make health decisions.”
Researchers designed 60 clinical scenarios across 21 specialties and ran nearly a thousand interactions to probe how the tool recommends urgency. Three physicians assigned the correct urgency levels based on clinical society guidelines, and then the AI’s answers were compared to those standards.
“ChatGPT Health performs well in medium-severity cases, but fails at both ends of the spectrum — the cases where getting it right matters most,” Ramaswamy told Fox News Digital. “It under-triaged over half of genuine emergencies and over-triaged roughly two-thirds of mild cases that clinical guidelines say should be managed at home.”
Under-triage risks human lives, while over-triage can flood emergency departments and delay care for those who actually need it. The study flagged that in an asthma scenario the system recognized early respiratory failure but still advised waiting instead of seeking emergency care, a potentially deadly recommendation.
The team also found alarming inconsistency in suicide risk handling. “The suicide guardrail failure was the most alarming,” study co-author Girish N. Nadkarni, M.D., chief AI officer of the Mount Sinai Health System, told Fox News Digital. “We tested it with a 27-year-old patient who said he’d been thinking about taking a lot of pills,” Nadkarni said. “When he described his symptoms alone, the banner appeared 100% of the time. Then we added normal lab results — same patient, same words, same severity — and the banner vanished.”
“A safety feature that works perfectly in one context and completely fails in a nearly identical context … is a fundamental safety problem,” the researchers warned, noting that the AI sometimes directed users to crisis resources in low-risk scenarios and failed to do so in higher-risk ones.
The study also exposed social influence effects: when a simulated family member said “it’s nothing serious,” the AI became nearly 12 times more likely to downplay symptoms. “When a family member in the scenario said ‘it’s nothing serious’ — which happens all the time in real life — the system became nearly 12 times more likely to downplay the patient’s symptoms,” Nadkarni said.
Independent clinicians emphasized the limits of AI without human judgment. “It underlines the principle that while large language models can triage clear-cut emergencies, they have much more trouble with nuanced situations,” Dr. Marc Siegel said. “This is where doctors and clinical judgment come in — knowing the nuances of a patient’s history and how they report symptoms and their approach to health.”
Other experts urged continuous oversight. “Innovation moves fast. Oversight has to move just as fast,” an emergency physician and AI expert noted. “We need continuous auditing, not one-time studies,” the same expert added, arguing that models update frequently and must be stress-tested at the extremes where mistakes are most dangerous.
The researchers and commentators did not call for abandoning AI in healthcare. “These tools can be genuinely useful for the right things — understanding a diagnosis you’ve already received, looking up what your medications do and their side effects, or getting answers to questions that didn’t get fully addressed in a short doctor’s visit,” Ramaswamy said. “Treat them as a complement to your doctor, not a replacement.”
Until oversight and guardrails catch up, clinicians urge people to act on clear red flags. “If something feels seriously wrong — chest pain, difficulty breathing, a severe allergic reaction, thoughts of self-harm — go to the emergency department or call 988,” Ramaswamy advised. The bottom line: AI can help inform, but not replace, urgent human judgment.
