How Safe Is ChatGPT Health Advice? Evidence from Real User Conversations in Four Global South Countries
One in five real ChatGPT health conversations in a four-country study was rated high-risk, rising further with personal context.
Researchers analyzed 202,590 real ChatGPT conversations donated by 1,252 users in India, Nigeria, Brazil, and Pakistan, isolating 15,555 health-related exchanges. About one in five were rated high-risk for unsafe medication guidance, harmful self-diagnosis, missing warnings, or failure to refer to a professional, and risk rose further when the model's own accumulated context about the user was factored in.
Why it matters
This is the first evaluation of LLM health advice safety using real, demographically linked conversations rather than synthetic vignettes, and it shows that safety testing built on scripted prompts likely understates real-world harm, especially when a model has accumulated context about a specific user. That has direct implications for how any LLM-based health-adjacent feature should be safety-tested before deployment.
What is verified
The study analyzed 202,590 real ChatGPT conversations from 1,252 consenting users in India, Nigeria, Brazil, and Pakistan collected between December 2022 and February 2026, identifying 15,555 health-related conversations. [1]
Roughly one in five health conversations in the dataset received a high-risk safety rating, with failure modes including unsafe medication or dosage guidance, harmful self-diagnosis, missing safety warnings, over-reliance encouragement, and failure to refer to professionals. [1]
When the user's own accumulated conversation context was supplied to the safety rubric, the high-risk share rose by 13-17 percentage points, and about one in four conversations rated safe in isolation were rated unsafe when evaluated with the specific user's context. [1]
The authors describe this as the first demographically grounded safety evaluation of real-world ChatGPT health conversations, in contrast to prior evaluations built on clinician-authored vignettes, synthetic prompts or simulated profiles. [1]
What is not
- The paper does not specify which ChatGPT model version(s) produced the evaluated responses, so findings may not generalize to current model behavior.
- The risk scoring methodology relies on an LLM classifier and an LLM-based safety rubric rather than clinician review of every case; inter-rater reliability or clinician validation detail is not given here.
- No information on how representative the 1,252 consenting users are of the broader ChatGPT user base in these countries, or how donation-based sampling might bias the conversation set.
- The evidence does not state whether OpenAI has responded to or reviewed these findings.
Sources
- 1arXiv cs.CRarxiv.org · Primary source
Original source published 13 October 2026 at 16:30.
An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.