‘I Tell People Not to Fully Trust AI for Health Concerns’

People used to talk about Dr. Google, but now it’s Dr. Chatbot. I tell people, if you want to use a chatbot for health-related concerns, go for it—but you need to use it very carefully. You must check the information with a doctor or another expert. You cannot just trust in the AI 100%. 

It can be dangerous going to a chatbot to get answers related to your health because it can give you misinformation. The chatbot gathers information not only from validated medical literature, but also from online forums, social media, platforms like Discord, and other sources. You don’t know what references the chatbot used. If you ask for references, the chatbot might start to hallucinate and provide made-up references. It can be hard for the layperson to tell whether references are made up, but your doctor would be able to see that a journal name is wrong or the volume number is way too high.

Why Chatbots Feel Trustworthy

We are already dealing with a trust paradox in medicine, with people placing more weight on information they get from social media than from experts. AI presents an even bigger challenge because its responses sound believable and trustworthy, while its sources remain hidden.

One reason chatbots feel trustworthy is that they have a tendency to agree with you, which is called sycophancy. People like that. Most people want someone saying, “You are right.” Human beings are always looking for empathy and AI does that very well. But you must know how to respond to that. If you’re not an expert in this field and you don’t have a lot of experience using chatbots, be careful and don’t let sycophancy override your own critical thinking skills. 

How Patients Can Use AI 

There are ways to use AI that minimize the risk of misinformation. Here are my suggestions:

• Start with a good prompt. If you don’t know how to describe your problem, you’re more likely to get a wrong answer.
• Create your own chatbot (such as “kidney expert”) or give the chatbot specific journal articles to use for information. If you do that, most of the time you get very good information. 
• Be wary if you’re not using English. If you write a question in another language, you might not get the same information.
• When you are in a chat for a long period, you might see the chatbot slow down and start to give random answers. If you want a more accurate answer, just start a new chat with a new prompt.
• Never trust information from a chatbot 100%. Your doctor is the expert and the person you should trust.

How Providers Can Use AI Together

Providers sometimes have patients coming in with misinformation they got from AI. If the patient believes what a chatbot has told them, a provider can use AI to show them the information is wrong by using an AI tool like NotebookLM to summarize information from specific scientific papers and resources. 

For providers using AI for clinical or research purposes, it’s critical to use a framework or protocol that are already validated in the literature, with the goal of reducing risks and preventing harm to patients. This is especially important in the case of large language models, since they can hallucinate and make mistakes. For that reason, at least at the present stage, human oversight by a qualified specialist remains essential to critically review and validate the information generated by artificial intelligence.

Carlos Fernando Mourão is an assistant professor in the Department of Basic & Clinical Translational Sciences at Tufts University School of Dental Medicine.