Artificial Intelligence in Medical Research

At MGH's Treadwell Library, we had a physician come to us with an interesting conundrum. They were reading a literature review but couldn't find the papers it referenced. As it turned out, the authors used ChatGPT to perform the literature search, and the majority of its citations were fake! To add insult to injury, one of the fake papers included an editorial board member from the journal as an author. 

Clearly, AI usage is not without risks. This guide will outline those, as well as best practices to follow if you use an AI tool in research or clinical practice. When it comes to AI, the key is to “just say know.” The more knowledge you have about its risks, the better prepared you will be if you use such a tool.

• That paper doesn't exist! Literature searches using AI result in fabricated and fake citations.
• Not provable. Experiments are provable because they are done once, then again, and produce the same results. This is not true for AI. Ask the question in a different location or browser, and you get a different answer.
• Not transparent. Transparency is a critical element of research. Generative AI's decision-making (even open source) is not transparent because users "cannot interpret what happens within each layer of the model when it’s active." 
• Probability not logic. AI's answers are "generated by predicting and inferring which word comes next in a sequence. It cannot apply logic or reasoning. The output is 'linguistically plausible' without regard for truth."
• Not objective. AI tools are not entirely unbiased or objective. For example, one study found AI-based commercial gender classification systems performed better on light skin, male faces. 
• Contradictory answers. A slight variation in your search query can produce wildly different answers:

• Citation issues. AI search engines are prone to citing syndicated sources like Yahoo News or AOL instead of original studies.

• Watch what you type! You cannot put any protected health information into an AI tool or feed it a paper you are peer reviewing.
• Copyright issues. Harvard librarians explain: "There are a number of lawsuits that allege Generative AI tools infringe copyright and it remains unclear if and how the fair use doctrine can be applied." 
• Legal liability. If an AI tool generates misinformation you pass on to your patient, you're opening yourself up to potential litigation.

• AI negatively impacts learning. Individuals develop "shallower knowledge' when learning about a topic from an AI summary versus a traditional web search. 
• Generative AI is not a search engine or a research database. Harvard librarians explain that "generative AI tools are designed to 'generate' their own content or output" unlike a repository of source material. 
• Requires disclosure. If you use an AI tool when writing a manuscript, you must disclose this to the publisher and agree to be responsible for the accuracy of whatever information the AI tool produces.
• Leave literature searching to the experts. Search strategies developed by ChatGPT have been found to be "unusable" and include fake MeSH terms. If you need a search done and are short on time, we can do it for you! 

• Pick the right tool. Some considerations:
  • Enterprise tools tend to have better privacy terms than free tools.
  • Some tools claim to have fewer hallucinations. For example, OpenEvidence claims not to respond if the evidence is inconclusive. But that doesn't mean it's perfect: "When OpenEvidence took the US Medical Licensing Exam... it was wrong 9% of the time." 
  • Premium chatbots have been found to provide more incorrect answers than free versions.
  • Be cautious which tool you select. DeepSeek is prohibited across MGB.
• Opt out. Omit AI results from your Google search by adding -ai to your query:

diabetes medication -ai

• Consider the source. AI tools often list citations that do not contain any reference to your question. Check the source by clicking the linked citations. Then, use the find feature to type in a word related to your question:

• It's all how you say it. One study used prompt engineering to guide ChatGPT in its search creation. While the search improved, it still included fake MeSH terms. So make sure to always check its work. 
• Keep it simple. Chatbots can be used for specific and simple tasks, such as generating synonyms for search like so:

What is another way of saying "delirium"?

• Good for grunt work. If prompted correctly, generative AI can help adapt a search from one database's syntax to another. But don't expect it to translate the entire thing for you or translate subject headings. You may also need to correct its work.

I'm going to give you a block of text I need formatted. Take each word or phrase excluding the word OR and put it in single quotation marks.

• MGB's rules for AI usage
• MGB guidelines for generative AI usage in research

• MGH Institute of Health Professions' Generative AI Guidelines for Faculty