Artificial Intelligence in Medical Research

At MGH's Treadwell Library, we recently had a physician come to us with an interesting conundrum. They were reading a literature review but couldn't find the papers it referenced. As it turned out, the authors used ChatGPT to perform the literature search, and a majority of the papers were fake! 

Clearly, AI usage, while holding promise for streamlining certain tasks, is not without risks. This guide will outline those, as well as best practices to follow if you use an AI tool in research or clinical practice. When it comes to AI, the key is to “just say know.” The more knowledge you have about its risks, the better prepared you will be if you use such a tool.

• That paper doesn't exist! Literature reviews that employ an AI tool result in fabricated and fake citations.
• Not provable or transparent. Scientific experiments are provable because an experiment is done once, then again, and produces the same results. This is not true for generative AI! Ask the question in a different room of your house, or browser, and you get a different answer. Additionally, generative AI's decision-making (even open source) is not transparent because users "cannot interpret what happens within each layer of the model when it’s active." 
• Probability not logic. Per librarian Brandy King: "AI is not a search engine, it's a prediction engine. The answers it provides are generated by predicting and inferring which word comes next in a sequence. It cannot apply logic or reasoning. The output is 'linguistically plausible' without regard for truth."
• Not objective. AI tools are not entirely unbiased or objective. For example, one group of researchers found that AI-based commercial gender classification systems performed better on light skin, male faces. 
• Contradictory answers. A slight variation in your search query can produce wildly different answers:

• Citation issues. AI search engines are prone to citing syndicated sources like Yahoo News or AOL instead of original studies.

• Watch what you type! You cannot put any protected health information into an AI tool or feed it a paper you are peer reviewing.
• Copyright issues. Harvard librarians explain: "There are a number of lawsuits that allege Generative AI tools infringe copyright and it remains unclear if and how the fair use doctrine can be applied." 
• Legal liability. If an AI tool generates misinformation you pass on to your patient, you're opening yourself up to potential litigation.

• Generative AI is not a search engine or a research database. Harvard librarians explain that "generative AI tools are designed to 'generate' their own content or output" unlike a repository of source material. 
• Requires disclosure. If you use an AI tool when writing a manuscript, you must disclose this to the publisher and agree to be responsible for the accuracy of whatever information the AI tool produces.
• Leave literature searching to the experts. Search strategies developed by ChatGPT have been found to be "unusable" and include fake MeSH terms. If you need a search done and are short on time, we can do it for you! 

• Pick the right tool. Some considerations:
  • Enterprise tools tend to have better privacy terms than free tools.
  • Some tools claim to have fewer hallucinations. For example, OpenEvidence claims not to respond if the evidence is inconclusive. But that doesn't mean it's perfect: "When OpenEvidence took the US Medical Licensing Exam... it was wrong 9% of the time." 
  • Premium chatbots aren't necessarily a better option. In fact, premium chatbots have been found to provide more incorrect answers than free versions.
  • Finally, be cautious which tool you select. DeepSeek is prohibited across MGB.
• Opt out. Omit AI results from your Google search by adding -ai to your query:

diabetes medication -ai

• Consider the source. AI tools often list citations that do not contain any reference to your question. Always check the source by clicking the linked citations. Then, use the find feature to type in a word related to your question:

• It's all how you say it. One study used prompt engineering to guide ChatGPT in its search creation. While the search improved, it still included fake MeSH terms. So make sure to always check its work. 
• Keep it simple. Chatbots can be used for specific and simple tasks, such as generating synonyms for search like so:

What is another way of saying "delirium"?

• Good for grunt work. If prompted correctly, generative AI can help adapt a search from one database's syntax to another. But don't expect it to translate the entire thing for you and don't expect it to translate subject headings. You may also need to correct its work.

I'm going to give you a block of text I need formatted. Take each word or phrase excluding the word OR and put it in single quotation marks.

• MGB's rules for AI usage
• MGB guidelines for generative AI usage in research

• MGH Institute of Health Professions' Generative AI Guidelines for Faculty