Can ChatGPT Prove Its Effectiveness in Health Care?

BAIVECTOR (top pattern) / Malchevska Studio (doctor) - stock.adobe.com

MANY IN HEALTH care have had breakthroughs with the integration of artificial intelligence (AI); however, the technology is less effective when physicians are not fully trained to use it. In a recent study conducted by University of Virginia (UVA) Health’s Andrew S. Parsons, MD, MPH, and colleagues, the researchers wanted to determine how technology such as ChatGPT Plus (or GPT-4) could be used more effectively to overcome this lack of training. The single-blind, randomized clinical study was conducted from November 29 to December 29, 2023, with results f irst reported in JAMA Network Open.¹

Parson’s team found that using ChatGPT Plus did not significantly improve the accuracy of doctors’ diagnoses compared with using traditional resources when studied across multiple academic medical institutions. These included UVA Health University Hospital, Stanford University, and Harvard Medical School’s Beth Israel Deaconess Medical Center.

During the study, Parsons and his colleagues randomly assigned 50 physicians in family medicine, internal medicine, and emergency medicine to use ChatGPT Plus and other sources to diagnose complex cases. There were 26 attending physicians and 24 resident physicians, and the median number of years in practice was 3. Of the physicians, 25 were assigned to GPT Plus and 25 to more traditional sources such as UpToDate and Google. After studying both groups, the researchers compared their resulting diagnoses, finding that accuracy across the 2 groups was similar.

However, ChatGPT Plus on its own outperformed both groups, highlighting that the technology has the potential to keep improving patient care. Yet the study findings also emphasize that physicians need more training and experience with the technology to become more skilled at using it in real-life patient situations.

“Our study shows that AI alone can be an effective and powerful tool for diagnosis,” Parsons said in a statement.² “We were surprised to find that adding a human physician to the mix actually reduced diagnostic accuracy though improved efficiency. These results likely mean that we need formal training in how best to use AI.”

The investigators noted that the commercially available chatbot did not improve diagnostic reasoning on challenging cases. It suggests that access alone to chatbots will not improve overall physician diagnostic reasoning in practice. The results were similar across subgroups of different training levels.

Parsons and his team suggest that ChatGPT Plus is better suited for augmenting the work of physicians, meaning it should assist and enhance their tasks rather than replace them. They also recommend the option for hospitals and healthcare organizations to purchase predefined AI prompts to integrate into clinical workflows and documentation, helping to streamline processes without replacing the need for human expertise and oversight.

“As AI becomes more embedded in health care, it’s essential to understand how we can leverage these tools to improve patient care and the physician experience,” Parsons said. “This study suggests there is much work to be done in terms of optimizing our partnership with AI in the clinical environment.”

For more info, visit https://bit.ly/3OsKdIH

REFERENCES

1. Goh E, Gallo R, Hom J, et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw Open. 2024;7(10):e2440969. doi:10.1001/ jamanetworkopen.2024.40969

2. Barney J. Does AI improve doctors’ diagnoses? study f inds out. UVA Health Newsroom. November 13, 2024. Accessed November 27, 2024. https://tinyurl.com/3dd4s3xm