ChatGPT Only Gets Diagnoses Correct Half of the Time

THURSDAY, Aug. 8, 2024 (HealthDay News) — ChatGPT is not accurate as a diagnostic tool, but does offer some medical educational benefits, according to a study published online July 31 in PLOS ONE.

Ali Hadi, from the Schulich School of Medicine and Dentistry at Western University in London, Ontario, Canada, and colleagues investigated ChatGPT’s diagnostic accuracy and utility in medical education. The analysis included 150 Medscape case challenges (September 2021 to January 2023) that were inputted into ChatGPT.

The researchers found that ChatGPT answered 49 percent of cases correctly, with an overall accuracy of 74 percent, a precision of 48.67 percent, sensitivity of 48.67 percent, specificity of 82.89 percent, and an area under the curve of 0.66. ChatGPT struggled with the interpretation of laboratory values and imaging results, but was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Just over half of answers were complete and relevant (52 percent) and a similar percentage of answers were characterized as low cognitive load (51 percent).

“Additional research should focus on enhancing the accuracy and dependability of ChatGPT as a diagnostic instrument,” the authors write. “Integrating ChatGPT into medical education and clinical practice necessitates a thorough examination of its educational and clinical limitations. Transparent guidelines should be established for ChatGPT’s clinical usage, and medical students and clinicians should be trained on how to effectively and responsibly employ the tool.”

Abstract/Full Text