ChatGPT Scores High for Genetic Counseling in Gynecologic Cancers

Jharna M. Patel, MD

A study tested ChatGPT, the artificial intelligence (AI) chatbot, on its ability to answer questions about genetic counseling in gynecologic oncology with a focus on ovarian cancer. The tool performed well, correctly answering 83% of questions asked by researchers.

Researchers gave the AI 40 questions that patients commonly ask about genetic counseling, and doctors then rated the AI's answers.

The AI did best with questions about genetic syndromes, answering all 20 perfectly. Answers regarding syndrome-specific questions, including about BRCA mutations and Lynch syndromes, delivered around 60% to 70% accuracy.

“It goes to show that there are limitations in this technology,and while patients may be using [ChatGPT], we have to caution them that it is not a tool that's able to be applied clinically just yet,” said Jharna M. Patel, MD, a gynecologic oncology fellow at NYU Langone Health, in an interview with Targeted Oncology^TM.

In the interview, Patel discussed the study and its findings presented at the 2024 Society of Gynecologic Oncology (SGO) 2024 Annual Meeting.

Targeted Oncology: What is the current landscape of genetic counseling in ovarian cancer?

Patel: The current landscape involves any patient that is diagnosed with ovarian cancer or has the high suspicion of a cancer that could be transmitted via familial syndromes. We recommend genetic counseling and genetic testing [for those patients]. Genetic testing can occur in 1 of 2 ways: It can occur on the tissue type itself, or [the patient] is referred to a genetic counselor who delves a little bit more into [their] family history. At that time, based on [their] risk assessment, or if there's only history that is divulged, then [the counselors] recommend germline testing.

What was the background or rationale of the study presented at SGO?

We know that ChatGPT is good at answering fact-based questions. In fact, it was able to pass the [United States Medical Licensing Examination according to] recent papers. We know that other medical fields have investigated its ability to conduct testing in their fields [and answer] fact-based questions in their field. We wanted to see if ChatGPT could handle more nuanced questions. As you can imagine, genetic counseling is extremely nuanced and very patient specific. That is what inspired this: We wanted to see if we can conduct personalized counseling [with ChatGPT].

Could you speak on the methodology of the analysis?

We generated a list of 40 questions in conjunction with the [gynecologic] oncologists, and we also adapted these questions from professional society websites. This was to emulate clinical questions and questions about genetic counseling that are most commonly encountered in clinical practice. We decided to formulate each question at an eighth-grade reading level, which would emulate an average United States user, and we used ChatGPT version 3.5. Although we recognize that there are versions of ChatGPT that could be a little bit more accurate, this is the version that is most commonly used by the population of the United States, as it is the free version. We queried ChatGPT with each question, and we recorded its answers, based on a scale of 1 to 4, and oncologists were asked to evaluate how comprehensive and correct the answers were. A 1 would denote us get an answer that is completely comprehensive and correct vs 4 would denote an answer that is completely incorrect. If there were any scoring discrepancies that needed to be resolved, we will send out the questions to additional [gynecologic] oncology attendings.

What were your findings?

The questions are broken up into 2 categories: general genetic counseling questions vs syndrome-specific questions. In the general genetic counseling category, this would encompass questions such as, will my insurance be affected if I test positive for syndromes? What is cascade testing? Do I have to tell my family members if I tested positive for something? These are just general questions, and in that question category, ChatGPT answered the questions correctly and comprehensively 100% of the time. Each answer received a score of 1; there were no scoring discrepancies that needed to be resolved. It was unanimous between the 2 [gynecologic] oncologists that these questions were comprehensive and correct.

The second category of questions was related more to syndromes. This was further divided into 2 different categories: questions about Lynch syndrome specifically, and questions about BRCA-associated hereditary breast and ovarian cancer syndromes. In the Lynch syndrome questions, we found that two-thirds of the time, questions were answered comprehensively and correctly. About 60% percent were correct in the BRCA-associated hereditary breast and ovarian cancer syndrome questions.

What are some of the implications of your findings for clinicians and patients?

For clinicians, they have to understand that ChatGPT is a tool that patients might already be using. As I said, although it was good and 100% of the time accurate and answering general genetic counseling questions, it was less accurate about syndrome-specific questions. It goes to show that there are limitations in this technologyand while patients may be using [ChatGPT], we have to caution them that it is not a tool that is able to be applied clinically just yet.

REFERENCE:

Patel JM, Hermann CE, Growdon WB, et al. ChatGPT accurately performs genetic counseling for gynecologic cancers. Presented at the Society of Gynecologic Oncology 2024 Annual Meeting on Women’s Cancer; San Diego, CA; March 16-18, 2024.