AI Showdown: ChatGPT vs OpenEvidence in ATA Guideline Evaluation

Sunday, October 12, 2025

1:54 PM - 2:00 PM EDT

Location: 127

Disclosure(s):

Delaney E.S Clark, BS:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Introduction: The use of artificial intelligence (AI) models has become more common recently, with ChatGPT (CGPT) being the most common household name. Despite consistent feature updates, CGPT remains unable to conduct literature searches. OpenEvidence (OE) is an AI system designed to synthesize evidence from journal articles to answer questions, presumably generating more accurate answers than the former.

Methods: Guidelines from the American Thyroid Association (ATA) on treatment of differentiated thyroid cancer (DTC), anaplastic thyroid cancer (ATC), and hyperthyroidism were made into a question. These were then used as input for CGPT and OE with the outputs aggregated into a scoring sheet. Three independent reviewers, all attending physicians, were blinded to the AI model and graded responses using a 4-point Likert scale for both completeness and accuracy. These scores were then analyzed for agreement between raters and variances of scores.

Results: Overall, the outputs from OE scored an average of 3.76 for completeness and 3.71 for accuracy, while CGPT scored 3.52 (p = 0.002) and 3.26 (p < 0.0001) respectively. When analyzed by guideline, OE significantly out-performed CGPT in average completeness for DTC (3.83/3.47, p = 0.001), but not for ATC (3.79/3.58, p = 0.08) or hyperthyroidism (3.63/3.53, p = 0.23). In terms of accuracy, OE performed significantly better than CGPT with DTC (3.64/3.19, p = 0.02), ATC (3.83/3.21, p = 0.005), and hyperthyroidism guidelines (3.7/3.37, p = 0.02). The average interclass correlation coefficient between raters was 0.51, indicating moderate reliability between graders.

Conclusions: Our study demonstrates that OE is overall more likely than CGPT to generate complete and accurate answers using evidence from peer-reviewed journals. Although 2/3 guidelines had no significant variances noted for completeness, OE significantly out-performed CGPT with output accuracy. These findings highlight OE’s capability to interpret surgical guidelines and respond accurately, indicating a potential use as a more appropriate educational tool for surgeons and the public.

Presenting Author(s)

Delaney Shiu Clark, BS

Medical Student
University of Texas Medical Branch At Galveston
Galveston, TX, United States

Disclosure(s):
faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Co-Author(s)

AT

Andrew LS Thornton, BA

Medical Student
UTMB John Sealy School of Medicine
Galveston, Texas, United States

Disclosure information not submitted.
Arati Bendapudi, BS

Medical Student
UTMB John Sealy School of Medicine
Celina, Texas, United States

Disclosure information not submitted.
Sepehr Shabani, MD

Assistant Professor
City of Hope National Medical Center
Duarte, California, United States

Disclosure information not submitted.
Orly M. Coblens, MD

Associate Professor & Residency Program Director
University of Texas Medical Branch Hospitals
Galveston, Texas, United States

Disclosure information not submitted.
Vanessa C. Stubbs, MD

Assistant Professor
Rush University Medical Center
CHICAGO, IL, United States

Disclosure information not submitted.

Senior Author(s)

Viran J. Ranasinghe, MD

Assistant Professor
University of Texas Medical Branch
Galveston, TX, United States

Disclosure information not submitted.