Large Language Models as Adjunctive Tools for Diagnosing Rare Diseases in Otolaryngology: A Controlled Study

Sunday, October 12, 2025

1:18 PM - 1:24 PM EDT

Location: 242

Disclosure(s):

Jerome R. Lechien, MD, PhD, FACS:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Lise Sogalow, MD:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Meryem Miri, MD:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Introduction:

Objective: Artificial Intelligence has been found to support practitioners in some rare disease diagnoses. This study compared the diagnostic accuracy of 3 LLMs between rare and common otolaryngology-head and neck conditions through a prospective controlled analysis.

Methods: Study design: Prospective controlled study.
Setting: University Study.
Patient history, clinical image, imaging, or pathological features of 35 published very rare conditions and 35 common otolaryngological conditions were presented to ChatGPT-4o, Google Gemini-1.5-pro, and Claude-3.5-Sonnet for primary and differential diagnoses. LLM responses were assessed by two blinded board-certified otolaryngologists with the Artificial Intelligence Performance Instrument (AIPI). The intraclass correlation coefficient (ICC) was used to measure the interrater agreement.

Results: Claude-3.5-Sonnet demonstrated superior performance in rare condition diagnosis (54.3%), significantly outperforming Gemini-1.5-Pro (28.6%, p=0.029) and surpassing ChatGPT-4o (45.7%). This superiority was validated by significantly higher AIPI diagnosis scores compared to other LLMs. For common conditions, all LLMs demonstrated significantly higher diagnostic accuracy versus rare conditions (p < 0.001), with Claude-3.5-Sonnet achieving 82.9% accuracy and superior AIPI scores across clinical features and diagnostic domains. The ICC demonstrated excellent overall inter-rater reliability (0.903) for AIPI scores.

Conclusions: LLMs demonstrate promising potential as supplementary diagnostic tools, with Claude-3.5-Sonnet exhibiting superior performance across both rare and common conditions.

Presenting Author(s)

JL

Jerome R. Lechien, MD, PhD, FACS

Chair
University of Mons
Sombreffe, Hainaut, Belgium

Disclosure(s):
faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.
LS

Lise Sogalow, MD

PhD student
University of Mons
Jurbise, Hainaut, Belgium

Disclosure(s):
faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Co-Author(s)

LS

Lise Sogalow, MD

PhD student
University of Mons
Jurbise, Hainaut, Belgium

Disclosure(s):
faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.
MM

Meryem Miri, MD

PhD candidate
UMONS
Mons, Hainaut, Belgium

Disclosure(s):
faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Senior Author(s)

AM

Antonino Maniaci, MD

MD, PhD, Assistant Professor
Enna Kore
Enna, Sicilia, Italy

Disclosure information not submitted.