Skip to main content
AAO-HNSF 2025 Annual Meeting & OTO EXPO Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.

Scientific Oral Presentations

Laryngology/Broncho-Esophagology

Session: Scientific Oral Presentation 1 - Laryngology/Broncho-Esophagology

AI in Clinical Decision-Making: ChatGPT-4 vs. Llama2 for Otolaryngology Cases

Saturday, October 11, 2025
9:30 AM - 9:36 AM EDT
Location: 139
Disclosure(s):
Lise Sogalow, MD:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Jerome R. Lechien, MD, PhD, FACS:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Introduction: To evaluate the diagnostic accuracy, appropriateness of additional examination
recommendations, and consistency of therapeutic regimens by ChatGPT-4 and
Llama2 based on real otolaryngology cases.

Methods: Study Design
Observational study.
Setting: Tertiary care academic medical center (University Hospital of Saint-Pierre,
Brussels, Belgium) and affiliated otolaryngology outpatient clinics between July and
October 2024.

Methods: A prospective controlled study was conducted on 98 anonymized
otolaryngology cases. Clinical information was entered in ChatGPT-4 and Llama2 for
reaching primary diagnoses, additional examination recommendations, and treatment
strategies. Two independent otolaryngologists evaluated the AI outputs using the
artificial intelligence performance instrument (AIPI), evaluating diagnostic accuracy,
appropriateness of examination, and adequacy of treatment. Statistical comparisons
were conducted between the AI systems and expert decisions. Interrater reliability was
evaluated with kappa statistics.

Results:

Results: ChatGPT-4 diagnosed 82% correctly, outperforming Llama2 at 76%. For
additional examinations, ChatGPT-4 suggested relevant and appropriate tests in 88%
of the studies, while Llama2 did so in 83%. Treatment appropriateness was achieved
in 80% of the cases through ChatGPT-4 and 72% through Llama2. Sometimes, both
systems suggested inappropriate tests. The interrater reliability was high for AIPI
scores (kappa=0.85).

Conclusions:

Conclusion: ChatGPT-4 and Llama2 have shown great potential as clinical decision-
support tools in otolaryngology, with ChatGPT-4 exhibiting superior performance. At
the same time, non-relevant recommendations indicate further refinement and human oversight to ensure safe application in clinical practice.

    Presenting Author(s)

  • LS

    Lise Sogalow, MD

    PhD student
    University of Mons
    Jurbise, Hainaut, Belgium

    Disclosure(s):

    faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

  • AM

    Antonino Maniaci, MD

    MD, PhD, Assistant Professor
    Enna Kore
    Enna, Sicilia, Italy

    Disclosure information not submitted.

    • Co-Author(s)

  • AM

    Antonino Maniaci, MD

    MD, PhD, Assistant Professor
    Enna Kore
    Enna, Sicilia, Italy

    Disclosure information not submitted.

  • Cosima C. Hoch, MD

    MD
    Department of Otolaryngology, Head and Neck Surgery, TUM School of Medicine and Health, Technical University of Munich (TUM), Munich, Germany
    Munich, Bayern, Germany

    Disclosure information not submitted.

    • Senior Author(s)

  • JL

    Jerome R. Lechien, MD, PhD, FACS

    Chair
    University of Mons
    Sombreffe, Hainaut, Belgium

    Disclosure(s):

    faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.