Can It Reason? Evaluation of Multimodal Models for Imaging Based Otolaryngology Board Preparation

Introduction: Given large language models (LLM) updates to powerful multimodal models (MM), we evaluate MM capability, reliability, and explainability as a study aid on otolaryngology residency board level questions associated with imaging. By grading insight of the explanations provided by MMs, we evaluate clinical reasoning capabilities, image interpretation and integration of associated questions.

Methods: Eighty-nine questions from the 2020 OTOQuest Knowledge Assessment, approved by AAO-HNS, were inputted into MM1 (Chat-GPT 4) and MM2 (Google Gemini) from 6/1/2024 to 11/13/2024 with a standardized prompt soliciting a definitive answer, confidence level (0-5), and explanation. Each question was asked thrice: question text only (Q), image only (I), and question text with image (Q+I). Question characteristics like image type, management vs. diagnosis, and vignette vs. non-vignette were noted. Explanations for Q+I were scored based on being non definitional, unique, valid, image interpretation, and integration to determine insight score (0-6).

Results: MM1 showed accuracy and confidence for Q+I (n=89) of 65.17% and 4.20, and for Q (n=89) of 64.04% and 4.73. MM2 showed accuracy and confidence for Q+I (n=30) of 50.00% and 3.97, and for Q (n=78) of 43.18% and 4.12. Hallucination rates for MM1 and MM2 were 34.83% and 16.7%, defined by inaccurate answers/total answers. For MM1, Q and Q+I had a correlation of 0.63 (CI: 0.49, 0.74) for correct answers. Average explanation insight was higher in MM1 than MM2 (4.5 vs. 4.0, p=0.2), and image interpretation and integration were significantly better in MM1 (p=0.01).

Conclusions: MM1 showed higher accuracy, image incorporation, and confidence in all three instances than MM2. However, MM1's higher hallucination rate highlights the need for caution. MM2's consistent requests for more information and lower hallucination rate suggest a reliable but conservative approach. MM1 demonstrated greater image integration and incorporation capabilities in its explanations, revealing more advanced clinical reasoning skills than MM2, indicating potential as a non-inferior educational tool.