(A414) Evaluating the Performance of Artificial Intelligence Models in Answering Facial Plastic Surgery Board Questions

Introduction: Rapid advancements in Language Learning Models (LLM) has led to the increase in the use of Artificial Intelligence (AI) tools in the field of medicine. This study particularly focuses on the effectiveness of three popular AI models, ChatGPT, Gemini, and Copilot, in answering questions designed to replicate facial plastics surgery board examinations.

Methods:

Methods:
A publicly available question bank was utilized to complete this study, with five hundred and nineteen text-based questions selected, all questions with images were excluded. For each question, the human performance and question characteristics were recorded. Then, the questions were given to all LLMs and the models were prompted to give their answer selection and their confidence level in each question choice. Data were evaluated using the Z-test.

Results: Across all recorded questions, LLMs collectively achieved an accuracy of 62.2%, while humans completed questions at an accuracy of 65.6% (p=0.254). ChatGPT performed the best with a 73% accuracy compared to Copilot (63.0%, p < 0.01) and Gemini (50.7%, p< 0.01).
For questions assigned a difficulty level of “1”, LLM performance was 100% compared to humans (65%), but for questions with a difficulty level of “4”, LLMs scored 47.7% compared to humans (62.1%)

Conclusions: ChatGPT surpassed the performance of Gemini and Copilot in answering the facial plastics board questions. The average LLM answer performed worse than humans on average, however when all three models agreed on the same answer, they outperformed humans. The average accuracy of LLM answers fell under human performance, especially in difficult questions, indicating the careful integration of AI into medical education.