Skip to main content
AAO-HNSF 2025 Annual Meeting & OTO EXPO Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Rate ePoster
Video

Professional and Personal Development

(A1017) Natural Language Processing for Reflection and Feedback Quality in a Nationwide Otolaryngologic Workplace-Based Assessment Database

Monday, October 13, 2025
12:00 PM - 1:00 PM EDT
Location: OTO EXPO, Posters

    Presenting Author(s)

  • Jeng-Wen Chen, MD, MSC

    Deputy Superintendent
    Cardinal Tien Hospital and Fu Jen Catholic University
    New Taipei City, New Taipei, Taiwan (Republic of China)

    • Co-Author(s)

  • HT

    Hai-Lun Tu, PhD

    Professor
    Fu Jen Catholic University, New Taipei, United States

    • Senior Author(s)

  • Wei-Chung Hsu, MD, PhD

    Professor
    National Taiwan University Hospital
    Taipei, Taipei, Taiwan (Republic of China)

    • Co-Author(s)

  • Pa-Chun Wang, MD, PhD

    Professor
    Cathay General Hospital
    Taipei, Taipei, United States

Disclosure(s):
Jeng-Wen Chen, MD, MSC: No relevant relationships to disclose.
Wei-Chung Hsu, MD, PhD: No relevant relationships to disclose.


Introduction: Competency-based medical education (CBME) aims to train specialists in a way aligned with societal expectations. The Taiwan Otolaryngology Society, collaborating with the Joint Commission of Taiwan, launched the EMYWAY platform in 2021 and commenced nationwide Entrustable Professional Activities (EPAs) assessments in 2022. Over three years, the platform amassed over 25,000 EPA records.

Methods: This study analyzed 300 entries from the EPA database, focusing on workplace context, EPA titles, clinical diagnoses, and narrative reflections and feedback. Two medical education experts performed expert coding on the quality of reflections and feedback, assessing relevance, specificity, and the presence of reflective or improvement suggestions. Conflicts were resolved through consensus or by the project leader. The data underwent deep learning natural language processing (NLP) to classify the quality of reflections and feedback into 'effective,' 'mediocre,' 'ineffective,' or 'other,' and further into 'high' and 'low' quality categories.

Results: Using the Bidirectional Encoder Representations from Transformers (BERT) algorithm, the study revealed a 90% accuracy for trainee reflection quality in a two-level model, with precision, sensitivity, and specificity at 67%, 50%, and 96%, respectively. The four-level model showed 63% accuracy, 69% precision, and 66% sensitivity. For faculty's feedback, accuracy was 80% in the two-level model and 67% in the four-level model, with respective precision and sensitivity values of 100% and 60%, and 70% and 83%. The F1 scores were favorable across models.

Conclusions: The results demonstrate the effectiveness of deep learning NLP in evaluating the quality of residents' reflections and faculty's feedback, yielding results comparable to traditional machine learning with a limited dataset. This underscores the potential of technology in enhancing feedback quality for workplace-based assessments, which is crucial for the advancement of CBME.