(A1027) To Bot or Not to Bot? The Impact of Generative AI on ERAS Personal Statements

Introduction: We explore the use of AI in crafting personal statements for ENT residency applications, comparing between genuine and AI-written statements while considering AI's impact on the residency selection process. AI detection tools and the judgement of Attorneys against that of ENTs are explored for detection of AI-written work.

Methods: 5 de-identified ENT personal statements were compared against 5 AI-written essays following prompting using ChatGPT 4.0. Eight blinded reviewers, four ENTs and four Attorneys, graded each statement for originality, readability, persuasiveness, and likelihood of interview extension using a Likert scale. Data were collected regarding the perceived source of each essay by the blinded reviewers. Statements were also analyzed in an online AI detector tool. Data were analyzed using descriptive statistics, paired t-tests for comparisons, and the intraclass correlation coefficient to calculate inter-rater reliability. Statistical significance was set at P < 0.05, with analyses performed in R (Version 4.4.2).

Results: There was no significant difference in the survey criteria between the AI-generated and human-written personal statements. The intraclass correlation coefficient was 0.66. Stratifying the data by the profession revealed no statistically significant difference regarding the assessed criteria. Attorneys correctly identified the source of statements half of the time while otolaryngologists were 90% correct. The AI detection tool detected an average of 93.6% AI-generated text in the fully AI-written personal statements, and an average of 4% of AI-generated text in the genuine personal statements.

Conclusions: AI can create personal statements that can be difficult to distinguish from human voices. While AI-generated personal statements compare similarly to genuine ones in certain aspects, there is something in these AI essays that an astute reader can pick up on. The debate on responsibility by individuals for AI-generated content serves to highlight the need for clear guidelines and ethical standards in using AI.