Assessing the Quality and Efficiency of an Artificial Intelligence Powered Scribe in Outpatient Clinic Visits

Introduction: Efficiency in the outpatient setting is essential to optimize patient care, improve physician productivity, and maximize revenue. Artificial intelligence (AI) powered scribes have been advertised as a method to improve efficiency, however their quality have not been formally assessed. In this study, we compare the quality and efficiency of an AI powered scribe versus human scribe in a busy head and neck clinic.

Methods: An AI powered scribe was utilized for outpatient clinic note generation by a single head and neck surgeon beginning June 2024. A total of 100 control notes generated by a human scribe and 100 notes generated by the AI scribe were collected retrospectively. Notes were standardized for blinded review. Two reviewers assessed note quality utilizing the 9-Item Physician Documentation Quality Instrument (PDQI-9). Reviewers also predicted the method of note generation (human vs. AI). PDQI-9 scores, rate of accurate prediction, rate of note attestation day of visit, and rate of note attestation during working hours day of visit were compared between the human and AI generated notes. Continuous variables were compared with Wilcoxon rank sum test, and categorical variables were compared with Pearson’s Chi-squared test.

Results: On the PDQI-9, AI scribe generated notes were significantly more organized (4.5±0.8 vs. 4.2±0.7, p< 0.001) but less succinct (4.5±0.7 vs. 4.8±0.4, p = 0.048). There were no significant differences between AI and human scribes in note quality for other criteria or in overall score (42.1±3.0 vs. 42.2±2.2, p=0.277). Reviewers predicted AI and human generated notes accurately at similar rates (56% vs. 70%, p=0.057). AI scribe notes were attested the day of visit (54% vs. 30%, p< 0.001) and during work hours day of visit (48% vs. 23%, p< 0.001) significantly more often.

Conclusions: AI scribe generated notes were attested earlier and with similar quality compared to human generated notes. AI scribe integration into the outpatient head and neck clinic setting provides similar quality and has the potential to improve clinic workflow. A more in-depth evaluation of efficiency metrics is warranted.