(A154) Validation of a Novel Mobile Application for the Assessment of Adult Tracheostomy Emergency Simulations

Introduction: Tracheostomy emergencies are time-critical crises associated with high morbidity and mortality. In situ simulation is an effective tool for evaluating team and system response to tracheostomy emergencies in the actual clinical environment but retrospective video review of time-critical actions is cumbersome and vulnerable to data loss. We evaluated whether a novel mobile application, previously validated for pediatric airway emergency scenarios, was a reliable option for real-time assessment of adult tracheostomy emergency simulations.

Methods: The NeoCHART+™ for PEAK II Trach-Adult application interface was developed for assessment of tracheostomy simulations involving cuffed adult tracheostomy tubes with inner cannulas. Four independent raters scored ten de-identified video-recordings of tracheostomy emergency simulations completed from August 2021 to August 2023 across inpatient and critical care units in a single, tertiary academic center. The raters consisted of two medical students, one otolaryngology attending, and one critical care attending. Inter-rater reliability was calculated based on absolute agreement of step occurrence and timestamps.

Results: Fleiss kappa for step occurrence among all app raters was 0.694 with a 95% confidence interval (CI) (0.629, 0.759), demonstrating a strong level of agreement. Time stamp agreement was nearly excellent, with intraclass correlation coefficient of 0.846 95% CI (0.776, 0.901).

Conclusions: The NeoCHART+™ for PEAK II Trach-Adult application interface demonstrates strong inter-rater reliability, supporting its reproducibility across raters with varying levels of training and clinical expertise. With further validation, this tool has the potential to enhance understanding and benchmarking of time-critical response to adult tracheostomy emergencies.