(A504) Adherence of Large Language Models with Guidelines on Osteoradionecrosis

Introduction: The ability of large language models (LLMs) to serve as clinical decision tools is being explored. We evaluated the alignment of information generated by LLMs against guidelines on the management of osteonecrosis in patients with head and neck cancer treated with radiation.

Methods: 42 recommendations were extracted from the “Prevention and Management of Osteoradionecrosis in Patients With Head and Neck Cancer Treated With Radiation Therapy” guideline from the International Society of Oral Oncology-Multinational Association for Supportive Care in Cancer and American Society of Clinical Oncology in December 2024. Each recommendation was translated into a question, which was then followed by a request to generate a resource used to support the response. Two LLMs, ChatGPT 4.0 and Gemini, were queried with each question. Reviewers graded each response for its alignment with the guideline, which was graded on a 3-point Likert scale. Credibility was deemed if the LLM provided a credible resource. Statistical analysis was performed.

Results: 42 questions (84 responses) were generated. ChatGPT responses had a mean alignment score of 1.43, indicating that the responses were reflective and non-reflective of the guidelines, and provided 48% credible resources. Gemini responses had a mean alignment score of 1.43 and provided 83% credible resources. 10% of ChatGPT and 14% of Gemini responses completely contradicted the guideline.

Conclusions: LLMs produced information that was both adherent and non-adherent to clinical guidelines on osteoradionecrosis. However, completely discordant information was produced in 10-14% of responses. LLMs display promise as clinical decision aids yet cannot be utilized as an independent tool.