TOP 13 - Utilizing Generative Artificial Intelligence to Evaluate the Quality of Pediatric Hospital Medicine Faculty Verbal Feedback

Publication Number: 1773.TOP 13

Madeline Parr, Cohen Children's Medical Center, Hackensack, NJ, United States; Pratichi K. Goenka, Northwell Health, Jericho, NY, United States; Alice Fornari, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, New Hyde park, NY, United States; Shahidul Islam, Northwell Health, New Hyde Park, NY, United States; Nina Kumar, Cohen Children's Medical Center, Glen Oaks, NY, United States; Sheila Liewehr, Cohen Children's Medical Center, Dix Hills, NY, United States; Nancy Palumbo, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, New Hyde Park, NY, United States; Adrita Khan, Cohen Children's Medical Center, Plainview, NY, United States

Background: Artificial intelligence (AI) is increasingly being used in healthcare in both clinical and educational contexts; however, its use to assess faculty-to-learner verbal feedback is underexplored. While effective feedback from attending physicians is crucial for medical trainee development, providing high-quality, timely feedback often presents a significant challenge for faculty. This study aims to assess the feasibility of using generative AI to score the quality of verbal feedback from faculty to residents.

Objective: To explore the ability of AI technology to assess the quality of verbal feedback provided by pediatric hospitalists to residents, focusing on adherence to the Self-Assessment, Feedback, Encouragement, Direction (SFED) feedback model.

Design/Methods: IRB approval was obtained for this study. Feedback sessions between faculty and resident physicians were transcribed using hospital-encrypted transcription software. The analysis of transcripts involves a 50-point scoring rubric using a 5-point Likert scale, which was developed from the SFED feedback model (Figure 1). The rubric comprises four categories (Self-assessment, Encouragement, Feedback, and Direction) and two to three subcategories based on the central tenets of each major category. Subcategories capture each domain's core elements and include criteria usable by both human and AI evaluators to judge feedback quality. To determine AI scoring performance, transcripts will be independently scored by two trained human raters and also by the AI model using the SFED rubric, and the scores will be compared. Human scorers received training by an SFED expert (author AF) in the application of the SFED model. AI scoring involves an encrypted Large Language Model (LLM) within the health system, trained via prompt engineering using the SFED scoring rubric, trial scoring, and example transcript scores. Descriptive statistics will be generated to summarize data. Inter-rater reliability among human assessors will be evaluated using the intraclass correlation coefficient (ICC). Human and AI-generated scores will be compared for equivalence using two one-sided tests of equivalence for paired samples (TOST-P). All data (transcripts, n = 80) have been collected; scoring and analysis are currently ongoing.

Self-Assessment, Feedback, Encouragement, Direction (SFED) Feedback Model Rubric
The SFED Feedback Model in rubric form for evaluation of verbal feedback between attending and resident physicians. Both human and AI scorers utilize this rubric to explore AI's ability to evaluate verbal feedback sessions.