143 - Harnessing Artificial Intelligence to Support Clinical Competency Committee Decision Making

Publication Number: 4140.143

Laura Chiel, Boston Children's Hospital, Boston, MA, United States; Ross Carson, UPMC Childrens Hospital of Pittsburgh, Pittsburgh, PA, United States; Meghan O'Connor, University of Utah, Salt Lake City, UT, United States; Sara Multerer, University of Louisville School of Medicine, Louisville, KY, United States; Srinivasan Suresh, University of Pittsburgh / UPMC Children's Hospital, Pittsburgh, PA, United States; David A. Turner, American Board of Pediatrics, Chapel Hill, NC, United States; Ariel Winn, Boston Children's Hospital, Concord, MA, United States

Background: Residency Clinical Competency Committee (CCC) members review resident assessment data to inform learning plans and high-stakes decisions about progression in training and readiness for graduation. Decisions are increasingly being made based on Entrustable Professional Activities (EPAs), the key activities of our profession, in line with a competency-based approach. Evidence supports the value of narrative comments in CCC processes, but a high volume of narrative text can be time consuming and difficult for CCC members to synthesize. Artificial intelligence (AI) may support synthesis of narrative comments but is being used in an ad hoc manner without full understanding of its capabilities, limitations, and risks.

Objective: We aim to define optimal strategies for prompt engineering within existing large language models to support CCC use of narrative comments, focusing on comments gathered from a multi-site study using an app-based, EPA-based assessment platform in pediatric residency programs.

Design/Methods: The research team, including experts in AI and assessment, conducted three focus groups with: (1) 5 researchers with expertise in CCC decision making; (2) 8 residency program leaders from programs of diverse sizes and geographies; and (3) 3 experts in bias in assessment, during which key considerations and possible prompts were presented. American Board of Pediatrics committee members provided further input.

Results: The Table summarizes considerations, potential options, stakeholder input, and consolidated decisions. Focus group participants agreed that there is a need to pursue this work but that there was not evidence to definitively suggest a single ideal approach, with any approach at risk for unintended consequences. Participants noted the need for flexibility and refinement as AI and assessment practices evolve. They preferred that AI not be used to generate decisions or summaries, but rather to organize original comments. Program leaders preferred organization along required reporting constructs (e.g, EPAs and milestones), rather than adding additional constructs, such as evidenced-based frameworks focused on learner entrustability, as discussed by experts in CCC decision making. Bias in assessment experts noted that the output should include information on bias in assessment and AI, and that while using AI to screen for bias is promising, it needs careful consideration to minimize risk.

Conclusion(s): Stakeholders support an AI-based approach to synthesis of narrative comments. This work will inform an initial approach to be piloted in residency programs.

Table: The key considerations, potential options, stakeholder input, and decisions derived from focus groups
Table_Harnessing AI.jpeg