144 - Establishing Validity Evidence for Assessment Tools in a Pediatric Resident Mastery Learning Curriculum

Publication Number: 4141.144

Michaela Mills-Rudy, Mayo Clinic Children's Center, Rochester, MN, United States; Stephanie Mavis, Mayo Clinic Children's Center, Rochester, MN, United States; Danielle Klima, Mayo Clinic Children's Center, Rochester, MN, United States

Background: Systematic and reliable assessment is at the heart of learning, and while this requires robust performance assessment tools, such tools are not well developed in pediatrics.

Objective: The aim of this study was to establish validity arguments for several updated assessment tools for use in a pediatric mastery learning procedural skills curriculum.

Design/Methods: Messick’s validity framework was used. Assessment tools for four core procedural skills (pediatric lumbar puncture, neonatal bag mask ventilation using a flow-inflating bag, tracheostomy tube replacement, and gastrostomy tube replacement) were developed through literature review to identify and critique existing performance assessment tools, with creation of new checklists when none existed. Each tool underwent iterative refinement with local content experts to ensure task definition, observability, and clarity until consensus was achieved. For each checklist, two mastery standards were set using the Mastery-Angoff method. Interrater reliability was determined during rater training by comparing rater scores for training videos demonstrating a range of simulated learner skill with those of content experts. To examine relationships to other variables and provide further consequential evidence, assessment tools were deployed for baseline learner testing then used in a longitudinal simulation environment. The correlation of learners achieving the minimum passing standard to key clinical outcomes for two skills are being examined (for LP, in success in obtaining CSF and in maintaining sterility during the pediatric lumbar puncture procedure, and for BMV, achieving visible chest rise during neonatal bag-mask ventilation).

Results: Four updated performance assessment tools were developed. Minimum passing scores were set between 73-85% for the first standard (entrustability to perform the skill safely under direct supervision) and 91-100% for the second (entrustability to perform the skill safely with supervisor in the next room). Inter-rater reliability ranged from 80-94%. Investigations of the correlation between achievement of a passing MPS and outcomes is ongoing.

Conclusion(s): We gathered initial validity evidence supporting the use of these assessment tools to evaluate pediatric residents’ simulated procedural competence. Integration of such checklists within a mastery learning framework could promote rigorous and transparent assessment of procedural skills in pediatric training.