179 - Implementation of an Automated Curriculum to Support Behavioral Health Counseling Skills Among Pediatric Residents

Publication Number: 4176.179

Francis Real, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Micah Goldson, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Sara Williams, University of Cincinnati College of Medicine, Cincinnati, OH, United States; Cassandra Conrad, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Andrea Meisman, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Madison Hansen, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Liam Fleck, Cincinnati Children's Hospital Medical Center, Nashville, TN, United States; Michael Glisson, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States; Rachel Herbst, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States

Background: Pediatric residents report gaps in behavioral health counseling skills. A prior human-facilitated, screen-based virtual reality (VR) curriculum demonstrated the capacity to enhance behavioral health anticipatory guidance (BHAG) and motivational interviewing (MI) skills. However, human facilitation of the intervention limited scale. Thus, the curriculum was automated to allow asynchronous, self-directed learning through a partnership with BreakAway Ltd. The automated curriculum allowed learners to verbally counsel graphical characters. Following a scenario, the users received specific feedback regarding each selection highlighting the relevant BHAG or MI competencies.

Objective: To assess usability, immersion, and implementation outcomes following incorporation of the automated curriculum into the developmental and behavioral pediatrics rotation.

Design/Methods: Following the automated curriculum, residents were invited to complete a survey instrument assessing platform usability via the System Usability Scale (SUS), immersion via the MEC-Spatial Presence Questionnaire (MEC-SPQ), and implementation outcomes of acceptability, appropriateness, and feasibility via the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM), respectively. The IRB was exempt. Survey data was analyzed with descriptive statistics.

Results: In total, 41 residents (77% of eligible population) completed the survey. Participants were mostly women (76%), white (78%), with a mean age of 30, and third-year pediatric residents (51%). The averaged SUS score was 73.5, which falls in the range typically interpreted as 'good' usability for digital interfaces. Two-thirds of participants agreed or strongly agreed that the VR scenario held their attention (67%), though fewer felt as though they were actively taking part in the action of the simulation (38%). A majority agreed or strongly agreed that the intervention was appropriate for behavioral health training (80%) and feasible to implement (83%), though acceptability was more mixed, with just over half agreeing the curriculum was acceptable (56%). Most participants (83%) relied primarily on clicking or a combination of clicking and verbalizing rather than solely verbalizing their responses.

Conclusion(s): An automated curriculum on BHAG and MI demonstrated good usability and fair acceptability among a cohort of pediatric residents. However, although the intervention was designed to support verbal counseling, most learners did not consistently verbalize responses.

Figure 1. The automated behavioral health curriculum
Participation in the automated behavioral health counseling curriculum included (1) a brief orientation familiarizing learners with the platform and (2) two separate self-guided counseling scenarios accessed through secure web links. To support reliable progression through the scenarios, possible answer choices were displayed on the screen (A). The learner was meant to verbalize their counseling behaviors with on-screen answer options providing scaffolding to inform potential statements. The answer option most closely aligned to the user's verbalizations was selected by the system. Alternatively, the learner had an option to manually click a response. The learner received immediate feedback regarding answer selection in the form of a green, yellow, or red emoji indicating an excellent, acceptable, or poor answer, respectively. Following completion of the scenario, the learner could review detailed feedback regarding their answer selections that aligned with training on evidence-based BHAG and MI skills (B). Completion of the curriculum took approximately 15-30 minutes.

Table 1. Outcome metrics
Outcome metrics assessing system usability (System Usability Scale [SUS]), spatial presence (MEC-Spatial Presence Questionnaire [MEC-SPQ]), and implementation outcomes (Acceptability of Intervention Measure [AIM], Intervention Appropriateness Measure [IAM], and Feasibility of Intervention Measure [FIM]). The SUS is a 10-item instrument using a 5-point Likert scale from strongly disagree to strongly agree (1-5), scored by adjusting odd-numbered items (response minus 1) and even-numbered items (5 minus response), summing the adjusted values, and multiplying by 2.5 to generate a final score from 0 to 100. We assessed a subset of MEC-SPQ items including 7-item using a 5-point Likert scale from strongly disagree to strongly agree (1-5). The AIM, IAM, and FIM are each 4-item instruments using a 5-point Likert scale from completely disagree to completely agree (1-5).

Figure 2. Scatterplot distribution of individual system usability scale (SUS) scores