TOP 8 - Validity Evidence for a Novel Tool for Evaluation of Clinical Information Synthesis

Publication Number: 1768.TOP 8

Brianne C. Drury, The Barbara Bush Children's Hospital at Maine Medical Center, Portland, ME, United States; Jamie Fey, Tufts University School of Medicine, Portland, ME, United States; Leah Mallory, The Barbara Bush Children's Hospital at Maine Medical Center, Portland, ME, United States; Amy Buczkowski, The Barbara Bush Children's Hospital at Maine Medical Center, Portland, ME, United States; Karyn Parsons, The Barbara Bush Children's Hospital at Maine Medical Center, Portland, ME, United States; Christopher Motyl, The Barbara Bush Children's Hospital at Maine Medical Center, Portland, ME, United States; Danielle Doctor, MaineHealth Barbara Bush Children’s Hospital, Portland, ME, United States; Amanda Gagnon, The Barbara Bush Children's Hospital at Maine Medical Center, Portland, ME, United States; Shannon M. Bennett, MaineHealth Maine Medical Center Barbara Bush Children's Hospital, Saco, ME, United States; Norah Emara, MassGeneral Hospital for Children, Boston, MA, United States; Cathleen Renzi Gulen, Tufts University School of Medicine, Southborough, MA, United States

Background: Information synthesis is a vital skill in clinical practice, commonly used by medical professionals to communicate concepts to other colleagues, patients, and families; in the setting of documentation; handoff; and in-person discussion. It is also critical to clinical reasoning. The cognitive process of synthesis - defined as the gathering and assembly of facts, ideas, or information to create a novel concept – is both complex and intuitive. As such, it is a difficult skill to assess in trainees and is difficult to teach. It has been identified as an area of weakness in trainees who have required remediation over the course of their education (1). Without valid methods to assess this critical skill, educators cannot accurately identify at-risk learners. This tool allows for early identification and intervention by educators to optimize learner success in multiple clinical realms.

Citations:
1. Grainger B, et al. Predictors of medical student remediation and their underlying causes: early lessons from a curriculum change in the University of Auckland Medical Programme. New Zealand Medical Journal. 2017; 130(1460): 73-82.

Objective: To develop an assessment tool to evaluate trainee ability to synthesize clinical information and present content and response process validity evidence according to Messick’s framework.

Design/Methods: This study was determined not human subjects research by the MaineHealth IRB. We developed a novel assessment tool, the Synthesis of Clinical Information (SOCI) rubric (Figure 1), using literature review, expert input, and iterative discussion, to assess information synthesis as a trainee skill in the context of patient handoffs.

The SOCI tool will be applied to fictional pre-recorded patient handoff exemplars created by our study team specifically for this pilot testing. We have created three discrete clinical cases, each with three associated tiered handoffs. The handoffs were stratified according to quality by senior clinicians into gold, silver, and bronze categories, using consistent criteria (Table 1). The handoffs will be audio recorded and presented to ten experienced pediatric hospitalists who are familiar with the cases. They will listen to them in random order and score each handoff using the SOCI tool. Figure 2 illustrates this process. Inter-rater reliability and repeated measures analysis of variance (ANOVA) comparing scores of the gold, silver and bronze exemplars will be calculated using R studio statistical software. We expect to be completed by mid-December.

Figure 1: Synthesis of Clinical Information (SOCI) Assessment Tool
Figure 1 SOCI rubric.pdfThis is an example of the worksheet with the novel assessment tool provided to clinical experts for use in scoring fictional pre-recorded patient handoffs.

Figure 2: Project Workflow
Synthesis Project Workflow.pdfThe above depicts the anticipated study workflow.

Table 1: Handoff Standard Stratification Criteria
Table 1 Exemplar Criteria.pdfTable 1 displays the criteria for stratification of case handoffs into gold, silver, and bronze categories, using the I-PASS (Illness severity, Patient summary, Action list, Situation awareness and contingency planning, and Synthesis by receiver).