157 - Evidence-Based Conversations: AI Chatbots to Improve Provider Communication on RSV Immunoprophylaxis

Publication Number: 4154.157

Eury Hong, University of Pennsylvania, Philadelphia, PA, United States; Victor N. Oboli, Yale University School of Medicine, New Haven, CT, United States; Marc Auerbach, Yale School of Medicine, New Haven, CT, United States; Douglas J.. Opel, Seattle Children's Research Institute, Seattle, WA, United States; Annika M. Hofstetter, University of Washington/Seattle Children's Research Institute, Seattle, WA, United States; Jaspreet Loyal, Yale School of Medicine, New Haven, CT, United States; Elena Aragona, Yale School of Medicine, New Haven, CT, United States; Sonny Tat, University of California, San Francisco, School of Medicine, San Francisco, CA, United States

Background: Bronchiolitis remains the leading cause of hospitalization in infants under 12 months. The American Academy of Pediatrics recommends RSV immunoprophylaxis to prevent RSV infections and reduce hospitalizations. Despite its proven benefits, parental uptake remains suboptimal, with some evidence to suggest that limited knowledge and awareness of RSV immunoprophylaxis among pediatricians may represent a barrier to uptake. Artificial intelligence (AI) chatbots may be a useful tool to improve vaccine communication between pediatricians and parents.

Objective: To determine whether a Retrieval-Augmented Generation (RAG)-powered AI chatbot (Figure 1) during standardized role-play encounters between pediatricians and caregivers improves pediatrician knowledge, self-efficacy, and the perceived usefulness of AI as a real-time support tool.

Design/Methods: This study was conducted via a video platform. Eligible participants were pediatric residents at two pediatric training programs in the U.S. Participants joined in 30-minute role-play sessions with a standardized caregiver, using a semi-structured script (Figure 2). Participants were given the option to engage with the AI chatbot, either during preparation and/or while interacting with the caregiver. Surveys on knowledge (benefits, administration, evidence), self-efficacy (addressing concerns, communicating evidence in plain language, recommending current sources), and chatbot acceptability were administered at baseline, immediately post-session, and two weeks later. Responses were analyzed using descriptive statistics and Wilcoxon Test. Chatbot interaction data (query type, frequency, duration) were also examined.

Results: Ten pediatric residents participated. 70% used the chatbot in both preparation and interaction with the standardized caregiver. Self-reported measures of knowledge (p=.004, ΔMedian= +.67) and self-efficacy (p=.004, ΔMedian= +.83) improved immediately post-session compared to baseline (Table 1). Among all items, providing evidence-based information showed the largest gain (p <.01, ΔMedian= +2.0). Participants valued the chatbot's trustworthiness and provision of verifiable source links, though some noted delays in responses. At the two-week follow-up survey, knowledge and self-efficacy declined slightly but remained above baseline levels (p=.03, ΔMedian=+.75; p.=02, ΔMedian=+.75, respectively). Half of the participants reused the chatbot for other clinical/educational purposes.

Conclusion(s): A RAG-based AI chatbot may bridge physicians' knowledge and confidence gaps in RSV immunoprophylaxis counseling by delivering real-time, trusted information.

Figure 1. AI Chatbot & Standardized Caregiver Interaction Timeline


Figure 2. Retrieval-Augmented Generation(RAG)-based Model


Table 1. Wilcoxon Signed-Rank Test on Survey Results