441 - ChatGPT vs. Pediatric Neurology Consultation: A Mixed-Methods Study of Diagnostic and Management Accuracy

Publication Number: 4432.441

Jessica K. Creedon, Boston Children's Hospital, Dover, MA, United States; MARVIN B. HARPER, Harvard Medical School, NORTH READING, MA, United States; Andrew J. Capraro, Boston Childrens Hospital, Boston, MA, United States; Andrew M.. Fine, Boston Children's Hospital, Boston, MA, United States

Background: Access to specialty consultation in community emergency departments (EDs) is often limited, leading to unnecessary transfers and delays. Large language models (LLMs) such as ChatGPT can generate complete and accurate clinical recommendations, but their role in specialty consultation in the pediatric ED setting remains untested.

Objective: To evaluate ChatGPT's performance in generating differential diagnoses and management recommendations for pediatric neurologic emergencies.

Design/Methods: We conducted a mixed-methods study of ED encounters for patients < 18 years who received neurology consultations (11/2024-5/2025) at a tertiary children's hospital. Real cases were randomly selected, and we abstracted and entered the history, physical exam, recent clinic notes, and basic diagnostics, into a HIPAA compliant version of ChatGPT-4. We developed and used a standardized prompt to simulate a community ED clinician seeking virtual advice without neurology on site. ChatGPT generated a case summary, differential diagnosis, and management recommendations. Four pediatric emergency medicine attendings independently rated GPT outputs on 0-10 Likert scales for accuracy, completeness, efficiency, readability, and overall quality. We conducted thematic analyses on open-ended feedback about the GPT output.

Results: Fourteen cases were analyzed (median age 11 years [IQR 4-17]; 64% female). Physician reviewers rated GPT recommendations favorably across all domains (figure), with highest scores for efficiency (median 10 [IQR 10-10]) and readability (10 (10-10) and very high scores for accuracy (9, 9-10), completeness (9,9-10) and overall quality (9, 8-9). Qualitative themes (table) highlighted GPT summaries as clear and accurate, but reviewers noted a conservative bias toward neuroimaging and tertiary care center transfer.

Conclusion(s): ChatGPT rapidly generated high-quality, readable recommendations for pediatric neurologic emergencies. LLMs may augment specialty consultation in resource-limited EDs but will require calibration and training to prevent overly cautious management.

Pediatric emergency medicine attending ratings of ChatGPT-generated neurology consults compared to in-person pediatric neurology consults
Pediatric emergency medicine attending ratings of ChatGPT-generated neurology consults compared to in-person pediatric neurology consults. Ratings were based on a 10-point Likert scale across five domains: accuracy, completeness, efficiency, readability, and overall quality. ChatGPT output was rated highest in the domains of efficiency and readability (n = 14).

Thematic analysis of pediatric emergency medicine attendings' qualitative feedback on ChatGPT-generated neurology consults
Thematic analysis of pediatric emergency medicine attendings' qualitative feedback on ChatGPT-generated neurology consults, highlighting key themes, limitations, and opportunities for improving LLM use in pediatric neurologic emergencies.