307 - Transparent AI-Powered Chart Review for Assessing Guideline-Concordant Treatment of Young Children with ADHD

Publication Number: 1293.307

Yair Bannett, Stanford University School of Medicine, Stanford, CA, United States; Malvika Pillai, Stanford University School of Medicine, Stanford, CA, United States; Fatma Gunturkun, Stanford University School of Medicine, Palo Alto, CA, United States; Ingrid Luo, Stanford University School of Medicine, San Francisco, CA, United States; Tracy Huang, Stanford University School of Medicine, Menlo Park, CA, United States; Heidi M. Feldman, Stanford University School of Medicine, Palo Alto, CA, United States

Background: Previous chart review-based studies found low rates of primary care pediatrician (PCP) guideline-concordant recommendations for parent training in behavior management (PTBM) as first-line treatment for young children with attention-deficit/hyperactivity disorder (ADHD). However, chart review methods do not enable timely data extraction for learning health systems to improve patient care.

Objective: In this study, we evaluated the use of large language models (LLMs), as a scalable and transparent method for extracting electronic health record (EHR) data to assess quality-of-care.

Design/Methods: We retrospectively extracted EHR data for children aged 4-6 years seen at least twice between 2020-2024 at one of 25 community-based primary-care clinics. The study cohort included children who had a diagnosis of ADHD or ADHD-related symptoms and available clinical notes from their first encounter with an ADHD diagnosis (n=541). Three LLMs (Claude3.5, LLaMA3.3-70B, and GPT-4o) were prompted to provide a binary classification of whether the "assessment and plan" section of the clinical note included recommendation for PTBM (yes/no) and to provide an explanation for their classification. To evaluate model performance, we sampled 112 notes, including all notes that the models disagreed on. These notes were manually annotated by two annotators; disagreements were reconciled. Annotators also ranked models (1-3) on their explainability (i.e., providing relevant evidence).

Results: All three LLMs achieved excellent performance in identifying notes that contained PTBM recommendations (Table 1). Claude showed the most balanced performance (sensitivity=0.89, positive predictive value=0.95, F1-Score=0.92) and scored highest in explainability (3/3) (Table 2). LLaMA and GPT-4o had comparable performance but ranked lower in explainability. Based on model classifications, PTBM was recommended in the first ADHD visit in 26.2% of all patients (142/541) and in 32.1% of patients prescribed ADHD medications (43/134).

Conclusion(s): LLMs have the potential to generate accurate, explainable, and scalable quality-of-care measurement for ADHD from EHR free text, offering innovative methods that can be widely applicable to other chronic physical and mental health conditions.

Table 1. Model performance evaluation on sample notes (n=112)
*Explainability scores ranked by annotators range from 1 (lowest explainability) to 3 (highest explainability).

Table 2. Model Reasoning Examples (Claude 3.5)