541 - Performance of Open-Source LLMs in Identifying Pediatric Pneumonia from Free-Text Chest Radiograph Reports

Publication Number: 2526.541

Saki Amagai, Northwestern University The Feinberg School of Medicine, Chicago, IL, United States; Elizabeth C. Powell, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, United States; Elizabeth Alpern, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, United States; Todd A. Florin, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, United States; Yikuan Li, George Mason University, Fairfax, VA, United States; David M.. Liebovitz, Northwestern University, Chicago, IL, United States; Yuan Luo, Northwestern University The Feinberg School of Medicine, Chicago, IL, United States; Juan C. Espinoza, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, United States; Anthony Wong, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, United States; Sriram Ramgopal, Ann & Robert H. Lurie Children's Hospital of Chicago, CHICAGO, IL, United States

Background: Performance of Open-Source LLMs in Identifying Pediatric Pneumonia from Free-Text Chest Radiograph Reports

Objective: To develop and internally validate an automated classification of chest radiograph (CXR) reports for community-acquired pneumonia in children.

Design/Methods: We performed a retrospective single-center study using 1,000 pediatric emergency department encounters (2016-2022) with CXR. Reports were adjudicated by two physicians as positive, negative, or indeterminate for pneumonia. We evaluated five open-source LLMs (Gemma2 9B, Gemma2 27B, Falcon 8B, DeepSeek R1 Distill Llama 8B, Llama3.1 8B) on a 70/30 train-test split for an outcome of pneumonia. We reported performance metrics for both three-class and binary classification (pneumonia and indeterminate vs. no pneumonia).

Results: The median patient age of 4.2 years (IQR 1.7-10.5), and 54.4% were admitted from the ED (Table). After clinician adjudication, 27.8% of reports were labeled pneumonia, 13.7% indeterminate, and 58.5% no pneumonia. Among the LLMs, Gemma2 9B demonstrated strong performance in identifying pneumonia, with a sensitivity of 0.82 (95% CI 0.73-0.90), specificity of 0.94 (95% CI 0.90-0.97), and F1 score of 0.82 (95% CI 0.75-0.88). It also performed well for no pneumonia (F1=0.97, 95% CI 0.95-0.99) and moderately for indeterminate (F1=0.66, 95% CI 0.55-0.75). Incorporating RAG did not improve performance for Gemma2 9B. Gemma2 27B showed slightly higher sensitivity (0.88, 95% CI 0.80-0.95) but lower specificity (0.90, 95% CI 0.86-0.94). Llama3.1 8B underperformed (pneumonia F1=0.54, 95% CI 0.45-0.62), while Falcon 7B had high sensitivity (0.98, 95% CI 0.93, 1.00) but lower specificity (0.74, 95% CI 0.67-0.80), and achieved perfect specificity for indeterminate cases (Figure). In the binary classification task (i.e., grouping indeterminate with pneumonia), Gemma2 9B and Gemma2 27B achieved F1 scores of 0.97 and 0.93, respectively, both higher than their original three-class F1 scores. The Gemma 29B model demonstrated both high sensitivity (0.95, 95% CI 0.92-0.98) and specificity (0.99, 95% CI 0.96-1.00). All LLMs substantially outperformed traditional NLP classifiers such as XGBoost, random forest, and logistic regression.

Conclusion(s): Open-source LLMs accurately classified pediatric pneumonia from CXR reports. LLMs showed strong agreement with expert reviewers and were effective at identifying both clear and borderline cases. These findings support the feasibility of integrating LLMs into decision support and quality improvement pipelines to enhance radiographic interpretation and improve pediatric emergency care.

Table. Demographics of study sample.
ED, emergency department; ICD-10-CM, International Classification of Disease, 10th revision

Figure 1. Sensitivity, specificity, and F1 scores for three-class analysis for outcome labels of (a) pneumonia, (b) no pneumonia, and (c) indeterminate.
Error bars represent 95% confidence intervals.