366 - Performance of GPT-5 for Extraction of Key Clinical Characteristics Around Oseltamivir Use in Hospitalized Pediatric Influenza Patients

Publication Number: 4359.366

Prachi M. Keni-McCray, Lucile Packard Children's Hospital Stanford, Menlo Park, CA, United States; Yair Bannett, Stanford University School of Medicine, Stanford, CA, United States; Christine Cunningham, Stanford University School of Medicine, Palo Alto, CA, United States; Fatma Gunturkun, Stanford University School of Medicine, Palo Alto, CA, United States; Alan Schroeder, Stanford University School of Medicine, Palo Alto, CA, United States; Hannah K. Bassett, Stanford University School of Medicine, Redwood City, CA, United States

Background: The use of oseltamivir for all hospitalized children with influenza remains a controversial topic because there is a lack of randomized trials and symptom duration is often >2 days in this population. Administrative database studies have suggested benefits of oseltamivir use, but these studies are biased by exposure misclassification and lack of symptom duration data. Use of large language models (LLMs) is emerging as an efficient approach to extracting relevant information from clinical documentation compared to manual chart review (MCR), but its utility has not been studied or validated for pediatric influenza research.

Objective: Assess the performance of a secure GPT-5 (SGPT) compared to MCR in extracting key clinical characteristics from admission and discharge documentation of children who were hospitalized with a diagnosis code for influenza. Characteristics include confirmation of influenza infection, reason for admission, symptom duration, and oseltamivir use prior to admission (Table 1).

Design/Methods: This single-center retrospective cohort study assessed 196 patients ≤ 18 years old, hospitalized after September 1, 2015, with an ICD-10 code for influenza. Two clinicians performed MCR. They jointly reviewed 40 charts to create reviewer annotation guidelines and iteratively developed a prompt for SGPT. 156 charts were then manually reviewed to create the ground truth testing dataset and the outputs from SGPT were compared for each clinical characteristic. Outcomes included accuracy, sensitivity, precision (positive predictive value) and F1 score.

Results: Compared to MCR as ground truth, SGPT accuracy on each characteristic ranged from 0.94 to 0.97, and F1 scores ranged from 0.95 to 0.99 (Figure 2). SGPT output for symptom duration differed from MCR on 14 (12%) charts, but SGPT was found to more accurately follow the annotation guidelines or use information missed by the manual reviewer in 7 of 14 charts (Table 3). SGPT accurately determined whether symptom duration was ≥ 2 days in 114 out of 116 charts (98%).

Conclusion(s): SGPT reliably extracts clinical characteristics from unstructured documentation of children hospitalized with influenza, including instances where SGPT outperforms MCR. In the absence of randomized trials of oseltamivir use in hospitalized children with influenza, clinicians need high quality retrospective comparative effectiveness data to guide decision making. This work shows how LLMs can expedite and enhance large EHR database (e.g. PEDSnet, Epic Cosmos, Kaiser) studies to further elucidate the efficacy of oseltamivir use in children hospitalized with influenza.

Table 1: Desired Clinical Characteristics to be Extracted Using GPT-5


Figure 2: SGPT Performance Compared to Manual Chart Review
Accuracy: # Correct Predictions by SGPT / Total Predictions. Sensitivity: # Correct Positive Predictions by SGPT / Total Positive Cases. Precision (Positive Predictive Value): True Positives / Total Positive Predictions by SGPT. F1 Score: A measure of SGPT's performance that balances positive predictive value and sensitivity.

Table 3: Analysis of Discrepancies between SGPT and MCR Output for Duration of Symptoms
This table highlights the discrepant charts for the symptom duration characteristic. For each discrepant chart, repeat manual review was conducted to determine whether SGPT or MCR more closely followed the annotation guidelines or prompt. The results are shown here.