300 - Zero-Shot, Two-Pass Prompting Improves EHR Cohort Identification for Pediatric ED Asthma/RAD

Publication Number: 1286.300

Assaf Landschaft, Boston Children's Hospital, Bergisch Gladbach, Nordrhein-Westfalen, Germany; Alexander H. Hogan, Connecticut Children's, Hartford, CT, United States; Alek C. Jorge, Connecticut Children's Medical Center, Berlin, CT, United States; Amir Kimia, Connecticut Children's Medical Center, Boston, MA, United States

Background: Cohort identification from pediatric EHR free text is increasingly done with LLMs, but accuracy drops when eligibility logic is multi-step. Pediatric ED asthma/reactive airway disease (RAD) notes mix active symptoms, historical problems, and heuristics, which challenge zero-shot classifiers. A pressing challenge is instruction-load: long, compositional prompts can reduce specificity in models. We examine whether decomposing instructions can stabilize performance for children's hospitals seeking scalable screening.

Objective: Test whether two zero-shot passes (inclusion then exclusion with deterministic fusion) outperform a single joint prompt for pediatric ED asthma/RAD cohort identification.

Design/Methods: We used clinician-annotated ED notes from children presenting to Connecticut Children's ED (July-December 2024). Following that study, we selected a 2,322 high-ranking subset at an NLP classifier sensitivity cutoff of 95% to emulate a real screening funnel with class imbalance. Two arms (Figure 1), both zero-shot: Arm 1 (single-pass): one prompt with four inclusion (I1-I4) and three exclusion (E5-E7) rules (Table 1). Arm 2 (two-pass + fusion): an inclusion-only prompt and an exclusion-only prompt, then the rule: Include if Inclusion=TRUE ∧ Exclusion=FALSE. Inference ran on llama3.3:70b-instruct-q4_K_M using affordable single-GPU hardware ( <$20k). Metrics: Sensitivity (TPR), Specificity (TNR), Precision (PPV), F1 (positive/negative), Balanced Accuracy, and Macro-F1 vs. clinician labels. We report three readouts: Arm 1; Arm 2 Step 1 (inclusion-only); Arm 2 fused decision.

Results: Arm 2 Step 1 achieved high Sensitivity 0.977 with lower Specificity 0.342, reflecting an aggressive screen. Arm 1 balanced Sensitivity 0.793 and Specificity 0.594 (PPV 0.800, Macro-F1 0.693). Arm 2 fused improved overall discrimination: Sensitivity 0.835, Specificity 0.689, PPV 0.846, Balanced Accuracy 0.762, Macro-F1 0.760: the strongest composite profile. These findings support the idea that decomposing multi-step eligibility instructions mitigates instruction-load and yields more reliable zero-shot classification.

Conclusion(s): Zero-shot prompting on affordable, single-GPU infrastructure produced clinically useful cohort filters for pediatric ED asthma/RAD notes. The two-pass + fusion strategy preserves screening sensitivity while recovering specificity, enabling scalable retrospective studies and pragmatic trials without fine-tuning. The approach generalizes to multi-rule pediatric phenotypes where temporal qualifiers and exclusions complicate simpler prompts, offering immediate value to the EHR research community.

Figure 1: Two‑Arm Evaluation of Zero‑Shot Prompting for EHR Eligibility Classification Using an Externally Annotated ED Asthma/RAD Corpus
Figure 1. Study design for zero-shot eligibility classification of pediatric ED asthma/RAD notes from Connecticut Children's Emergency Department (July-December 2024). Arm 1: single-pass joint prompt applying four inclusion (I1-I4) and three exclusion (E5-E7) rules. Arm 2: two-pass prompting: Step 1 inclusion-only, Step 2 exclusion-only, followed by deterministic fusion (Include if Inclusion = TRUE ∧ Exclusion = FALSE). Performance is reported at three points against clinician adjudication: Arm 1, Arm 2 Step 1, and Arm 2 fused.

Table 1: ED Asthma/RAD Cohort Eligibility Criteria - Operational definitions with example EHR narratives. Positive cases and near-miss negatives
Table 1. Eligibility criteria for pediatric ED asthma/RAD cohort identification. Inclusion (I1-I4) and Exclusion (E5-E7) with representative positive and challenging negative EHR narrative snippets. Negative examples are selected to be similar to positives (e.g., historical diagnoses, rule-out language, non-ED contexts) to probe zero-shot specificity and instruction-load sensitivity.

Table 2: Comparative results across three reporting points
Table 2. Comparative performance of zero-shot eligibility classification across three reporting points: ARM 1 (single-pass), ARM 2 Step 1 (inclusion-only), and ARM 2 fused (inclusion∧¬exclusion), evaluated against clinician classification. Each block shows the 2×2 confusion matrix (rows = Human/Reference; columns = Model) and summary metrics: Sensitivity (TPR), Specificity (TNR), Precision (PPV), F1 (positive/negative), Balanced Accuracy, and Macro-F1.