142 - Advancing Large Language Models in Solving Pediatric Medical Questions: Comparing Recent Models, Prior Models, and Residents on Pediatric In-Training Examinations

Publication Number: 4139.142

Eunseong Seo, Asan Medical Center, Seoul, Seoul, Seoul-t'ukpyolsi, Republic of Korea; Jun Sung Park, asan medical center, seoul, Seoul-t'ukpyolsi, Republic of Korea

Background: Recent vision-enabled multimodal large language models (LLMs) have achieved strong performance on high-stakes medical examinations, yet their capabilities in pediatrics, particularly for image-based questions, remain underexplored.

Objective: To evaluate the performance of recent and prior multimodal LLMs on pediatric in-training examinations (ITEs), focusing on their ability to handle both text and image-based questions.

Design/Methods: Institutional review board approval and informed consent requirements were waived as the study did not involve patient data or human participants (#2025-0722).
Analyzed 498 unique questions with Korean-English terminologies, taken from 10 pediatric in-training examinations (ITEs) conducted by a single pediatric department between 2016 and 2023.
Approximately 22% of items contained medical images.
Three recent publicly accessible LLMs (GPT-4.1, Gemini-2.5-Pro, Claude-4.1-Opus) and three prior models (GPT-4o, Gemini-1.5-Pro, Claude-3.5-Sonnet) were tested.
Performance was compared between the LLMs and fourth-year residents (R4).

Results: Recent LLMs significantly outperformed fourth-year residents (R4) on text-only items, achieving a proportion correct (PC) of 77.7-78.9% versus 70.1% for R4 (all P < 0.008).
On text-only items, recent LLMs showed a superior PC compared with R4 (80.1-81.0% vs. 69.6%).
However, none of the evaluated LLMs surpassed R4 performance on image-included questions, with both prior and recent models consistently performing worse on image-included items than text-only questions.
Outputs from the models demonstrated high repeatability (intraclass correlation coefficient >0.98) across most models.

Conclusion(s): Multimodal LLMs demonstrated high performance on the Pediatric ITE, with improvements over the past year that surpassed the performance of senior residents in text-only items. However, their performance on image-included questions was inferior to that of text-only questions and did not exceed that of senior residents. LLMs can be used as simulation tools for diagnosing and treating various conditions, providing instant feedback for self-directed learning. However, their performance on image-based questions lags behind text-based tasks, indicating the need for further improvement. A hybrid model combining text and image-based learning could enhance training, and LLMs could also support continuous assessment, helping residents stay updated in rapidly evolving medical fields.

Flow diagram of LLMs, with published performance on pediatric field in-training and board-examination questions and the LLM release timeline.
Abbreviations: LLM, large language model; GPT, Generative Pre-trained Transformer.
Text-only LLMs excluded
Bold: LLMs Used in This Study