303 - From Zero-Shot to Chain-of-Thought: Testing Advanced Prompting on Neonatology Board-Style Questions

Publication Number: 1289.303

Ashley Cowan, University of Colorado School of Medicine, Denver, CO, United States; Reece Barry, university of california santa cruz, centennial, CO, United States; Atticus Conn, Northeastern University, Centennial, CO, United States; Julia A. Pisc, University of Colorado School of Medicine, Wheat Ridge, CO, United States; James s. Barry, University of Colorado School of Medicine, centennial, CO, United States

Background: ChatGPT, Claude, and Gemini are publicly available large language models (LLMs) capable of generating human-like responses and have achieved passing scores on multiple medical licensing exams. Recent studies suggest advanced prompting techniques can enhance model accuracy and reasoning transparency. We sought to better understand how advanced prompting strategies influence accuracy on neonatology board-style text- and image- based questions.

Objective: Compare correct response rates of multiple LLMs on neonatology board-style questions and evaluate whether advanced prompting improves performance compared to zero-shot prompting.

Design/Methods: This comparative study evaluated ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Flash on 920 questions from Neonatology Review: Q&A (3rd ed). Each model was tested using zero-shot prompting. 85 questions missed by all 3 models were retested using advanced prompts including role specification, ensemble reasoning, chain-of-thought analysis, temperature adjustment, and step-back reflection. ChatGPT-5, a newly released reason-enabled multimodal foundational model, was tested using zero-shot prompting for comparison. Scores were analyzed with descriptive statistics and chi-square testing.

Results: Accuracy differed significantly across models (χ²(3) = 77.8, p < 0.001). With simple zero-shot prompting, ChatGPT-4o scored 81%, Claude 71%, and Gemini 67%. ChatGPT-5 achieved 82%. Both ChatGPT models outperformed Claude and Gemini (p < 0.001); differences between ChatGPT models were not significant. For the 85 questions incorrectly answered by all models with zero-shot prompting, advanced prompting increased correct responses: ChatGPT-4o 29.4%, Claude 25.9%, Gemini 36.5%, and ChatGPT-5 35%. Model performance varied by topic domain: Ethics had the highest accuracy (>92%), Gastroenterology and Inborn Errors of Metabolism the lowest.

Conclusion(s): ChatGPT-5 answered more neonatology board-style questions correctly than other LLMs. Advanced prompting improved performance across all models compared to zero-shot prompting, highlighting the potential of prompt engineering to enhance subspecialty knowledge retrieval and application. However, differences between prompting technique and model types did not reach statistical significance, suggesting limitations related to relative scarcity of neonatal-perinatal content in LLM training datasets. Although various prompting techniques may increase accuracy, LLMs remain insufficiently, reliable for educational or clinical use without independent verification of their output.

Table 1. Zero-Shot Prompting Correct Response Rates Across Models
Percentage of correct answers across the models tested using simple, zero-shot prompting

Table 2. Large Language Models Percentage Correct by Neonatology Topic
Percentage correct divided by neonatology topics across the models tested. Ethics/Teaching achieved the highest correct response rates across models while Gastroenterology and Inborn Errors of Metabolism were the lowest.