159 - Are Large Language Models as Competent as Healthcare Professionals in Neonatal Resuscitation? A Controlled Comparative Study Using Multi-Source Assessment

Publication Number: 4156.159

Chenguang Xu, The University of Hong Kong - Shenzhen Hospital, Shenzhen, Guangdong, China (People's Republic); Yihua Chen, Hongkong university Shenzhen hosptial, Shenzhen, Guangdong, China (People's Republic); Yin Xue, The University of Hong Kong-Shenzhen Hospital, ShenZhen, Guangdong, China (People's Republic); Shelley Skelding, University of Alberta, Edmonton, AB, Canada; Dianna Wang, University of Alberta Faculty of Medicine and Dentistry, Edmonton, AB, Canada; Qianshen Zhang, The University of Hongkong, Shenzhen Hospital, SHENZHEN, Guangdong, China (People's Republic); Georg Schmolzer, University of Alberta Faculty of Medicine and Dentistry, Edmonton, AB, Canada; Po-Yin Cheung, University of Alberta Faculty of Medicine and Dentistry, Edmonton, AB, Canada

Background: Artificial intelligence and large language models (LLMs) have developed rapidly in recent years and involved in medical education, in addition to clinical care. However, It remains unknown how the performance of LLMs in neonatal resuscitation compares to that of healthcare professionals (HCPs).

Objective: To investigate and compare (1) the capability of LLMs with those of participants in the neonatal resuscitation training in Canada and China; and (2) the performance of LLMs on 3 sources of neonatal resuscitation examinations.

Design/Methods: This study evaluated the performance of ChatGPT-5 and DeepSeek-R1 on neonatal resuscitation questions from neonatal resuscitation workshop (NRW) in China (38 questions, Chinese), the NRP® 8th edition textbook (75 questions, English), and Kahoot quizzes in Canada (21 questions, English). Each LLM was tested three times per question with one-week intervals in August and September, 2025. Questions were posed in their original language using simple prompt: "Please answer the following questions." The scores of 286 HCPs in the NRW in China (2022-2024), and Kahoot scores of 251 HCPs in NRP training in Canada (2022-2025) were reviewed. The primary outcome was correctness and reliability of responses. The secondary outcomes included correctness for scenario-based questions, MCQs and open-ended questions.

Results: In the Chinese NRW examinations, both ChatGPT and DeepSeek achieved high accuracy rates (90% and 92%) , matching the performance of HCPs after completing the workshop (92%). These results were significantly higher than the HCPs pre-training accuracy of 82% (P < 0.05). In the Kahoot quizzes, ChatGPT demonstrated significantly higher accuracy rate than DeepSeek (94% vs. 78%, P=0.011) and that of HCPs in NRP training (94% vs.78%, P=0.003). ChatGPT achieved higher scores in scenario-based questions than non-scenario-based questions (100% vs. 83%, P=0.004). ChatGPT and DeepSeek showed similar accuracy in NRP® 8th edition textbook questions (96% vs. 94%). Both LLMs scored significantly higher on MCQs than open-ended questions in the NRP® 8th edition textbook. Simple calculation mistakes and cross-lingual inconsistency were noted in some answers by both LLMs.

Conclusion(s): Both ChatGPT and DeepSeek showed high correctness and consistency in answering neonatal resuscitation questions from different sources in English and Chinese. ChatGPT's accuracy rate in the Kahoot quiz exceeded that of HCPs and its superior performance in answering scenario-based questions suggests the potential for LLMs to serve as valuable tools in neonatal resuscitation training and education.

Table 1 Performance comparison (overall accuracy) between LLMs and healthcare professionals (HCPs).
＊: n/N represents the proportion of total correct answers over the total number of questions.
ɸ: P<0.05 vs. overall accuracy of HCPs pre-training
LLMs: large language models
NRW: neonatal resuscitation workshop

Table 2 Accuracy and reliability analysis of LLMs across repeated tests.
LLMs: large language models
NRW: neonatal resuscitation workshop

Table 3 Accuracy rates by question format, model, and data source.
LLMs: large language models
NRW: neonatal resuscitation workshop