744 - Use of Generative Artificial Intelligence in Pediatric Residency Personal Statements

Publication Number: 3720.744

Lisa DelSignore, Yale School of Medicine, New Haven, CT, United States; Gary K. Soffer, Yale School of Medicine, New Haven, CT, United States; Brooke B. Redmond, Yale School of Medicine, New Haven, CT, United States; Uma Phatak, Yale School of Medicine, New Haven, CT, United States; Farzana Pashankar, Yale School of Medicine, New Haven, CT, United States; Alexander Koral, Yale-New Haven Children's Hospital, New Haven, CT, United States; Ruchika Karnik, Yale School of Medicine, New Haven, CT, United States; Lindsay Johnston, Yale-New Haven Children's Hospital, New Haven, CT, United States; Robert Elder, Yale University, School of Medicine, New Haven, CT, United States; Laura Chen, Yale School of Medicine, New Haven, CT, United States; Christie J. Bruno, Yale School of Medicine, Glastonbury, CT, United States; Adam Berkwitt, Yale New Haven Health Hospital - - New Haven, CT, New Haven, CT, United States; Leah K. Amster, Vanderbilt University, Fairfield, CT, United States; Melissa L.. Langhan, Yale university, New Haven, CT, United States

Background: The widespread use of large language models (i.e. ChatGPT) has caused concern about personal statement authenticity in residency applications. This may threaten holistic reviews and assessment of "goodness of fit" between applicants and residency programs. Previous studies describe challenges by human reviewers in determining which personal statements are written by AI versus those that are human-generated.

Objective: To assess the percent of text generated by AI as determined by AI detection platforms in personal statements for pediatric residency applications and its frequency of use between 2022 (pre-ChatGPT release) to 2024.

Design/Methods: Retrospective cross-sectional study of de-identified Electronic Residency Application System (ERAS) applications submitted to a single pediatric residency program from 2022-2024. Demographic information was abstracted (gender, self-identity, medical school country) and interview status. Personal statements were entered into two AI detectors (Quillbot and ZeroGPT) to assess % AI generated text. The % AI generated text was compared across applicant characteristics and time.

Results: 3521 applications were assessed over the study period (1255 in 2022, 1139 in 2023, 1157 in 2024). Applicant data are presented in Table 1. AI detection in each platform and by applicant characteristics are presented in Table 2. The correlation between the total AI detected in each platform was .68 (95% CI .66, .7). Median AI detection in Quillbot was 0 across all years (83% applicants with 0 detection in 2022, 36% in 2023, 51% in 2024). There was a significant increase in mean AI detection in both platforms between 2022 and 2024 (p <.001). Significant differences were found by self-identity and medical school country across platforms (p <.001). After adjusting for other applicant characteristics, there was no statistically significant difference in AI detection among applicants invited to interview and those who did not receive an invitation to interview (mean difference .5, 95% CI -1.8, 2.8).

Conclusion(s): Generative AI use in pediatric residency personal statements is increasing. Applicants may use AI to overcome English language barriers. There is significant variability across platforms in what text is considered AI generated, questioning their accuracy. Generating consensus among stakeholders on how to integrate and monitor use of this technology for applicants may be beneficial.

Table 1. Applicant demographics and characteristics by subspecialty


Mean AI detection in each platform by applicant characteristics