138 - Use of AI-generated Content in Personal Statements Among Pediatric Residency Applicants

Publication Number: 4135.138

Cole A. Labhart, University of Louisville School of Medicine, Louisville, KY, United States; Sara Multerer, University of Louisville School of Medicine, Louisville, KY, United States; Rebecca Hart, University of Louisville School of Medicine, Floyds Knobs, IN, United States; Erin Owen, University of Louisville School of Medicine, Louisville, KY, United States; Dakota J. Williams, University of Louisville School of Medicine, Louisville, KY, United States

Background: The use of generative artificial intelligence (AI) has simplified the process of academic writing, yet raised concerns about academic integrity. Little is known about the prevalence of AI use among pediatric residency applicants in development of critical application components such as personal statements.

Objective: To describe the proportion of applicants to a categorical pediatrics residency program whose personal statements had high likelihood of AI use across three application cycles, and to compare likely AI use based on applicant demographics.

Design/Methods: Personal statements (PS) of applicants to the University of Louisville Pediatric Residency Program were collected from the Electronic Residency Application Service (ERAS) and de-identified across three academic years from 2021-2024. Demographic information including medical school type (MD/DO, public/private), location (US/non-US), USMLE or COMLEX scores, and quartile rank was also collected. Statements were scanned using AI detection software Originality (version Lite 1.0.0, Originality.AI, Ontario, CA) and percent confidence for generation by AI vs. "original" (non-AI) was recorded. Proportion of PS deemed likely to be AI-generated (with >50% and > 75% confidence) were compared across demographic groups using standard descriptive statistics.

Results: A total of 836 PS were reviewed (71% MD, 65% public, 81% US); 110 (13.2%) were flagged as AI-generated in some capacity with minimum 50% confidence and 76 (9.1%) flagged AI-generated with 75% confidence. Overall, there were no significant differences in AI use based on applicant demographics at either threshold: 12.5% MD vs 14.5% DO (OR 1.19, p=0.44), 12.9% public vs 20.8% private (OR 1.077, p=0.73),12.9% US vs 18.4% non-US (OR 1.52, p=0.33), 13.2% for 1st and 2nd rank quartile vs 13.2% 3rd and 4th quartile (OR 0.997, P=989). There was, however, a significant increase in proportion of statements screening positive for AI use each year (21-22: 4.2%, 22-23 10.4%, 23-24: 12.4%, p=0.002).

Conclusion(s): Since the public release of ChatGPT in 2022, AI detection software indicated an increase in proportion of PS that were likely AI-generated among applicants to a single categorical pediatrics residency program. AI use was not associated with any specific demographic factors, indicating that applicants from any school/location type and across academic levels are likely utilizing AI for applications with increasing frequency. Further studies evaluating the use of AI among residency applicants, as well as guidelines for both applicants and programs in this use, are needed.

Proportion of Al-Positive Personal Statements at 75% Confidence
Proportion of personal statements flagged positive for AI use by both year and demographic.