155 - AI vs Proctor: Comparing the Detectability and Accuracy of AI-Based and Traditional Cheating Methods

Publication Number: 4152.155

Caroline Howard, Northwell Health, New York, NY, United States; Joseph Mekhail, Northwell Health, New York, NY, United States; Lillian M. Ravikoff, Northwell Health, Bronx, NY, United States; Ruth Milanaik, Northwell Health, Great Neck, NY, United States

Background: The rapid growth of AI has heightened long-standing concerns about academic integrity in medical education. In 2013, up to 58% of medical students reported academic dishonesty, including the use of unauthorized notes during exams. Cheating concerns have intensified as smartphone AI tools can read a photographed exam question and return an answer in seconds. By 2025, some AI systems were even able to achieve a perfect score on the USMLE. Both note-based and AI-based cheating methods can be difficult to detect with typical proctoring practices, yet there is limited comparison of how detectable these methods are or how much they affect exam performance.

Objective: To assess the detectability and efficacy of AI-based and note-based cheating methods during exams.

Design/Methods: Each exam included 25 NYS standardized exam questions across topics in Biology, Global History and Geography, Earth Science, US History and Government, Chemistry, and Sequential Mathematics II. 18 students participated in 6 testing rounds, each 20 minutes long. Pilot testing identified the most effective AI cheating method (Snapchat AI, powered by ChatGPT and Gemini) and typical proctor-behavior (2 proctors located at the front of the classroom, roaming infrequently). Each round, students were randomly assigned to use the AI-based cheating method, note-based cheating method, or no cheating. Proctor-perceived cheating reports and exam scores were recorded. Scores were normalized to account for variability in test difficulty between rounds.

Results: Welch's two-sample t-test indicated that the mean test score was higher for AI-based cheating methods compared to no cheating (p=.031); however, no difference in mean test score was found between note-based cheating methods and no cheating (Figure 1). Note-based cheating methods were less detectable than AI-based cheating methods (p=.044), though both cheating methods went widely undetected by proctors (new: 32.1% detection; old: 7.14% detection) (Figure 2).

Conclusion(s): AI-based cheating significantly increased exam performance, highlighting the effectiveness of this method and the need for robust preventative measures. Although AI-based cheating was detected more frequently than traditional methods, overall detection rates remained low, indicating that typical proctoring practices may be insufficient to maintain academic integrity. These findings emphasize the growing challenge AI poses for medical education, where high-stakes exams are critical for licensure and competency assessment. Future studies should evaluate additional AI tools and explore strategies for improving proctor detection.

Figure 1. Difference in test score between cheating and not cheating conditions.
Boxplot shows the difference in test score between AI-based cheating methods, note-based cheating methods, and no cheating conditions. AI-based cheating is associated with higher test scores compared to no cheating (p=.031).

Figure 2. Proctor Detection Rates by Cheating Method
Proctors identified whether students were cheating. Bars show the percentage of times cheating was detected (hits). AI-based cheating (new) was detected more frequently than note-based cheating methods (old) (p=.044), though both went largely undetected.