388 - Using Reverse Image Search and Artificial Intelligence Technology to Identify Prescription Pills

Publication Number: 3375.388

Lillian M. Ravikoff, Northwell Health, Bronx, NY, United States; Sherry Rizkalla, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Lake Success, NY, United States; Joseph Mekhail, Northwell Health, New York, NY, United States; Caroline Howard, Northwell Health, New York, NY, United States; Ruth Milanaik, Northwell Health, Great Neck, NY, United States

Background: Every day, approximately 165 children are treated in U.S. emergency rooms for medication-related poisoning. When a medication is accidentally ingested, it is vital to identify the medication and seek emergency care. While the proper course of action upon accidental ingestion is to contact Poison Control and go to the emergency room, not every caregiver may take this action. In recent years, publicly accessible artificial intelligence (AI) platforms have evolved to include image upload features for visual content analysis. These features have been advertised as capable of identifying information on plants, tools, or products. Members of the public may be tempted to use this as a pill identification tool given recent survey data showing 57% of U.S. adults use AI at least several times per week.

Objective: To assess the accuracy of ChatGPT, Gemini, Claude, Copilot, and Google Lens in identifying prescription medications from images.

Design/Methods: 49 of the most commonly prescribed medications as determined by the Medical Expenditure Panel Survey and Vivo Health Pharmacy were photographed and run through ChatGPT, Gemini, Claude, Copilot, and Google Lens. Only medications in tablet and capsule form were used. Photographs were taken on a paper towel using an iPhone 13 Pro. AIs were prompted with photos including front and back of the tablet or markings on the capsule and no typed text prompt [Figure 1].

Results: AI outputs were categorized as correct, incorrect, or no-answer. The no-answer category included outputs indicating that the pill could not be identified, a refusal to identify the pill, or an identification of something other than a medication. In total, 18.37% of responses across all AI platforms were correct identifications with 44.49% of trials resulting in no-answer. Claude refused to attempt to identify any medications, noting safety concerns. ChatGPT, Gemini, and Copilot included warnings along with their responses; Google Lens did not. The following statistics exclude Claude. 22.96% correct, 46.43% incorrect, 30.61% no answer. Copilot had the lowest success rate at 8.16% and Gemini had the highest success rate at 36.73% [Figure 2].

Conclusion(s): Artificial intelligence tools are unreliable in identifying prescription medications from images and should not be used for this purpose. Physicians and pharmacists should inform patients and caregivers of the risks of accidentally ingesting medications and should encourage emergency room visits for any suspected cases. All AI platforms should be designed to consistently provide an error message when presented with a request to identify a medication.

Figure 1: Cyclobenzaprine 5mg Tablet
Cyclobenzaprine was unidentified by ChatGPT, Copilot, and Google Lens. Claude refused to answer. Gemini identified it as a combination of Levonorgestrel 0.1 mg and Ethinyl Estradiol 0.02 mg.

Figure 2: Summary of Responses Across Artificial Intelligence Platforms