Cedars-Sinai Study Finds AI Tool Could Improve Care in Virtual Urgent Care Settings

A new study from one of the nation’s leading medical centers suggests use of artificial intelligence (AI) in virtual visits could improve decision-making and adherence to guideline-directed care in virtual urgent care settings.

Led by investigators at Cedars-Sinai Medical Center, results of the study, which were presented at the American College of Physicians Internal Medicine Meeting and published simultaneously in the Annals of Internal Medicine, suggest AI recommendations were more often rated better quality in instances when AI and physician recommendations differed. According to investigators, the AI tool outperformed physicians in identifying opportunities to support guideline-adherent care, but physicians were better at adapting recommendations to changing information during consultations.1

“We found that initial AI recommendations for common complaints in an urgent care setting were rated higher than final physician recommendations,” said Joshua Pevnick, MD, MSHS, codirector of the Cedars-Sinai Division of Informatics, associate professor of Medicine and cosenior investigator of the study.2 “Artificial intelligence, as an example, was especially successful in flagging urinary tract infections potentially caused by antibiotic-resistant bacteria and suggesting a culture be ordered before prescribing medications.”

The explosion in AI tools and how to best incorporate these new capabilities into medical care has been a prominent point of discussion in medical circles in recent years. A retrospective cohort study conducted within Cedars-Sinai Connect, a virtual urgent care clinic using an AI-powered chat intake system, the current study assessed the concordance and quality of initial AI-generated recommendations compared to final physician decisions for adults presenting with common symptoms, including respiratory, urinary, vaginal, eye, or dental concerns.1,2

The AI tool leveraged demographic information provided via a mobile app at initiate visits and conducted a structured dynamic interview with the intent of gathering symptom information and medical history. According to investigators, patients answer an average of 25 questions in 5 minutes.1

Using the aforementioned data and electronic health records, the algorithm presents patients with possible diagnoses to explain their symptoms and then allows patients to initiate a video visit with a physician.1

The analysis included 461 physician-managed visits between June and July 2024 where the AI tool generated diagnosis and management recommendations with sufficient confidence. Final physician decisions were compared to AI recommendations on diagnoses, prescriptions, lab orders, and referrals.1

Results showed AI and physician recommendations were concordant in 56.8% of visits. However, when independent physician adjudicators scored the quality of recommendations, AI recommendations were rated as optimal more often than physician decisions (77.1% vs 67.1%). In most cases (67.9%), quality scores between AI and physicians were equivalent. AI recommendations were rated higher in 20.8% of cases, while physician decisions scored higher in 11.3%.1

Investigators acknowledged key limitations within their study design, including the retrospective single-center nature of the study and their inability to confirm whether physicians viewed the AI recommendations before finalizing care decisions.1

“The major uncertainty of this study is whether physicians scrolled down to view the prescribing, ordering, referral or other management suggestions made by AI, and whether they incorporated these recommendations into their clinical decision-making,” said Caroline Goldzweig, MD, Cedars-Sinai Medical Network chief medical officer and cosenior author of the study.2 “The fact that the AI recommendations were often rated as higher quality than physician decisions, however, suggests that AI decision support, when implemented effectively at the point of care, has the potential to improve clinical decision-making for common and acute conditions.”

References:

1. Zeltzer D, Kugler Z, Hayat L, et al. Comparison of Initial Artificial Intelligence (AI) and Final Physician Recommendations in AI-Assisted Virtual Urgent Care Visits. Annals of Internal Medicine. Published online April 4, 2025. doi: 10.7326/annals-24-03283

2. Center CSM. Artificial Intelligence Has Potential to Aid Physician Decisions During Virtual Urgent Care. Artificial Intelligence Has Potential to Aid Physician Decisions During Virtual Urgent Care . Published April 4, 2025. Accessed April 6, 2025. https://www.cedars-sinai.org/newsroom/artificial-intelligence-has-potential-to-aid-physician-decisions-during-virtual-urgent-care/.