A fresh research assessment has intensified the debate over which artificial intelligence systems deliver superior results for clinical decision-making. OpenEvidence, a specialized medical platform, demonstrated stronger performance than OpenAI's GPT-5.5, Google's Gemini 3.1 Pro, and Anthropic's Claude Opus 4.8 when evaluated by practicing physicians on real medical questions, according to preliminary findings posted this month.
The study, posted as a preprint on arXiv without yet undergoing peer review, directly challenges conclusions published in Nature Medicine just weeks earlier. That June article argued general-purpose large language models had surpassed dedicated clinical tools. The new research tells a markedly different story.
Study Design and Results
According to Becker's Hospital Review, researchers recruited 149 physicians across 36 states to evaluate responses to clinical questions. The evaluation included 620 inquiries sourced from OpenEvidence's platform and an additional 187 from HealthBench, a standard clinical benchmark. Physicians scored each response across five dimensions: accuracy, clinical utility, source quality, verifiability, and completeness.
OpenEvidence achieved advantages ranging from 25 to 39 percentage points over the three general-purpose models on all five assessment criteria. Claude Opus 4.8 and Gemini 3.1 Pro performed comparably to each other, while GPT-5.5 recorded the lowest success rates across every measure.
The Contradiction Problem
The divergence from the Nature Medicine study has created confusion among healthcare administrators navigating AI adoption. That earlier research examined GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6 against OpenEvidence and UpToDate Expert AI from Wolters Kluwer, reaching opposite conclusions about the superiority of general-purpose systems.
OpenEvidence subsequently requested that Nature Medicine retract the earlier findings, claiming methodological flaws. The journal declined the request and directed the company toward its formal rebuttal process instead.
Methodological Differences Explain Variance
The research teams attribute the conflicting outcomes to fundamental design choices:
- The new study employed 149 physicians from multiple states in specialty-matched, head-to-head comparisons
- The Nature Medicine study used 12 clinicians at a single institution scoring responses against predefined rubrics
- Researchers note different question sources and evaluation frameworks between studies
The new paper's authors are affiliated with University of California San Francisco, Harvard Medical School, Stanford University, and other institutions. Notably, while OpenEvidence helped design the data collection methodology, administered the survey, and compensated participating physicians, the academic researchers report no formal relationship with the company.
Implications for Healthcare IT Decisions
The conflicting evidence creates a challenging situation for hospital systems and health networks currently evaluating AI tools. Administrators seeking to enhance clinical decision support face contradictory independent assessments of how these systems perform on actual medical questions.
As adoption of both general-purpose AI models and specialized clinical platforms accelerates across healthcare organizations, leadership teams must navigate these competing claims carefully. The divergence underscores broader questions about AI evaluation methodologies, the importance of study design choices, and the influence of different assessment frameworks on outcomes.
This research episode also highlights the stakes involved when independent studies reach opposite conclusions about emerging medical technology. Transparency in methodology, particularly around data sources and physician selection, appears critical for establishing credibility in this rapidly evolving space.
