Microsoft MAI-DxO in Science: what the 83% vs 36% result against doctors on hard cases shows, and what it doesn't

Microsoft MAI-DxO, the Microsoft AI diagnostic system first described in a June 2025 preprint, now has a peer-reviewed paper. On October 22, 2026, Microsoft AI said its work was "published today in Science." On 304 hard case records from the New England Journal of Medicine, MAI-DxO named the right diagnosis 83% of the time, while generalist physicians scored 36% with textbooks and the internet and 22% without them. Microsoft AI chief Mustafa Suleyman called it a step toward "medical superintelligence," TechCrunch reported. It is a research result on simulated cases, not a product doctors or patients can use.

This story is based on Microsoft AI's October 22, 2026 announcement, the open-source SDBench benchmark page, the June 2025 arXiv preprint and Microsoft's June 30, 2025 post, and TechCrunch's October 22 report, checked October 26, 2026. The Science article's full text couldn't be read for this story. Its abstract gives the same results more precisely: 83.2% on the NEJM cases and 80.3% on the emergency cases, against 36.1% and 46.8% for generalist physicians. Figures in neither Microsoft's post nor the abstract are attributed to TechCrunch. Microsoft's other recent model work is covered in our MAI-Base-2 story.

The short version

• The result: 83% accuracy on hard NEJM cases, against 36% (with references) and 22% (without) for generalist physicians. On more typical emergency-department cases from the MIMIC-IV-ED dataset, 80% against 47%.
• The comparison: primary-care physicians and general internists, working alone. No specialists or physician teams, who usually handle cases this hard.
• The system: an orchestration layer over existing models, including OpenAI's GPT-5 and GPT-5.5, not a new model.
• Cost: Microsoft says one configuration hit 81% "at less than half the cost of comparative systems," based on estimated test prices.
• Status: no regulatory clearance has been announced; a prospective hospital study with Beth Israel Deaconess Medical Center is starting.

How does the MAI-DxO benchmark work?

Most medical AI results come from exam-style questions, which Microsoft says "are designed to assess physicians' knowledge – not the realities of making a clinical diagnosis." Its Sequential Diagnosis Benchmark, SDBench, turns published cases into step-by-step workups instead.

A case starts with a short summary. The physician or AI then asks questions or orders tests, and a "gatekeeper" model reveals only what was asked for. At the end, a judge model scores the final diagnosis. Each test has an estimated price, so the benchmark measures both accuracy and how much the workup would have cost.

The hard cases are 304 Clinicopathological Conference cases from the NEJM, which are chosen for being diagnostically difficult. The 2026 paper adds emergency-department presentations from MIMIC-IV-ED, a database of de-identified hospital records. Microsoft has released the benchmark code, gatekeeper, judge and cost estimator. The NEJM cases must be licensed from the journal, and the MIMIC data requires credentialed access.

What exactly did MAI-DxO score against doctors?

Diagnostic accuracy on hard NEJM cases (SDBench)

share of cases with the correct final diagnosis, %

MAI-DxO83

Generalist physicians, with textbooks and internet36

Generalist physicians, without reference resources22

Show the data

Who	Accuracy
MAI-DxO	83
Generalist physicians, with textbooks and internet	36
Generalist physicians, without reference resources	22

These are Microsoft's figures from its own study. Microsoft says MAI-DxO is "more than twice as accurate as physicians using textbooks and the internet" on the NEJM cases and "1.7 times more accurate" on the emergency cases. It also says that across both datasets "no other human or AI approach was both more accurate and less costly than MAI-DxO."

Microsoft's post doesn't say how many physicians took part in the 2026 study, and TechCrunch's report doesn't either. In the 2025 preprint there were 21: practicing doctors in the US and UK with a median of 12 years' experience, 17 in primary care and 4 hospital generalists.

How does the 2026 paper differ from the 2025 preprint?

The preprint, "Sequential Diagnosis with Language Models," went up on arXiv on June 27, 2025. It reported that MAI-DxO with o3 was about four times as accurate as the physicians, who worked under tight restrictions: Microsoft's 2025 post said the clinicians "worked without access to colleagues, textbooks, or even generative AI."

The with-references arm loosens those restrictions only in part. The physicians still worked alone, without colleagues, specialists or the ability to see a patient.

How much of the gain comes from the orchestrator?

Less than the headline suggests, according to TechCrunch's reading of the study data. TechCrunch reported that GPT-5 on its own, with a short basic prompt, scored 78.9% on the hard cases at about $8,400 per case. Adding the orchestrator cut that cost by about $3,000 and raised accuracy by about four points. GPT-5.5 alone scored 72%, and nearly 87% with the orchestrator, at slightly lower cost. On emergency cases the orchestrator added little: GPT-5.5 scored about 85% with or without it. A smaller model gained most, Gemini 2.5 Flash going from 46.4% to nearly 69%.

TechCrunch's point is that the study's own data suggests most of the improvement comes from the underlying models, with the orchestrator adding gains on top. Microsoft researchers told TechCrunch that gains naturally shrink as models approach the maximum possible score, and a spokesperson said the orchestrator helps optimize both accuracy and cost.

The numbers above come from TechCrunch's report. How they map to the 83% headline configuration isn't explained in Microsoft's post, and couldn't be checked against the paper for this story.

What does the cost figure measure?

Not real bills. Each question and test in SDBench has an estimated price, and a run's cost is the sum of what it ordered. The 2025 preprint called its estimates "first-order approximations" that vary by geography, payer and provider. In that version, physicians averaged $2,963 per case, and the budget-limited MAI-DxO configuration about $2,396, at 79.9% accuracy. Lower estimated cost means fewer or cheaper tests were ordered in simulation; it doesn't show savings in a hospital.

What doesn't the study show?

The 2025 preprint listed limits that the setup still has:

• Case mix: the hard cases don't resemble real practice, and "there are no cases where the patients are in fact healthy or have benign syndromes." The authors couldn't measure false positives.
• Generalization: they didn't know whether gains on hard cases carry over "to common, everyday clinical conditions." The emergency-department set addresses this partly.
• The comparison group: generalists only, no specialists.
• Simulation: the gatekeeper is itself a model, and no patient was examined.

None of this tells patients anything about their own symptoms. The result is about a benchmark, not about using chatbots for diagnosis. Whether benchmarks like this capture clinical work at all is a wider problem, discussed in our story on Mercor's benchmark map.

Is MAI-DxO a clinical product?

No. Microsoft hasn't announced any regulatory clearance, and its October 2026 post makes no claim that MAI-DxO is ready for clinical use. In June 2025, Microsoft wrote that the work "is not yet approved for clinical use" and "would only be approved after rigorous safety testing, clinical validation, and regulatory reviews." The October post says "the true test is performance in real clinical environments."

That test is the Beth Israel Deaconess partnership. In a prospective study, patients who consent will interact with the system alongside standard care, and its output will be compared with physicians' independent assessments. Microsoft gave no timeline. It also announced work with Mayo Clinic on a healthcare model, which TechCrunch reported is planned for release in 2027. In the EU, AI in regulated products such as medical devices faces high-risk rules on a timeline now pushed to August 2, 2028, as our EU AI Act guide explains.

Suleyman told TechCrunch he expects healthcare models to get "an order of magnitude better" within 24 to 36 months. That's a forecast, not a finding. Google has made similar claims for its AMIE system, which it tested in a study with Beth Israel published in The Lancet in October 2026, according to TechCrunch.

What's confirmed, reported and uncertain

• Confirmed (Microsoft): the Science publication on October 22, 2026; 304 NEJM cases plus MIMIC-IV-ED; 83%, 80%, 81%, 22%, 36% and 47%; the open-source benchmark; the Beth Israel and Mayo partnerships.
• Confirmed (Science abstract): 83.2% and 80.3% for the system, against 36.1% and 46.8% for generalist physicians, in what the authors call "a controlled benchmark setting."
• Confirmed (2025 preprint and post): 21 physicians, 19.9% average, the o3 results, the cost method and the "not yet approved for clinical use" statement.
• Reported (TechCrunch): the per-model results with and without the orchestrator, the $8,400 per-case figure, and the exclusion of specialists and teams.
• Uncertain: the 2026 physician count and conditions beyond "textbooks and the internet," how the per-model figures relate to the 83% configuration, and how the system performs with real patients.