Can AI diagnose like a team of doctors? New research puts it to the test

Microsoft has developed MAI-DxO, an AI diagnostic system designed to assist clinicians by improving diagnostic accuracy while reducing costs and unnecessary tests. In evaluations using MIMIC-IV-ED cases, MAI-DxO was 1.7 times more accurate than physicians. On complex cases from The New England Journal of Medicine, it achieved more than twice the accuracy of physicians using standard resources and nearly four times the accuracy without those resources.

To assess step-by-step diagnostic reasoning, Microsoft introduced the Sequential Diagnosis Benchmark (SDBench), which tests AI and physician performance on interactive cases. SDBench includes 304 complex cases from NEJM and typical emergency department presentations from MIMIC-IV-ED, measuring diagnostic accuracy, evidence requested, and estimated test costs. The benchmark evaluates both the final diagnosis and the reasoning process, providing a more realistic assessment than traditional licensing exams.

MAI-DxO operates like a virtual panel of physicians, with multiple AI models proposing diagnoses, selecting tests, and debating findings over several rounds. The system coordinates this exchange, updating its assessment as new evidence emerges and weighing test costs against diagnostic value. Microsoft has open-sourced SDBench for select NEJM cases, including evaluation tools, to enable broader research and validation of the findings.

The broader vision extends beyond diagnosis to include treatment selection, adapting to changing patient conditions, and aligning care with patient values and access. Microsoft is collaborating with Mayo Clinic and Beth Israel Deaconess Medical Center to test MAI-DxO in real clinical settings, including a prospective study where patients interact directly with the system. The goal is to integrate AI into clinical workflows and improve outcomes across diverse healthcare environments.