MIT Study Finds AI Explanations Mislead Non-Experts, While Clinicians Do Best With 1 Prediction
Updated
Updated · MIT News · Aug 4
MIT Study Finds AI Explanations Mislead Non-Experts, While Clinicians Do Best With 1 Prediction
3 articles · Updated · MIT News · Aug 4
Summary
Nature Medicine published a study showing AI diagnostic help improved skin-disease accuracy overall, but the gains varied sharply by expertise: non-experts benefited through deference to the model, while clinicians performed best with a prediction alone.
LLM-based explanations proved the riskiest for non-experts, who trusted them whether right or wrong and grew more confident in incorrect answers when the rationale sounded plausible but vague.
Primary care clinicians were far less vulnerable to bad AI guidance because they checked outputs against their own judgment; among explanation types, LLMs boosted their accuracy the least.
The researchers also found deference was strongest among the weakest unaided performers and increased when explanations appeared before users formed their own diagnosis.
In skin-cancer screening, a fairness-constrained model improved accuracy and reduced disparities across darker skin tones, but the broader lesson was that explainability must be tailored to users to avoid automation bias.