A new study from the Google AI research group shows how physicians and algorithms become more effective when they work together. This expands on previous work showing that an algorithm works roughly as well as human experts in screening patients for diabetic retinopathy. The findings will be published in the April edition of Ophthalmology, the journal of the American Academy of Ophthalmology.
For their latest study, the researchers wanted to see if their algorithm could do more than simply diagnose disease. They wanted to create a new computer-assisted system that could “explain” the algorithm’s diagnosis. They found that this system not only improved the ophthalmologists’ diagnostic accuracy, but it also improved algorithm’s accuracy.
To test this theory, the researchers developed two types of assistance to help physicians read the algorithm’s predictions.
- Grades: A set of five scores that represent the strength of evidence for the algorithm’s prediction.
- Grades + heatmap: Enhance the grading system with a heatmap that measures the contribution of each pixel in the image to the algorithm’s prediction.
Ten ophthalmologists (four general ophthalmologists, one trained outside the US, four retina specialists, and one retina specialist in training) were asked to read each image once under one of three conditions: unassisted, grades only, and grades + heatmap. Both types of assistance improved physicians’ diagnostic accuracy. It also improved their confidence in the diagnosis. But the degree of improvement depended on the physician’s level of expertise.
Without assistance, general ophthalmologists are significantly less accurate than the algorithm, while retina specialists are not significantly more accurate than the algorithm. With assistance, general ophthalmologists match but do not exceed the model’s accuracy, while retina specialists start to exceed the model’s performance.
“What we found is that AI can do more than simply automate eye screening, it can assist physicians in more accurately diagnosing diabetic retinopathy,” said lead researcher, Rory Sayres, PhD.. “AI and physicians working together can be more accurate than either alone.”
Read the full news release here.