A study conducted in a primary care clinic in Spain highlights the need for improvement of AI algorithms for interpreting chest x-rays, despite being commercially approved by European regulators, according to an article published March 3 in Scientific Reports.
A group at a university in Catalonia tested an AI product called ChestEye in a local clinic that was approved by the European Union (EU) in 2019. They found that while it compared well to a radiologist’s diagnoses in 278 images and reports, it does not yet perform well enough for clinical use.
“External validation, crucial for ensuring nondiscrimination and equity in healthcare, should be a key requirement for the widespread implementation of AI algorithms. However, it is not yet specifically mandated by European legislation,” noted lead author Queralt Catalina, of University of Vic-Central University of Catalonia.
ChestEye uses a computer-aided diagnosis (CAD) algorithm that analyzes x-rays for 75 different findings and localizes the features on images as heatmaps. It can also generate preliminary text reports that incorporate relevant findings in chest x-ray images.
The algorithm was developed by a Lithuanian company called Oxipit and was trained on more than 300,000 images during its development, the authors noted.
In this study, the researchers aimed to externally validate the technology in a prospective observational study in patients who were scheduled for chest x-rays. They obtained a radiologist’s report for each patient (considered the gold standard) and subsequently compared the findings to the AI algorithm’s findings on the same reports.
Image of patient (upper-left) in which, according to the radiologist’s report, there is only consolidation, but the algorithm detects an abnormal rib (upper-right), consolidation (lower-left), and two nodules (lower-right). It is worth noting the confusion of a consolidation with mammary tissue and of two nodules with the two mammary areolae. Image courtesy of Scientific Reports.
The study was performed with a sample of 278 images and reports, 51.8% of which showed no radiological abnormalities according to the radiologist’s report. An analysis revealed that the AI algorithm achieved an average accuracy 95%, a sensitivity of 48%, and a specificity of 98%, according to the researchers.
On the plus side, the conditions where the algorithm was most sensitive were in detecting external, upper abdominal, and cardiac and/or valvular implants, the group noted. On the other hand, the conditions where the algorithm was less sensitive were located in the mediastinum, vessels, and bone, they wrote.
“The algorithm has been validated in the primary care setting and has proven to be useful when identifying images with or without conditions,” the authors noted.
However, in order to be a valuable tool to help and support radiologists, it requires additional real-world training to enhance its diagnostic capabilities for some of the conditions analyzed, the group added.
Ultimately, the implementation of AI in healthcare appears to be an imminent reality that can offer significant benefits to both professionals and the general population, yet it is essential to validate tools in real clinical settings to balance a phenomenon called “digital exceptionalism,” which they may achieve in development, the researchers wrote.
“Our study emphasizes the need for continuous improvement to ensure the algorithm’s effectiveness in primary care,” the group concluded.
The full article is available here.
link