Abstract
Purpose or Learning Objective
Reported accuracies and workforce shortages have increased integration of AI into clinical environments. Furthermore, radiographer reporting helps ease the burden of image reporting. ‘System trust’ is identified as a challenge to clinical AI integration. To the authors’ knowledge, no research has been conducted on the factors impacting reporting radiographers’ trust and decision making when using different forms of AI feedback.
Methods or Background
Twelve reporting radiographers, three from each region of the UK, participated in this study. The Qualtrics® platform was used to randomly allocate 18 radiographic examinations to each participant. Participants were asked to locate any pathology and indicate their agreement with the AI localisation, represented by GradCAM heatmaps and the AI binary diagnosis. Spearman’s rho and Kendall’s tau were used to investigate any correlation between trust and agreement with various forms of AI feedback and initial image quality.
Results or Findings
Participants disagreed with the AI heatmaps for the abnormal examinations 45.8% (n=66 of 144 individual images) of the time and agreed with binary feedback on 86.7% of examinations (26 of 30 cases). 0.7% (n=2) indicated that they would decision switch following AI feedback. 22.2% (n=32) agreed with the localisation of pathology from the heatmap. Agreement with AI feedback was correlated with trust (-.515; -.584, significant large negative correlation (p=<.01) and -.309; -.369, significant medium negative correlation (p=<.01) for GradCAM and binary diagnosis respectively).
Conclusion
The extent of agreement with both AI binary diagnosis and heatmap is correlated with trust in AI, where greater agreement with AI feedback is associated with greater trust, with a large effect size in agreement with GradCAM feedback.
Limitations
The Qualtrics® platform may not allow for an accurate simulation of the clinical setting. This will be further investigated in subsequent studies.
Reported accuracies and workforce shortages have increased integration of AI into clinical environments. Furthermore, radiographer reporting helps ease the burden of image reporting. ‘System trust’ is identified as a challenge to clinical AI integration. To the authors’ knowledge, no research has been conducted on the factors impacting reporting radiographers’ trust and decision making when using different forms of AI feedback.
Methods or Background
Twelve reporting radiographers, three from each region of the UK, participated in this study. The Qualtrics® platform was used to randomly allocate 18 radiographic examinations to each participant. Participants were asked to locate any pathology and indicate their agreement with the AI localisation, represented by GradCAM heatmaps and the AI binary diagnosis. Spearman’s rho and Kendall’s tau were used to investigate any correlation between trust and agreement with various forms of AI feedback and initial image quality.
Results or Findings
Participants disagreed with the AI heatmaps for the abnormal examinations 45.8% (n=66 of 144 individual images) of the time and agreed with binary feedback on 86.7% of examinations (26 of 30 cases). 0.7% (n=2) indicated that they would decision switch following AI feedback. 22.2% (n=32) agreed with the localisation of pathology from the heatmap. Agreement with AI feedback was correlated with trust (-.515; -.584, significant large negative correlation (p=<.01) and -.309; -.369, significant medium negative correlation (p=<.01) for GradCAM and binary diagnosis respectively).
Conclusion
The extent of agreement with both AI binary diagnosis and heatmap is correlated with trust in AI, where greater agreement with AI feedback is associated with greater trust, with a large effect size in agreement with GradCAM feedback.
Limitations
The Qualtrics® platform may not allow for an accurate simulation of the clinical setting. This will be further investigated in subsequent studies.
| Original language | English |
|---|---|
| Pages | C-23132 |
| DOIs | |
| Publication status | Published (in print/issue) - 27 Mar 2024 |
| Event | European Congress of Radiology (2024) - Vienna Duration: 28 Feb 2024 → 3 Mar 2024 https://www.myesr.org/congress/programme/ |
Conference
| Conference | European Congress of Radiology (2024) |
|---|---|
| Abbreviated title | ECR 2024 |
| Period | 28/02/24 → 3/03/24 |
| Internet address |
Keywords
- Artificial intelligence,
- AI
- decision hygiene
- trust
- radiography
