Automation Bias and Interpreter Uncertainty when Reading 12-lead Electrocardiograms

Description

Electrocardiogram (ECG) interpretation can be a difficult task due to the complexity of the data being presented. The interpretation is normally assisted with an automated diagnosis/diagnoses (AD/s). This can facilitate an ‘automation bias’ where interpreters can be anchored. This can result in an ‘automation paradox’ where humans loose their skills, vigilance and a reliable sense of certainty in decision making. We studied, 1) the effect of an incorrect AD on interpreter accuracy and confidence (a proxy for uncertainty), and 2) how predictive the interpreter’s confidence is for accuracy. A total of 9000 interpretations were collected from 30 physicians (15 cardiology fellows [CFs] and 15 non-CFs). One third involved no ADs, one third with ADs (half as incorrect) and one third had multiple ADs. Confidence ratings were scaled for each individual. Spearman coefficient was used for correlation analysis and Wilcoxon test for significance. C5.0 decision trees were used for predictive modelling (10-fold CV and 20% validation set). A total of 84.86% of CF interpretations were correct when a correct AD was presented but this dropped to 41.66% when an incorrect AD was presented (p<0.001). 86.38% of non-CF interpretations were correct when a correct AD was presented but this dropped to 27.43% when an incorrect AD was presented (p<0.001). Correlation between confidence and accuracy increased when using standardised confidence ratings (from 0.229 to 0.249). This correlation was higher amongst CFs (0.285 vs. 0.221). Confidence ratings amongst both groups dropped by 1 unit when reading ECGs with an incorrect AD (8±4 vs. 7±4 [for non-CFs], p<0.001). For CFs, correlation between confidence and accuracy decreased from 0.256 to 0.198 when reading ECGs with incorrect ADs. For non-CFs, correlation between confidence and accuracy decreased from 0.190 to 0.151 when reading ECGs with incorrect ADs. Using confidence with a rule based C5.0 model, correct interpretation was predicted at a 62.78% accuracy (CI: 60.50, 65.02, kappa=0.211, Sens.=42.76, Spec.= 77.64). Whilst a significant model (p<0.001), the accuracy is only 4% greater than the no-information rate (58.11%). Adding age, experience, and whether the reader was a CF, did not substantially improve the predictive power of the model (Acc.= 64.11%, kappa=0.260). Whilst CFs are influenced by incorrect ADs, non-CFs are more affected. Whilst incorrect ADs affect accuracy, it does also on a lesser scale affect confidence. Confidence ratings from CFs are more predictive of accuracy than non-CFs. However, interpreter intuition of uncertainty is generally a poor predictor for accuracy.

Period	26 Apr 2018
Held at	43rd Annual Conference of the International Society for Computersied Electrocardiology
Event type	Conference
Conference number	43
Location	Park City, United States
Degree of Recognition	International