SPICED-ACS: Study of the Potential Impact of a Computer-generated ECG Diagnostic Algorithmic Certainty Index in STEMI diagnosis: towards transparent AI

Charles Knoery, RR Bond, Aleeha Iftikhar, Khaled Rjoob, V. E. McGilligan, Aaron Peace, Janet Heaton, Stephen Leslie

Research output: Contribution to journalArticle

Abstract

Background: Computerised electrocardiogram (ECG) interpretation diagnostic algorithms have been developed to guide clinical decisions like with ST segment elevation myocardial infarction (STEMI) where time in decision making is critical. These computer-generated diagnoses have been proven to strongly influence the final ECG diagnosis by the clinician; often called automation bias. However, the computerised diagnosis may be inaccurate and could result in a wrong or delayed treatment harm to the patient. We hypothesise that an algorithmic certainty index alongside a computer-generated diagnosis might mitigate automation bias. The impact of reporting a certainty index on the final diagnosis is not known. Purpose: To ascertain whether knowledge of the computer-generated ECG algorithmic certainty index influences operator diagnostic accuracy. Methodology: Clinicians who regularly analyse ECGs such as cardiology or acute care doctors, cardiac nurses and ambulance staff were invited to complete an online anonymous survey between March and April 2019. The survey had 36 ECGs with a clinical vignette of a typical chest pain and which were either a STEMI, normal, or borderline (but do not fit the STEMI criteria) along with an artificially created certainty index that was either high, medium, low or none. Participants were asked whether the ECG showed a STEMI and their confidence in the diagnosis. The primary outcomes were whether a computer-generated certainty index influenced interpreter's diagnostic decisions and improved their diagnostic accuracy. Secondary outcomes were influence of certainty index between different types of clinicians and influence of certainty index on user's own-diagnostic confidence. Results: A total of 91 participants undertook the survey and submitted 3262 ECG interpretations of which 75% of ECG interpretations were correct. Presence of a certainty index significantly increased the odds ratio of a correct ECG interpretation (OR 1.063, 95% CI 1.022–1.106, p = 0.004) but there was no significant difference between correct certainty index and incorrect certainty index (OR 1.028, 95% CI 0.923–1.145, p = 0.615). There was a trend for low certainty index to increase odds ratio compared to no certainty index (OR 1.153, 95% CI 0.898–1.482, p = 0.264) but a high certainty index significantly decreased the odds ratio of a correct ECG interpretation (OR 0.492, 95% CI 0.391–0.619, p < 0.001). There was no impact of presence of a certainty index (p = 0.528) or correct certainty index (p = 0.812) on interpreters' confidence in their ECG interpretation. Conclusions: Our results show that the presence of an ECG certainty index improves the users ECG interpretation accuracy. This effect is not seen with differing levels of confidence within a certainty index, with reduced ECG interpretation success with a high certainty index compared with a trend for increased success with a low certainty index. This suggests that a certainty index improves interpretation when there is an increased element of doubt, possibly forcing the ECG user to spend more time and effort analysing the ECG. Further research is needed looking at time spent analysing differing certainty indices with alternate ECG diagnoses.

LanguageEnglish
JournalJournal of Electrocardiology
Early online date13 Aug 2019
DOIs
Publication statusE-pub ahead of print - 13 Aug 2019

Fingerprint

Electrocardiography
Myocardial Infarction
Automation
Odds Ratio
Patient Harm
Ambulances
Cardiology
Chest Pain
Decision Making
Nurses

Keywords

  • Algorithm
  • Automated decision-making
  • Automation bias
  • Electrocardiogram
  • Ischaemic heart disease
  • Myocardial infarction

Cite this

@article{ad6f1c865dc64ee4a9870e9a717e5e88,
title = "SPICED-ACS: Study of the Potential Impact of a Computer-generated ECG Diagnostic Algorithmic Certainty Index in STEMI diagnosis: towards transparent AI",
abstract = "Background: Computerised electrocardiogram (ECG) interpretation diagnostic algorithms have been developed to guide clinical decisions like with ST segment elevation myocardial infarction (STEMI) where time in decision making is critical. These computer-generated diagnoses have been proven to strongly influence the final ECG diagnosis by the clinician; often called automation bias. However, the computerised diagnosis may be inaccurate and could result in a wrong or delayed treatment harm to the patient. We hypothesise that an algorithmic certainty index alongside a computer-generated diagnosis might mitigate automation bias. The impact of reporting a certainty index on the final diagnosis is not known. Purpose: To ascertain whether knowledge of the computer-generated ECG algorithmic certainty index influences operator diagnostic accuracy. Methodology: Clinicians who regularly analyse ECGs such as cardiology or acute care doctors, cardiac nurses and ambulance staff were invited to complete an online anonymous survey between March and April 2019. The survey had 36 ECGs with a clinical vignette of a typical chest pain and which were either a STEMI, normal, or borderline (but do not fit the STEMI criteria) along with an artificially created certainty index that was either high, medium, low or none. Participants were asked whether the ECG showed a STEMI and their confidence in the diagnosis. The primary outcomes were whether a computer-generated certainty index influenced interpreter's diagnostic decisions and improved their diagnostic accuracy. Secondary outcomes were influence of certainty index between different types of clinicians and influence of certainty index on user's own-diagnostic confidence. Results: A total of 91 participants undertook the survey and submitted 3262 ECG interpretations of which 75{\%} of ECG interpretations were correct. Presence of a certainty index significantly increased the odds ratio of a correct ECG interpretation (OR 1.063, 95{\%} CI 1.022–1.106, p = 0.004) but there was no significant difference between correct certainty index and incorrect certainty index (OR 1.028, 95{\%} CI 0.923–1.145, p = 0.615). There was a trend for low certainty index to increase odds ratio compared to no certainty index (OR 1.153, 95{\%} CI 0.898–1.482, p = 0.264) but a high certainty index significantly decreased the odds ratio of a correct ECG interpretation (OR 0.492, 95{\%} CI 0.391–0.619, p < 0.001). There was no impact of presence of a certainty index (p = 0.528) or correct certainty index (p = 0.812) on interpreters' confidence in their ECG interpretation. Conclusions: Our results show that the presence of an ECG certainty index improves the users ECG interpretation accuracy. This effect is not seen with differing levels of confidence within a certainty index, with reduced ECG interpretation success with a high certainty index compared with a trend for increased success with a low certainty index. This suggests that a certainty index improves interpretation when there is an increased element of doubt, possibly forcing the ECG user to spend more time and effort analysing the ECG. Further research is needed looking at time spent analysing differing certainty indices with alternate ECG diagnoses.",
keywords = "Algorithm, Automated decision-making, Automation bias, Electrocardiogram, Ischaemic heart disease, Myocardial infarction",
author = "Charles Knoery and RR Bond and Aleeha Iftikhar and Khaled Rjoob and McGilligan, {V. E.} and Aaron Peace and Janet Heaton and Stephen Leslie",
year = "2019",
month = "8",
day = "13",
doi = "10.1016/j.jelectrocard.2019.08.006",
language = "English",
journal = "Journal of Electrocardiology",
issn = "0022-0736",
publisher = "Elsevier",

}

SPICED-ACS: Study of the Potential Impact of a Computer-generated ECG Diagnostic Algorithmic Certainty Index in STEMI diagnosis: towards transparent AI. / Knoery, Charles; Bond, RR; Iftikhar, Aleeha; Rjoob, Khaled; McGilligan, V. E.; Peace, Aaron; Heaton, Janet; Leslie, Stephen.

In: Journal of Electrocardiology, 13.08.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - SPICED-ACS: Study of the Potential Impact of a Computer-generated ECG Diagnostic Algorithmic Certainty Index in STEMI diagnosis: towards transparent AI

AU - Knoery, Charles

AU - Bond, RR

AU - Iftikhar, Aleeha

AU - Rjoob, Khaled

AU - McGilligan, V. E.

AU - Peace, Aaron

AU - Heaton, Janet

AU - Leslie, Stephen

PY - 2019/8/13

Y1 - 2019/8/13

N2 - Background: Computerised electrocardiogram (ECG) interpretation diagnostic algorithms have been developed to guide clinical decisions like with ST segment elevation myocardial infarction (STEMI) where time in decision making is critical. These computer-generated diagnoses have been proven to strongly influence the final ECG diagnosis by the clinician; often called automation bias. However, the computerised diagnosis may be inaccurate and could result in a wrong or delayed treatment harm to the patient. We hypothesise that an algorithmic certainty index alongside a computer-generated diagnosis might mitigate automation bias. The impact of reporting a certainty index on the final diagnosis is not known. Purpose: To ascertain whether knowledge of the computer-generated ECG algorithmic certainty index influences operator diagnostic accuracy. Methodology: Clinicians who regularly analyse ECGs such as cardiology or acute care doctors, cardiac nurses and ambulance staff were invited to complete an online anonymous survey between March and April 2019. The survey had 36 ECGs with a clinical vignette of a typical chest pain and which were either a STEMI, normal, or borderline (but do not fit the STEMI criteria) along with an artificially created certainty index that was either high, medium, low or none. Participants were asked whether the ECG showed a STEMI and their confidence in the diagnosis. The primary outcomes were whether a computer-generated certainty index influenced interpreter's diagnostic decisions and improved their diagnostic accuracy. Secondary outcomes were influence of certainty index between different types of clinicians and influence of certainty index on user's own-diagnostic confidence. Results: A total of 91 participants undertook the survey and submitted 3262 ECG interpretations of which 75% of ECG interpretations were correct. Presence of a certainty index significantly increased the odds ratio of a correct ECG interpretation (OR 1.063, 95% CI 1.022–1.106, p = 0.004) but there was no significant difference between correct certainty index and incorrect certainty index (OR 1.028, 95% CI 0.923–1.145, p = 0.615). There was a trend for low certainty index to increase odds ratio compared to no certainty index (OR 1.153, 95% CI 0.898–1.482, p = 0.264) but a high certainty index significantly decreased the odds ratio of a correct ECG interpretation (OR 0.492, 95% CI 0.391–0.619, p < 0.001). There was no impact of presence of a certainty index (p = 0.528) or correct certainty index (p = 0.812) on interpreters' confidence in their ECG interpretation. Conclusions: Our results show that the presence of an ECG certainty index improves the users ECG interpretation accuracy. This effect is not seen with differing levels of confidence within a certainty index, with reduced ECG interpretation success with a high certainty index compared with a trend for increased success with a low certainty index. This suggests that a certainty index improves interpretation when there is an increased element of doubt, possibly forcing the ECG user to spend more time and effort analysing the ECG. Further research is needed looking at time spent analysing differing certainty indices with alternate ECG diagnoses.

AB - Background: Computerised electrocardiogram (ECG) interpretation diagnostic algorithms have been developed to guide clinical decisions like with ST segment elevation myocardial infarction (STEMI) where time in decision making is critical. These computer-generated diagnoses have been proven to strongly influence the final ECG diagnosis by the clinician; often called automation bias. However, the computerised diagnosis may be inaccurate and could result in a wrong or delayed treatment harm to the patient. We hypothesise that an algorithmic certainty index alongside a computer-generated diagnosis might mitigate automation bias. The impact of reporting a certainty index on the final diagnosis is not known. Purpose: To ascertain whether knowledge of the computer-generated ECG algorithmic certainty index influences operator diagnostic accuracy. Methodology: Clinicians who regularly analyse ECGs such as cardiology or acute care doctors, cardiac nurses and ambulance staff were invited to complete an online anonymous survey between March and April 2019. The survey had 36 ECGs with a clinical vignette of a typical chest pain and which were either a STEMI, normal, or borderline (but do not fit the STEMI criteria) along with an artificially created certainty index that was either high, medium, low or none. Participants were asked whether the ECG showed a STEMI and their confidence in the diagnosis. The primary outcomes were whether a computer-generated certainty index influenced interpreter's diagnostic decisions and improved their diagnostic accuracy. Secondary outcomes were influence of certainty index between different types of clinicians and influence of certainty index on user's own-diagnostic confidence. Results: A total of 91 participants undertook the survey and submitted 3262 ECG interpretations of which 75% of ECG interpretations were correct. Presence of a certainty index significantly increased the odds ratio of a correct ECG interpretation (OR 1.063, 95% CI 1.022–1.106, p = 0.004) but there was no significant difference between correct certainty index and incorrect certainty index (OR 1.028, 95% CI 0.923–1.145, p = 0.615). There was a trend for low certainty index to increase odds ratio compared to no certainty index (OR 1.153, 95% CI 0.898–1.482, p = 0.264) but a high certainty index significantly decreased the odds ratio of a correct ECG interpretation (OR 0.492, 95% CI 0.391–0.619, p < 0.001). There was no impact of presence of a certainty index (p = 0.528) or correct certainty index (p = 0.812) on interpreters' confidence in their ECG interpretation. Conclusions: Our results show that the presence of an ECG certainty index improves the users ECG interpretation accuracy. This effect is not seen with differing levels of confidence within a certainty index, with reduced ECG interpretation success with a high certainty index compared with a trend for increased success with a low certainty index. This suggests that a certainty index improves interpretation when there is an increased element of doubt, possibly forcing the ECG user to spend more time and effort analysing the ECG. Further research is needed looking at time spent analysing differing certainty indices with alternate ECG diagnoses.

KW - Algorithm

KW - Automated decision-making

KW - Automation bias

KW - Electrocardiogram

KW - Ischaemic heart disease

KW - Myocardial infarction

UR - https://www.sciencedirect.com/science/article/abs/pii/S0022073619303772

UR - http://www.scopus.com/inward/record.url?scp=85071366480&partnerID=8YFLogxK

U2 - 10.1016/j.jelectrocard.2019.08.006

DO - 10.1016/j.jelectrocard.2019.08.006

M3 - Article

JO - Journal of Electrocardiology

T2 - Journal of Electrocardiology

JF - Journal of Electrocardiology

SN - 0022-0736

ER -