Variability of human-annotations of 12-lead ECG features collected using a web system: Students vs. practitioners

Andrew W. Cairns, Raymond Bond, Cathal Breen, Dewar Finlay, Daniel Guldenring, Aaron Peace

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Introduction: The electrocardiogram (ECG) is often interpreted incorrectly with up to 33% of interpretations containing a significant error. The difficulty in ECG interpretation is two-fold; 1) it demands an extensive knowledge of cardiac physiology, and 2) the ECG inflates cognitive workload due to the complex nature of its presentation. To make a diagnosis, the reader is required to measure ECG features in order to contrast these annotations with diagnostic criteria. Whilst signal processing algorithms can provide automated measurements, they are often imprecise. Based on these observations, a web-based system was developed to allow the interpreter to measure and input their own ECG annotations. These annotations are then processed by a rule-based algorithm, which presents a set of suggested diagnoses.However, imprecise and inconsistent human annotations would affect both the reader's diagnostic decision making and also the accuracy of the diagnoses suggested by the algorithm (junk in = junk out). Our study measures the variability of manual annotations collected using our web system. Clinical physiology students (n = 10) and medical practitioners (n = 11) participated in our study.Results: Annotations for the same ECG are as follows (all participants = A, students = S, medical practitioners = P):Heart Rate (A[mean = 89.39 bpm, SD = 9.95], S[mean = 88.7 bpm, SD = 4.27], P[mean = 91.4 bpm, SD = 14.68], p = 0.73), P-wave duration (A[mean = 0.08 s, SD = 0.02], S[mean = 0.09 s, SD = 0.03], P[mean = 0.08 s, SD = 0.01], p = 0.39), P-wave amplitude (A[mean = 0.18 mv, SD = 0.04], S[mean = 0.19 mv, SD = 0.05], P[mean = 0.18 mv, SD = 0.3], p = 0.38), P-R interval (A[mean = 0.16 s, SD = 0.04], S[mean = 0.18 s, SD = 0.05], P[mean = 0.16 s, SD = 0.03], p = 0.49), cardiac axis (A[mean = 58.11°, SD = 13.23], S[mean = 60°, SD = 0], P[mean = 51.5°, SD = 18.8], p = 0.46), Q-T interval (A[mean = 0.32 s, SD = 0.14], S[mean = 0.41 s, SD = 0.06], P[mean = 0.24 s, SD = 0.17], p <0.01), R-R interval (A[mean = 0.63 s, SD = 0.21], S[mean = 0.72 s, SD = 0.13], P[mean = 0.53 s, SD = 0.27], p = 0.06) and QTc (A[mean = 0.4 s, SD = 0.15], S[mean = 0.48 s, SD = 0.09], P[mean = 0.33 s, SD = 0.2], p = 0.02).Discussion: Students annotated more features (5/8) with less variance. Students annotate interval measurements with 47% less variation than medical practitioners (Σ interval measurement; students SD = 0.36, practitioners SD = 0.68). Students also had less variation in measuring heart rate, P-wave amplitude and cardiac axis. Two of the annotated features (QT-interval and QTc) from both cohorts were statistically different (p ≤ 0.05).Conclusion: In order to make an accurate diagnosis precise ECG annotations are required. This study determined the variability of manual ECG annotations on a cohort containing both students and practitioners.
LanguageEnglish
Title of host publicationUnknown Host Publication
PublisherElsevier
Number of pages1
DOIs
Publication statusE-pub ahead of print - Nov 2017
EventInternational Society for Computerised Electrocardiology - St. Simons Island
Duration: 1 Nov 2017 → …

Conference

ConferenceInternational Society for Computerised Electrocardiology
Period1/11/17 → …

Fingerprint

Electrocardiography
Students
Heart Rate
Lead
Workload
Medical Students
Decision Making

Keywords

  • ECG
  • medical informatics
  • cardiology

Cite this

@inproceedings{4d7d1ca5a3a84bc9b8496b7847c035cc,
title = "Variability of human-annotations of 12-lead ECG features collected using a web system: Students vs. practitioners",
abstract = "Introduction: The electrocardiogram (ECG) is often interpreted incorrectly with up to 33{\%} of interpretations containing a significant error. The difficulty in ECG interpretation is two-fold; 1) it demands an extensive knowledge of cardiac physiology, and 2) the ECG inflates cognitive workload due to the complex nature of its presentation. To make a diagnosis, the reader is required to measure ECG features in order to contrast these annotations with diagnostic criteria. Whilst signal processing algorithms can provide automated measurements, they are often imprecise. Based on these observations, a web-based system was developed to allow the interpreter to measure and input their own ECG annotations. These annotations are then processed by a rule-based algorithm, which presents a set of suggested diagnoses.However, imprecise and inconsistent human annotations would affect both the reader's diagnostic decision making and also the accuracy of the diagnoses suggested by the algorithm (junk in = junk out). Our study measures the variability of manual annotations collected using our web system. Clinical physiology students (n = 10) and medical practitioners (n = 11) participated in our study.Results: Annotations for the same ECG are as follows (all participants = A, students = S, medical practitioners = P):Heart Rate (A[mean = 89.39 bpm, SD = 9.95], S[mean = 88.7 bpm, SD = 4.27], P[mean = 91.4 bpm, SD = 14.68], p = 0.73), P-wave duration (A[mean = 0.08 s, SD = 0.02], S[mean = 0.09 s, SD = 0.03], P[mean = 0.08 s, SD = 0.01], p = 0.39), P-wave amplitude (A[mean = 0.18 mv, SD = 0.04], S[mean = 0.19 mv, SD = 0.05], P[mean = 0.18 mv, SD = 0.3], p = 0.38), P-R interval (A[mean = 0.16 s, SD = 0.04], S[mean = 0.18 s, SD = 0.05], P[mean = 0.16 s, SD = 0.03], p = 0.49), cardiac axis (A[mean = 58.11°, SD = 13.23], S[mean = 60°, SD = 0], P[mean = 51.5°, SD = 18.8], p = 0.46), Q-T interval (A[mean = 0.32 s, SD = 0.14], S[mean = 0.41 s, SD = 0.06], P[mean = 0.24 s, SD = 0.17], p <0.01), R-R interval (A[mean = 0.63 s, SD = 0.21], S[mean = 0.72 s, SD = 0.13], P[mean = 0.53 s, SD = 0.27], p = 0.06) and QTc (A[mean = 0.4 s, SD = 0.15], S[mean = 0.48 s, SD = 0.09], P[mean = 0.33 s, SD = 0.2], p = 0.02).Discussion: Students annotated more features (5/8) with less variance. Students annotate interval measurements with 47{\%} less variation than medical practitioners (Σ interval measurement; students SD = 0.36, practitioners SD = 0.68). Students also had less variation in measuring heart rate, P-wave amplitude and cardiac axis. Two of the annotated features (QT-interval and QTc) from both cohorts were statistically different (p ≤ 0.05).Conclusion: In order to make an accurate diagnosis precise ECG annotations are required. This study determined the variability of manual ECG annotations on a cohort containing both students and practitioners.",
keywords = "ECG, medical informatics, cardiology",
author = "Cairns, {Andrew W.} and Raymond Bond and Cathal Breen and Dewar Finlay and Daniel Guldenring and Aaron Peace",
year = "2017",
month = "11",
doi = "10.1016/j.jelectrocard.2017.08.041",
language = "English",
booktitle = "Unknown Host Publication",
publisher = "Elsevier",
address = "Netherlands",

}

Cairns, AW, Bond, R, Breen, C, Finlay, D, Guldenring, D & Peace, A 2017, Variability of human-annotations of 12-lead ECG features collected using a web system: Students vs. practitioners. in Unknown Host Publication. Elsevier, International Society for Computerised Electrocardiology, 1/11/17. https://doi.org/10.1016/j.jelectrocard.2017.08.041

Variability of human-annotations of 12-lead ECG features collected using a web system: Students vs. practitioners. / Cairns, Andrew W.; Bond, Raymond; Breen, Cathal; Finlay, Dewar; Guldenring, Daniel; Peace, Aaron.

Unknown Host Publication. Elsevier, 2017.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Variability of human-annotations of 12-lead ECG features collected using a web system: Students vs. practitioners

AU - Cairns, Andrew W.

AU - Bond, Raymond

AU - Breen, Cathal

AU - Finlay, Dewar

AU - Guldenring, Daniel

AU - Peace, Aaron

PY - 2017/11

Y1 - 2017/11

N2 - Introduction: The electrocardiogram (ECG) is often interpreted incorrectly with up to 33% of interpretations containing a significant error. The difficulty in ECG interpretation is two-fold; 1) it demands an extensive knowledge of cardiac physiology, and 2) the ECG inflates cognitive workload due to the complex nature of its presentation. To make a diagnosis, the reader is required to measure ECG features in order to contrast these annotations with diagnostic criteria. Whilst signal processing algorithms can provide automated measurements, they are often imprecise. Based on these observations, a web-based system was developed to allow the interpreter to measure and input their own ECG annotations. These annotations are then processed by a rule-based algorithm, which presents a set of suggested diagnoses.However, imprecise and inconsistent human annotations would affect both the reader's diagnostic decision making and also the accuracy of the diagnoses suggested by the algorithm (junk in = junk out). Our study measures the variability of manual annotations collected using our web system. Clinical physiology students (n = 10) and medical practitioners (n = 11) participated in our study.Results: Annotations for the same ECG are as follows (all participants = A, students = S, medical practitioners = P):Heart Rate (A[mean = 89.39 bpm, SD = 9.95], S[mean = 88.7 bpm, SD = 4.27], P[mean = 91.4 bpm, SD = 14.68], p = 0.73), P-wave duration (A[mean = 0.08 s, SD = 0.02], S[mean = 0.09 s, SD = 0.03], P[mean = 0.08 s, SD = 0.01], p = 0.39), P-wave amplitude (A[mean = 0.18 mv, SD = 0.04], S[mean = 0.19 mv, SD = 0.05], P[mean = 0.18 mv, SD = 0.3], p = 0.38), P-R interval (A[mean = 0.16 s, SD = 0.04], S[mean = 0.18 s, SD = 0.05], P[mean = 0.16 s, SD = 0.03], p = 0.49), cardiac axis (A[mean = 58.11°, SD = 13.23], S[mean = 60°, SD = 0], P[mean = 51.5°, SD = 18.8], p = 0.46), Q-T interval (A[mean = 0.32 s, SD = 0.14], S[mean = 0.41 s, SD = 0.06], P[mean = 0.24 s, SD = 0.17], p <0.01), R-R interval (A[mean = 0.63 s, SD = 0.21], S[mean = 0.72 s, SD = 0.13], P[mean = 0.53 s, SD = 0.27], p = 0.06) and QTc (A[mean = 0.4 s, SD = 0.15], S[mean = 0.48 s, SD = 0.09], P[mean = 0.33 s, SD = 0.2], p = 0.02).Discussion: Students annotated more features (5/8) with less variance. Students annotate interval measurements with 47% less variation than medical practitioners (Σ interval measurement; students SD = 0.36, practitioners SD = 0.68). Students also had less variation in measuring heart rate, P-wave amplitude and cardiac axis. Two of the annotated features (QT-interval and QTc) from both cohorts were statistically different (p ≤ 0.05).Conclusion: In order to make an accurate diagnosis precise ECG annotations are required. This study determined the variability of manual ECG annotations on a cohort containing both students and practitioners.

AB - Introduction: The electrocardiogram (ECG) is often interpreted incorrectly with up to 33% of interpretations containing a significant error. The difficulty in ECG interpretation is two-fold; 1) it demands an extensive knowledge of cardiac physiology, and 2) the ECG inflates cognitive workload due to the complex nature of its presentation. To make a diagnosis, the reader is required to measure ECG features in order to contrast these annotations with diagnostic criteria. Whilst signal processing algorithms can provide automated measurements, they are often imprecise. Based on these observations, a web-based system was developed to allow the interpreter to measure and input their own ECG annotations. These annotations are then processed by a rule-based algorithm, which presents a set of suggested diagnoses.However, imprecise and inconsistent human annotations would affect both the reader's diagnostic decision making and also the accuracy of the diagnoses suggested by the algorithm (junk in = junk out). Our study measures the variability of manual annotations collected using our web system. Clinical physiology students (n = 10) and medical practitioners (n = 11) participated in our study.Results: Annotations for the same ECG are as follows (all participants = A, students = S, medical practitioners = P):Heart Rate (A[mean = 89.39 bpm, SD = 9.95], S[mean = 88.7 bpm, SD = 4.27], P[mean = 91.4 bpm, SD = 14.68], p = 0.73), P-wave duration (A[mean = 0.08 s, SD = 0.02], S[mean = 0.09 s, SD = 0.03], P[mean = 0.08 s, SD = 0.01], p = 0.39), P-wave amplitude (A[mean = 0.18 mv, SD = 0.04], S[mean = 0.19 mv, SD = 0.05], P[mean = 0.18 mv, SD = 0.3], p = 0.38), P-R interval (A[mean = 0.16 s, SD = 0.04], S[mean = 0.18 s, SD = 0.05], P[mean = 0.16 s, SD = 0.03], p = 0.49), cardiac axis (A[mean = 58.11°, SD = 13.23], S[mean = 60°, SD = 0], P[mean = 51.5°, SD = 18.8], p = 0.46), Q-T interval (A[mean = 0.32 s, SD = 0.14], S[mean = 0.41 s, SD = 0.06], P[mean = 0.24 s, SD = 0.17], p <0.01), R-R interval (A[mean = 0.63 s, SD = 0.21], S[mean = 0.72 s, SD = 0.13], P[mean = 0.53 s, SD = 0.27], p = 0.06) and QTc (A[mean = 0.4 s, SD = 0.15], S[mean = 0.48 s, SD = 0.09], P[mean = 0.33 s, SD = 0.2], p = 0.02).Discussion: Students annotated more features (5/8) with less variance. Students annotate interval measurements with 47% less variation than medical practitioners (Σ interval measurement; students SD = 0.36, practitioners SD = 0.68). Students also had less variation in measuring heart rate, P-wave amplitude and cardiac axis. Two of the annotated features (QT-interval and QTc) from both cohorts were statistically different (p ≤ 0.05).Conclusion: In order to make an accurate diagnosis precise ECG annotations are required. This study determined the variability of manual ECG annotations on a cohort containing both students and practitioners.

KW - ECG

KW - medical informatics

KW - cardiology

U2 - 10.1016/j.jelectrocard.2017.08.041

DO - 10.1016/j.jelectrocard.2017.08.041

M3 - Conference contribution

BT - Unknown Host Publication

PB - Elsevier

ER -