Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial

Ruben De Groote, Stefano Puliatti, Marco Amato, Elio Mazzone, Alessandro Larcher, Rui Farinha, Artur Paludo, Liesbeth Desender, Nicolas Hubert, Ben Van Cleynenbreugel, Brendan P. Bunting, Alexandre Mottrie, Anthony G. Gallagher, Giuseppe Rosiello, Pieter Uvin, Jasper Decoene, Tom Tuyten, Mathieu D’Hondt, Charles Chatzopoulos, Bart De TroyerFilippo Turri, Paolo Dell’Oglio, Nikolaos Liakos, Carlo Andrea Bravi, Edward Lambert, Iulia Andras, Fabrizio Di Maida, Wouter Everaerts

Research output: Contribution to journalArticlepeer-review

15 Downloads (Pure)

Abstract

Objective:
To compare binary metrics and Global Evaluative Assessment of Robotic Skills (GEARS) evaluations of training outcome assessments for reliability, sensitivity, and specificity.

Background:
GEARS–Likert-scale skills assessment are a widely accepted tool for robotic surgical training outcome evaluations. Proficiency-based progression (PBP) training is another methodology but uses binary performance metrics for evaluations.

Methods:
In a prospective, randomized, and blinded study, we compared conventional with PBP training for a robotic suturing, knot-tying anastomosis task. Thirty-six surgical residents from 16 Belgium residency programs were randomized. In the skills laboratory, the PBP group trained until they demonstrated a quantitatively defined proficiency benchmark. The conventional group were yoked to the same training time but without the proficiency requirement. The final trial was video recorded and assessed with binary metrics and GEARS by robotic surgeons blinded to individual, group, and residency program. Sensitivity and specificity of the two assessment methods were evaluated with area under the curve (AUC) and receiver operating characteristics (ROC) curves.

Results:
The PBP group made 42% fewer objectively assessed performance errors than the conventional group (P < 0.001) and scored 15% better on the GEARS assessment (P = 0.033). The mean interrater reliability for binary metrics and GEARS was 0.87 and 0.38, respectively. Binary total error metrics AUC was 97% and for GEARS 85%. With a sensitivity threshold of 0.8, false positives rates were 3% and 25% for, respectively, the binary and GEARS assessments.

Conclusions:
Binary metrics for scoring a robotic VUA task demonstrated better psychometric properties than the GEARS assessment.
Original languageEnglish
Pages (from-to)1-7
Number of pages7
JournalAnnals of Surgery
Volume4
Issue number3
Early online date16 Aug 2023
DOIs
Publication statusPublished (in print/issue) - 30 Sept 2023

Bibliographical note

Copyright © 2023 The Author(s). Published by Wolters Kluwer Health, Inc.

Keywords

  • GEARS
  • basic skill training
  • Proficiency based progression
  • Robotic surgery
  • Training

Fingerprint

Dive into the research topics of 'Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial'. Together they form a unique fingerprint.

Cite this