Virtual reality simulator metrics cannot be used to assess competence in ureteronephroscopy and stone removal - a validation study

• Published in: Scandinavian journal of urology Vol. 55; no. 5; p. 399
• Main Authors: Dagnaes-Hansen, Julia, Konge, Lars, Andreassen, Kim Hovgaard, Hansen, Rikke Bølling
• Format: Journal Article
• Language: English
• Published: England 03-09-2021
• Subjects: Benchmarking; Clinical Competence; Computer Simulation; Humans; Simulation Training; Virtual Reality; virtual reality; urology urolithiasis; Ureteroscopy; medical education
• ISSN: 2168-1813
• DOI: 10.1080/21681805.2021.1960599

The growing use of simulation-based training makes it necessary to develop efficient training programs in order to ensure optimal use of time and resources. Our aim was to develop and gather validity evidence for a simulation-based test in ureteronephroscopy and set a pass/fail standard for the test that will allow future mastery learning.
This study is a validation study. A test in ureteronephroscopy and stone removal on the URO Mentor™ virtual reality simulator (3D Systems, USA) was developed by two experienced urologists in order to ensure
. Participants with different experience completed three standardized tasks on the simulator and simulator-generated metrics were used as outcome parameters to minimize bias and ensure a fair
.
Twenty novices, 15 intermediates, and 8 experienced urologists were included in the study. Validity evidence for
and
was questionable with weak and mostly insignificant correlations across all four metrics (Cronbach's alpha = 0.14,
 = 0.15) and across the three modules (Cronbach's alpha = 0.41 (
 = 0.02), 0.35 (
 = 0.06), 0.10 (
 = 0.35), and 0.30 (
 = 0.09) for each metric, respectively). It was not possible to establish a pass/fail score for the simulation test with meaningful
.
Our study showed that automatically generated simulator metrics cannot be used as a valid way of assessing competence in ureteronephroscopy. Virtual-reality simulator training could still be a valuable and patient-safe way to practice these skills, but an experienced supervisor is needed to determine when the trainee is ready to continue to supervised practice on patients.