The need to guarantee a uniform and consistent surgical performance justifies the creation of a structured and validated training program, that should contemplate a technical and a procedural component, having in mind the final goal to achieve the best possible patient outcomes.
Due to legal and ethical concerns, surgical training is being driven out of the operating room (OR) in the early stage of the surgeon’s learning curve. This change is imposing a paradigm shift from the Halsted training model to a Proficiency-Based-Progression training model (PBP).8 This shift will alter the way surgeons are trained, potentially producing an improvement in surgical outcomes and avoiding complications that most frequently happen during those early stages of training.
Grounded in a solid scientific methodology, PBP training has as its cornerstone, the development of procedural performance metrics. Nowadays, surgical performance is usually assessed using Likert scales like GEARS, OSATS, or GOALS.9,10 Although these scales use concepts that are largely understood by the surgical community, they are not clearly defined, and therefore, their use introduces a high degree of subjectivity, opacity, and unfairness in the evaluation of surgical performance.
With the goal to develop highly detailed, explicit, binary surgical performance metrics, that allow an objective, transparent, and fair assessment, the authors gathered a Core Team Metrics group, that developed a first set of metrics, defining all the relevant Phases, Steps, Errors and Critical Errors that can potentially be performed during a RAPN procedure.12
Afterward, a modified Delphi meeting was organized, where the previous set of metrics was presented to a larger group of RAPN experts, from different countries and continents. The metrics were presented as an adequate way to train but not as the only way to do the procedure. The results of this face and content validation study were published in European Urology Focus, reporting 100% consensus on the presented metrics.12
The second study describes the process of construct and discriminative validation of the approved metrics.13 The authors trained two assessors in the use of the RAPN metrics, by evaluating several RAPN videos of novices and experts. The assessors were considered prepared to assess the “Construct Validation Videos” when they consistently reached an Inter-Rater-reliability (IRR) above 0.8 in the assessment of the “training” videos.13
The” Construct Validation Videos” included anonymized full RAPN recordings of novices and experts given to the trained assessors. They were blinded to the level of skills, surgeon, and center of origin of those videos. IRR was constantly monitored after each video assessment to guarantee that any drift from the metric´s definitions was immediately detected and corrected.13
The results of this assessment show that the number of phases and steps was not the main discriminator of performance between novices and experts. It was the number of total errors (errors plus critical errors) that determined a proficiency benchmark for RAPN performance, distinguished between novice and expert surgeon performances, and discriminated subgroup performances inside each one of the groups.13
The proficiency benchmark was quantitatively defined and derived from the performance of real-life RAPN experts. In the future, trainees enrolled in a PBP training program will have to consistently reach this benchmark, before they are considered proficient.13
Using the number of total errors, these RAPN metrics clearly differentiate surgical performance between novice and expert groups, establishing their construct validation.13 Most important, was the fact that they discriminated subgroups inside the novice and expert groups, determining who were the “real novices” and the “real experts”. They also identified a subgroup of experts that performed worse than a subgroup of novices, suggesting the importance of these metrics to train, certify and recertify different subgroups of surgeons.13
The face, content, construct and discriminative validation of these RAPN metrics, make them useful for training novice surgeons, allowing trainers to give explicit, formative, and immediate feedback on the surgical performance of the trainees. Besides training, these metrics might also be useful for credentialing robotic surgeons that want to perform RAPN and will play a crucial role in the reaccreditation of surgeons already performing RAPN.
Nowadays, it is no longer acceptable for novice surgeons to acquire their initial RAPN surgical skills using the patient as a training model. Their learning curve should start in the laboratory, and this is the reason why huge efforts are being placed in the development of animal, 3D printed, and/or Virtual Reality RAPN training models.
Although the trend is to produce training models that look good and recreate the same feeling as in real surgery, the authors are investing their time and effort in the development of a new training model that will be used to deliver a metric-based training curriculum.8 The human RAPN metrics validated by the authors, will inform the development of performance metrics to be used in this new RAPN training model.12,13 This will enable its use in a PBP training program, where the trainees will be trained up to a proficiency level, using a training model equipped with assessment metrics that are objective, transparent, and fair.
Written by: Rui Farinha1,2,3,4 Alberto Breda5 James Porter6 Alexandre Mottrie1,2 Ben Van Cleynenbreugel7,8 Jozef Vander Sloten9 Angelo Mottaran10,11 Anthony G. Gallagher12,13
- Orsi Academy, Melle, Belgium
- Department of Urology, Onze-Lieve-Vrouw Ziekenhuis, Aalst, Belgium
- Department of Urology, São José Hospital, Lisbon, Portugal
- Lusíadas Hospital, Lisbon, Portugal
- Department of Urology, Fundació Puigvert, Universitat Autonoma de Barcelona, Spain
- Swedish Urology Group, Swedish Medical Center, Seattle, WA, USA
- Department of Urology, University Hospitals Leuven, Leuven, Belgium
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Mechanical Engineering, Section of Biomechanics, KU Leuven, Leuven, Belgium
- Division of Urology, IRCCS Azienda Ospedaliero-Universitaris di Bologna, Italy
- University of Bologna, Bologna, Italy
- Faculty of Medicine, KU Leuven, Leuven, Belgium
- Faculty of Life and Health Sciences, Ulster University, Derry, Northern Ireland, United Kingdom
References:
- Znaor A, Lortet-Tieulent J, Laversanne M, Jemal A, Bray F. International variations and trends in renal cell carcinoma incidence and mortality. Eur Urol 2015;67:519–30.
- Rossi SH, Klatte T, Usher-Smith J, Stewart GD. Epidemiology and screening for renal cancer. World J Urol 2018;36:1341–53.
- Ljungberg B, Bensalah K, Canfield S, Dabestani S, Hofmann F, Hora M, et al. EAU guidelines on renal cell carcinoma: 2014 update. Eur Urol 2015;67:913–24.
- Thompson RH, Boorjian SA, Lohse CM, Leibovich BC, Kwon ED, Cheville JC, et al. Radical nephrectomy for pT1a renal masses may be associated with decreased overall survival compared with partial nephrectomy. J Urol 2008;179:463–8.
- Roos FC, Steffens S, Junker K, Janssen M, Becker F, Wegener G, et al. Survival advantage of partial over radical nephrectomy in patients presenting with localized renal cell carcinoma. BMC Cancer 2014;14:372.
- Farinha R, Puliatti S, Mazzone E, Amato M, Rosiello G, Yadav S, et al. Potential Contenders for the Leadership in Robotic Surgery. J Endourol 2022;36:317–26.
- Sarchi L, Mottaran A, Bravi CA, Paciotti M, Farinha R, Piazza P, et al. Robot-assisted radical prostatectomy feasibility and setting with the HugoTM robot-assisted surgery system. BJU Int 2022.
- Gallagher AG OGC. Fundamentals of Surgical Simulation: Principles and Practice. Springer Publishing Company, Incorporated; 2011.
- Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ. Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 2012;187:247–52.
- Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84:273–8
- Mottrie A, Mazzone E, Wiklund P, Graefen M, Collins JW, de Groote R, et al. Objective assessment of intraoperative skills for robot-assisted radical prostatectomy (RARP): results from the ERUS Scientific and Educational Working Groups Metrics Initiative. BJU Int 2020.
- Farinha R, Breda A, Porter J, Mottrie A, van Cleynenbreugel B, vander Sloten J, et al. International Expert Consensus on Metric-based Characterization of Robot-assisted Partial Nephrectomy. Eur Urol Focus 2022.
- Farinha R, Breda A, Porter J, Mottrie A, van Cleynenbreugel B, vander Sloten J, et al. Objective assessment of intraoperative skills for robot-assisted partial nephrectomy (RAPN). J Robot Surg 2023.