Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark
Authors:
Martin Wagner,
Beat-Peter Müller-Stich,
Anna Kisilenko,
Duc Tran,
Patrick Heger,
Lars Mündermann,
David M Lubotsky,
Benjamin Müller,
Tornike Davitashvili,
Manuela Capek,
Annika Reinke,
Tong Yu,
Armine Vardazaryan,
Chinedu Innocent Nwoye,
Nicolas Padoy,
Xinyang Liu,
Eung-Joo Lee,
Constantin Disch,
Hans Meine,
Tong Xia,
Fucang Jia,
Satoshi Kondo,
Wolfgang Reiter,
Yueming **,
Yonghao Long
, et al. (16 additional authors not shown)
Abstract:
PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported fo…
▽ More
PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data single-center dataset. In this work we investigated the generalizability of phase recognition algorithms in a multi-center setting including more difficult recognition tasks such as surgical action and surgical skill. METHODS: To achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 hours was created. Labels included annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 teams submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment. RESULTS: F1-scores were achieved for phase recognition between 23.9% and 67.7% (n=9 teams), for instrument presence detection between 38.5% and 63.8% (n=8 teams), but for action recognition only between 21.8% and 23.3% (n=5 teams). The average absolute error for skill assessment was 0.78 (n=1 team). CONCLUSION: Surgical workflow and skill analysis are promising technologies to support the surgical team, but are not solved yet, as shown by our comparison of algorithms. This novel benchmark can be used for comparable evaluation and validation of future work.
△ Less
Submitted 30 September, 2021;
originally announced September 2021.
Robust Medical Instrument Segmentation Challenge 2019
Authors:
Tobias Ross,
Annika Reinke,
Peter M. Full,
Martin Wagner,
Hannes Kenngott,
Martin Apitz,
Hellena Hempe,
Diana Mindroc Filimon,
Patrick Scholz,
Thuy Nuong Tran,
Pierangela Bruno,
Pablo Arbeláez,
Gui-Bin Bian,
Sebastian Bodenstedt,
Jon Lindström Bolmgren,
Laura Bravo-Sánchez,
Hua-Bin Chen,
Cristina González,
Dong Guo,
Pål Halvorsen,
Pheng-Ann Heng,
Enes Hosgor,
Zeng-Guang Hou,
Fabian Isensee,
Debesh Jha
, et al. (25 additional authors not shown)
Abstract:
Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art meth…
▽ More
Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions.
In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).
△ Less
Submitted 19 May, 2020; v1 submitted 23 March, 2020;
originally announced March 2020.
