Self-distillation for surgical action recognition
Authors:
Amine Yamlahi,
Thuy Nuong Tran,
Patrick Godau,
Melanie Schellenberg,
Dominik Michael,
Finn-Henri Smidt,
Jan-Hinrich Noelke,
Tim Adler,
Minu Dietlinde Tizabi,
Chinedu Nwoye,
Nicolas Padoy,
Lena Maier-Hein
Abstract:
Surgical scene understanding is a key prerequisite for contextaware decision support in the operating room. While deep learning-based approaches have already reached or even surpassed human performance in various fields, the task of surgical action recognition remains a major challenge. With this contribution, we are the first to investigate the concept of self-distillation as a means of addressin…
▽ More
Surgical scene understanding is a key prerequisite for contextaware decision support in the operating room. While deep learning-based approaches have already reached or even surpassed human performance in various fields, the task of surgical action recognition remains a major challenge. With this contribution, we are the first to investigate the concept of self-distillation as a means of addressing class imbalance and potential label ambiguity in surgical video analysis. Our proposed method is a heterogeneous ensemble of three models that use Swin Transfomers as backbone and the concepts of self-distillation and multi-task learning as core design choices. According to ablation studies performed with the CholecT45 challenge data via cross-validation, the biggest performance boost is achieved by the usage of soft labels obtained by self-distillation. External validation of our method on an independent test set was achieved by providing a Docker container of our inference model to the challenge organizers. According to their analysis, our method outperforms all other solutions submitted to the latest challenge in the field. Our approach thus shows the potential of self-distillation for becoming an important tool in medical image analysis applications.
△ Less
Submitted 22 March, 2023;
originally announced March 2023.
CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection
Authors:
Chinedu Innocent Nwoye,
Tong Yu,
Saurav Sharma,
Aditya Murali,
Deepak Alapatt,
Armine Vardazaryan,
Kun Yuan,
Jonas Hajek,
Wolfgang Reiter,
Amine Yamlahi,
Finn-Henri Smidt,
Xiaoyang Zou,
Guoyan Zheng,
Bruno Oliveira,
Helena R. Torres,
Satoshi Kondo,
Satoshi Kasai,
Felix Holm,
Ege Özsoy,
Shuangchun Gui,
Han Li,
Sista Raviteja,
Rachana Sathish,
Pranav Poudel,
Binod Bhattarai
, et al. (24 additional authors not shown)
Abstract:
Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier effor…
▽ More
Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of <instrument, verb, target> triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.
△ Less
Submitted 14 July, 2023; v1 submitted 13 February, 2023;
originally announced February 2023.
