-
Labeling instructions matter in biomedical image analysis
Authors:
Tim Rädsch,
Annika Reinke,
Vivienn Weru,
Minu D. Tizabi,
Nicholas Schreck,
A. Emre Kavur,
Bünyamin Pekdemir,
Tobias Roß,
Annette Kopp-Schneider,
Lena Maier-Hein
Abstract:
Biomedical image analysis algorithm validation depends on high-quality annotation of reference datasets, for which labeling instructions are key. Despite their importance, their optimization remains largely unexplored. Here, we present the first systematic study of labeling instructions and their impact on annotation quality in the field. Through comprehensive examination of professional practice…
▽ More
Biomedical image analysis algorithm validation depends on high-quality annotation of reference datasets, for which labeling instructions are key. Despite their importance, their optimization remains largely unexplored. Here, we present the first systematic study of labeling instructions and their impact on annotation quality in the field. Through comprehensive examination of professional practice and international competitions registered at the MICCAI Society, we uncovered a discrepancy between annotators' needs for labeling instructions and their current quality and availability. Based on an analysis of 14,040 images annotated by 156 annotators from four professional companies and 708 Amazon Mechanical Turk (MTurk) crowdworkers using instructions with different information density levels, we further found that including exemplary images significantly boosts annotation performance compared to text-only descriptions, while solely extending text descriptions does not. Finally, professional annotators constantly outperform MTurk crowdworkers. Our study raises awareness for the need of quality standards in biomedical image analysis labeling instructions.
△ Less
Submitted 20 July, 2022;
originally announced July 2022.
-
How can we learn (more) from challenges? A statistical approach to driving future algorithm development
Authors:
Tobias Roß,
Pierangela Bruno,
Annika Reinke,
Manuel Wiesenfarth,
Lisa Koeppel,
Peter M. Full,
Bünyamin Pekdemir,
Patrick Godau,
Darya Trofimova,
Fabian Isensee,
Sara Moccia,
Francesco Calimeri,
Beat P. Müller-Stich,
Annette Kopp-Schneider,
Lena Maier-Hein
Abstract:
Challenges have become the state-of-the-art approach to benchmark image analysis algorithms in a comparative manner. While the validation on identical data sets was a great step forward, results analysis is often restricted to pure ranking tables, leaving relevant questions unanswered. Specifically, little effort has been put into the systematic investigation on what characterizes images in which…
▽ More
Challenges have become the state-of-the-art approach to benchmark image analysis algorithms in a comparative manner. While the validation on identical data sets was a great step forward, results analysis is often restricted to pure ranking tables, leaving relevant questions unanswered. Specifically, little effort has been put into the systematic investigation on what characterizes images in which state-of-the-art algorithms fail. To address this gap in the literature, we (1) present a statistical framework for learning from challenges and (2) instantiate it for the specific task of instrument instance segmentation in laparoscopic videos. Our framework relies on the semantic meta data annotation of images, which serves as foundation for a General Linear Mixed Models (GLMM) analysis. Based on 51,542 meta data annotations performed on 2,728 images, we applied our approach to the results of the Robust Medical Instrument Segmentation Challenge (ROBUST-MIS) challenge 2019 and revealed underexposure, motion and occlusion of instruments as well as the presence of smoke or other objects in the background as major sources of algorithm failure. Our subsequent method development, tailored to the specific remaining issues, yielded a deep learning model with state-of-the-art overall performance and specific strengths in the processing of images in which previous methods tended to fail. Due to the objectivity and generic applicability of our approach, it could become a valuable tool for validation in the field of medical image analysis and beyond. and segmentation of small, crossing, moving and transparent instrument(s) (parts).
△ Less
Submitted 17 June, 2021;
originally announced June 2021.
-
Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room
Authors:
Lena Maier-Hein,
Martin Wagner,
Tobias Ross,
Annika Reinke,
Sebastian Bodenstedt,
Peter M. Full,
Hellena Hempe,
Diana Mindroc-Filimon,
Patrick Scholz,
Thuy Nuong Tran,
Pierangela Bruno,
Anna Kisilenko,
Benjamin Müller,
Tornike Davitashvili,
Manuela Capek,
Minu Tizabi,
Matthias Eisenmann,
Tim J. Adler,
Janek Gröhl,
Melanie Schellenberg,
Silvia Seidlitz,
T. Y. Emmy Lai,
Bünyamin Pekdemir,
Veith Roethlingshoefer,
Fabian Both
, et al. (8 additional authors not shown)
Abstract:
Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper in…
▽ More
Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a specific emphasis on method robustness and generalization capabilities. Our data set comprises 30 laparoscopic videos and corresponding sensor data from medical devices in the operating room for three different types of laparoscopic surgery. Annotations include surgical phase labels for all video frames as well as information on instrument presence and corresponding instance-wise segmentation masks for surgical instruments (if any) in more than 10,000 individual frames. The data has successfully been used to organize international competitions within the Endoscopic Vision Challenges 2017 and 2019.
△ Less
Submitted 23 February, 2021; v1 submitted 7 May, 2020;
originally announced May 2020.
