Robust deep learning-based semantic organ segmentation in hyperspectral images
Authors:
Silvia Seidlitz,
Jan Sellner,
Jan Odenthal,
Berkin Özdemir,
Alexander Studier-Fischer,
Samuel Knödler,
Leonardo Ayala,
Tim J. Adler,
Hannes G. Kenngott,
Minu Tizabi,
Martin Wagner,
Felix Nickel,
Beat P. Müller-Stich,
Lena Maier-Hein
Abstract:
Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literat…
▽ More
Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literature, we are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting: (1) What is an adequate representation of HSI data for neural network-based fully automated organ segmentation, especially with respect to the spatial granularity of the data (pixels vs. superpixels vs. patches vs. full images)? (2) Is there a benefit of using HSI data compared to other modalities, namely RGB data and processed HSI data (e.g. tissue parameters like oxygenation), when performing semantic organ segmentation? According to a comprehensive validation study based on 506 HSI images from 20 pigs, annotated with a total of 19 classes, deep learning-based segmentation performance increases, consistently across modalities, with the spatial context of the input data. Unprocessed HSI data offers an advantage over RGB data or processed data from the camera provider, with the advantage increasing with decreasing size of the input to the neural network. Maximum performance (HSI applied to whole images) yielded a mean DSC of 0.90 ((standard deviation (SD)) 0.04), which is in the range of the inter-rater variability (DSC of 0.89 ((standard deviation (SD)) 0.07)). We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding with many advantages over traditional imaging, including the ability to recover additional functional tissue information. Code and pre-trained models: https://github.com/IMSY-DKFZ/htc.
△ Less
Submitted 10 July, 2022; v1 submitted 9 November, 2021;
originally announced November 2021.
Machine learning-based analysis of hyperspectral images for automated sepsis diagnosis
Authors:
Maximilian Dietrich,
Silvia Seidlitz,
Nicholas Schreck,
Manuel Wiesenfarth,
Patrick Godau,
Minu Tizabi,
Jan Sellner,
Sebastian Marx,
Samuel Knödler,
Michael M. Allers,
Leonardo Ayala,
Karsten Schmidt,
Thorsten Brenner,
Alexander Studier-Fischer,
Felix Nickel,
Beat P. Müller-Stich,
Annette Kopp-Schneider,
Markus A. Weigand,
Lena Maier-Hein
Abstract:
Sepsis is a leading cause of mortality and critical illness worldwide. While robust biomarkers for early diagnosis are still missing, recent work indicates that hyperspectral imaging (HSI) has the potential to overcome this bottleneck by monitoring microcirculatory alterations. Automated machine learning-based diagnosis of sepsis based on HSI data, however, has not been explored to date. Given thi…
▽ More
Sepsis is a leading cause of mortality and critical illness worldwide. While robust biomarkers for early diagnosis are still missing, recent work indicates that hyperspectral imaging (HSI) has the potential to overcome this bottleneck by monitoring microcirculatory alterations. Automated machine learning-based diagnosis of sepsis based on HSI data, however, has not been explored to date. Given this gap in the literature, we leveraged an existing data set to (1) investigate whether HSI-based automated diagnosis of sepsis is possible and (2) put forth a list of possible confounders relevant for HSI-based tissue classification. While we were able to classify sepsis with an accuracy of over $98\,\%$ using the existing data, our research also revealed several subject-, therapy- and imaging-related confounders that may lead to an overestimation of algorithm performance when not balanced across the patient groups. We conclude that further prospective studies, carefully designed with respect to these confounders, are necessary to confirm the preliminary results obtained in this study.
△ Less
Submitted 15 June, 2021;
originally announced June 2021.
