-
Video-rate multispectral imaging in laparoscopic surgery: First-in-human application
Authors:
Leonardo Ayala,
Sebastian Wirkert,
Anant Vemuri,
Tim Adler,
Silvia Seidlitz,
Sebastian Pirmann,
Christina Engels,
Dogu Teber,
Lena Maier-Hein
Abstract:
Multispectral and hyperspectral imaging (MSI/HSI) can provide clinically relevant information on morphological and functional tissue properties. Application in the operating room (OR), however, has so far been limited by complex hardware setups and slow acquisition times. To overcome these limitations, we propose a novel imaging system for video-rate spectral imaging in the clinical workflow. The…
▽ More
Multispectral and hyperspectral imaging (MSI/HSI) can provide clinically relevant information on morphological and functional tissue properties. Application in the operating room (OR), however, has so far been limited by complex hardware setups and slow acquisition times. To overcome these limitations, we propose a novel imaging system for video-rate spectral imaging in the clinical workflow. The system integrates a small snapshot multispectral camera with a standard laparoscope and a clinically commonly used light source, enabling the recording of multispectral images with a spectral dimension of 16 at a frame rate of 25 Hz. An ongoing in patient study shows that multispectral recordings from this system can help detect perfusion changes in partial nephrectomy surgery, thus opening the doors to a wide range of clinical applications.
△ Less
Submitted 28 May, 2021;
originally announced May 2021.
-
Out of distribution detection for intra-operative functional imaging
Authors:
Tim J. Adler,
Leonardo Ayala,
Lynton Ardizzone,
Hannes G. Kenngott,
Anant Vemuri,
Beat P. Müller-Stich,
Carsten Rother,
Ullrich Köthe,
Lena Maier-Hein
Abstract:
Multispectral optical imaging is becoming a key tool in the operating room. Recent research has shown that machine learning algorithms can be used to convert pixel-wise reflectance measurements to tissue parameters, such as oxygenation. However, the accuracy of these algorithms can only be guaranteed if the spectra acquired during surgery match the ones seen during training. It is therefore of gre…
▽ More
Multispectral optical imaging is becoming a key tool in the operating room. Recent research has shown that machine learning algorithms can be used to convert pixel-wise reflectance measurements to tissue parameters, such as oxygenation. However, the accuracy of these algorithms can only be guaranteed if the spectra acquired during surgery match the ones seen during training. It is therefore of great interest to detect so-called out of distribution (OoD) spectra to prevent the algorithm from presenting spurious results. In this paper we present an information theory based approach to OoD detection based on the widely applicable information criterion (WAIC). Our work builds upon recent methodology related to invertible neural networks (INN). Specifically, we make use of an ensemble of INNs as we need their tractable Jacobians in order to compute the WAIC. Comprehensive experiments with in silico, and in vivo multispectral imaging data indicate that our approach is well-suited for OoD detection. Our method could thus be an important step towards reliable functional imaging in the operating room.
△ Less
Submitted 5 November, 2019;
originally announced November 2019.
-
Band selection for oxygenation estimation with multispectral/hyperspectral imaging
Authors:
Leonardo A. Ayala,
Fabian Isensee,
Sebastian J. Wirkert,
Anant S. Vemuri,
Klaus H. Maier-Hein,
Baowei Fei,
Lena Maier-Hein
Abstract:
Multispectral imaging provides valuable information on tissue composition such as hemoglobin oxygen saturation. However, the real-time application of this technique in interventional medicine can be challenging due to the long acquisition times needed for large amounts of hyperspectral data with hundreds of bands. While this challenge can partially be addressed by choosing a discriminative subset…
▽ More
Multispectral imaging provides valuable information on tissue composition such as hemoglobin oxygen saturation. However, the real-time application of this technique in interventional medicine can be challenging due to the long acquisition times needed for large amounts of hyperspectral data with hundreds of bands. While this challenge can partially be addressed by choosing a discriminative subset of bands, the band selection methods proposed to date are mainly restricted by the availability of often hard to obtain reference measurements. We address this bottleneck with a new approach to band selection that leverages highly accurate Monte Carlo (MC) simulations. We hypothesize that a so chosen small subset of bands can reproduce or even improve upon the results of a quasi continuous spectral measurement. We further investigate whether novel domain adaptation techniques can address the inevitable domain shift stemming from the use of simulations. Initial results based on in silico and in vivo experiments suggest that 10-20 bands are sufficient to closely reproduce results from spectral measurements with 101 bands in the 500-700 nm range. The investigated domain adaptation technique, which only requires unlabeled in vivo measurements, yielded better results than the pure in silico band selection method. Overall, our method could guide development of fast multispectral imaging systems suited for interventional use without relying on complex hardware setups or manually labeled data
△ Less
Submitted 20 August, 2021; v1 submitted 27 May, 2019;
originally announced May 2019.
-
Survey of Computer Vision and Machine Learning in Gastrointestinal Endoscopy
Authors:
Anant S. Vemuri
Abstract:
This paper attempts to provide the reader a place to begin studying the application of computer vision and machine learning to gastrointestinal (GI) endoscopy. They have been classified into 18 categories. It should be be noted by the reader that this is a review from pre-deep learning era. A lot of deep learning based applications have not been covered in this thesis.
This paper attempts to provide the reader a place to begin studying the application of computer vision and machine learning to gastrointestinal (GI) endoscopy. They have been classified into 18 categories. It should be be noted by the reader that this is a review from pre-deep learning era. A lot of deep learning based applications have not been covered in this thesis.
△ Less
Submitted 26 April, 2019;
originally announced April 2019.
-
Hyperspectral Camera Selection for Interventional Health-care
Authors:
Anant S. Vemuri,
Sebastian Wirkert,
Lena Maier-Hein
Abstract:
Hyperspectral imaging (HSI) is an emerging modality in health-care applications for disease diagnosis, tissue assessment and image-guided surgery. Tissue reflectances captured by a HSI camera encode physiological properties including oxygenation and blood volume fraction. Optimal camera properties such as filter responses depend crucially on the application, and choosing a suitable HSI camera for…
▽ More
Hyperspectral imaging (HSI) is an emerging modality in health-care applications for disease diagnosis, tissue assessment and image-guided surgery. Tissue reflectances captured by a HSI camera encode physiological properties including oxygenation and blood volume fraction. Optimal camera properties such as filter responses depend crucially on the application, and choosing a suitable HSI camera for a research project and/or a clinical problem is not straightforward. We propose a generic framework for quantitative and application-specific performance assessment of HSI cameras and optical subsystem without the need for any physical setup. Based on user input about the camera characteristics and properties of the target domain, our framework quantifies the performance of the given camera configuration using large amounts of simulated data and a user-defined metric. The application of the framework to commercial camera selection and band selection in the context of oxygenation monitoring in interventional health-care demonstrates its integration into the design work-flow of an engineer. The advantage of being able to test the desired configuration without the need for purchasing expensive components may save system engineers valuable resources.
△ Less
Submitted 4 April, 2019;
originally announced April 2019.
-
Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks
Authors:
Tim J. Adler,
Lynton Ardizzone,
Anant Vemuri,
Leonardo Ayala,
Janek Gröhl,
Thomas Kirchner,
Sebastian Wirkert,
Jakob Kruse,
Carsten Rother,
Ullrich Köthe,
Lena Maier-Hein
Abstract:
Purpose: Optical imaging is evolving as a key technique for advanced sensing in the operating room. Recent research has shown that machine learning algorithms can be used to address the inverse problem of converting pixel-wise multispectral reflectance measurements to underlying tissue parameters, such as oxygenation. Assessment of the specific hardware used in conjunction with such algorithms, ho…
▽ More
Purpose: Optical imaging is evolving as a key technique for advanced sensing in the operating room. Recent research has shown that machine learning algorithms can be used to address the inverse problem of converting pixel-wise multispectral reflectance measurements to underlying tissue parameters, such as oxygenation. Assessment of the specific hardware used in conjunction with such algorithms, however, has not properly addressed the possibility that the problem may be ill-posed.
Methods: We present a novel approach to the assessment of optical imaging modalities, which is sensitive to the different types of uncertainties that may occur when inferring tissue parameters. Based on the concept of invertible neural networks, our framework goes beyond point estimates and maps each multispectral measurement to a full posterior probability distribution which is capable of representing ambiguity in the solution via multiple modes. Performance metrics for a hardware setup can then be computed from the characteristics of the posteriors.
Results: Application of the assessment framework to the specific use case of camera selection for physiological parameter estimation yields the following insights: (1) Estimation of tissue oxygenation from multispectral images is a well-posed problem, while (2) blood volume fraction may not be recovered without ambiguity. (3) In general, ambiguity may be reduced by increasing the number of spectral bands in the camera.
Conclusion: Our method could help to optimize optical camera design in an application-specific manner.
△ Less
Submitted 8 March, 2019;
originally announced March 2019.
-
Exploiting the potential of unlabeled endoscopic video data with self-supervised learning
Authors:
Tobias Ross,
David Zimmerer,
Anant Vemuri,
Fabian Isensee,
Manuel Wiesenfarth,
Sebastian Bodenstedt,
Fabian Both,
Philip Kessler,
Martin Wagner,
Beat Müller,
Hannes Kenngott,
Stefanie Speidel,
Annette Kopp-Schneider,
Klaus Maier-Hein,
Lena Maier-Hein
Abstract:
Surgical data science is a new research field that aims to observe all aspects of the patient treatment process in order to provide the right assistance at the right time. Due to the breakthrough successes of deep learning-based solutions for automatic image annotation, the availability of reference annotations for algorithm training is becoming a major bottleneck in the field. The purpose of this…
▽ More
Surgical data science is a new research field that aims to observe all aspects of the patient treatment process in order to provide the right assistance at the right time. Due to the breakthrough successes of deep learning-based solutions for automatic image annotation, the availability of reference annotations for algorithm training is becoming a major bottleneck in the field. The purpose of this paper was to investigate the concept of self-supervised learning to address this issue.
Our approach is guided by the hypothesis that unlabeled video data can be used to learn a representation of the target domain that boosts the performance of state-of-the-art machine learning algorithms when used for pre-training. Core of the method is an auxiliary task based on raw endoscopic video data of the target domain that is used to initialize the convolutional neural network (CNN) for the target task. In this paper, we propose the re-colorization of medical images with a generative adversarial network (GAN)-based architecture as auxiliary task. A variant of the method involves a second pre-training step based on labeled data for the target task from a related domain. We validate both variants using medical instrument segmentation as target task.
The proposed approach can be used to radically reduce the manual annotation effort involved in training CNNs. Compared to the baseline approach of generating annotated data from scratch, our method decreases exploratively the number of labeled images by up to 75% without sacrificing performance. Our method also outperforms alternative methods for CNN pre-training, such as pre-training on publicly available non-medical or medical data using the target task (in this instance: segmentation).
As it makes efficient use of available (non-)public and (un-)labeled data, the approach has the potential to become a valuable tool for CNN (pre-)training.
△ Less
Submitted 31 January, 2018; v1 submitted 27 November, 2017;
originally announced November 2017.
-
Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy
Authors:
S. Moccia,
S. J. Wirkert,
H. Kenngott,
A. S. Vemuri,
M. Apitz,
B. Mayer,
E. De Momi,
L. S. Mattos,
L. Maier-Hein
Abstract:
Objective: Surgical data science is evolving into a research field that aims to observe everything occurring within and around the treatment process to provide situation-aware data-driven assistance. In the context of endoscopic video analysis, the accurate classification of organs in the field of view of the camera proffers a technical challenge. Herein, we propose a new approach to anatomical st…
▽ More
Objective: Surgical data science is evolving into a research field that aims to observe everything occurring within and around the treatment process to provide situation-aware data-driven assistance. In the context of endoscopic video analysis, the accurate classification of organs in the field of view of the camera proffers a technical challenge. Herein, we propose a new approach to anatomical structure classification and image tagging that features an intrinsic measure of confidence to estimate its own performance with high reliability and which can be applied to both RGB and multispectral imaging (MI) data. Methods: Organ recognition is performed using a superpixel classification strategy based on textural and reflectance information. Classification confidence is estimated by analyzing the dispersion of class probabilities. Assessment of the proposed technology is performed through a comprehensive in vivo study with seven pigs. Results: When applied to image tagging, mean accuracy in our experiments increased from 65% (RGB) and 80% (MI) to 90% (RGB) and 96% (MI) with the confidence measure. Conclusion: Results showed that the confidence measure had a significant influence on the classification accuracy, and MI data are better suited for anatomical structure labeling than RGB data. Significance: This work significantly enhances the state of art in automatic labeling of endoscopic videos by introducing the use of the confidence metric, and by being the first study to use MI data for in vivo laparoscopic tissue classification. The data of our experiments will be released as the first in vivo MI dataset upon publication of this paper.
△ Less
Submitted 19 October, 2018; v1 submitted 21 June, 2017;
originally announced June 2017.
-
Automatic View-Point Selection for Inter-Operative Endoscopic Surveillance
Authors:
Anant S. Vemuri,
Stephane A. Nicolau,
Jacques Marescaux,
Luc Soler,
Nicholas Ayache
Abstract:
Esophageal adenocarcinoma arises from Barrett's esophagus, which is the most serious complication of gastroesophageal reflux disease. Strategies for screening involve periodic surveillance and tissue biopsies. A major challenge in such regular examinations is to record and track the disease evolution and re-localization of biopsied sites to provide targeted treatments. In this paper, we extend our…
▽ More
Esophageal adenocarcinoma arises from Barrett's esophagus, which is the most serious complication of gastroesophageal reflux disease. Strategies for screening involve periodic surveillance and tissue biopsies. A major challenge in such regular examinations is to record and track the disease evolution and re-localization of biopsied sites to provide targeted treatments. In this paper, we extend our original inter-operative relocalization framework to provide a constrained image based search for obtaining the best view-point match to the live view. Within this context we investigate the effect of: the choice of feature descriptors and color-space; filtering of uninformative frames and endoscopic modality, for view-point localization. Our experiments indicate an improvement in the best view-point retrieval rate to [92%,87%] from [73%,76%] (in our previous approach) for NBI and WL.
△ Less
Submitted 13 October, 2016;
originally announced October 2016.
