-
Creating a Digital Twin of Spinal Surgery: A Proof of Concept
Authors:
Jonas Hein,
Frédéric Giraud,
Lilian Calvet,
Alexander Schwarz,
Nicola Alessandro Cavalcanti,
Sergey Prokudin,
Mazda Farshad,
Siyu Tang,
Marc Pollefeys,
Fabio Carrillo,
Philipp Fürnstahl
Abstract:
Surgery digitalization is the process of creating a virtual replica of real-world surgery, also referred to as a surgical digital twin (SDT). It has significant applications in various fields such as education and training, surgical planning, and automation of surgical tasks. In addition, SDTs are an ideal foundation for machine learning methods, enabling the automatic generation of training data.…
▽ More
Surgery digitalization is the process of creating a virtual replica of real-world surgery, also referred to as a surgical digital twin (SDT). It has significant applications in various fields such as education and training, surgical planning, and automation of surgical tasks. In addition, SDTs are an ideal foundation for machine learning methods, enabling the automatic generation of training data. In this paper, we present a proof of concept (PoC) for surgery digitalization that is applied to an ex-vivo spinal surgery. The proposed digitalization focuses on the acquisition and modelling of the geometry and appearance of the entire surgical scene. We employ five RGB-D cameras for dynamic 3D reconstruction of the surgeon, a high-end camera for 3D reconstruction of the anatomy, an infrared stereo camera for surgical instrument tracking, and a laser scanner for 3D reconstruction of the operating room and data fusion. We justify the proposed methodology, discuss the challenges faced and further extensions of our prototype. While our PoC partially relies on manual data curation, its high quality and great potential motivate the development of automated methods for the creation of SDTs.
△ Less
Submitted 22 May, 2024; v1 submitted 25 March, 2024;
originally announced March 2024.
-
Domain adaptation strategies for 3D reconstruction of the lumbar spine using real fluoroscopy data
Authors:
Sascha Jecklin,
Youyang Shen,
Amandine Gout,
Daniel Suter,
Lilian Calvet,
Lukas Zingg,
Jennifer Straub,
Nicola Alessandro Cavalcanti,
Mazda Farshad,
Philipp Fürnstahl,
Hooman Esfandiari
Abstract:
This study tackles key obstacles in adopting surgical navigation in orthopedic surgeries, including time, cost, radiation, and workflow integration challenges. Recently, our work X23D showed an approach for generating 3D anatomical models of the spine from only a few intraoperative fluoroscopic images. This negates the need for conventional registration-based surgical navigation by creating a dire…
▽ More
This study tackles key obstacles in adopting surgical navigation in orthopedic surgeries, including time, cost, radiation, and workflow integration challenges. Recently, our work X23D showed an approach for generating 3D anatomical models of the spine from only a few intraoperative fluoroscopic images. This negates the need for conventional registration-based surgical navigation by creating a direct intraoperative 3D reconstruction of the anatomy. Despite these strides, the practical application of X23D has been limited by a domain gap between synthetic training data and real intraoperative images.
In response, we devised a novel data collection protocol for a paired dataset consisting of synthetic and real fluoroscopic images from the same perspectives. Utilizing this dataset, we refined our deep learning model via transfer learning, effectively bridging the domain gap between synthetic and real X-ray data. A novel style transfer mechanism also allows us to convert real X-rays to mirror the synthetic domain, enabling our in-silico-trained X23D model to achieve high accuracy in real-world settings.
Our results demonstrated that the refined model can rapidly generate accurate 3D reconstructions of the entire lumbar spine from as few as three intraoperative fluoroscopic shots. It achieved an 84% F1 score, matching the accuracy of our previous synthetic data-based research. Additionally, with a computational time of only 81.1 ms, our approach provides real-time capabilities essential for surgery integration.
Through examining ideal imaging setups and view angle dependencies, we've further confirmed our system's practicality and dependability in clinical settings. Our research marks a significant step forward in intraoperative 3D reconstruction, offering enhancements to surgical planning, navigation, and robotics.
△ Less
Submitted 18 June, 2024; v1 submitted 29 January, 2024;
originally announced January 2024.
-
Automatic registration with continuous pose updates for marker-less surgical navigation in spine surgery
Authors:
Florentin Liebmann,
Marco von Atzigen,
Dominik Stütz,
Julian Wolf,
Lukas Zingg,
Daniel Suter,
Laura Leoty,
Hooman Esfandiari,
Jess G. Snedeker,
Martin R. Oswald,
Marc Pollefeys,
Mazda Farshad,
Philipp Fürnstahl
Abstract:
Established surgical navigation systems for pedicle screw placement have been proven to be accurate, but still reveal limitations in registration or surgical guidance. Registration of preoperative data to the intraoperative anatomy remains a time-consuming, error-prone task that includes exposure to harmful radiation. Surgical guidance through conventional displays has well-known drawbacks, as inf…
▽ More
Established surgical navigation systems for pedicle screw placement have been proven to be accurate, but still reveal limitations in registration or surgical guidance. Registration of preoperative data to the intraoperative anatomy remains a time-consuming, error-prone task that includes exposure to harmful radiation. Surgical guidance through conventional displays has well-known drawbacks, as information cannot be presented in-situ and from the surgeon's perspective. Consequently, radiation-free and more automatic registration methods with subsequent surgeon-centric navigation feedback are desirable. In this work, we present an approach that automatically solves the registration problem for lumbar spinal fusion surgery in a radiation-free manner. A deep neural network was trained to segment the lumbar spine and simultaneously predict its orientation, yielding an initial pose for preoperative models, which then is refined for each vertebra individually and updated in real-time with GPU acceleration while handling surgeon occlusions. An intuitive surgical guidance is provided thanks to the integration into an augmented reality based navigation system. The registration method was verified on a public dataset with a mean of 96\% successful registrations, a target registration error of 2.73 mm, a screw trajectory error of 1.79° and a screw entry point error of 2.43 mm. Additionally, the whole pipeline was validated in an ex-vivo surgery, yielding a 100\% screw accuracy and a registration accuracy of 1.20 mm. Our results meet clinical demands and emphasize the potential of RGB-D data for fully automatic registration approaches in combination with augmented reality guidance.
△ Less
Submitted 5 August, 2023;
originally announced August 2023.
-
Synthetic white balancing for intra-operative hyperspectral imaging
Authors:
Anisha Bahl,
Conor C. Horgan,
Mirek Janatka,
Oscar J. MacCormac,
Philip Noonan,
Yi**g Xie,
Jianrong Qiu,
Nicola Cavalcanti,
Philipp Fürnstahl,
Michael Ebner,
Mads S. Bergholt,
Jonathan Shapey,
Tom Vercauteren
Abstract:
Hyperspectral imaging shows promise for surgical applications to non-invasively provide spatially-resolved, spectral information. For calibration purposes, a white reference image of a highly-reflective Lambertian surface should be obtained under the same imaging conditions. Standard white references are not sterilizable, and so are unsuitable for surgical environments. We demonstrate the necessit…
▽ More
Hyperspectral imaging shows promise for surgical applications to non-invasively provide spatially-resolved, spectral information. For calibration purposes, a white reference image of a highly-reflective Lambertian surface should be obtained under the same imaging conditions. Standard white references are not sterilizable, and so are unsuitable for surgical environments. We demonstrate the necessity for in situ white references and address this by proposing a novel, sterile, synthetic reference construction algorithm. The use of references obtained at different distances and lighting conditions to the subject were examined. Spectral and color reconstructions were compared with standard measurements qualitatively and quantitatively, using $ΔE$ and normalised RMSE respectively. The algorithm forms a composite image from a video of a standard sterile ruler, whose imperfect reflectivity is compensated for. The reference is modelled as the product of independent spatial and spectral components, and a scalar factor accounting for gain, exposure, and light intensity. Evaluation of synthetic references against ideal but non-sterile references is performed using the same metrics alongside pixel-by-pixel errors. Finally, intraoperative integration is assessed though cadaveric experiments. Improper white balancing leads to increases in all quantitative and qualitative errors. Synthetic references achieve median pixel-by-pixel errors lower than 6.5% and produce similar reconstructions and errors to an ideal reference. The algorithm integrated well into surgical workflow, achieving median pixel-by-pixel errors of 4.77%, while maintaining good spectral and color reconstruction.
△ Less
Submitted 24 July, 2023;
originally announced July 2023.
-
Safe Deep RL for Intraoperative Planning of Pedicle Screw Placement
Authors:
Yunke Ao,
Hooman Esfandiari,
Fabio Carrillo,
Yarden As,
Mazda Farshad,
Benjamin F. Grewe,
Andreas Krause,
Philipp Fuernstahl
Abstract:
Spinal fusion surgery requires highly accurate implantation of pedicle screw implants, which must be conducted in critical proximity to vital structures with a limited view of anatomy. Robotic surgery systems have been proposed to improve placement accuracy, however, state-of-the-art systems suffer from the limitations of open-loop approaches, as they follow traditional concepts of preoperative pl…
▽ More
Spinal fusion surgery requires highly accurate implantation of pedicle screw implants, which must be conducted in critical proximity to vital structures with a limited view of anatomy. Robotic surgery systems have been proposed to improve placement accuracy, however, state-of-the-art systems suffer from the limitations of open-loop approaches, as they follow traditional concepts of preoperative planning and intraoperative registration, without real-time recalculation of the surgical plan. In this paper, we propose an intraoperative planning approach for robotic spine surgery that leverages real-time observation for drill path planning based on Safe Deep Reinforcement Learning (DRL). The main contributions of our method are (1) the capability to guarantee safe actions by introducing an uncertainty-aware distance-based safety filter; and (2) the ability to compensate for incomplete intraoperative anatomical information, by encoding a-priori knowledge about anatomical structures with a network pre-trained on high-fidelity anatomical models. Planning quality was assessed by quantitative comparison with the gold standard (GS) drill planning. In experiments with 5 models derived from real magnetic resonance imaging (MRI) data, our approach was capable of achieving 90% bone penetration with respect to the GS while satisfying safety requirements, even under observation and motion uncertainty. To the best of our knowledge, our approach is the first safe DRL approach focusing on orthopedic surgeries.
△ Less
Submitted 10 May, 2023; v1 submitted 9 May, 2023;
originally announced May 2023.
-
Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments
Authors:
Jonas Hein,
Nicola Cavalcanti,
Daniel Suter,
Lukas Zingg,
Fabio Carrillo,
Lilian Calvet,
Mazda Farshad,
Marc Pollefeys,
Nassir Navab,
Philipp Fürnstahl
Abstract:
State-of-the-art research of traditional computer vision is increasingly leveraged in the surgical domain. A particular focus in computer-assisted surgery is to replace marker-based tracking systems for instrument localization with pure image-based 6DoF pose estimation using deep-learning methods. However, state-of-the-art single-view pose estimation methods do not yet meet the accuracy required f…
▽ More
State-of-the-art research of traditional computer vision is increasingly leveraged in the surgical domain. A particular focus in computer-assisted surgery is to replace marker-based tracking systems for instrument localization with pure image-based 6DoF pose estimation using deep-learning methods. However, state-of-the-art single-view pose estimation methods do not yet meet the accuracy required for surgical navigation. In this context, we investigate the benefits of multi-view setups for highly accurate and occlusion-robust 6DoF pose estimation of surgical instruments and derive recommendations for an ideal camera system that addresses the challenges in the operating room.
The contributions of this work are threefold. First, we present a multi-camera capture setup consisting of static and head-mounted cameras, which allows us to study the performance of pose estimation methods under various camera configurations. Second, we publish a multi-view RGB-D video dataset of ex-vivo spine surgeries, captured in a surgical wet lab and a real operating theatre and including rich annotations for surgeon, instrument, and patient anatomy. Third, we evaluate three state-of-the-art single-view and multi-view methods for the task of 6DoF pose estimation of surgical instruments and analyze the influence of camera configurations, training data, and occlusions on the pose accuracy and generalization ability. The best method utilizes five cameras in a multi-view pose optimization and achieves an average position and orientation error of 1.01 mm and 0.89°for a surgical drill as well as 2.79 mm and 3.33°for a screwdriver under optimal conditions. Our results demonstrate that marker-less tracking of surgical instruments is becoming a feasible alternative to existing marker-based systems.
△ Less
Submitted 22 December, 2023; v1 submitted 5 May, 2023;
originally announced May 2023.
-
Automatic breach detection during spine pedicle drilling based on vibroacoustic sensing
Authors:
Aidana Massalimova,
Maikel Timmermans,
Nicola Cavalcanti,
Daniel Suter,
Matthias Seibold,
Fabio Carrillo,
Christoph J. Laux,
Reto Sutter,
Mazda Farshad,
Kathleen Denis,
Philipp Fürnstahl
Abstract:
Pedicle drilling is a complex and critical spinal surgery task. Detecting breach or penetration of the surgical tool to the cortical wall during pilot-hole drilling is essential to avoid damage to vital anatomical structures adjacent to the pedicle, such as the spinal cord, blood vessels, and nerves. Currently, the guidance of pedicle drilling is done using image-guided methods that are radiation…
▽ More
Pedicle drilling is a complex and critical spinal surgery task. Detecting breach or penetration of the surgical tool to the cortical wall during pilot-hole drilling is essential to avoid damage to vital anatomical structures adjacent to the pedicle, such as the spinal cord, blood vessels, and nerves. Currently, the guidance of pedicle drilling is done using image-guided methods that are radiation intensive and limited to the preoperative information. This work proposes a new radiation-free breach detection algorithm leveraging a non-visual sensor setup in combination with deep learning approach. Multiple vibroacoustic sensors, such as a contact microphone, a free-field microphone, a tri-axial accelerometer, a uni-axial accelerometer, and an optical tracking system were integrated into the setup. Data were collected on four cadaveric human spines, ranging from L5 to T10. An experienced spine surgeon drilled the pedicles relying on optical navigation. A new automatic labeling method based on the tracking data was introduced. Labeled data was subsequently fed to the network in mel-spectrograms, classifying the data into breach and non-breach. Different sensor types, sensor positioning, and their combinations were evaluated. The best results in breach recall for individual sensors could be achieved using contact microphones attached to the dorsal skin (85.8\%) and uni-axial accelerometers clamped to the spinous process of the drilled vertebra (81.0\%). The best-performing data fusion model combined the latter two sensors with a breach recall of 98\%. The proposed method shows the great potential of non-visual sensor fusion for avoiding screw misplacement and accidental bone breaches during pedicle drilling and could be extended to further surgical applications.
△ Less
Submitted 27 March, 2023;
originally announced March 2023.
-
Improved Techniques for the Conditional Generative Augmentation of Clinical Audio Data
Authors:
Mane Margaryan,
Matthias Seibold,
Indu Joshi,
Mazda Farshad,
Philipp Fürnstahl,
Nassir Navab
Abstract:
Data augmentation is a valuable tool for the design of deep learning systems to overcome data limitations and stabilize the training process. Especially in the medical domain, where the collection of large-scale data sets is challenging and expensive due to limited access to patient data, relevant environments, as well as strict regulations, community-curated large-scale public datasets, pretraine…
▽ More
Data augmentation is a valuable tool for the design of deep learning systems to overcome data limitations and stabilize the training process. Especially in the medical domain, where the collection of large-scale data sets is challenging and expensive due to limited access to patient data, relevant environments, as well as strict regulations, community-curated large-scale public datasets, pretrained models, and advanced data augmentation methods are the main factors for develo** reliable systems to improve patient care. However, for the development of medical acoustic sensing systems, an emerging field of research, the community lacks large-scale publicly available data sets and pretrained models. To address the problem of limited data, we propose a conditional generative adversarial neural network-based augmentation method which is able to synthesize mel spectrograms from a learned data distribution of a source data set. In contrast to previously proposed fully convolutional models, the proposed model implements residual Squeeze and Excitation modules in the generator architecture. We show that our method outperforms all classical audio augmentation techniques and previously published generative methods in terms of generated sample quality and a performance improvement of 2.84% of Macro F1-Score for a classifier trained on the augmented data set, an enhancement of $1.14\%$ in relation to previous work. By analyzing the correlation of intermediate feature spaces, we show that the residual Squeeze and Excitation modules help the model to reduce redundancy in the latent features. Therefore, the proposed model advances the state-of-the-art in the augmentation of clinical audio data and improves the data bottleneck for the design of clinical acoustic sensing systems.
△ Less
Submitted 5 November, 2022;
originally announced November 2022.
-
A new sensing paradigm for the vibroacoustic detection of pedicle screw loosening
Authors:
Matthias Seibold,
Bastian Sigrist,
Tobias Götschi,
Jonas Widmer,
Sandro Hodel,
Mazda Farshad,
Nassir Navab,
Philipp Fürnstahl,
Christoph J. Laux
Abstract:
There is an unmet clinical need for develo** novel methods to complement and replace the current radiation-emitting imaging-based methods for the detection of loose pedicle screws as a complication after spinal fusion surgery which fail to identify a substantial amount of loose implants. In this work, we propose a new methodology and paradigm for the radiation-free, non-destructive, and easy-to-…
▽ More
There is an unmet clinical need for develo** novel methods to complement and replace the current radiation-emitting imaging-based methods for the detection of loose pedicle screws as a complication after spinal fusion surgery which fail to identify a substantial amount of loose implants. In this work, we propose a new methodology and paradigm for the radiation-free, non-destructive, and easy-to-integrate detection of pedicle screw loosening based on vibroacoustic sensing. Furthermore, we propose a novel simulation technique for pedicle screw loosening, which is biomechanically validated. For the detection of a loose implant, we excite the vertebra of interest with a sine sweep vibration at the spinous process and attach a custom highly-sensitive piezo contact microphone to the screw head to capture the propagated vibration characteristics which are analyzed using a detection pipeline based on spectrogram features and a SE-ResNet-18. To validate the proposed approach, we conducted experiments using four human cadaveric lumbar spine specimens and evaluate our algorithm in a cross validation experiment. Our method reaches a sensitivity of 91.50 +- 6.58% and a specificity of 91.10 +- 2.27%. The proposed system shows great potentials for the development of alternative assessment methods for implant loosening based on vibroacoustic sensing.
△ Less
Submitted 27 October, 2022;
originally announced October 2022.
-
Sonification as a Reliable Alternative to Conventional Visual Surgical Navigation
Authors:
Sasan Matinfar,
Mehrdad Salehi,
Daniel Suter,
Matthias Seibold,
Navid Navab,
Shervin Dehghani,
Florian Wanivenhaus,
Philipp Fürnstahl,
Mazda Farshad,
Nassir Navab
Abstract:
Despite the undeniable advantages of image-guided surgical assistance systems in terms of accuracy, such systems have not yet fully met surgeons' needs or expectations regarding usability, time efficiency, and their integration into the surgical workflow. On the other hand, perceptual studies have shown that presenting independent but causally correlated information via multimodal feedback involvi…
▽ More
Despite the undeniable advantages of image-guided surgical assistance systems in terms of accuracy, such systems have not yet fully met surgeons' needs or expectations regarding usability, time efficiency, and their integration into the surgical workflow. On the other hand, perceptual studies have shown that presenting independent but causally correlated information via multimodal feedback involving different sensory modalities can improve task performance. This article investigates an alternative method for computer-assisted surgical navigation, introduces a novel sonification methodology for navigated pedicle screw placement, and discusses advanced solutions based on multisensory feedback. The proposed method comprises a novel sonification solution for alignment tasks in four degrees of freedom based on frequency modulation (FM) synthesis. We compared the resulting accuracy and execution time of the proposed sonification method with visual navigation, which is currently considered the state of the art. We conducted a phantom study in which 17 surgeons executed the pedicle screw placement task in the lumbar spine, guided by either the proposed sonification-based or the traditional visual navigation method. The results demonstrated that the proposed method is as accurate as the state of the art while decreasing the surgeon's need to focus on visual navigation displays instead of the natural focus on surgical tools and targeted anatomy during task execution.
△ Less
Submitted 30 June, 2022;
originally announced June 2022.
-
Conditional Generative Data Augmentation for Clinical Audio Datasets
Authors:
Matthias Seibold,
Armando Hoch,
Mazda Farshad,
Nassir Navab,
Philipp Fürnstahl
Abstract:
In this work, we propose a novel data augmentation method for clinical audio datasets based on a conditional Wasserstein Generative Adversarial Network with Gradient Penalty (cWGAN-GP), operating on log-mel spectrograms. To validate our method, we created a clinical audio dataset which was recorded in a real-world operating room during Total Hip Arthroplasty (THA) procedures and contains typical s…
▽ More
In this work, we propose a novel data augmentation method for clinical audio datasets based on a conditional Wasserstein Generative Adversarial Network with Gradient Penalty (cWGAN-GP), operating on log-mel spectrograms. To validate our method, we created a clinical audio dataset which was recorded in a real-world operating room during Total Hip Arthroplasty (THA) procedures and contains typical sounds which resemble the different phases of the intervention. We demonstrate the capability of the proposed method to generate realistic class-conditioned samples from the dataset distribution and show that training with the generated augmented samples outperforms classical audio augmentation methods in terms of classification performance. The performance was evaluated using a ResNet-18 classifier which shows a mean Macro F1-score improvement of 1.70% in a 5-fold cross validation experiment using the proposed augmentation method. Because clinical data is often expensive to acquire, the development of realistic and high-quality data augmentation methods is crucial to improve the robustness and generalization capabilities of learning-based algorithms which is especially important for safety-critical medical applications. Therefore, the proposed data augmentation method is an important step towards improving the data bottleneck for clinical audio-based machine learning systems.
△ Less
Submitted 24 June, 2022; v1 submitted 22 March, 2022;
originally announced March 2022.
-
Three-Dimensional Automated Assessment of the Distal Radioulnar Joint Morphology according to Sigmoid Notch Surface Orientation
Authors:
Simon Roner,
Philipp Fürnstahl,
Anne-Gita Scheibler,
Ladislav Nagy,
Fabio Carrillo
Abstract:
The aim of this study was to develop a new method for generating reproducible 3D measurements for the quantification of the distal radioulnar joint morphology. We hypothesized that automated 3D measurement of the ulnar variance and the sigmoid notch angle are comparable to those of the gold standard, while overcoming some of the drawbacks of conventional 2D measurements. Radiological data of healt…
▽ More
The aim of this study was to develop a new method for generating reproducible 3D measurements for the quantification of the distal radioulnar joint morphology. We hypothesized that automated 3D measurement of the ulnar variance and the sigmoid notch angle are comparable to those of the gold standard, while overcoming some of the drawbacks of conventional 2D measurements. Radiological data of healthy forearm bones of 53 adult subjects were included in the study. Automated measurements for the assessment of the sigmoid-notch morphology based on 3D landmarks were developed incorporating the subject-specific estimation of the cartilage surface orientation. A common anatomical reference was defined among the different imaging modalities and a comparison between the sigmoid notch angle and UV measurements was performed in radiographs, CT scans and 3D models. Finally, the developed UV measurements in 3D were compared to the method by radiographs in an experimental setup with 3D printed bone models. The proposed automated 3D analysis of notch subtype showed a significantly larger notch radius for negative notch angle compared to positive sigmoid notch angle subjects. Similar UV measurements were obtained in healthy joint morphologies with a high correlation between the radiographs and 3D measurements, for sigmoid notch angle and UV . In the experimental setup with a modified radial inclination, the UV was on average 1.13 mm larger in the radiographs compared to the 3D measurements, and 1.30 mm larger in the cases with a modified palmar tilt. The developed 3D measurements allowed to reliably quantify differences in the sigmoid notch subtypes.
△ Less
Submitted 27 March, 2020;
originally announced March 2020.
-
Digitalization of the IOM: A comprehensive cadaveric study for obtaining three-dimensional models and morphological properties of the forearm's interosseous membrane
Authors:
Fabio Carrillo,
Simon Suter,
Fabio A. Casari,
Reto Sutter,
Ladislav Nagy,
Jess G. Snedeker,
Philipp Fürnstahl
Abstract:
State-of-the-art of preoperative planning for forearm orthopaedic surgeries is currently limited to simple bone procedures. The increasing interest of clinicians for more comprehensive analysis of complex pathologies often requires dynamic models, able to include the soft tissue influence into the preoperative process. Previous studies have shown that the interosseous membrane (IOM) influences for…
▽ More
State-of-the-art of preoperative planning for forearm orthopaedic surgeries is currently limited to simple bone procedures. The increasing interest of clinicians for more comprehensive analysis of complex pathologies often requires dynamic models, able to include the soft tissue influence into the preoperative process. Previous studies have shown that the interosseous membrane (IOM) influences forearm motion and stability, but due to the lack of morphological and biomechanical data, existing simulation models of the IOM are either too simple or clinically unreliable. This work aims to address this problematic by generating 3D morphological and tensile properties of the individual IOM structures. First, micro- and standard-CT acquisitions were performed on five fresh-frozen annotated cadaveric forearms for the generation of 3D models of the radius, ulna and each of the individual ligaments of the IOM. Afterwards, novel 3D methods were developed for the measurement of common morphological features, which were validated against established optical ex-vivo measurements. Finally, we investigated the individual tensile properties of each IOM ligament. The generated 3D morphological features can provide the basis for the future development of functional planning simulation of the forearm.
△ Less
Submitted 26 March, 2020;
originally announced March 2020.
-
Automatic Modelling of Human Musculoskeletal Ligaments -- Framework Overview and Model Quality Evaluation
Authors:
Noura Hamze,
Lukas Nocker,
Nikolaus Rauch,
Markus Walzthöni,
Fabio Carrillo,
Philipp Fürnstahl,
Matthias Harders
Abstract:
Accurate segmentation of connective soft tissues is still a challenging task, which hinders the generation of corresponding geometric models for biomechanical computations. Alternatively, one could predict ligament insertion sites and then approximate the shapes, based on anatomical knowledge and morphological studies. Here, we describe a corresponding integrated framework for the automatic modell…
▽ More
Accurate segmentation of connective soft tissues is still a challenging task, which hinders the generation of corresponding geometric models for biomechanical computations. Alternatively, one could predict ligament insertion sites and then approximate the shapes, based on anatomical knowledge and morphological studies. Here, we describe a corresponding integrated framework for the automatic modelling of human musculoskeletal ligaments. We combine statistical shape modelling with geometric algorithms to automatically identify insertion sites, based on which geometric surface and volume meshes are created. For demonstrating a clinical use case, the framework has been applied to generate models of the interosseous membrane in the forearm. For the adoption to the forearm anatomy, ligament insertion sites in the statistical model were defined according to anatomical predictions following an approach proposed in prior work. For evaluation we compared the generated sites, as well as the ligament shapes, to data obtained from a cadaveric study, involving five forearms with a total of 15 ligaments. Our framework permitted the creation of 3D models approximating ligaments' shapes with good fidelity. However, we found that the statistical model trained with the state-of-the-art prediction of the insertion sites was not always reliable. Using that model, average mean square errors as well as Hausdorff distances of the meshes increased by more than one order of magnitude, as compared to employing the known insertion locations of the cadaveric study. Using the latter an average mean square error of 0.59 mm and an average Hausdorff distance of less than 7 mm resulted, for the complete set of ligaments. In conclusion, the presented approach for generating ligament shapes from insertion points appears to be feasible but the detection of the insertion sites with a SSM is too inaccurate.
△ Less
Submitted 24 March, 2020;
originally announced March 2020.
-
Registration made easy -- standalone orthopedic navigation with HoloLens
Authors:
Florentin Liebmann,
Simon Roner,
Marco von Atzigen,
Florian Wanivenhaus,
Caroline Neuhaus,
José Spirig,
Davide Scaramuzza,
Reto Sutter,
Jess Snedeker,
Mazda Farshad,
Philipp Fürnstahl
Abstract:
In surgical navigation, finding correspondence between preoperative plan and intraoperative anatomy, the so-called registration task, is imperative. One promising approach is to intraoperatively digitize anatomy and register it with the preoperative plan. State-of-the-art commercial navigation systems implement such approaches for pedicle screw placement in spinal fusion surgery. Although these sy…
▽ More
In surgical navigation, finding correspondence between preoperative plan and intraoperative anatomy, the so-called registration task, is imperative. One promising approach is to intraoperatively digitize anatomy and register it with the preoperative plan. State-of-the-art commercial navigation systems implement such approaches for pedicle screw placement in spinal fusion surgery. Although these systems improve surgical accuracy, they are not gold standard in clinical practice. Besides economical reasons, this may be due to their difficult integration into clinical workflows and unintuitive navigation feedback. Augmented Reality has the potential to overcome these limitations. Consequently, we propose a surgical navigation approach comprising intraoperative surface digitization for registration and intuitive holographic navigation for pedicle screw placement that runs entirely on the Microsoft HoloLens. Preliminary results from phantom experiments suggest that the method may meet clinical accuracy requirements.
△ Less
Submitted 17 January, 2020;
originally announced January 2020.
-
Pivot calibration concept for sensor attached mobile c-arms
Authors:
Sing Chun Lee,
Matthias Seibold,
Philipp Fürnstahl,
Mazda Farshad,
Nassir Navab
Abstract:
Medical augmented reality has been actively studied for decades and many methods have been proposed torevolutionize clinical procedures. One example is the camera augmented mobile C-arm (CAMC), which providesa real-time video augmentation onto medical images by rigidly mounting and calibrating a camera to the imagingdevice. Since then, several CAMC variations have been suggested by calibrating 2D/…
▽ More
Medical augmented reality has been actively studied for decades and many methods have been proposed torevolutionize clinical procedures. One example is the camera augmented mobile C-arm (CAMC), which providesa real-time video augmentation onto medical images by rigidly mounting and calibrating a camera to the imagingdevice. Since then, several CAMC variations have been suggested by calibrating 2D/3D cameras, trackers, andmore recently a Microsoft HoloLens to the C-arm. Different calibration methods have been applied to establishthe correspondence between the rigidly attached sensor and the imaging device. A crucial step for these methodsis the acquisition of X-Ray images or 3D reconstruction volumes; therefore, requiring the emission of ionizingradiation. In this work, we analyze the mechanical motion of the device and propose an alternatative methodto calibrate sensors to the C-arm without emitting any radiation. Given a sensor is rigidly attached to thedevice, we introduce an extended pivot calibration concept to compute the fixed translation from the sensor tothe C-arm rotation center. The fixed relationship between the sensor and rotation center can be formulated as apivot calibration problem with the pivot point moving on a locus. Our method exploits the rigid C-arm motiondescribing a Torus surface to solve this calibration problem. We explain the geometry of the C-arm motion andits relation to the attached sensor, propose a calibration algorithm and show its robustness against noise, as wellas trajectory and observed pose density by computer simulations. We discuss this geometric-based formulationand its potential extensions to different C-arm applications.
△ Less
Submitted 9 January, 2020;
originally announced January 2020.
-
Active Learning for Segmentation Based on Bayesian Sample Queries
Authors:
Firat Ozdemir,
Zixuan Peng,
Philipp Fuernstahl,
Christine Tanner,
Orcun Goksel
Abstract:
Segmentation of anatomical structures is a fundamental image analysis task for many applications in the medical field. Deep learning methods have been shown to perform well, but for this purpose large numbers of manual annotations are needed in the first place, which necessitate prohibitive levels of resources that are often unavailable. In an active learning framework of selecting informed sample…
▽ More
Segmentation of anatomical structures is a fundamental image analysis task for many applications in the medical field. Deep learning methods have been shown to perform well, but for this purpose large numbers of manual annotations are needed in the first place, which necessitate prohibitive levels of resources that are often unavailable. In an active learning framework of selecting informed samples for manual labeling, expert clinician time for manual annotation can be optimally utilized, enabling the establishment of large labeled datasets for machine learning. In this paper, we propose a novel method that combines representativeness with uncertainty in order to estimate ideal samples to be annotated, iteratively from a given dataset. Our novel representativeness metric is based on Bayesian sampling, by using information-maximizing autoencoders. We conduct experiments on a shoulder magnetic resonance imaging (MRI) dataset for the segmentation of four musculoskeletal tissue classes. Quantitative results show that the annotation of representative samples selected by our proposed querying method yields an improved segmentation performance at each active learning iteration, compared to a baseline method that also employs uncertainty and representativeness metrics. For instance, with only 10% of the dataset annotated, our method reaches within 5% of Dice score expected from the upper bound scenario of all the dataset given as annotated (an impractical scenario due to resource constraints), and this gap drops down to a mere 2% when less than a fifth of the dataset samples are annotated. Such active learning approach to selecting samples to annotate enables an optimal use of the expert clinician time, being often the bottleneck in realizing machine learning solutions in medicine.
△ Less
Submitted 22 December, 2019;
originally announced December 2019.
-
An Automatic Genetic Algorithm Framework for the Optimization of Three-dimensional Surgical Plans of Forearm Corrective Osteotomies
Authors:
Fabio Carrillo,
Simon Roner,
Marco von Atzigen,
Andreas Schweizer,
Ladislav Nagy,
Lazaros Vlachopoulos,
Jess G. Snedeker,
Philipp Fürnstahl
Abstract:
3D computer-assisted corrective osteotomy has become the state-of-the-art for surgical treatment of complex bone deformities. Despite available technologies, the automatic generation of clinically acceptable, ready-to-use preoperative planning solutions is currently not possible for such pathologies. Multiple contradicting and mutually dependent objectives have to be considered, as well as clinica…
▽ More
3D computer-assisted corrective osteotomy has become the state-of-the-art for surgical treatment of complex bone deformities. Despite available technologies, the automatic generation of clinically acceptable, ready-to-use preoperative planning solutions is currently not possible for such pathologies. Multiple contradicting and mutually dependent objectives have to be considered, as well as clinical and technical constraints, generally requiring iterative manual adjustments. This leads to unnecessary efforts and unbearable clinical costs, hindering also the quality of patient treatment. In this paper, we propose an optimization framework for the generation of ready-to-use preoperative planning solutions in a fully automatic fashion. An automatic diagnostic assessment using patient-specific 3D models is performed for 3D malunion quantification and definition of the optimization parameters. Afterward, clinical objectives are translated into the optimization module, and controlled through tailored fitness functions based on a weighted and multi-staged optimization approach. The optimization is based on a genetic algorithm capable of solving multi-objective optimization problems with non-linear constraints. The framework outputs a complete preoperative planning solution including position and orientation of the osteotomy plane, transformation to achieve the bone reduction, and position and orientation of the fixation plate and screws. A qualitative validation was performed on 36 consecutive cases of radius osteotomy where solutions generated by the optimization algorithm (OA) were compared against the gold standard (GS) solutions generated by experienced surgeons. Solutions were blinded and presented to 6 readers, who voted OA solutions to be better in 55% of the time. The quantitative evaluation was based on different error measurements, showing average improvements with respect to the GS.
△ Less
Submitted 16 September, 2019;
originally announced September 2019.
-
Active Learning for Segmentation by Optimizing Content Information for Maximal Entropy
Authors:
Firat Ozdemir,
Zixuan Peng,
Christine Tanner,
Philipp Fuernstahl,
Orcun Goksel
Abstract:
Segmentation is essential for medical image analysis tasks such as intervention planning, therapy guidance, diagnosis, treatment decisions. Deep learning is becoming increasingly prominent for segmentation, where the lack of annotations, however, often becomes the main limitation. Due to privacy concerns and ethical considerations, most medical datasets are created, curated, and allow access only…
▽ More
Segmentation is essential for medical image analysis tasks such as intervention planning, therapy guidance, diagnosis, treatment decisions. Deep learning is becoming increasingly prominent for segmentation, where the lack of annotations, however, often becomes the main limitation. Due to privacy concerns and ethical considerations, most medical datasets are created, curated, and allow access only locally. Furthermore, current deep learning methods are often suboptimal in translating anatomical knowledge between different medical imaging modalities. Active learning can be used to select an informed set of image samples to request for manual annotation, in order to best utilize the limited annotation time of clinical experts for optimal outcomes, which we focus on in this work. Our contributions herein are two fold: (1) we enforce domain-representativeness of selected samples using a proposed penalization scheme to maximize information at the network abstraction layer, and (2) we propose a Borda-count based sample querying scheme for selecting samples for segmentation. Comparative experiments with baseline approaches show that the samples queried with our proposed method, where both above contributions are combined, result in significantly improved segmentation performance for this active learning task.
△ Less
Submitted 18 July, 2018;
originally announced July 2018.
-
Learn the new, keep the old: Extending pretrained models with new anatomy and images
Authors:
Firat Ozdemir,
Philipp Fuernstahl,
Orcun Goksel
Abstract:
Deep learning has been widely accepted as a promising solution for medical image segmentation, given a sufficiently large representative dataset of images with corresponding annotations. With ever increasing amounts of annotated medical datasets, it is infeasible to train a learning method always with all data from scratch. This is also doomed to hit computational limits, e.g., memory or runtime f…
▽ More
Deep learning has been widely accepted as a promising solution for medical image segmentation, given a sufficiently large representative dataset of images with corresponding annotations. With ever increasing amounts of annotated medical datasets, it is infeasible to train a learning method always with all data from scratch. This is also doomed to hit computational limits, e.g., memory or runtime feasible for training. Incremental learning can be a potential solution, where new information (images or anatomy) is introduced iteratively. Nevertheless, for the preservation of the collective information, it is essential to keep some "important" (i.e. representative) images and annotations from the past, while adding new information. In this paper, we introduce a framework for applying incremental learning for segmentation and propose novel methods for selecting representative data therein. We comparatively evaluate our methods in different scenarios using MR images and validate the increased learning capacity with using our methods.
△ Less
Submitted 1 June, 2018;
originally announced June 2018.