Showing 1–46 of 46 results for author: Armand, M

Uncertainty-Aware Shape Estimation of a Surgical Continuum Manipulator in Constrained Environments using Fiber Bragg Grating Sensors

Authors: Alexander Schwarz, Arian Mehrfard, Golchehr Amirkhani, Henry Phalen, Justin H. Ma, Robert B. Grupp, Alejandro Martin-Gomez, Mehran Armand

Abstract: Continuum Dexterous Manipulators (CDMs) are well-suited tools for minimally invasive surgery due to their inherent dexterity and reachability. Nonetheless, their flexible structure and non-linear curvature pose significant challenges for shape-based feedback control. The use of Fiber Bragg Grating (FBG) sensors for shape sensing has shown great potential in estimating the CDM's tip position and su… ▽ More Continuum Dexterous Manipulators (CDMs) are well-suited tools for minimally invasive surgery due to their inherent dexterity and reachability. Nonetheless, their flexible structure and non-linear curvature pose significant challenges for shape-based feedback control. The use of Fiber Bragg Grating (FBG) sensors for shape sensing has shown great potential in estimating the CDM's tip position and subsequently reconstructing the shape using optimization algorithms. This optimization, however, is under-constrained and may be ill-posed for complex shapes, falling into local minima. In this work, we introduce a novel method capable of directly estimating a CDM's shape from FBG sensor wavelengths using a deep neural network. In addition, we propose the integration of uncertainty estimation to address the critical issue of uncertainty in neural network predictions. Neural network predictions are unreliable when the input sample is outside the training distribution or corrupted by noise. Recognizing such deviations is crucial when integrating neural networks within surgical robotics, as inaccurate estimations can pose serious risks to the patient. We present a robust method that not only improves the precision upon existing techniques for FBG-based shape estimation but also incorporates a mechanism to quantify the models' confidence through uncertainty estimation. We validate the uncertainty estimation through extensive experiments, demonstrating its effectiveness and reliability on out-of-distribution (OOD) data, adding an additional layer of safety and precision to minimally invasive surgical robotics. △ Less

Submitted 11 May, 2024; originally announced May 2024.

On the Fly Robotic-Assisted Medical Instrument Planning and Execution Using Mixed Reality

Authors: Letian Ai, Yihao Liu, Mehran Armand, Amir Kheradmand, Alejandro Martin-Gomez

Abstract: Robotic-assisted medical systems (RAMS) have gained significant attention for their advantages in alleviating surgeons' fatigue and improving patients' outcomes. These systems comprise a range of human-computer interactions, including medical scene monitoring, anatomical target planning, and robot manipulation. However, despite its versatility and effectiveness, RAMS demands expertise in robotics,… ▽ More Robotic-assisted medical systems (RAMS) have gained significant attention for their advantages in alleviating surgeons' fatigue and improving patients' outcomes. These systems comprise a range of human-computer interactions, including medical scene monitoring, anatomical target planning, and robot manipulation. However, despite its versatility and effectiveness, RAMS demands expertise in robotics, leading to a high learning cost for the operator. In this work, we introduce a novel framework using mixed reality technologies to ease the use of RAMS. The proposed framework achieves real-time planning and execution of medical instruments by providing 3D anatomical image overlay, human-robot collision detection, and robot programming interface. These features, integrated with an easy-to-use calibration method for head-mounted display, improve the effectiveness of human-robot interactions. To assess the feasibility of the framework, two medical applications are presented in this work: 1) coil placement during transcranial magnetic stimulation and 2) drill and injector device positioning during femoroplasty. Results from these use cases demonstrate its potential to extend to a wider range of medical scenarios. △ Less

Submitted 8 April, 2024; originally announced April 2024.

Comments: This paper has been accepted to IEEE ICRA 2024 as a contributed paper

GBEC: Geometry-Based Hand-Eye Calibration

Authors: Yihao Liu, Jiaming Zhang, Zhangcong She, Amir Kheradmand, Mehran Armand

Abstract: Hand-eye calibration is the problem of solving the transformation from the end-effector of a robot to the sensor attached to it. Commonly employed techniques, such as AXXB or AXZB formulations, rely on regression methods that require collecting pose data from different robot configurations, which can produce low accuracy and repeatability. However, the derived transformation should solely depend o… ▽ More Hand-eye calibration is the problem of solving the transformation from the end-effector of a robot to the sensor attached to it. Commonly employed techniques, such as AXXB or AXZB formulations, rely on regression methods that require collecting pose data from different robot configurations, which can produce low accuracy and repeatability. However, the derived transformation should solely depend on the geometry of the end-effector and the sensor attachment. We propose Geometry-Based End-Effector Calibration (GBEC) that enhances the repeatability and accuracy of the derived transformation compared to traditional hand-eye calibrations. To demonstrate improvements, we apply the approach to two different robot-assisted procedures: Transcranial Magnetic Stimulation (TMS) and femoroplasty. We also discuss the generalizability of GBEC for camera-in-hand and marker-in-hand sensor mounting methods. In the experiments, we perform GBEC between the robot end-effector and an optical tracker's rigid body marker attached to the TMS coil or femoroplasty drill guide. Previous research documents low repeatability and accuracy of the conventional methods for robot-assisted TMS hand-eye calibration. When compared to some existing methods, the proposed method relies solely on the geometry of the flange and the pose of the rigid-body marker, making it independent of workspace constraints or robot accuracy, without sacrificing the orthogonality of the rotation matrix. Our results validate the accuracy and applicability of the approach, providing a new and generalizable methodology for obtaining the transformation from the end-effector to a sensor. △ Less

Submitted 8 April, 2024; originally announced April 2024.

Comments: This paper has been accepted to IEEE ICRA 2024 as a contributed paper

Segment Any Medical Model Extended

Authors: Yihao Liu, Jiaming Zhang, Andres Diaz-Pinto, Haowei Li, Alejandro Martin-Gomez, Amir Kheradmand, Mehran Armand

Abstract: The Segment Anything Model (SAM) has drawn significant attention from researchers who work on medical image segmentation because of its generalizability. However, researchers have found that SAM may have limited performance on medical images compared to state-of-the-art non-foundation models. Regardless, the community sees potential in extending, fine-tuning, modifying, and evaluating SAM for anal… ▽ More The Segment Anything Model (SAM) has drawn significant attention from researchers who work on medical image segmentation because of its generalizability. However, researchers have found that SAM may have limited performance on medical images compared to state-of-the-art non-foundation models. Regardless, the community sees potential in extending, fine-tuning, modifying, and evaluating SAM for analysis of medical imaging. An increasing number of works have been published focusing on the mentioned four directions, where variants of SAM are proposed. To this end, a unified platform helps push the boundary of the foundation model for medical images, facilitating the use, modification, and validation of SAM and its variants in medical image segmentation. In this work, we introduce SAMM Extended (SAMME), a platform that integrates new SAM variant models, adopts faster communication protocols, accommodates new interactive modes, and allows for fine-tuning of subcomponents of the models. These features can expand the potential of foundation models like SAM, and the results can be translated to applications such as image-guided therapy, mixed reality interaction, robotic navigation, and data augmentation. △ Less

Submitted 26 March, 2024; originally announced March 2024.

Comments: The content of the manuscript has been presented in SPIE Medical Imaging 2024, and had been accepted to appear in the proceedings of the conference

Realtime Robust Shape Estimation of Deformable Linear Object

Authors: Jiaming Zhang, Zhaomeng Zhang, Yihao Liu, Yaqian Chen, Amir Kheradmand, Mehran Armand

Abstract: Realtime shape estimation of continuum objects and manipulators is essential for develo** accurate planning and control paradigms. The existing methods that create dense point clouds from camera images, and/or use distinguishable markers on a deformable body have limitations in realtime tracking of large continuum objects/manipulators. The physical occlusion of markers can often compromise accur… ▽ More Realtime shape estimation of continuum objects and manipulators is essential for develo** accurate planning and control paradigms. The existing methods that create dense point clouds from camera images, and/or use distinguishable markers on a deformable body have limitations in realtime tracking of large continuum objects/manipulators. The physical occlusion of markers can often compromise accurate shape estimation. We propose a robust method to estimate the shape of linear deformable objects in realtime using scattered and unordered key points. By utilizing a robust probability-based labeling algorithm, our approach identifies the true order of the detected key points and then reconstructs the shape using piecewise spline interpolation. The approach only relies on knowing the number of the key points and the interval between two neighboring points. We demonstrate the robustness of the method when key points are partially occluded. The proposed method is also integrated into a simulation in Unity for tracking the shape of a cable with a length of 1m and a radius of 5mm. The simulation results show that our proposed approach achieves an average length error of 1.07% over the continuum's centerline and an average cross-section error of 2.11mm. The real-world experiments of tracking and estimating a heavy-load cable prove that the proposed approach is robust under occlusion and complex entanglement scenarios. △ Less

Submitted 24 March, 2024; originally announced March 2024.

Comments: This paper has been accepted to IEEE ICRA 2024 as a contributed paper

A Roadmap Towards Automated and Regulated Robotic Systems

Authors: Yihao Liu, Mehran Armand

Abstract: The rapid development of generative technology opens up possibility for higher level of automation, and artificial intelligence (AI) embodiment in robotic systems is imminent. However, due to the blackbox nature of the generative technology, the generation of the knowledge and workflow scheme is uncontrolled, especially in a dynamic environment and a complex scene. This poses challenges to regulat… ▽ More The rapid development of generative technology opens up possibility for higher level of automation, and artificial intelligence (AI) embodiment in robotic systems is imminent. However, due to the blackbox nature of the generative technology, the generation of the knowledge and workflow scheme is uncontrolled, especially in a dynamic environment and a complex scene. This poses challenges to regulations in safety-demanding applications such as medical scenes. We argue that the unregulated generative processes from AI is fitted for low level end tasks, but intervention in the form of manual or automated regulation should happen post-workflow-generation and pre-robotic-execution. To address this, we propose a roadmap that can lead to fully automated and regulated robotic systems. In this paradigm, the high level policies are generated as structured graph data, enabling regulatory oversight and reusability, while the code base for lower level tasks is generated by generative models. Our approach aims the transitioning from expert knowledge to regulated action, akin to the iterative processes of study, practice, scrutiny, and execution in human tasks. We identify the generative and deterministic processes in a design cycle, where generative processes serve as a text-based world simulator and the deterministic processes generate the executable system. We propose State Machine Seralization Language (SMSL) to be the conversion point between text simulator and executable workflow control. From there, we analyze the modules involved based on the current literature, and discuss human in the loop. As a roadmap, this work identifies the current possible implementation and future work. This work does not provide an implemented system but envisions to inspire the researchers working on the direction in the roadmap. We implement the SMSL and D-SFO paradigm that serve as the starting point of the roadmap. △ Less

Submitted 20 March, 2024; originally announced March 2024.

Comments: 17 pages, 9 figures

FluoroSAM: A Language-aligned Foundation Model for X-ray Image Segmentation

Authors: Benjamin D. Killeen, Liam J. Wang, Han Zhang, Mehran Armand, Russell H. Taylor, Dave Dreizin, Greg Osgood, Mathias Unberath

Abstract: Automated X-ray image segmentation would accelerate research and development in diagnostic and interventional precision medicine. Prior efforts have contributed task-specific models capable of solving specific image analysis problems, but the utility of these models is restricted to their particular task domain, and expanding to broader use requires additional data, labels, and retraining efforts.… ▽ More Automated X-ray image segmentation would accelerate research and development in diagnostic and interventional precision medicine. Prior efforts have contributed task-specific models capable of solving specific image analysis problems, but the utility of these models is restricted to their particular task domain, and expanding to broader use requires additional data, labels, and retraining efforts. Recently, foundation models (FMs) -- machine learning models trained on large amounts of highly variable data thus enabling broad applicability -- have emerged as promising tools for automated image analysis. Existing FMs for medical image analysis focus on scenarios and modalities where objects are clearly defined by visually apparent boundaries, such as surgical tool segmentation in endoscopy. X-ray imaging, by contrast, does not generally offer such clearly delineated boundaries or structure priors. During X-ray image formation, complex 3D structures are projected in transmission onto the imaging plane, resulting in overlap** features of varying opacity and shape. To pave the way toward an FM for comprehensive and automated analysis of arbitrary medical X-ray images, we develop FluoroSAM, a language-aligned variant of the Segment-Anything Model, trained from scratch on 1.6M synthetic X-ray images. FluoroSAM is trained on data including masks for 128 organ types and 464 non-anatomical objects, such as tools and implants. In real X-ray images of cadaveric specimens, FluoroSAM is able to segment bony anatomical structures based on text-only prompting with 0.51 and 0.79 DICE with point-based refinement, outperforming competing SAM variants for all structures. FluoroSAM is also capable of zero-shot generalization to segmenting classes beyond the training set thanks to its language alignment, which we demonstrate for full lung segmentation on real chest X-rays. △ Less

Submitted 27 March, 2024; v1 submitted 12 March, 2024; originally announced March 2024.

Data-Driven Shape Sensing in Continuum Manipulators via Sliding Resistive Flex Sensors

Authors: Chenhan Zhang, Shaopeng Jiang, Heyun Wang, Joshua Liu, Amit Jain, Mehran Armand

Abstract: We introduce a novel shape-sensing method using Resistive Flex Sensors (RFS) embedded in cable-driven Continuum Dexterous Manipulators (CDMs). The RFS is predominantly sensitive to deformation rather than direct forces, making it a distinctive tool for shape sensing. The RFS unit we designed is a considerably less expensive and robust alternative, offering comparable accuracy and real-time perform… ▽ More We introduce a novel shape-sensing method using Resistive Flex Sensors (RFS) embedded in cable-driven Continuum Dexterous Manipulators (CDMs). The RFS is predominantly sensitive to deformation rather than direct forces, making it a distinctive tool for shape sensing. The RFS unit we designed is a considerably less expensive and robust alternative, offering comparable accuracy and real-time performance to existing shape sensing methods used for the CDMs proposed for minimally-invasive surgery. Our design allows the RFS to move along and inside the CDM conforming to its curvature, offering the ability to capture resistance metrics from various bending positions without the need for elaborate sensor setups. The RFS unit is calibrated using an overhead camera and a ResNet machine learning framework. Experiments using a 3D printed prototype of the CDM achieved an average shape estimation error of 0.968 mm with a standard error of 0.275 mm. The response time of the model was approximately 1.16 ms, making real-time shape sensing feasible. While this preliminary study successfully showed the feasibility of our approach for C-shape CDM deformations with non-constant curvatures, we are currently extending the results to show the feasibility for adapting to more complex CDM configurations such as S-shape created in obstructed environments or in presence of the external forces. △ Less

Submitted 29 November, 2023; originally announced November 2023.

Toward Process Controlled Medical Robotic System

Authors: Yihao Liu, Amir Kheradmand, Mehran Armand

Abstract: Medical errors, defined as unintended acts either of omission or commission that cause the failure of medical actions, are the third leading cause of death in the United States. The application of autonomy and robotics can alleviate some causes of medical errors by improving accuracy and providing means to follow planned procedures. However, for the robotic applications to improve safety, they mus… ▽ More Medical errors, defined as unintended acts either of omission or commission that cause the failure of medical actions, are the third leading cause of death in the United States. The application of autonomy and robotics can alleviate some causes of medical errors by improving accuracy and providing means to follow planned procedures. However, for the robotic applications to improve safety, they must maintain constant operating conditions in the presence of disturbances, and provide reliable measurements, evaluation, and control for each state of the procedure. This article addresses the need for process control in medical robotic systems, and proposes a standardized design cycle toward its automation. Monitoring and controlling the changing conditions in a medical or surgical environment necessitates a clear definition of workflows and their procedural dependencies. We propose integrating process control into medical robotic workflows to identify change in states of the system and environment, possible operations, and transitions to new states. Therefore, the system translates clinician experiences and procedure workflows into machine-interpretable languages. The design cycle using hFSM formulation can be a deterministic process, which opens up possibilities for higher-level automation in medical robotics. Shown in our work, with a standardized design cycle and software paradigm, we pave the way toward controlled workflows that can be automatically generated. Additionally, a modular design for a robotic system architecture that integrates hFSM can provide easy software and hardware integration. This article discusses the system design, software implementation, and example application to Robot-Assisted Transcranial Magnetic Stimulation and robot-assisted femoroplasty. We provide assessments using performance in terms of robotic tool placement and response to failure injections. △ Less

Submitted 19 February, 2024; v1 submitted 10 August, 2023; originally announced August 2023.

Comments: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Skin Lesion Correspondence Localization in Total Body Photography

Authors: Wei-Lun Huang, Davood Tashayyod, Jun Kang, Amir Gandjbakhche, Michael Kazhdan, Mehran Armand

Abstract: Longitudinal tracking of skin lesions - finding correspondence, changes in morphology, and texture - is beneficial to the early detection of melanoma. However, it has not been well investigated in the context of full-body imaging. We propose a novel framework combining geometric and texture information to localize skin lesion correspondence from a source scan to a target scan in total body photogr… ▽ More Longitudinal tracking of skin lesions - finding correspondence, changes in morphology, and texture - is beneficial to the early detection of melanoma. However, it has not been well investigated in the context of full-body imaging. We propose a novel framework combining geometric and texture information to localize skin lesion correspondence from a source scan to a target scan in total body photography (TBP). Body landmarks or sparse correspondence are first created on the source and target 3D textured meshes. Every vertex on each of the meshes is then mapped to a feature vector characterizing the geodesic distances to the landmarks on that mesh. Then, for each lesion of interest (LOI) on the source, its corresponding location on the target is first coarsely estimated using the geometric information encoded in the feature vectors and then refined using the texture information. We evaluated the framework quantitatively on both a public and a private dataset, for which our success rates (at 10 mm criterion) are comparable to the only reported longitudinal study. As full-body 3D capture becomes more prevalent and has higher quality, we expect the proposed method to constitute a valuable step in the longitudinal tracking of skin lesions. △ Less

Submitted 22 August, 2023; v1 submitted 18 July, 2023; originally announced July 2023.

Comments: MICCAI-2023

Pelphix: Surgical Phase Recognition from X-ray Images in Percutaneous Pelvic Fixation

Authors: Benjamin D. Killeen, Han Zhang, Jan Mangulabnan, Mehran Armand, Russel H. Taylor, Greg Osgood, Mathias Unberath

Abstract: Surgical phase recognition (SPR) is a crucial element in the digital transformation of the modern operating theater. While SPR based on video sources is well-established, incorporation of interventional X-ray sequences has not yet been explored. This paper presents Pelphix, a first approach to SPR for X-ray-guided percutaneous pelvic fracture fixation, which models the procedure at four levels of… ▽ More Surgical phase recognition (SPR) is a crucial element in the digital transformation of the modern operating theater. While SPR based on video sources is well-established, incorporation of interventional X-ray sequences has not yet been explored. This paper presents Pelphix, a first approach to SPR for X-ray-guided percutaneous pelvic fracture fixation, which models the procedure at four levels of granularity -- corridor, activity, view, and frame value -- simulating the pelvic fracture fixation workflow as a Markov process to provide fully annotated training data. Using added supervision from detection of bony corridors, tools, and anatomy, we learn image representations that are fed into a transformer model to regress surgical phases at the four granularity levels. Our approach demonstrates the feasibility of X-ray-based SPR, achieving an average accuracy of 93.8% on simulated sequences and 67.57% in cadaver across all granularity levels, with up to 88% accuracy for the target corridor in real data. This work constitutes the first step toward SPR for the X-ray domain, establishing an approach to categorizing phases in X-ray-guided surgery, simulating realistic image sequences to enable machine learning model development, and demonstrating that this approach is feasible for the analysis of real procedures. As X-ray-based SPR continues to mature, it will benefit procedures in orthopedic surgery, angiography, and interventional radiology by equip** intelligent surgical systems with situational awareness in the operating room. △ Less

Submitted 18 April, 2023; originally announced April 2023.

SAMM (Segment Any Medical Model): A 3D Slicer Integration to SAM

Authors: Yihao Liu, Jiaming Zhang, Zhangcong She, Amir Kheradmand, Mehran Armand

Abstract: The Segment Anything Model (SAM) is a new image segmentation tool trained with the largest available segmentation dataset. The model has demonstrated that, with prompts, it can create high-quality masks for general images. However, the performance of the model on medical images requires further validation. To assist with the development, assessment, and application of SAM on medical images, we int… ▽ More The Segment Anything Model (SAM) is a new image segmentation tool trained with the largest available segmentation dataset. The model has demonstrated that, with prompts, it can create high-quality masks for general images. However, the performance of the model on medical images requires further validation. To assist with the development, assessment, and application of SAM on medical images, we introduce Segment Any Medical Model (SAMM), an extension of SAM on 3D Slicer - an image processing and visualization software extensively used by the medical imaging community. This open-source extension to 3D Slicer and its demonstrations are posted on GitHub (https://github.com/bingogome/samm). SAMM achieves 0.6-second latency of a complete cycle and can infer image masks in nearly real-time. △ Less

Submitted 3 February, 2024; v1 submitted 12 April, 2023; originally announced April 2023.

Comments: 5 pages, 4 figures. We added editorial changes in the text

Design and Fabrication of a Fiber Bragg Grating Shape Sensor for Shape Reconstruction of a Continuum Manipulator

Authors: Golchehr Amirkhani, Anna Goodridge, Mojtaba Esfandiari, Henry Phalen, Justin H. Ma, Iulian Iordachita, Mehran Armand

Abstract: Continuum dexterous manipulators (CDMs) are suitable for performing tasks in a constrained environment due to their high dexterity and maneuverability. Despite the inherent advantages of CDMs in minimally invasive surgery, real-time control of CDMs' shape during non-constant curvature bending is still challenging. This study presents a novel approach for the design and fabrication of a large defle… ▽ More Continuum dexterous manipulators (CDMs) are suitable for performing tasks in a constrained environment due to their high dexterity and maneuverability. Despite the inherent advantages of CDMs in minimally invasive surgery, real-time control of CDMs' shape during non-constant curvature bending is still challenging. This study presents a novel approach for the design and fabrication of a large deflection fiber Bragg grating (FBG) shape sensor embedded within the lumens inside the walls of a CDM with a large instrument channel. The shape sensor consisted of two fibers, each with three FBG nodes. A shape-sensing model was introduced to reconstruct the centerline of the CDM based on FBG wavelengths. Different experiments, including shape sensor tests and CDM shape reconstruction tests, were conducted to assess the overall accuracy of the shape sensing. The FBG sensor evaluation results revealed the linear curvature-wavelength relationship with the large curvature detection of 0.045 mm at a 90 degrees bending angle and a sensitivity of up to 5.50 nm/mm in each bending direction. The CDM's shape reconstruction experiments in a free environment demonstrated the shape tracking accuracy of 0.216+-0.126 mm for positive/negative deflections. Also, the CDM shape reconstruction error for three cases of bending with obstacles were observed to be 0.436+-0.370 mm for the proximal case, 0.485+-0.418 mm for the middle case, and 0.312+-0.261 mm for the distal case. This study indicates the adequate performance of the FBG sensor and the effectiveness of the model for tracking the shape of the large-deflection CDM with nonconstant-curvature bending for minimally-invasive orthopaedic applications. △ Less

Submitted 6 March, 2023; originally announced March 2023.

STTAR: Surgical Tool Tracking using off-the-shelf Augmented Reality Head-Mounted Displays

Authors: Alejandro Martin-Gomez, Haowei Li, Tianyu Song, Sheng Yang, Guangzhi Wang, Hui Ding, Nassir Navab, Zhe Zhao, Mehran Armand

Abstract: The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the task performance. Existing medical-grade tracking systems use infrared cameras placed inside the Operat… ▽ More The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the task performance. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers, such as those used in surgical procedures, without the need to integrate any additional components. This framework is also capable of simultaneously tracking multiple tools. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09 +- 0.06 mm on lateral translation, 0.42 +- 0.32 mm on longitudinal translation, and 0.80 +- 0.39 deg for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, two surgeons and one biomedical researcher were provided with visual navigation, each performing 21 injections. Results from this use case provide comparable accuracy to those reported in the literature for AR-based navigation procedures. △ Less

Submitted 16 August, 2022; originally announced August 2022.

Comments: 14 pages, 12 figures, 1 table

SyntheX: Scaling Up Learning-based X-ray Image Analysis Through In Silico Experiments

Authors: Cong Gao, Benjamin D. Killeen, Yicheng Hu, Robert B. Grupp, Russell H. Taylor, Mehran Armand, Mathias Unberath

Abstract: Artificial intelligence (AI) now enables automated interpretation of medical images for clinical use. However, AI's potential use for interventional images (versus those involved in triage or diagnosis), such as for guidance during surgery, remains largely untapped. This is because surgical AI systems are currently trained using post hoc analysis of data collected during live surgeries, which has… ▽ More Artificial intelligence (AI) now enables automated interpretation of medical images for clinical use. However, AI's potential use for interventional images (versus those involved in triage or diagnosis), such as for guidance during surgery, remains largely untapped. This is because surgical AI systems are currently trained using post hoc analysis of data collected during live surgeries, which has fundamental and practical limitations, including ethical considerations, expense, scalability, data integrity, and a lack of ground truth. Here, we demonstrate that creating realistic simulated images from human models is a viable alternative and complement to large-scale in situ data collection. We show that training AI image analysis models on realistically synthesized data, combined with contemporary domain generalization or adaptation techniques, results in models that on real data perform comparably to models trained on a precisely matched real data training set. Because synthetic generation of training data from human-based models scales easily, we find that our model transfer paradigm for X-ray image analysis, which we refer to as SyntheX, can even outperform real data-trained models due to the effectiveness of training on a larger dataset. We demonstrate the potential of SyntheX on three clinical tasks: Hip image analysis, surgical robotic tool detection, and COVID-19 lung lesion segmentation. SyntheX provides an opportunity to drastically accelerate the conception, design, and evaluation of intelligent systems for X-ray-based medicine. In addition, simulated image environments provide the opportunity to test novel instrumentation, design complementary surgical approaches, and envision novel techniques that improve outcomes, save time, or mitigate human error, freed from the ethical and practical considerations of live human data collection. △ Less

Submitted 13 June, 2022; originally announced June 2022.

The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective

Authors: Mathias Unberath, Cong Gao, Yicheng Hu, Max Judish, Russell H Taylor, Mehran Armand, Robert Grupp

Abstract: Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflo… ▽ More Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Further, it is expected that image-based navigation will play a major role in enabling mixed reality environments and autonomous, robotic workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., volumetric imagery or tool models, and 2D images thereof, such as fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness of the optimization objective, hyperparameter selection, and initialization, difficulties around inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional map**, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps. △ Less

Submitted 4 August, 2021; originally announced August 2021.

A Robotic System for Implant Modification in Single-stage Cranioplasty

Authors: Shuya Liu, Wei-Lun Huang, Chad Gordon, Mehran Armand

Abstract: Craniomaxillofacial reconstruction with patient-specific customized craniofacial implants (CCIs) is most commonly performed for large-sized skeletal defects. Because the exact size of skull resection may not be known prior to the surgery, in the single-stage cranioplasty, a large CCI is prefabricated and resized intraoperatively with a manual-cutting process provided by a surgeon. The manual resiz… ▽ More Craniomaxillofacial reconstruction with patient-specific customized craniofacial implants (CCIs) is most commonly performed for large-sized skeletal defects. Because the exact size of skull resection may not be known prior to the surgery, in the single-stage cranioplasty, a large CCI is prefabricated and resized intraoperatively with a manual-cutting process provided by a surgeon. The manual resizing, however, may be inaccurate and significantly add to the operating time. This paper introduces a fast and non-contact approach for intraoperatively determining the exact contour of the skull resection and automatically resizing the implant to fit the resection area. Our approach includes four steps: First, a patient's defect information is acquired by a 3D scanner. Second, the scanned defect is aligned to the CCI by registering the scanned defect to the reconstructed CT model. Third, a cutting toolpath is generated from the contour of the scanned defect. Lastly, the large CCI is resized by a cutting robot to fit the resection area according to the given toolpath. To evaluate the resizing performance of our method, six different resection shapes were used in the cutting experiments. We compared the performance of our method to the performances of surgeon's manual resizing and an existing technique which collects the defect contour with an optical tracking system and projects the contour on the CCI to guide the manual modification. The results show that our proposed method improves the resizing accuracy by 56% compared to the surgeon's manual modification and 42% compared to the projection method. △ Less

Submitted 17 March, 2021; v1 submitted 12 January, 2021; originally announced January 2021.

Comments: Submitted to ICRA 2021

A Versatile Data-Driven Framework for Model-Independent Control of Continuum Manipulators Interacting With Obstructed Environments With Unknown Geometry and Stiffness

Authors: Farshid Alambeigi, Zerui Wang, Yun-Hui Liu, Russell H. Taylor, Mehran Armand

Abstract: This paper addresses the problem of controlling a continuum manipulator (CM) in free or obstructed environments with no prior knowledge about the deformation behavior of the CM and the stiffness and geometry of the interacting obstructed environment. We propose a versatile data-driven priori-model-independent (PMI) control framework, in which various control paradigms (e.g. CM's position or shap… ▽ More This paper addresses the problem of controlling a continuum manipulator (CM) in free or obstructed environments with no prior knowledge about the deformation behavior of the CM and the stiffness and geometry of the interacting obstructed environment. We propose a versatile data-driven priori-model-independent (PMI) control framework, in which various control paradigms (e.g. CM's position or shape control) can be defined based on the provided feedback. This optimal iterative algorithm learns the deformation behavior of the CM in interaction with an unknown environment, in real time, and then accomplishes the defined control objective. To evaluate the scalability of the proposed framework, we integrated two different CMs, designed for medical applications, with the da Vinci Research Kit (dVRK). The performance and learning capability of the framework was investigated in 11 sets of experiments including PMI position and shape control in free and unknown obstructed environments as well as during manipulation of an unknown deformable object. We also evaluated the performance of our algorithm in an ex-vivo experiment with a lamb heart.The theoretical and experimental results demonstrate the adaptivity, versatility, and accuracy of the proposed framework and, therefore, its suitability for a variety of applications involving continuum manipulators. △ Less

Submitted 5 May, 2020; originally announced May 2020.

Comments: 28 pages, 15 Figures

Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration

Authors: Cong Gao, Xingtong Liu, Wenhao Gu, Benjamin Killeen, Mehran Armand, Russell Taylor, Mathias Unberath

Abstract: Differentiable rendering is a technique to connect 3D scenes with corresponding 2D images. Since it is differentiable, processes during image formation can be learned. Previous approaches to differentiable rendering focus on mesh-based representations of 3D scenes, which is inappropriate for medical applications where volumetric, voxelized models are used to represent anatomy. We propose a novel P… ▽ More Differentiable rendering is a technique to connect 3D scenes with corresponding 2D images. Since it is differentiable, processes during image formation can be learned. Previous approaches to differentiable rendering focus on mesh-based representations of 3D scenes, which is inappropriate for medical applications where volumetric, voxelized models are used to represent anatomy. We propose a novel Projective Spatial Transformer module that generalizes spatial transformers to projective geometry, thus enabling differentiable volume rendering. We demonstrate the usefulness of this architecture on the example of 2D/3D registration between radiographs and CT scans. Specifically, we show that our transformer enables end-to-end learning of an image processing and projection model that approximates an image similarity function that is convex with respect to the pose parameters, and can thus be optimized effectively using conventional gradient descent. To the best of our knowledge, this is the first time that spatial transformers have been described for projective geometry. The source code will be made public upon publication of this manuscript and we hope that our developments will benefit related 3D research applications. △ Less

Submitted 24 March, 2020; originally announced March 2020.

From Perspective X-ray Imaging to Parallax-Robust Orthographic Stitching

Authors: Javad Fotouhi, Xingtong Liu, Mehran Armand, Nassir Navab, Mathias Unberath

Abstract: Stitching images acquired under perspective projective geometry is a relevant topic in computer vision with multiple applications ranging from smartphone panoramas to the construction of digital maps. Image stitching is an equally prominent challenge in medical imaging, where the limited field-of-view captured by single images prohibits holistic analysis of patient anatomy. The barrier that preven… ▽ More Stitching images acquired under perspective projective geometry is a relevant topic in computer vision with multiple applications ranging from smartphone panoramas to the construction of digital maps. Image stitching is an equally prominent challenge in medical imaging, where the limited field-of-view captured by single images prohibits holistic analysis of patient anatomy. The barrier that prevents straight-forward mosaicing of 2D images is depth mismatch due to parallax. In this work, we leverage the Fourier slice theorem to aggregate information from multiple transmission images in parallax-free domains using fundamental principles of X-ray image formation. The semantics of the stitched image are restored using a novel deep learning strategy that exploits similarity measures designed around frequency, as well as dense and sparse spatial image content. Our pipeline, not only stitches images, but also provides orthographic reconstruction that enables metric measurements of clinically relevant quantities directly on the 2D image plane. △ Less

Submitted 5 March, 2020; originally announced March 2020.

Exploring Partial Intrinsic and Extrinsic Symmetry in 3D Medical Imaging

Authors: Javad Fotouhi, Giacomo Taylor, Mathias Unberath, Alex Johnson, Sing Chun Lee, Greg Osgood, Mehran Armand, Nassir Navab

Abstract: We present a novel methodology to detect imperfect bilateral symmetry in CT of human anatomy. In this paper, the structurally symmetric nature of the pelvic bone is explored and is used to provide interventional image augmentation for treatment of unilateral fractures in patients with traumatic injuries. The mathematical basis of our solution is on the incorporation of attributes and characteristi… ▽ More We present a novel methodology to detect imperfect bilateral symmetry in CT of human anatomy. In this paper, the structurally symmetric nature of the pelvic bone is explored and is used to provide interventional image augmentation for treatment of unilateral fractures in patients with traumatic injuries. The mathematical basis of our solution is on the incorporation of attributes and characteristics that satisfy the properties of intrinsic and extrinsic symmetry and are robust to outliers. In the first step, feature points that satisfy intrinsic symmetry are automatically detected in the Möbius space defined on the CT data. These features are then pruned via a two-stage RANSAC to attain correspondences that satisfy also the extrinsic symmetry. Then, a disparity function based on Tukey's biweight robust estimator is introduced and minimized to identify a symmetry plane parametrization that yields maximum contralateral similarity. Finally, a novel regularization term is introduced to enhance similarity between bone density histograms across the partial symmetry plane, relying on the important biological observation that, even if injured, the dislocated bone segments remain within the body. Our extensive evaluations on various cases of common fracture types demonstrate the validity of the novel concepts and the robustness and accuracy of the proposed method. △ Less

Submitted 26 September, 2020; v1 submitted 4 March, 2020; originally announced March 2020.

Spatiotemporal-Aware Augmented Reality: Redefining HCI in Image-Guided Therapy

Authors: Javad Fotouhi, Arian Mehrfard, Tianyu Song, Alex Johnson, Greg Osgood, Mathias Unberath, Mehran Armand, Nassir Navab

Abstract: Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. As a co… ▽ More Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. As a consequence, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. The main contribution of this paper is to reveal how exemplary workflows are redefined by taking full advantage of head-mounted displays when entirely co-registered with the imaging system at all times. The proposed AR landscape is enabled by co-localizing the users and the imaging devices via the operating room environment and exploiting all involved frustums to move spatial information between different bodies. The awareness of the system from the geometric and physical characteristics of X-ray imaging allows the redefinition of different human-machine interfaces. We demonstrate that this AR paradigm is generic, and can benefit a wide variety of procedures. Our system achieved an error of $4.76\pm2.91$ mm for placing K-wire in a fracture management procedure, and yielded errors of $1.57\pm1.16^\circ$ and $1.46\pm1.00^\circ$ in the abduction and anteversion angles, respectively, for total hip arthroplasty. We hope that our holistic approach towards improving the interface of surgery not only augments the surgeon's capabilities but also augments the surgical team's experience in carrying out an effective intervention with reduced complications and provide novel approaches of documenting procedures for training purposes. △ Less

Submitted 4 March, 2020; originally announced March 2020.

Automatic Annotation of Hip Anatomy in Fluoroscopy for Robust and Efficient 2D/3D Registration

Authors: Robert Grupp, Mathias Unberath, Cong Gao, Rachel Hegeman, Ryan Murphy, Clayton Alexander, Yoshito Otake, Benjamin McArthur, Mehran Armand, Russell Taylor

Abstract: Fluoroscopy is the standard imaging modality used to guide hip surgery and is therefore a natural sensor for computer-assisted navigation. In order to efficiently solve the complex registration problems presented during navigation, human-assisted annotations of the intraoperative image are typically required. This manual initialization interferes with the surgical workflow and diminishes any advan… ▽ More Fluoroscopy is the standard imaging modality used to guide hip surgery and is therefore a natural sensor for computer-assisted navigation. In order to efficiently solve the complex registration problems presented during navigation, human-assisted annotations of the intraoperative image are typically required. This manual initialization interferes with the surgical workflow and diminishes any advantages gained from navigation. We propose a method for fully automatic registration using annotations produced by a neural network. Neural networks are trained to simultaneously segment anatomy and identify landmarks in fluoroscopy. Training data is obtained using an intraoperatively incompatible 2D/3D registration of hip anatomy. Ground truth 2D labels are established using projected 3D annotations. Intraoperative registration couples an intensity-based strategy with annotations inferred by the network and requires no human assistance. Ground truth labels were obtained in 366 fluoroscopic images across 6 cadaveric specimens. In a leave-one-subject-out experiment, networks obtained mean dice coefficients for left and right hemipelves, left and right femurs of 0.86, 0.87, 0.90, and 0.84. The mean 2D landmark error was 5.0 mm. The pelvis was registered within 1 degree for 86% of the images when using the proposed intraoperative approach with an average runtime of 7 seconds. In comparison, an intensity-only approach without manual initialization, registered the pelvis to 1 degree in 18% of images. We have created the first accurately annotated, non-synthetic, dataset of hip fluoroscopy. By using these annotations as training data for neural networks, state of the art performance in fluoroscopic segmentation and landmark localization was achieved. Integrating these annotations allows for a robust, fully automatic, and efficient intraoperative registration during fluoroscopic navigation of the hip. △ Less

Submitted 18 March, 2020; v1 submitted 16 November, 2019; originally announced November 2019.

Comments: Revised article to address reviewer comments. Accepted to IPCAI 2020. Supplementary video at https://youtu.be/5AwGlNkcp9o and dataset/code at https://github.com/rg2/DeepFluoroLabeling-IPCAI2020

Journal ref: International Journal of Computer Assisted Radiology and Surgery 15 (2020) 759-769

Fast and Automatic Periacetabular Osteotomy Fragment Pose Estimation Using Intraoperatively Implanted Fiducials and Single-View Fluoroscopy

Authors: Robert Grupp, Ryan Murphy, Rachel Hegeman, Clayton Alexander, Mathias Unberath, Yoshito Otake, Benjamin McArthur, Mehran Armand, Russell Taylor

Abstract: Accurate and consistent mental interpretation of fluoroscopy to determine the position and orientation of acetabular bone fragments in 3D space is difficult. We propose a computer assisted approach that uses a single fluoroscopic view and quickly reports the pose of an acetabular fragment without any user input or initialization. Intraoperatively, but prior to any osteotomies, two constellations o… ▽ More Accurate and consistent mental interpretation of fluoroscopy to determine the position and orientation of acetabular bone fragments in 3D space is difficult. We propose a computer assisted approach that uses a single fluoroscopic view and quickly reports the pose of an acetabular fragment without any user input or initialization. Intraoperatively, but prior to any osteotomies, two constellations of metallic ball-bearings (BBs) are injected into the wing of a patient's ilium and lateral superior pubic ramus. One constellation is located on the expected acetabular fragment, and the other is located on the remaining, larger, pelvis fragment. The 3D locations of each BB are reconstructed using three fluoroscopic views and 2D/3D registrations to a preoperative CT scan of the pelvis. The relative pose of the fragment is established by estimating the movement of the two BB constellations using a single fluoroscopic view taken after osteotomy and fragment relocation. BB detection and inter-view correspondences are automatically computed throughout the processing pipeline. The proposed method was evaluated on a multitude of fluoroscopic images collected from six cadaveric surgeries performed bilaterally on three specimens. Mean fragment rotation error was 2.4 +/- 1.0 degrees, mean translation error was 2.1 +/- 0.6 mm, and mean 3D lateral center edge angle error was 1.0 +/- 0.5 degrees. The average runtime of the single-view pose estimation was 0.7 +/- 0.2 seconds. The proposed method demonstrates accuracy similar to other state of the art systems which require optical tracking systems or multiple-view 2D/3D registrations with manual input. The errors reported on fragment poses and lateral center edge angles are within the margins required for accurate intraoperative evaluation of femoral head coverage. △ Less

Submitted 12 June, 2020; v1 submitted 22 October, 2019; originally announced October 2019.

Comments: Revised article to address reviewer comments. Under review for Physics in Medicine and Biology. Supplementary video at https://youtu.be/0E0U9G81q8g

Journal ref: 2020 Phys. Med. Biol. 65 245019

Pelvis Surface Estimation From Partial CT for Computer-Aided Pelvic Osteotomies

Authors: Robert Grupp, Yoshito Otake, Ryan Murphy, Javad Parvizi, Mehran Armand, Russell Taylor

Abstract: Computer-aided surgical systems commonly use preoperative CT scans when performing pelvic osteotomies for intraoperative navigation. These systems have the potential to improve the safety and accuracy of pelvic osteotomies, however, exposing the patient to radiation is a significant drawback. In order to reduce radiation exposure, we propose a new smooth extrapolation method leveraging a partial p… ▽ More Computer-aided surgical systems commonly use preoperative CT scans when performing pelvic osteotomies for intraoperative navigation. These systems have the potential to improve the safety and accuracy of pelvic osteotomies, however, exposing the patient to radiation is a significant drawback. In order to reduce radiation exposure, we propose a new smooth extrapolation method leveraging a partial pelvis CT and a statistical shape model (SSM) of the full pelvis in order to estimate a patient's complete pelvis. A SSM of normal, complete, female pelvis anatomy was created and evaluated from 42 subjects. A leave-one-out test was performed to characterise the inherent generalisation capability of the SSM. An additional leave-one-out test was conducted to measure performance of the smooth extrapolation method and an existing "cut-and-paste" extrapolation method. Unknown anatomy was simulated by kee** the axial slices of the patient's acetabulum intact and varying the amount of the superior iliac crest retained; from 0% to 15% of the total pelvis extent. The smooth technique showed an average improvement over the cut-and-paste method of 1.31 mm and 3.61 mm, in RMS and maximum surface error, respectively. With 5% of the iliac crest retained, the smoothly estimated surface had an RMS surface error of 2.21 mm, an improvement of 1.25 mm when retaining none of the iliac crest. This anatomical estimation method creates the possibility of a patient and surgeon benefiting from the use of a CAS system and simultaneously reducing the patient's radiation exposure. △ Less

Submitted 23 September, 2019; originally announced September 2019.

Comments: CAOS 2015 Extended Paper

Journal ref: In Orthopaedic Proceedings 2016 Feb (Vol. 98, No. SUPP_5, pp. 55-55). The British Editorial Society of Bone & Joint Surgery

Patch-Based Image Similarity for Intraoperative 2D/3D Pelvis Registration During Periacetabular Osteotomy

Authors: Robert Grupp, Mehran Armand, Russell Taylor

Abstract: Periacetabular osteotomy is a challenging surgical procedure for treating developmental hip dysplasia, providing greater coverage of the femoral head via relocation of a patient's acetabulum. Since fluoroscopic imaging is frequently used in the surgical workflow, computer-assisted X-Ray navigation of osteotomes and the relocated acetabular fragment should be feasible. We use intensity-based 2D/3D… ▽ More Periacetabular osteotomy is a challenging surgical procedure for treating developmental hip dysplasia, providing greater coverage of the femoral head via relocation of a patient's acetabulum. Since fluoroscopic imaging is frequently used in the surgical workflow, computer-assisted X-Ray navigation of osteotomes and the relocated acetabular fragment should be feasible. We use intensity-based 2D/3D registration to estimate the pelvis pose with respect to fluoroscopic images, recover relative poses of multiple views, and triangulate landmarks which may be used for navigation. Existing similarity metrics are unable to consistently account for the inherent mismatch between the preoperative intact pelvis, and the intraoperative reality of a fractured pelvis. To mitigate the effect of this mismatch, we continuously estimate the relevance of each pixel to solving the registration and use these values as weightings in a patch-based similarity metric. Limiting computation to randomly selected subsets of patches results in faster runtimes than existing patch-based methods. A simulation study was conducted with random fragment shapes, relocations, and fluoroscopic views, and the proposed method achieved a 1.7 mm mean triangulation error over all landmarks, compared to mean errors of 3 mm and 2.8 mm for the non-patched and image-intensity-variance-weighted patch similarity metrics, respectively. △ Less

Submitted 23 September, 2019; originally announced September 2019.

Comments: Presented at MICCAI CLIP Workshop 2018

Journal ref: In OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis 2018 Sep 16 (pp. 153-163). Springer, Cham

Smooth Extrapolation of Unknown Anatomy via Statistical Shape Models

Authors: Robert Grupp, Hsin-Hong Chiang, Yoshito Otake, Ryan Murphy, Chad Gordon, Mehran Armand, Russell Taylor

Abstract: Several methods to perform extrapolation of unknown anatomy were evaluated. The primary application is to enhance surgical procedures that may use partial medical images or medical images of incomplete anatomy. Le Fort-based, face-jaw-teeth transplant is one such procedure. From CT data of 36 skulls and 21 mandibles separate Statistical Shape Models of the anatomical surfaces were created. Using t… ▽ More Several methods to perform extrapolation of unknown anatomy were evaluated. The primary application is to enhance surgical procedures that may use partial medical images or medical images of incomplete anatomy. Le Fort-based, face-jaw-teeth transplant is one such procedure. From CT data of 36 skulls and 21 mandibles separate Statistical Shape Models of the anatomical surfaces were created. Using the Statistical Shape Models, incomplete surfaces were projected to obtain complete surface estimates. The surface estimates exhibit non-zero error in regions where the true surface is known; it is desirable to keep the true surface and seamlessly merge the estimated unknown surface. Existing extrapolation techniques produce non-smooth transitions from the true surface to the estimated surface, resulting in additional error and a less aesthetically pleasing result. The three extrapolation techniques evaluated were: copying and pasting of the surface estimate (non-smooth baseline), a feathering between the patient surface and surface estimate, and an estimate generated via a Thin Plate Spline trained from displacements between the surface estimate and corresponding vertices of the known patient surface. Feathering and Thin Plate Spline approaches both yielded smooth transitions. However, feathering corrupted known vertex values. Leave-one-out analyses were conducted, with 5% to 50% of known anatomy removed from the left-out patient and estimated via the proposed approaches. The Thin Plate Spline approach yielded smaller errors than the other two approaches, with an average vertex error improvement of 1.46 mm and 1.38 mm for the skull and mandible respectively, over the baseline approach. △ Less

Submitted 23 September, 2019; originally announced September 2019.

Comments: SPIE Medical Imaging Conference 2015 Paper

Journal ref: In Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling 2015 Mar 18 (Vol. 9415, p. 941524). International Society for Optics and Photonics

Learning to Detect Collisions for Continuum Manipulators without a Prior Model

Authors: Shahriar Sefati, Shahin Sefati, Iulian Iordachita, Russell H. Taylor, Mehran Armand

Abstract: Due to their flexibility, dexterity, and compact size, Continuum Manipulators (CMs) can enhance minimally invasive interventions. In these procedures, the CM may be operated in proximity of sensitive organs; therefore, requiring accurate and appropriate feedback when colliding with their surroundings. Conventional CM collision detection algorithms rely on a combination of exact CM constrained kine… ▽ More Due to their flexibility, dexterity, and compact size, Continuum Manipulators (CMs) can enhance minimally invasive interventions. In these procedures, the CM may be operated in proximity of sensitive organs; therefore, requiring accurate and appropriate feedback when colliding with their surroundings. Conventional CM collision detection algorithms rely on a combination of exact CM constrained kinematics model, geometrical assumptions such as constant curvature behavior, a priori knowledge of the environmental constraint geometry, and/or additional sensors to scan the environment or sense contacts. In this paper, we propose a data-driven machine learning approach using only the available sensory information, without requiring any prior geometrical assumptions, model of the CM or the surrounding environment. The proposed algorithm is implemented and evaluated on a non-constant curvature CM, equipped with Fiber Bragg Grating (FBG) optical sensors for shape sensing purposes. Results demonstrate successful detection of collisions in constrained environments with soft and hard obstacles with unknown stiffness and location. △ Less

Submitted 12 August, 2019; originally announced August 2019.

Comments: Accepted for publication in International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2019

Reflective-AR Display: An Interaction Methodology for Virtual-Real Alignment in Medical Robotics

Authors: Javad Fotouhi, Tianyu Song, Arian Mehrfard, Giacomo Taylor, Qiaochu Wang, Fengfang Xian, Alejandro Martin-Gomez, Bernhard Fuerst, Mehran Armand, Mathias Unberath, Nassir Navab

Abstract: Robot-assisted minimally invasive surgery has shown to improve patient outcomes, as well as reduce complications and recovery time for several clinical applications. While increasingly configurable robotic arms can maximize reach and avoid collisions in cluttered environments, positioning them appropriately during surgery is complicated because safety regulations prevent automatic driving. We prop… ▽ More Robot-assisted minimally invasive surgery has shown to improve patient outcomes, as well as reduce complications and recovery time for several clinical applications. While increasingly configurable robotic arms can maximize reach and avoid collisions in cluttered environments, positioning them appropriately during surgery is complicated because safety regulations prevent automatic driving. We propose a head-mounted display (HMD) based augmented reality (AR) system designed to guide optimal surgical arm set up. The staff equipped with HMD aligns the robot with its planned virtual counterpart. In this user-centric setting, the main challenge is the perspective ambiguities hindering such collaborative robotic solution. To overcome this challenge, we introduce a novel registration concept for intuitive alignment of AR content to its physical counterpart by providing a multi-view AR experience via reflective-AR displays that simultaneously show the augmentations from multiple viewpoints. Using this system, users can visualize different perspectives while actively adjusting the pose to determine the registration transformation that most closely superimposes the virtual onto the real. The experimental results demonstrate improvement in the interactive alignment of a virtual and real robot when using a reflective-AR display. We also present measurements from configuring a robotic manipulator in a simulated trocar placement surgery using the AR guidance methodology. △ Less

Submitted 4 March, 2020; v1 submitted 23 July, 2019; originally announced July 2019.

Pose Estimation of Periacetabular Osteotomy Fragments with Intraoperative X-Ray Navigation

Authors: Robert B. Grupp, Rachel A. Hegeman, Ryan J. Murphy, Clayton P. Alexander, Yoshito Otake, Benjamin A. McArthur, Mehran Armand, Russell H. Taylor

Abstract: Objective: State of the art navigation systems for pelvic osteotomies use optical systems with external fiducials. We propose the use of X-Ray navigation for pose estimation of periacetabular fragments without fiducials. Methods: A 2D/3D registration pipeline was developed to recover fragment pose. This pipeline was tested through an extensive simulation study and 6 cadaveric surgeries. Using oste… ▽ More Objective: State of the art navigation systems for pelvic osteotomies use optical systems with external fiducials. We propose the use of X-Ray navigation for pose estimation of periacetabular fragments without fiducials. Methods: A 2D/3D registration pipeline was developed to recover fragment pose. This pipeline was tested through an extensive simulation study and 6 cadaveric surgeries. Using osteotomy boundaries in the fluoroscopic images, the preoperative plan is refined to more accurately match the intraoperative shape. Results: In simulation, average fragment pose errors were 1.3°/1.7 mm when the planned fragment matched the intraoperative fragment, 2.2°/2.1 mm when the plan was not updated to match the true shape, and 1.9°/2.0 mm when the fragment shape was intraoperatively estimated. In cadaver experiments, the average pose errors were 2.2°/2.2 mm, 3.8°/2.5 mm, and 3.5°/2.2 mm when registering with the actual fragment shape, a preoperative plan, and an intraoperatively refined plan, respectively. Average errors of the lateral center edge angle were less than 2° for all fragment shapes in simulation and cadaver experiments. Conclusion: The proposed pipeline is capable of accurately reporting femoral head coverage within a range clinically identified for long-term joint survivability. Significance: Human interpretation of fragment pose is challenging and usually restricted to rotation about a single anatomical axis. The proposed pipeline provides an intraoperative estimate of rigid pose with respect to all anatomical axes, is compatible with minimally invasive incisions, and has no dependence on external fiducials. △ Less

Submitted 9 May, 2019; v1 submitted 21 March, 2019; originally announced March 2019.

Comments: Accepted for publication in IEEE Transactions on Biomedical Engineering

Journal ref: IEEE Transactions on Biomedical Engineering, vol. 67, no. 2, pp. 441-452, Feb. 2020

A Biomechanical Study on the Use of Curved Drilling Technique for Treatment of Osteonecrosis of Femoral Head

Authors: Mahsan Bakhtiarinejad, Farshid Alambeigi, Alireza Chamani, Mathias Unberath, Harpal Khanuja, Mehran Armand

Abstract: Osteonecrosis occurs due to the loss of blood supply to the bone, leading to spontaneous death of the trabecular bone. Delayed treatment of the involved patients results in collapse of the femoral head, which leads to a need for total hip arthroplasty surgery. Core decompression, as the most popular technique for treatment of the osteonecrosis, includes removal of the lesion area by drilling a str… ▽ More Osteonecrosis occurs due to the loss of blood supply to the bone, leading to spontaneous death of the trabecular bone. Delayed treatment of the involved patients results in collapse of the femoral head, which leads to a need for total hip arthroplasty surgery. Core decompression, as the most popular technique for treatment of the osteonecrosis, includes removal of the lesion area by drilling a straight tunnel to the lesion, debriding the dead bone and replacing it with bone substitutes. However, there are two drawbacks for this treatment method. First, due to the rigidity of the instruments currently used during core decompression, lesions cannot be completely removed and/or excessive healthy bone may also be removed with the lesion. Second, the use of bone substitutes, despite its biocompatibility and osteoconductivity, may not provide sufficient mechanical strength and support for the bone. To address these shortcomings, a novel robot-assisted curved core decompression (CCD) technique is introduced to provide surgeons with direct access to the lesions causing minimal damage to the healthy bone. In this study, with the aid of finite element (FE) simulations, we investigate biomechanical performance of core decompression using the curved drilling technique in the presence of normal gait loading. In this regard, we compare the result of the CCD using bone substitutes and flexible implants with other conventional core decompression techniques. The study finding shows that the maximum principal stress occurring at the superior domain of the neck is smaller in the CCD techniques (i.e. 52.847 MPa) compared to the other core decompression methods. △ Less

Submitted 14 January, 2019; originally announced January 2019.

Comments: Accepted for 2018 MICCAI Workshop on Computational Biomechanics for Medicine XIII

Localizing dexterous surgical tools in X-ray for image-based navigation

Authors: Cong Gao, Mathias Unberath, Russell Taylor, Mehran Armand

Abstract: X-ray image based surgical tool navigation is fast and supplies accurate images of deep seated structures. Typically, recovering the 6 DOF rigid pose and deformation of tools with respect to the X-ray camera can be accurately achieved through intensity-based 2D/3D registration of 3D images or models to 2D X-rays. However, the capture range of image-based 2D/3D registration is inconveniently small… ▽ More X-ray image based surgical tool navigation is fast and supplies accurate images of deep seated structures. Typically, recovering the 6 DOF rigid pose and deformation of tools with respect to the X-ray camera can be accurately achieved through intensity-based 2D/3D registration of 3D images or models to 2D X-rays. However, the capture range of image-based 2D/3D registration is inconveniently small suggesting that automatic and robust initialization strategies are of critical importance. This manuscript describes a first step towards leveraging semantic information of the imaged object to initialize 2D/3D registration within the capture range of image-based registration by performing concurrent segmentation and localization of dexterous surgical tools in X-ray images. We presented a learning-based strategy to simultaneously localize and segment dexterous surgical tools in X-ray images and demonstrate promising performance on synthetic and ex vivo data. We currently investigate methods to use semantic information extracted by the proposed network to reliably and robustly initialize image-based 2D/3D registration. While image-based 2D/3D registration has been an obvious focus of the CAI community, robust initialization thereof (albeit critical) has largely been neglected. This manuscript discusses learning-based retrieval of semantic information on imaged-objects as a step** stone for such initialization and may therefore be of interest to the IPCAI community. Since results are still preliminary and only focus on localization, we target the Long Abstract category. △ Less

Submitted 4 July, 2019; v1 submitted 20 January, 2019; originally announced January 2019.

Comments: 4 pages

FBG-Based Position Estimation of Highly Deformable Continuum Manipulators: Model-Dependent vs. Data-Driven Approaches

Authors: Shahriar Sefati, Rachel Hegeman, Farshid Alambeigi, Iulian Iordachita, Mehran Armand

Abstract: Conventional shape sensing techniques using Fiber Bragg Grating (FBG) involve finding the curvature at discrete FBG active areas and integrating curvature over the length of the continuum dexterous manipulator (CDM) for tip position estimation (TPE). However, due to limited number of sensing locations and many geometrical assumptions, these methods are prone to large error propagation especially w… ▽ More Conventional shape sensing techniques using Fiber Bragg Grating (FBG) involve finding the curvature at discrete FBG active areas and integrating curvature over the length of the continuum dexterous manipulator (CDM) for tip position estimation (TPE). However, due to limited number of sensing locations and many geometrical assumptions, these methods are prone to large error propagation especially when the CDM undergoes large deflections. In this paper, we study the complications of using the conventional TPE methods that are dependent on sensor model and propose a new data-driven method that overcomes these challenges. The proposed method consists of a regression model that takes FBG wavelength raw data as input and directly estimates the CDM's tip position. This model is pre-operatively (off-line) trained on position information from optical trackers/cameras (as the ground truth) and it intra-operatively (on-line) estimates CDM tip position using only the FBG wavelength data. The method's performance is evaluated on a CDM developed for orthopedic applications, and the results are compared to conventional model-dependent methods during large deflection bendings. Mean absolute TPE error (and standard deviation) of 1.52 (0.67) mm and 0.11 (0.1) mm with maximum absolute errors of 3.63 mm and 0.62 mm for the conventional and the proposed data-driven techniques were obtained, respectively. These results demonstrate a significant out-performance of the proposed data-driven approach versus the conventional estimation technique. △ Less

Submitted 20 December, 2018; originally announced December 2018.

Comments: Accepted for International Symposium on Medical Robotics (ISMR) 2019

FBG-Based Control of a Continuum Manipulator Interacting With Obstacles

Authors: Shahriar Sefati, Ryan Murphy, Farshid Alambeigi, Michael Pozin, Iulian Iordachita, Russell Taylor, Mehran Armand

Abstract: Tracking and controlling the shape of continuum dexterous manipulators (CDM) in constraint environments is a challenging task. The imposed constraints and interaction with unknown obstacles may conform the CDM's shape and therefore demands for shape sensing methods which do not rely on direct line of sight. To address these issues, we integrate a novel Fiber Bragg Grating (FBG) shape sensing unit… ▽ More Tracking and controlling the shape of continuum dexterous manipulators (CDM) in constraint environments is a challenging task. The imposed constraints and interaction with unknown obstacles may conform the CDM's shape and therefore demands for shape sensing methods which do not rely on direct line of sight. To address these issues, we integrate a novel Fiber Bragg Grating (FBG) shape sensing unit into a CDM, reconstruct the shape in real-time, and develop an optimization-based control algorithm using FBG tip position feedback. The CDM is designed for less-invasive treatment of osteolysis (bone degradation). To evaluate the performance of the feedback control algorithm when the CDM interacts with obstacles, we perform a set of experiments similar to the real scenario of the CDM interaction with soft and hard lesions during the treatment of osteolysis. In addition, we propose methods for identification of the CDM collisions with soft or hard obstacles using the jacobian information. Results demonstrate successful control of the CDM tip based on the FBG feedback and indicate repeatability and robustness of the proposed method when interacting with unknown obstacles. △ Less

Submitted 1 July, 2018; originally announced July 2018.

Comments: Accepted for IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2018

Inroads Toward Robot-Assisted Internal Fixation of Bone Fractures Using a Bendable Medical Screw and the Curved Drilling Technique

Authors: Farshid Alambeigi, Mahsan Bakhtiarinejad, Armina Azizi, Rachel Hegeman, Iulian Iordachita, Harpal Khanuja, Mehran Armand

Abstract: Internal fixation is a common orthopedic procedure in which a rigid screw is used to fix fragments of a fractured bone together and expedite the healing process. However, the rigidity of the screw, geometry of the fractured anatomy (e.g. femur and pelvis), and patient age can cause an array of complications during screw placement, such as improper fracture healing due to misalignment of the bone f… ▽ More Internal fixation is a common orthopedic procedure in which a rigid screw is used to fix fragments of a fractured bone together and expedite the healing process. However, the rigidity of the screw, geometry of the fractured anatomy (e.g. femur and pelvis), and patient age can cause an array of complications during screw placement, such as improper fracture healing due to misalignment of the bone fragments, lengthy procedure time and subsequently high radiation exposure. To address these issues, we propose a minimally invasive robot-assisted procedure comprising of a continuum robot, called ortho-snake, together with a novel bendable medical screw (BMS) for fixating the fractures. We describe the implementation of a curved drilling technique and focus on the design, manufacturing, and evaluation of a novel BMS, which can passively morph into the drilled curved tunnels with various curvatures. We evaluate the performance and efficacy of the proposed BMS using both finite element simulations as well as experiments conducted on synthetic bone samples. △ Less

Submitted 30 June, 2018; originally announced July 2018.

Comments: 6 pages, To be appeared in the 7th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics (BIOROB 2018)

Augmented Reality-based Feedback for Technician-in-the-loop C-arm Repositioning

Authors: Mathias Unberath, Javad Fotouhi, Jonas Hajek, Andreas Maier, Greg Osgood, Russell Taylor, Mehran Armand, Nassir Navab

Abstract: Interventional C-arm imaging is crucial to percutaneous orthopedic procedures as it enables the surgeon to monitor the progress of surgery on the anatomy level. Minimally invasive interventions require repeated acquisition of X-ray images from different anatomical views to verify tool placement. Achieving and reproducing these views often comes at the cost of increased surgical time and radiation… ▽ More Interventional C-arm imaging is crucial to percutaneous orthopedic procedures as it enables the surgeon to monitor the progress of surgery on the anatomy level. Minimally invasive interventions require repeated acquisition of X-ray images from different anatomical views to verify tool placement. Achieving and reproducing these views often comes at the cost of increased surgical time and radiation dose to both patient and staff. This work proposes a marker-free "technician-in-the-loop" Augmented Reality (AR) solution for C-arm repositioning. The X-ray technician operating the C-arm interventionally is equipped with a head-mounted display capable of recording desired C-arm poses in 3D via an integrated infrared sensor. For C-arm repositioning to a particular target view, the recorded C-arm pose is restored as a virtual object and visualized in an AR environment, serving as a perceptual reference for the technician. We conduct experiments in a setting simulating orthopedic trauma surgery. Our proof-of-principle findings indicate that the proposed system can decrease the 2.76 X-ray images required per desired view down to zero, suggesting substantial reductions of radiation dose during C-arm repositioning. The proposed AR solution is a first step towards facilitating communication between the surgeon and the surgical staff, improving the quality of surgical image acquisition, and enabling context-aware guidance for surgery rooms of the future. The concept of technician-in-the-loop design will become relevant to various interventions considering the expected advancements of sensing and wearable computing in the near future. △ Less

Submitted 22 June, 2018; originally announced June 2018.

Exploiting Partial Structural Symmetry For Patient-Specific Image Augmentation in Trauma Interventions

Authors: Javad Fotouhi, Mathias Unberath, Giacomo Taylor, Arash Ghaani Farashahi, Bastian Bier, Russell H. Taylor, Greg M. Osgood, M. D., Mehran Armand, Nassir Navab

Abstract: In unilateral pelvic fracture reductions, surgeons attempt to reconstruct the bone fragments such that bilateral symmetry in the bony anatomy is restored. We propose to exploit this "structurally symmetric" nature of the pelvic bone, and provide intra-operative image augmentation to assist the surgeon in repairing dislocated fragments. The main challenge is to automatically estimate the desired pl… ▽ More In unilateral pelvic fracture reductions, surgeons attempt to reconstruct the bone fragments such that bilateral symmetry in the bony anatomy is restored. We propose to exploit this "structurally symmetric" nature of the pelvic bone, and provide intra-operative image augmentation to assist the surgeon in repairing dislocated fragments. The main challenge is to automatically estimate the desired plane of symmetry within the patient's pre-operative CT. We propose to estimate this plane using a non-linear optimization strategy, by minimizing Tukey's biweight robust estimator, relying on the partial symmetry of the anatomy. Moreover, a regularization term is designed to enforce the similarity of bone density histograms on both sides of this plane, relying on the biological fact that, even if injured, the dislocated bone segments remain within the body. The experimental results demonstrate the performance of the proposed method in estimating this "plane of partial symmetry" using CT images of both healthy and injured anatomy. Examples of unilateral pelvic fractures are used to show how intra-operative X-ray images could be augmented with the forward-projections of the mirrored anatomy, acting as objective road-map for fracture reduction procedures. △ Less

Submitted 9 April, 2018; originally announced April 2018.

Comments: JF, MU, and GT have contributed equally

Closing the Calibration Loop: An Inside-out-tracking Paradigm for Augmented Reality in Orthopedic Surgery

Authors: Jonas Hajek, Mathias Unberath, Javad Fotouhi, Bastian Bier, Sing Chun Lee, Greg Osgood, Andreas Maier, Mehran Armand, Nassir Navab

Abstract: In percutaneous orthopedic interventions the surgeon attempts to reduce and fixate fractures in bony structures. The complexity of these interventions arises when the surgeon performs the challenging task of navigating surgical tools percutaneously only under the guidance of 2D interventional X-ray imaging. Moreover, the intra-operatively acquired data is only visualized indirectly on external dis… ▽ More In percutaneous orthopedic interventions the surgeon attempts to reduce and fixate fractures in bony structures. The complexity of these interventions arises when the surgeon performs the challenging task of navigating surgical tools percutaneously only under the guidance of 2D interventional X-ray imaging. Moreover, the intra-operatively acquired data is only visualized indirectly on external displays. In this work, we propose a flexible Augmented Reality (AR) paradigm using optical see-through head mounted displays. The key technical contribution of this work includes the marker-less and dynamic tracking concept which closes the calibration loop between patient, C-arm and the surgeon. This calibration is enabled using Simultaneous Localization and Map** of the environment of the operating theater. In return, the proposed solution provides in situ visualization of pre- and intra-operative 3D medical data directly at the surgical site. We demonstrate pre-clinical evaluation of a prototype system, and report errors for calibration and target registration. Finally, we demonstrate the usefulness of the proposed inside-out tracking system in achieving "bull's eye" view for C-arm-guided punctures. This AR solution provides an intuitive visualization of the anatomy and can simplify the hand-eye coordination for the orthopedic surgeon. △ Less

Submitted 22 March, 2018; originally announced March 2018.

Comments: JH, MU, and JF have contributed equally

X-ray-transform Invariant Anatomical Landmark Detection for Pelvic Trauma Surgery

Authors: Bastian Bier, Mathias Unberath, Jan-Nico Zaech, Javad Fotouhi, Mehran Armand, Greg Osgood, Nassir Navab, Andreas Maier

Abstract: X-ray image guidance enables percutaneous alternatives to complex procedures. Unfortunately, the indirect view onto the anatomy in addition to projective simplification substantially increase the task-load for the surgeon. Additional 3D information such as knowledge of anatomical landmarks can benefit surgical decision making in complicated scenarios. Automatic detection of these landmarks in tran… ▽ More X-ray image guidance enables percutaneous alternatives to complex procedures. Unfortunately, the indirect view onto the anatomy in addition to projective simplification substantially increase the task-load for the surgeon. Additional 3D information such as knowledge of anatomical landmarks can benefit surgical decision making in complicated scenarios. Automatic detection of these landmarks in transmission imaging is challenging since image-domain features characteristic to a certain landmark change substantially depending on the viewing direction. Consequently and to the best of our knowledge, the above problem has not yet been addressed. In this work, we present a method to automatically detect anatomical landmarks in X-ray images independent of the viewing direction. To this end, a sequential prediction framework based on convolutional layers is trained on synthetically generated data of the pelvic anatomy to predict 23 landmarks in single X-ray images. View independence is contingent on training conditions and, here, is achieved on a spherical segment covering (120 x 90) degrees in LAO/RAO and CRAN/CAUD, respectively, centered around AP. On synthetic data, the proposed approach achieves a mean prediction error of 5.6 +- 4.5 mm. We demonstrate that the proposed network is immediately applicable to clinically acquired data of the pelvis. In particular, we show that our intra-operative landmark detection together with pre-operative CT enables X-ray pose estimation which, ultimately, benefits initialization of image-based 2D/3D registration. △ Less

Submitted 22 March, 2018; originally announced March 2018.

Comments: BB and MU have contributed equally and are listed alphabetically

DeepDRR -- A Catalyst for Machine Learning in Fluoroscopy-guided Procedures

Authors: Mathias Unberath, Jan-Nico Zaech, Sing Chun Lee, Bastian Bier, Javad Fotouhi, Mehran Armand, Nassir Navab

Abstract: Machine learning-based approaches outperform competing methods in most disciplines relevant to diagnostic radiology. Interventional radiology, however, has not yet benefited substantially from the advent of deep learning, in particular because of two reasons: 1) Most images acquired during the procedure are never archived and are thus not available for learning, and 2) even if they were available,… ▽ More Machine learning-based approaches outperform competing methods in most disciplines relevant to diagnostic radiology. Interventional radiology, however, has not yet benefited substantially from the advent of deep learning, in particular because of two reasons: 1) Most images acquired during the procedure are never archived and are thus not available for learning, and 2) even if they were available, annotations would be a severe challenge due to the vast amounts of data. When considering fluoroscopy-guided procedures, an interesting alternative to true interventional fluoroscopy is in silico simulation of the procedure from 3D diagnostic CT. In this case, labeling is comparably easy and potentially readily available, yet, the appropriateness of resulting synthetic data is dependent on the forward model. In this work, we propose DeepDRR, a framework for fast and realistic simulation of fluoroscopy and digital radiography from CT scans, tightly integrated with the software platforms native to deep learning. We use machine learning for material decomposition and scatter estimation in 3D and 2D, respectively, combined with analytic forward projection and noise injection to achieve the required performance. On the example of anatomical landmark detection in X-ray images of the pelvis, we demonstrate that machine learning models trained on DeepDRRs generalize to unseen clinically acquired data without the need for re-training or domain adaptation. Our results are promising and promote the establishment of machine learning in fluoroscopy-guided procedures. △ Less

Submitted 22 March, 2018; originally announced March 2018.

Comments: MU and JNZ have contributed equally

On The Effect of Vibration on Shape Sensing of Continuum Manipulators Using Fiber Bragg Gratings

Authors: Shahriar Sefati, Farshid Alambeigi, Iulian Iordachita, Russell Taylor, Mehran Armand

Abstract: Fiber Bragg Grating (FBG) has shown great potential in shape and force sensing of continuum manipulators (CM) and biopsy needles. In the recent years, many researchers have studied different manufacturing and modeling techniques of FBG-based force and shape sensors for medical applications. These studies mainly focus on obtaining shape and force information in a static (or quasi-static) environmen… ▽ More Fiber Bragg Grating (FBG) has shown great potential in shape and force sensing of continuum manipulators (CM) and biopsy needles. In the recent years, many researchers have studied different manufacturing and modeling techniques of FBG-based force and shape sensors for medical applications. These studies mainly focus on obtaining shape and force information in a static (or quasi-static) environment. In this paper, however, we study and evaluate dynamic environments where the FBG data is affected by vibration caused by a harmonic force e.g. a rotational debriding tool harmonically exciting the CM and the FBG-based shape sensor. In such situations, appropriate pre-processing of the FBG signal is necessary in order to infer correct information from the raw signal. We look at an example of such dynamic environments in the less invasive treatment of osteolysis by studying the FBG data both in time- and frequency-domain in presence of vibration due to a debriding tool rotating inside the lumen of the CM. △ Less

Submitted 22 January, 2018; originally announced January 2018.

Comments: Accepted for International Symposium on Medical Robotics 2018

A Convex Optimization Framework for Constrained Concurrent Motion Control of a Hybrid Redundant Surgical System

Authors: Farshid Alambeigi, Shahriar Sefati, Mehran Armand

Abstract: We present a constrained motion control framework for a redundant surgical system designed for minimally invasive treatment of pelvic osteolysis. The framework comprises a kinematics model of a six Degrees-of-Freedom (DoF) robotic arm integrated with a one DoF continuum manipulator as well as a novel convex optimization redundancy resolution controller. To resolve the redundancy resolution problem… ▽ More We present a constrained motion control framework for a redundant surgical system designed for minimally invasive treatment of pelvic osteolysis. The framework comprises a kinematics model of a six Degrees-of-Freedom (DoF) robotic arm integrated with a one DoF continuum manipulator as well as a novel convex optimization redundancy resolution controller. To resolve the redundancy resolution problem, formulated as a constrained l2-regularized quadratic minimization, we study and evaluate the potential use of an optimally tuned alternating direction method of multipliers (ADMM) algorithm. To this end, we prove global convergence of the algorithm at linear rate and propose expressions for the involved parameters resulting in a fast convergence. Simulations on the robotic system verified our analytical derivations and showed the capability and robustness of the ADMM algorithm in constrained motion control of our redundant surgical system. △ Less

Submitted 21 January, 2018; originally announced January 2018.

Comments: 8 pages, 7 figures, To be appeared in IEEE American Control Conference (ACC) 2018

Plan in 2D, execute in 3D: An augmented reality solution for cup placement in total hip arthroplasty

Authors: Javad Fotouhi, Clayton P. Alexander, Mathias Unberath, Giacomo Taylor, Sing Chun Lee, Bernhard Fuerst, Alex Johnson, Greg Osgood, Russell H. Taylor, Harpal Khanuja, Mehran Armand, Nassir Navab

Abstract: Reproducibly achieving proper implant alignment is a critical step in total hip arthroplasty (THA) procedures that has been shown to substantially affect patient outcome. In current practice, correct alignment of the acetabular cup is verified in C-arm X-ray images that are acquired in an anterior-posterior (AP) view. Favorable surgical outcome is, therefore, heavily dependent on the surgeon's exp… ▽ More Reproducibly achieving proper implant alignment is a critical step in total hip arthroplasty (THA) procedures that has been shown to substantially affect patient outcome. In current practice, correct alignment of the acetabular cup is verified in C-arm X-ray images that are acquired in an anterior-posterior (AP) view. Favorable surgical outcome is, therefore, heavily dependent on the surgeon's experience in understanding the 3D orientation of a hemispheric implant from 2D AP projection images. This work proposes an easy to use intra-operative component planning system based on two C-arm X-ray images that is combined with 3D augmented reality (AR) visualization that simplifies impactor and cup placement according to the planning by providing a real-time RGBD data overlay. We evaluate the feasibility of our system in a user study comprising four orthopedic surgeons at the Johns Hopkins Hospital, and also report errors in translation, anteversion, and abduction as low as 1.98 mm, 1.10 degrees, and 0.53 degrees, respectively. The promising performance of this AR solution shows that deploying this system could eliminate the need for excessive radiation, simplify the intervention, and enable reproducibly accurate placement of acetabular implants. △ Less

Submitted 4 January, 2018; originally announced January 2018.

Journal ref: J. Med. Imag. 5(2), 021205 (2018)

On Capacity Formulation with Stationary Inputs and Application to a Bit-Patterned Media Recording Channel Model

Authors: Phan-Minh Nguyen, Marc A. Armand

Abstract: In this correspondence, we illustrate among other things the use of the stationarity property of the set of capacity-achieving inputs in capacity calculations. In particular, as a case study, we consider a bit-patterned media recording channel model and formulate new lower and upper bounds on its capacity that yield improvements over existing results. Inspired by the observation that the new bound… ▽ More In this correspondence, we illustrate among other things the use of the stationarity property of the set of capacity-achieving inputs in capacity calculations. In particular, as a case study, we consider a bit-patterned media recording channel model and formulate new lower and upper bounds on its capacity that yield improvements over existing results. Inspired by the observation that the new bounds are tight at low noise levels, we also characterize the capacity of this model as a series expansion in the low-noise regime. The key to these results is the realization of stationarity in the supremizing input set in the capacity formula. While the property is prevalent in capacity formulations in the ergodic-theoretic literature, we show that this realization is possible in the Shannon-theoretic framework where a channel is defined as a sequence of finite-dimensional conditional probabilities, by defining a new class of consistent stationary and ergodic channels. △ Less

Submitted 8 September, 2015; originally announced September 2015.

Comments: 25 pages, 10 figures, to appear in IEEE Transactions on Information Theory

Nearly MDS expander codes with reduced alphabet size

Authors: Marc A. Armand, Jianwen Zhang

Abstract: Recently, Roth and Skachek proposed two methods for constructing nearly maximum-distance separable (MDS) expander codes. We show that through the simple modification of using mixed-alphabet codes derived from MDS codes as constituent codes in their code designs, one can obtain nearly MDS codes of significantly smaller alphabet size, albeit at the expense of a (very slight) reduction in code rate… ▽ More Recently, Roth and Skachek proposed two methods for constructing nearly maximum-distance separable (MDS) expander codes. We show that through the simple modification of using mixed-alphabet codes derived from MDS codes as constituent codes in their code designs, one can obtain nearly MDS codes of significantly smaller alphabet size, albeit at the expense of a (very slight) reduction in code rate. △ Less

Submitted 8 August, 2007; originally announced August 2007.

Retrieving Reed-Solomon coded data under interpolation-based list decoding

Authors: Jianwen Zhang, Marc A. Armand

Abstract: A transform that enables generator-matrix-based Reed-Solomon (RS) coded data to be recovered under interpolation-based list decoding is presented. The transform matrix needs to be computed only once and the transformation of an element from the output list to the desired RS coded data block incurs $k^{2}$ field multiplications, given a code of dimension $k$. A transform that enables generator-matrix-based Reed-Solomon (RS) coded data to be recovered under interpolation-based list decoding is presented. The transform matrix needs to be computed only once and the transformation of an element from the output list to the desired RS coded data block incurs $k^{2}$ field multiplications, given a code of dimension $k$. △ Less

Submitted 12 December, 2006; originally announced December 2006.

Comments: 10 pages. Submitted to IEEE information theory for possible publication