-
Visual Highlighting for Situated Brushing and Linking
Authors:
Nina Doerr,
Benjamin Lee,
Katarina Baricova,
Dieter Schmalstieg,
Michael Sedlmair
Abstract:
Brushing and linking is widely used for visual analytics in desktop environments. However, using this approach to link many data items between situated (e.g., a virtual screen with data) and embedded views (e.g., highlighted objects in the physical environment) is largely unexplored. To this end, we study the effectiveness of visual highlighting techniques in hel** users identify and link physic…
▽ More
Brushing and linking is widely used for visual analytics in desktop environments. However, using this approach to link many data items between situated (e.g., a virtual screen with data) and embedded views (e.g., highlighted objects in the physical environment) is largely unexplored. To this end, we study the effectiveness of visual highlighting techniques in hel** users identify and link physical referents to brushed data marks in a situated scatterplot. In an exploratory virtual reality user study (N=20), we evaluated four highlighting techniques under different physical layouts and tasks. We discuss the effectiveness of these techniques, as well as implications for the design of brushing and linking operations in situated analytics.
△ Less
Submitted 11 June, 2024; v1 submitted 22 March, 2024;
originally announced March 2024.
-
Deep Medial Voxels: Learned Medial Axis Approximations for Anatomical Shape Modeling
Authors:
Antonio Pepe,
Richard Schussnig,
Jianning Li,
Christina Gsaxner,
Dieter Schmalstieg,
Jan Egger
Abstract:
Shape reconstruction from imaging volumes is a recurring need in medical image analysis. Common workflows start with a segmentation step, followed by careful post-processing and,finally, ad hoc meshing algorithms. As this sequence can be timeconsuming, neural networks are trained to reconstruct shapes through template deformation. These networks deliver state-ofthe-art results without manual inter…
▽ More
Shape reconstruction from imaging volumes is a recurring need in medical image analysis. Common workflows start with a segmentation step, followed by careful post-processing and,finally, ad hoc meshing algorithms. As this sequence can be timeconsuming, neural networks are trained to reconstruct shapes through template deformation. These networks deliver state-ofthe-art results without manual intervention, but, so far, they have primarily been evaluated on anatomical shapes with little topological variety between individuals. In contrast, other works favor learning implicit shape models, which have multiple benefits for meshing and visualization. Our work follows this direction by introducing deep medial voxels, a semi-implicit representation that faithfully approximates the topological skeleton from imaging volumes and eventually leads to shape reconstruction via convolution surfaces. Our reconstruction technique shows potential for both visualization and computer simulations.
△ Less
Submitted 18 March, 2024;
originally announced March 2024.
-
DeepDR: Deep Structure-Aware RGB-D Inpainting for Diminished Reality
Authors:
Christina Gsaxner,
Shohei Mori,
Dieter Schmalstieg,
Jan Egger,
Gerhard Paar,
Werner Bailer,
Denis Kalkofen
Abstract:
Diminished reality (DR) refers to the removal of real objects from the environment by virtually replacing them with their background. Modern DR frameworks use inpainting to hallucinate unobserved regions. While recent deep learning-based inpainting is promising, the DR use case is complicated by the need to generate coherent structure and 3D geometry (i.e., depth), in particular for advanced appli…
▽ More
Diminished reality (DR) refers to the removal of real objects from the environment by virtually replacing them with their background. Modern DR frameworks use inpainting to hallucinate unobserved regions. While recent deep learning-based inpainting is promising, the DR use case is complicated by the need to generate coherent structure and 3D geometry (i.e., depth), in particular for advanced applications, such as 3D scene editing. In this paper, we propose DeepDR, a first RGB-D inpainting framework fulfilling all requirements of DR: Plausible image and geometry inpainting with coherent structure, running at real-time frame rates, with minimal temporal artifacts. Our structure-aware generative network allows us to explicitly condition color and depth outputs on the scene semantics, overcoming the difficulty of reconstructing sharp and consistent boundaries in regions with complex backgrounds. Experimental results show that the proposed framework can outperform related work qualitatively and quantitatively.
△ Less
Submitted 1 December, 2023;
originally announced December 2023.
-
Design Patterns for Situated Visualization in Augmented Reality
Authors:
Benjamin Lee,
Michael Sedlmair,
Dieter Schmalstieg
Abstract:
Situated visualization has become an increasingly popular research area in the visualization community, fueled by advancements in augmented reality (AR) technology and immersive analytics. Visualizing data in spatial proximity to their physical referents affords new design opportunities and considerations not present in traditional visualization, which researchers are now beginning to explore. How…
▽ More
Situated visualization has become an increasingly popular research area in the visualization community, fueled by advancements in augmented reality (AR) technology and immersive analytics. Visualizing data in spatial proximity to their physical referents affords new design opportunities and considerations not present in traditional visualization, which researchers are now beginning to explore. However, the AR research community has an extensive history of designing graphics that are displayed in highly physical contexts. In this work, we leverage the richness of AR research and apply it to situated visualization. We derive design patterns which summarize common approaches of visualizing data in situ. The design patterns are based on a survey of 293 papers published in the AR and visualization communities, as well as our own expertise. We discuss design dimensions that help to describe both our patterns and previous work in the literature. This discussion is accompanied by several guidelines which explain how to apply the patterns given the constraints imposed by the real world. We conclude by discussing future research directions that will help establish a complete understanding of the design of situated visualization, including the role of interactivity, tasks, and workflows.
△ Less
Submitted 23 November, 2023; v1 submitted 18 July, 2023;
originally announced July 2023.
-
The HoloLens in Medicine: A systematic Review and Taxonomy
Authors:
Christina Gsaxner,
Jianning Li,
Antonio Pepe,
Yuan **,
Jens Kleesiek,
Dieter Schmalstieg,
Jan Egger
Abstract:
The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality display, is the main player in the recent boost in medical augmented reality research. In medical settings, the HoloLens enables the physician to obtain immediate insight into patient information, directly overlaid with their view of the clinical scenario, the medical student to gain a better understa…
▽ More
The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality display, is the main player in the recent boost in medical augmented reality research. In medical settings, the HoloLens enables the physician to obtain immediate insight into patient information, directly overlaid with their view of the clinical scenario, the medical student to gain a better understanding of complex anatomies or procedures, and even the patient to execute therapeutic tasks with improved, immersive guidance. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021, were attention is shifting towards it's successor, the HoloLens 2. We identified 171 relevant publications through a systematic search of the PubMed and Scopus databases. We analyze these publications in regard to their intended use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation. We find that, although the feasibility of using the HoloLens in various medical scenarios has been shown, increased efforts in the areas of precision, reliability, usability, workflow and perception are necessary to establish AR in clinical practice.
△ Less
Submitted 6 September, 2022;
originally announced September 2022.
-
Automated cross-sectional view selection in CT angiography of aortic dissections with uncertainty awareness and retrospective clinical annotations
Authors:
Antonio Pepe,
Jan Egger,
Marina Codari,
Martin J. Willemink,
Christina Gsaxner,
Jianning Li,
Peter M. Roth,
Gabriel Mistelbauer,
Dieter Schmalstieg,
Dominik Fleischmann
Abstract:
Objective: Surveillance imaging of chronic aortic diseases, such as dissections, relies on obtaining and comparing cross-sectional diameter measurements at predefined aortic landmarks, over time. Due to a lack of robust tools, the orientation of the cross-sectional planes is defined manually by highly trained operators. We show how manual annotations routinely collected in a clinic can be efficien…
▽ More
Objective: Surveillance imaging of chronic aortic diseases, such as dissections, relies on obtaining and comparing cross-sectional diameter measurements at predefined aortic landmarks, over time. Due to a lack of robust tools, the orientation of the cross-sectional planes is defined manually by highly trained operators. We show how manual annotations routinely collected in a clinic can be efficiently used to ease this task, despite the presence of a non-negligible interoperator variability in the measurements.
Impact: Ill-posed but repetitive imaging tasks can be eased or automated by leveraging imperfect, retrospective clinical annotations.
Methodology: In this work, we combine convolutional neural networks and uncertainty quantification methods to predict the orientation of such cross-sectional planes. We use clinical data randomly processed by 11 operators for training, and test on a smaller set processed by 3 independent operators to assess interoperator variability.
Results: Our analysis shows that manual selection of cross-sectional planes is characterized by 95% limits of agreement (LOA) of $10.6^\circ$ and $21.4^\circ$ per angle. Our method showed to decrease static error by $3.57^\circ$ ($40.2$%) and $4.11^\circ$ ($32.8$%) against state of the art and LOA by $5.4^\circ$ ($49.0$%) and $16.0^\circ$ ($74.6$%) against manual processing.
Conclusion: This suggests that pre-existing annotations can be an inexpensive resource in clinics to ease ill-posed and repetitive tasks like cross-section extraction for surveillance of aortic dissections.
△ Less
Submitted 22 November, 2021;
originally announced November 2021.
-
Evaluating Mixed and Augmented Reality: A Systematic Literature Review (2009-2019)
Authors:
Leonel Merino,
Magdalena Schwarzl,
Matthias Kraus,
Michael Sedlmair,
Dieter Schmalstieg,
Daniel Weiskopf
Abstract:
We present a systematic review of 458 papers that report on evaluations in mixed and augmented reality (MR/AR) published in ISMAR, CHI, IEEE VR, and UIST over a span of 11 years (2009-2019). Our goal is to provide guidance for future evaluations of MR/AR approaches. To this end, we characterize publications by paper type (e.g., technique, design study), research topic (e.g., tracking, rendering),…
▽ More
We present a systematic review of 458 papers that report on evaluations in mixed and augmented reality (MR/AR) published in ISMAR, CHI, IEEE VR, and UIST over a span of 11 years (2009-2019). Our goal is to provide guidance for future evaluations of MR/AR approaches. To this end, we characterize publications by paper type (e.g., technique, design study), research topic (e.g., tracking, rendering), evaluation scenario (e.g., algorithm performance, user performance), cognitive aspects (e.g., perception, emotion), and the context in which evaluations were conducted (e.g., lab vs. in-the-wild). We found a strong coupling of types, topics, and scenarios. We observe two groups: (a) technology-centric performance evaluations of algorithms that focus on improving tracking, displays, reconstruction, rendering, and calibration, and (b) human-centric studies that analyze implications of applications and design, human factors on perception, usability, decision making, emotion, and attention. Amongst the 458 papers, we identified 248 user studies that involved 5,761 participants in total, of whom only 1,619 were identified as female. We identified 43 data collection methods used to analyze 10 cognitive aspects. We found nine objective methods, and eight methods that support qualitative analysis. A majority (216/248) of user studies are conducted in a laboratory setting. Often (138/248), such studies involve participants in a static way. However, we also found a fair number (30/248) of in-the-wild studies that involve participants in a mobile fashion. We consider this paper to be relevant to academia and industry alike in presenting the state-of-the-art and guiding the steps to designing, conducting, and analyzing results of evaluations in MR/AR.
△ Less
Submitted 12 October, 2020;
originally announced October 2020.
-
DetectFusion: Detecting and Segmenting Both Known and Unknown Dynamic Objects in Real-time SLAM
Authors:
Ryo Hachiuma,
Christian Pirchheim,
Dieter Schmalstieg,
Hideo Saito
Abstract:
We present DetectFusion, an RGB-D SLAM system that runs in real-time and can robustly handle semantically known and unknown objects that can move dynamically in the scene. Our system detects, segments and assigns semantic class labels to known objects in the scene, while tracking and reconstructing them even when they move independently in front of the monocular camera. In contrast to related work…
▽ More
We present DetectFusion, an RGB-D SLAM system that runs in real-time and can robustly handle semantically known and unknown objects that can move dynamically in the scene. Our system detects, segments and assigns semantic class labels to known objects in the scene, while tracking and reconstructing them even when they move independently in front of the monocular camera. In contrast to related work, we achieve real-time computational performance on semantic instance segmentation with a novel method combining 2D object detection and 3D geometric segmentation. In addition, we propose a method for detecting and segmenting the motion of semantically unknown objects, thus further improving the accuracy of camera tracking and map reconstruction. We show that our method performs on par or better than previous work in terms of localization and object reconstruction accuracy, while achieving about 20 FPS even if the objects are segmented in each frame.
△ Less
Submitted 22 July, 2019;
originally announced July 2019.
-
On-the-fly Vertex Reuse for Massively-Parallel Software Geometry Processing
Authors:
Michael Kenzel,
Bernhard Kerbl,
Wolfgang Tatzgern,
Elena Ivanchenko,
Dieter Schmalstieg,
Markus Steinberger
Abstract:
Compute-mode rendering is becoming more and more attractive for non-standard rendering applications, due to the high flexibility of compute-mode execution. These newly designed pipelines often include streaming vertex and geometry processing stages. In typical triangle meshes, the same transformed vertex is on average required six times during rendering. To avoid redundant computation, a post-tran…
▽ More
Compute-mode rendering is becoming more and more attractive for non-standard rendering applications, due to the high flexibility of compute-mode execution. These newly designed pipelines often include streaming vertex and geometry processing stages. In typical triangle meshes, the same transformed vertex is on average required six times during rendering. To avoid redundant computation, a post-transform cache is traditionally suggested to enable reuse of vertex processing results. However, traditional caching neither scales well as the hardware becomes more parallel, nor can be efficiently implemented in a software design. We investigate alternative strategies to reusing vertex shading results on-the-fly for massively parallel software geometry processing. Forming static and dynamic batching on the data input stream, we analyze the effectiveness of identifying potential local reuse based on sorting, hashing, and efficient intra-thread-group communication. Altogether, we present four vertex reuse strategies, tailored to modern parallel architectures. Our simulations showcase that our batch-based strategies significantly outperform parallel caches in terms of reuse. On actual GPU hardware, our evaluation shows that our strategies not only lead to good reuse of processing results, but also boost performance by $2-3\times$ compared to naïvely ignoring reuse in a variety of practical applications.
△ Less
Submitted 22 May, 2018;
originally announced May 2018.
-
Clinical evaluation of semi-automatic opensource algorithmic software segmentation of the mandibular bone: Practical feasibility and assessment of a new course of action
Authors:
Jürgen Wallner,
Kerstin Hochegger,
Xiaojun Chen,
Irene Mischak,
Knut Reinbacher,
Mauro Pau,
Tomislav Zrnc,
Katja Schwenzer-Zimmerer,
Wolfgang Zemann,
Dieter Schmalstieg,
Jan Egger
Abstract:
Computer assisted technologies based on algorithmic software segmentation are an increasing topic of interest in complex surgical cases. However - due to functional instability, time consuming software processes, personnel resources or licensed-based financial costs many segmentation processes are often outsourced from clinical centers to third parties and the industry. Therefore, the aim of this…
▽ More
Computer assisted technologies based on algorithmic software segmentation are an increasing topic of interest in complex surgical cases. However - due to functional instability, time consuming software processes, personnel resources or licensed-based financial costs many segmentation processes are often outsourced from clinical centers to third parties and the industry. Therefore, the aim of this trial was to assess the practical feasibility of an easy available, functional stable and licensed-free segmentation approach to be used in the clinical practice. In this retrospective, randomized, controlled trail the accuracy and accordance of the open-source based segmentation algorithm GrowCut (GC) was assessed through the comparison to the manually generated ground truth of the same anatomy using 10 CT lower jaw data-sets from the clinical routine. Assessment parameters were the segmentation time, the volume, the voxel number, the Dice Score (DSC) and the Hausdorff distance (HD). Overall segmentation times were about one minute. Mean DSC values of over 85% and HD below 33.5 voxel could be achieved. Statistical differences between the assessment parameters were not significant (p<0.05) and correlation coefficients were close to the value one (r > 0.94). Complete functional stable and time saving segmentations with high accuracy and high positive correlation could be performed by the presented interactive open-source based approach. In the cranio-maxillofacial complex the used method could represent an algorithmic alternative for image-based segmentation in the clinical practice for e.g. surgical treatment planning or visualization of postoperative results and offers several advantages. Systematic comparisons to other segmentation approaches or with a greater data amount are areas of future works.
△ Less
Submitted 11 May, 2018;
originally announced May 2018.
-
In-depth Assessment of an Interactive Graph-based Approach for the Segmentation for Pancreatic Metastasis in Ultrasound Acquisitions of the Liver with two Specialists in Internal Medicine
Authors:
Jan Egger,
Xiaojun Chen,
Lucas Bettac,
Mark Hänle,
Tilmann Gräter,
Wolfram Zoller,
Dieter Schmalstieg,
Alexander Hann
Abstract:
The manual outlining of hepatic metastasis in (US) ultrasound acquisitions from patients suffering from pancreatic cancer is common practice. However, such pure manual measurements are often very time consuming, and the results repeatedly differ between the raters. In this contribution, we study the in-depth assessment of an interactive graph-based approach for the segmentation for pancreatic meta…
▽ More
The manual outlining of hepatic metastasis in (US) ultrasound acquisitions from patients suffering from pancreatic cancer is common practice. However, such pure manual measurements are often very time consuming, and the results repeatedly differ between the raters. In this contribution, we study the in-depth assessment of an interactive graph-based approach for the segmentation for pancreatic metastasis in US images of the liver with two specialists in Internal Medicine. Thereby, evaluating the approach with over one hundred different acquisitions of metastases. The two physicians or the algorithm had never assessed the acquisitions before the evaluation. In summary, the physicians first performed a pure manual outlining followed by an algorithmic segmentation over one month later. As a result, the experts satisfied in up to ninety percent of algorithmic segmentation results. Furthermore, the algorithmic segmentation was much faster than manual outlining and achieved a median Dice Similarity Coefficient (DSC) of over eighty percent. Ultimately, the algorithm enables a fast and accurate segmentation of liver metastasis in clinical US images, which can support the manual outlining in daily practice.
△ Less
Submitted 12 March, 2018;
originally announced March 2018.
-
Algorithm guided outlining of 105 pancreatic cancer liver metastases in Ultrasound
Authors:
Alexander Hann,
Lucas Bettac,
Mark M. Haenle,
Tilmann Graeter,
Andreas W. Berger,
Jens Dreyhaupt,
Dieter Schmalstieg,
Wolfram G. Zoller,
Jan Egger
Abstract:
Manual segmentation of hepatic metastases in ultrasound images acquired from patients suffering from pancreatic cancer is common practice. Semiautomatic measurements promising assistance in this process are often assessed using a small number of lesions performed by examiners who already know the algorithm. In this work, we present the application of an algorithm for the segmentation of liver meta…
▽ More
Manual segmentation of hepatic metastases in ultrasound images acquired from patients suffering from pancreatic cancer is common practice. Semiautomatic measurements promising assistance in this process are often assessed using a small number of lesions performed by examiners who already know the algorithm. In this work, we present the application of an algorithm for the segmentation of liver metastases due to pancreatic cancer using a set of 105 different images of metastases. The algorithm and the two examiners had never assessed the images before. The examiners first performed a manual segmentation and, after five weeks, a semiautomatic segmentation using the algorithm. They were satisfied in up to 90% of the cases with the semiautomatic segmentation results. Using the algorithm was significantly faster and resulted in a median Dice similarity score of over 80%. Estimation of the inter-operator variability by using the intra class correlation coefficient was good with 0.8. In conclusion, the algorithm facilitates fast and accurate segmentation of liver metastases, comparable to the current gold standard of manual segmentation.
△ Less
Submitted 9 October, 2017;
originally announced October 2017.
-
Computer-aided position planning of miniplates to treat facial bone defects
Authors:
Jan Egger,
Jürgen Wallner,
Markus Gall,
Xiaojun Chen,
Katja Schwenzer-Zimmerer,
Knut Reinbacher,
Dieter Schmalstieg
Abstract:
In this contribution, a software system for computer-aided position planning of miniplates to treat facial bone defects is proposed. The intra-operatively used bone plates have to be passively adapted on the underlying bone contours for adequate bone fragment stabilization. However, this procedure can lead to frequent intra-operatively performed material readjustments especially in complex surgica…
▽ More
In this contribution, a software system for computer-aided position planning of miniplates to treat facial bone defects is proposed. The intra-operatively used bone plates have to be passively adapted on the underlying bone contours for adequate bone fragment stabilization. However, this procedure can lead to frequent intra-operatively performed material readjustments especially in complex surgical cases. Our approach is able to fit a selection of common implant models on the surgeon's desired position in a 3D computer model. This happens with respect to the surrounding anatomical structures, always including the possibility of adjusting both the direction and the position of the used osteosynthesis material. By using the proposed software, surgeons are able to pre-plan the out coming implant in its form and morphology with the aid of a computer-visualized model within a few minutes. Further, the resulting model can be stored in STL file format, the commonly used format for 3D printing. Using this technology, surgeons are able to print the virtual generated implant, or create an individually designed bending tool. This method leads to adapted osteosynthesis materials according to the surrounding anatomy and requires further a minimum amount of money and time.
△ Less
Submitted 17 August, 2017;
originally announced August 2017.
-
Interactive Outlining of Pancreatic Cancer Liver Metastases in Ultrasound Images
Authors:
Jan Egger,
Dieter Schmalstieg,
Xiaojun Chen,
Wolfram G. Zoller,
Alexander Hann
Abstract:
Ultrasound (US) is the most commonly used liver imaging modality worldwide. Due to its low cost, it is increasingly used in the follow-up of cancer patients with metastases localized in the liver. In this contribution, we present the results of an interactive segmentation approach for liver metastases in US acquisitions. A (semi-) automatic segmentation is still very challenging because of the low…
▽ More
Ultrasound (US) is the most commonly used liver imaging modality worldwide. Due to its low cost, it is increasingly used in the follow-up of cancer patients with metastases localized in the liver. In this contribution, we present the results of an interactive segmentation approach for liver metastases in US acquisitions. A (semi-) automatic segmentation is still very challenging because of the low image quality and the low contrast between the metastasis and the surrounding liver tissue. Thus, the state of the art in clinical practice is still manual measurement and outlining of the metastases in the US images. We tackle the problem by providing an interactive segmentation approach providing real-time feedback of the segmentation results. The approach has been evaluated with typical US acquisitions from the clinical routine, and the datasets consisted of pancreatic cancer metastases. Even for difficult cases, satisfying segmentations results could be achieved because of the interactive real-time behavior of the approach. In total, 40 clinical images have been evaluated with our method by comparing the results against manual ground truth segmentations. This evaluation yielded to an average Dice Score of 85% and an average Hausdorff Distance of 13 pixels.
△ Less
Submitted 18 April, 2017;
originally announced April 2017.
-
How Sensemaking Tools Influence Display Space Usage
Authors:
Thomas Geymayer,
Manuela Waldner,
Alexander Lex,
Dieter Schmalstieg
Abstract:
We explore how the availability of a sensemaking tool influences users' knowledge externalization strategies. On a large display, users were asked to solve an intelligence analysis task with or without a bidirectionally linked concept-graph (BLC) to organize insights into concepts (nodes) and relations (edges). In BLC, both nodes and edges maintain "deep links" to the exact source phrases and sect…
▽ More
We explore how the availability of a sensemaking tool influences users' knowledge externalization strategies. On a large display, users were asked to solve an intelligence analysis task with or without a bidirectionally linked concept-graph (BLC) to organize insights into concepts (nodes) and relations (edges). In BLC, both nodes and edges maintain "deep links" to the exact source phrases and sections in associated documents. In our control condition, we were able to reproduce previously described spatial organization behaviors using document windows on the large display. When using BLC, however, we found that analysts apply spatial organization to BLC nodes instead, use significantly less display space and have significantly fewer open windows.
△ Less
Submitted 12 April, 2017;
originally announced April 2017.
-
Adaptive User Perspective Rendering for Handheld Augmented Reality
Authors:
Peter Mohr,
Markus Tatzgern,
Jens Grubert,
Dieter Schmalstieg,
Denis Kalkofen
Abstract:
Handheld Augmented Reality commonly implements some variant of magic lens rendering, which turns only a fraction of the user's real environment into AR while the rest of the environment remains unaffected. Since handheld AR devices are commonly equipped with video see-through capabilities, AR magic lens applications often suffer from spatial distortions, because the AR environment is presented fro…
▽ More
Handheld Augmented Reality commonly implements some variant of magic lens rendering, which turns only a fraction of the user's real environment into AR while the rest of the environment remains unaffected. Since handheld AR devices are commonly equipped with video see-through capabilities, AR magic lens applications often suffer from spatial distortions, because the AR environment is presented from the perspective of the camera of the mobile device. Recent approaches counteract this distortion based on estimations of the user's head position, rendering the scene from the user's perspective. To this end, approaches usually apply face-tracking algorithms on the front camera of the mobile device. However, this demands high computational resources and therefore commonly affects the performance of the application beyond the already high computational load of AR applications. In this paper, we present a method to reduce the computational demands for user perspective rendering by applying lightweight optical flow tracking and an estimation of the user's motion before head tracking is started. We demonstrate the suitability of our approach for computationally limited mobile devices and we compare it to device perspective rendering, to head tracked user perspective rendering, as well as to fixed point of view user perspective rendering.
△ Less
Submitted 22 March, 2017;
originally announced March 2017.
-
HTC Vive MeVisLab integration via OpenVR for medical applications
Authors:
Jan Egger,
Markus Gall,
Jürgen Wallner,
Pedro Boechat,
Alexander Hann,
Xing Li,
Xiaojun Chen,
Dieter Schmalstieg
Abstract:
Virtual Reality, an immersive technology that replicates an environment via computer-simulated reality, gets a lot of attention in the entertainment industry. However, VR has also great potential in other areas, like the medical domain, Examples are intervention planning, training and simulation. This is especially of use in medical operations, where an aesthetic outcome is important, like for fac…
▽ More
Virtual Reality, an immersive technology that replicates an environment via computer-simulated reality, gets a lot of attention in the entertainment industry. However, VR has also great potential in other areas, like the medical domain, Examples are intervention planning, training and simulation. This is especially of use in medical operations, where an aesthetic outcome is important, like for facial surgeries. Alas, importing medical data into Virtual Reality devices is not necessarily trivial, in particular, when a direct connection to a proprietary application is desired. Moreover, most researcher do not build their medical applications from scratch, but rather leverage platforms like MeVisLab, MITK, OsiriX or 3D Slicer. These platforms have in common that they use libraries like ITK and VTK, and provide a convenient graphical interface. However, ITK and VTK do not support Virtual Reality directly. In this study, the usage of a Virtual Reality device for medical data under the MeVisLab platform is presented. The OpenVR library is integrated into the MeVisLab platform, allowing a direct and uncomplicated usage of the head mounted display HTC Vive inside the MeVisLab platform. Medical data coming from other MeVisLab modules can directly be connected per drag-and-drop to the Virtual Reality module, rendering the data inside the HTC Vive for immersive virtual reality inspection.
△ Less
Submitted 22 March, 2017;
originally announced March 2017.
-
Interactive reconstructions of cranial 3D implants under MeVisLab as an alternative to commercial planning software
Authors:
Jan Egger,
Markus Gall,
Alois Tax,
Muammer Ücal,
Ulrike Zefferer,
Xing Li,
Gord von Campe,
Ute Schäfer,
Dieter Schmalstieg,
Xiaojun Chen
Abstract:
In this publication, the interactive planning and reconstruction of cranial 3D Implants under the medical prototy** platform MeVisLab as alternative to commercial planning software is introduced. In doing so, a MeVisLab prototype consisting of a customized data-flow network and an own C++ module was set up. As a result, the Computer-Aided Design (CAD) software prototype guides a user through the…
▽ More
In this publication, the interactive planning and reconstruction of cranial 3D Implants under the medical prototy** platform MeVisLab as alternative to commercial planning software is introduced. In doing so, a MeVisLab prototype consisting of a customized data-flow network and an own C++ module was set up. As a result, the Computer-Aided Design (CAD) software prototype guides a user through the whole workflow to generate an implant. Therefore, the workflow begins with loading and mirroring the patients head for an initial curvature of the implant. Then, the user can perform an additional Laplacian smoothing, followed by a Delaunay triangulation. The result is an aesthetic looking and well-fitting 3D implant, which can be stored in a CAD file format, e.g. STereoLithography (STL), for 3D printing. The 3D printed implant can finally be used for an in-depth pre-surgical evaluation or even as a real implant for the patient. In a nutshell, our research and development shows that a customized MeVisLab software prototype can be used as an alternative to complex commercial planning software, which may also not be available in every clinic. Finally, not to conform ourselves directly to available commercial software and look for other options that might improve the workflow.
△ Less
Submitted 9 March, 2017;
originally announced March 2017.
-
A feasibility study on SSVEP-based interaction with motivating and immersive virtual and augmented reality
Authors:
Josef Faller,
Brendan Z. Allison,
Clemens Brunner,
Reinhold Scherer,
Dieter Schmalstieg,
Gert Pfurtscheller,
Christa Neuper
Abstract:
Non-invasive steady-state visual evoked potential (SSVEP) based brain-computer interface (BCI) systems offer high bandwidth compared to other BCI types and require only minimal calibration and training. Virtual reality (VR) has been already validated as effective, safe, affordable and motivating feedback modality for BCI experiments. Augmented reality (AR) enhances the physical world by superimpos…
▽ More
Non-invasive steady-state visual evoked potential (SSVEP) based brain-computer interface (BCI) systems offer high bandwidth compared to other BCI types and require only minimal calibration and training. Virtual reality (VR) has been already validated as effective, safe, affordable and motivating feedback modality for BCI experiments. Augmented reality (AR) enhances the physical world by superimposing informative, context sensitive, computer generated content. In the context of BCI, AR can be used as a friendlier and more intuitive real-world user interface, thereby facilitating a more seamless and goal directed interaction. This can improve practicality and usability of BCI systems and may help to compensate for their low bandwidth. In this feasibility study, three healthy participants had to finish a complex navigation task in immersive VR and AR conditions using an online SSVEP BCI. Two out of three subjects were successful in all conditions. To our knowledge, this is the first work to present an SSVEP BCI that operates using target stimuli integrated in immersive VR and AR (head-mounted display and camera). This research direction can benefit patients by introducing more intuitive and effective real-world interaction (e.g. smart home control). It may also be relevant for user groups that require or benefit from hands free operation (e.g. due to temporary situational disability).
△ Less
Submitted 14 January, 2017;
originally announced January 2017.
-
Towards Interaction Around Unmodified Camera-equipped Mobile Devices
Authors:
Jens Grubert,
Eyal Ofek,
Michel Pahud,
Matthias Kranz,
Dieter Schmalstieg
Abstract:
Around-device interaction promises to extend the input space of mobile and wearable devices beyond the common but restricted touchscreen. So far, most around-device interaction approaches rely on instrumenting the device or the environment with additional sensors. We believe, that the full potential of ordinary cameras, specifically user-facing cameras, which are integrated in most mobile devices…
▽ More
Around-device interaction promises to extend the input space of mobile and wearable devices beyond the common but restricted touchscreen. So far, most around-device interaction approaches rely on instrumenting the device or the environment with additional sensors. We believe, that the full potential of ordinary cameras, specifically user-facing cameras, which are integrated in most mobile devices today, are not used to their full potential, yet. We To this end, we present a novel approach for extending the input space around unmodified mobile devices using built-in front-facing cameras of unmodified handheld devices. Our approach estimates hand poses and gestures through reflections in sunglasses, ski goggles or visors. Thereby, GlassHands creates an enlarged input space, rivaling input reach on large touch displays. We discuss the idea, its limitations and future work.
△ Less
Submitted 14 January, 2017;
originally announced January 2017.
-
US-Cut: Interactive Algorithm for rapid Detection and Segmentation of Liver Tumors in Ultrasound Acquisitions
Authors:
Jan Egger,
Philip Voglreiter,
Mark Dokter,
Michael Hofmann,
Xiaojun Chen,
Wolfram G. Zoller,
Dieter Schmalstieg,
Alexander Hann
Abstract:
Ultrasound (US) is the most commonly used liver imaging modality worldwide. It plays an important role in follow-up of cancer patients with liver metastases. We present an interactive segmentation approach for liver tumors in US acquisitions. Due to the low image quality and the low contrast between the tumors and the surrounding tissue in US images, the segmentation is very challenging. Thus, the…
▽ More
Ultrasound (US) is the most commonly used liver imaging modality worldwide. It plays an important role in follow-up of cancer patients with liver metastases. We present an interactive segmentation approach for liver tumors in US acquisitions. Due to the low image quality and the low contrast between the tumors and the surrounding tissue in US images, the segmentation is very challenging. Thus, the clinical practice still relies on manual measurement and outlining of the tumors in the US images. We target this problem by applying an interactive segmentation algorithm to the US data, allowing the user to get real-time feedback of the segmentation results. The algorithm has been developed and tested hand-in-hand by physicians and computer scientists to make sure a future practical usage in a clinical setting is feasible. To cover typical acquisitions from the clinical routine, the approach has been evaluated with dozens of datasets where the tumors are hyperechoic (brighter), hypoechoic (darker) or isoechoic (similar) in comparison to the surrounding liver tissue. Due to the interactive real-time behavior of the approach, it was possible even in difficult cases to find satisfying segmentations of the tumors within seconds and without parameter settings, and the average tumor deviation was only 1.4mm compared with manual measurements. However, the long term goal is to ease the volumetric acquisition of liver tumors in order to evaluate for treatment response. Additional aim is the registration of intraoperative US images via the interactive segmentations to the patient's pre-interventional CT acquisitions.
△ Less
Submitted 1 March, 2016;
originally announced March 2016.
-
Interactive Volumetry Of Liver Ablation Zones
Authors:
Jan Egger,
Harald Busse,
Philipp Brandmaier,
Daniel Seider,
Matthias Gawlitza,
Steffen Strocka,
Philip Voglreiter,
Mark Dokter,
Michael Hofmann,
Bernhard Kainz,
Alexander Hann,
Xiaojun Chen,
Tuomas Alhonnoro,
Mika Pollari,
Dieter Schmalstieg,
Michael Moche
Abstract:
Percutaneous radiofrequency ablation (RFA) is a minimally invasive technique that destroys cancer cells by heat. The heat results from focusing energy in the radiofrequency spectrum through a needle. Amongst others, this can enable the treatment of patients who are not eligible for an open surgery. However, the possibility of recurrent liver cancer due to incomplete ablation of the tumor makes pos…
▽ More
Percutaneous radiofrequency ablation (RFA) is a minimally invasive technique that destroys cancer cells by heat. The heat results from focusing energy in the radiofrequency spectrum through a needle. Amongst others, this can enable the treatment of patients who are not eligible for an open surgery. However, the possibility of recurrent liver cancer due to incomplete ablation of the tumor makes post-interventional monitoring via regular follow-up scans mandatory. These scans have to be carefully inspected for any conspicuousness. Within this study, the RF ablation zones from twelve post-interventional CT acquisitions have been segmented semi-automatically to support the visual inspection. An interactive, graph-based contouring approach, which prefers spherically shaped regions, has been applied. For the quantitative and qualitative analysis of the algorithm's results, manual slice-by-slice segmentations produced by clinical experts have been used as the gold standard (which have also been compared among each other). As evaluation metric for the statistical validation, the Dice Similarity Coefficient (DSC) has been calculated. The results show that the proposed tool provides lesion segmentation with sufficient accuracy much faster than manual segmentation. The visual feedback and interactivity make the proposed tool well suitable for the clinical workflow.
△ Less
Submitted 21 October, 2015;
originally announced December 2015.
