-
From Virtual Reality to the Emerging Discipline of Perception Engineering
Authors:
Steven M. LaValle,
Evan G. Center,
Timo Ojala,
Matti Pouke,
Nicoletta Prencipe,
Basak Sakcak,
Markku Suomalainen,
Kalle G. Timperi,
Vadim K. Weinstein
Abstract:
This paper makes the case that a powerful new discipline, which we term perception engineering, is steadily emerging. It follows from a progression of ideas that involve creating illusions, from historical paintings and film, to video games and virtual reality in modern times. Rather than creating physical artifacts such as bridges, airplanes, or computers, perception engineers create illusory per…
▽ More
This paper makes the case that a powerful new discipline, which we term perception engineering, is steadily emerging. It follows from a progression of ideas that involve creating illusions, from historical paintings and film, to video games and virtual reality in modern times. Rather than creating physical artifacts such as bridges, airplanes, or computers, perception engineers create illusory perceptual experiences. The scope is defined over any agent that interacts with the physical world, including both biological organisms (humans, animals) and engineered systems (robots, autonomous systems). The key idea is that an agent, called a producer, alters the environment with the intent to alter the perceptual experience of another agent, called a receiver. Most importantly, the paper introduces a precise mathematical formulation of this process, based on the von Neumann-Morgenstern notion of information, to help scope and define the discipline. It is then applied to the cases of engineered and biological agents with discussion of its implications on existing fields such as virtual reality, robotics, and even social media. Finally, open challenges and opportunities for involvement are identified.
△ Less
Submitted 27 March, 2024;
originally announced March 2024.
-
Exploring the Efficacy of Base Data Augmentation Methods in Deep Learning-Based Radiograph Classification of Knee Joint Osteoarthritis
Authors:
Fabi Prezja,
Leevi Annala,
Sampsa Kiiskinen,
Timo Ojala
Abstract:
Diagnosing knee joint osteoarthritis (KOA), a major cause of disability worldwide, is challenging due to subtle radiographic indicators and the varied progression of the disease. Using deep learning for KOA diagnosis requires broad, comprehensive datasets. However, obtaining these datasets poses significant challenges due to patient privacy concerns and data collection restrictions. Additive data…
▽ More
Diagnosing knee joint osteoarthritis (KOA), a major cause of disability worldwide, is challenging due to subtle radiographic indicators and the varied progression of the disease. Using deep learning for KOA diagnosis requires broad, comprehensive datasets. However, obtaining these datasets poses significant challenges due to patient privacy concerns and data collection restrictions. Additive data augmentation, which enhances data variability, emerges as a promising solution. Yet, it's unclear which augmentation techniques are most effective for KOA. This study explored various data augmentation methods, including adversarial augmentations, and their impact on KOA classification model performance. While some techniques improved performance, others commonly used underperformed. We identified potential confounding regions within the images using adversarial augmentation. This was evidenced by our models' ability to classify KL0 and KL4 grades accurately, with the knee joint omitted. This observation suggested a model bias, which might leverage unrelated features for classification currently present in radiographs. Interestingly, removing the knee joint also led to an unexpected improvement in KL1 classification accuracy. To better visualize these paradoxical effects, we employed Grad-CAM, highlighting the associated regions. Our study underscores the need for careful technique selection for improved model performance and identifying and managing potential confounding regions in radiographic KOA deep learning.
△ Less
Submitted 10 November, 2023;
originally announced November 2023.
-
Adaptive Variance Thresholding: A Novel Approach to Improve Existing Deep Transfer Vision Models and Advance Automatic Knee-Joint Osteoarthritis Classification
Authors:
Fabi Prezja,
Leevi Annala,
Sampsa Kiiskinen,
Suvi Lahtinen,
Timo Ojala
Abstract:
Knee-Joint Osteoarthritis (KOA) is a prevalent cause of global disability and is inherently complex to diagnose due to its subtle radiographic markers and individualized progression. One promising classification avenue involves applying deep learning methods; however, these techniques demand extensive, diversified datasets, which pose substantial challenges due to medical data collection restricti…
▽ More
Knee-Joint Osteoarthritis (KOA) is a prevalent cause of global disability and is inherently complex to diagnose due to its subtle radiographic markers and individualized progression. One promising classification avenue involves applying deep learning methods; however, these techniques demand extensive, diversified datasets, which pose substantial challenges due to medical data collection restrictions. Existing practices typically resort to smaller datasets and transfer learning. However, this approach often inherits unnecessary pre-learned features that can clutter the classifier's vector space, potentially hampering performance. This study proposes a novel paradigm for improving post-training specialized classifiers by introducing adaptive variance thresholding (AVT) followed by Neural Architecture Search (NAS). This approach led to two key outcomes: an increase in the initial accuracy of the pre-trained KOA models and a 60-fold reduction in the NAS input vector space, thus facilitating faster inference speed and a more efficient hyperparameter search. We also applied this approach to an external model trained for KOA classification. Despite its initial performance, the application of our methodology improved its average accuracy, making it one of the top three KOA classification models.
△ Less
Submitted 9 November, 2023;
originally announced November 2023.
-
Synthesizing Bidirectional Temporal States of Knee Osteoarthritis Radiographs with Cycle-Consistent Generative Adversarial Neural Networks
Authors:
Fabi Prezja,
Leevi Annala,
Sampsa Kiiskinen,
Suvi Lahtinen,
Timo Ojala
Abstract:
Knee Osteoarthritis (KOA), a leading cause of disability worldwide, is challenging to detect early due to subtle radiographic indicators. Diverse, extensive datasets are needed but are challenging to compile because of privacy, data collection limitations, and the progressive nature of KOA. However, a model capable of projecting genuine radiographs into different OA stages could augment data pools…
▽ More
Knee Osteoarthritis (KOA), a leading cause of disability worldwide, is challenging to detect early due to subtle radiographic indicators. Diverse, extensive datasets are needed but are challenging to compile because of privacy, data collection limitations, and the progressive nature of KOA. However, a model capable of projecting genuine radiographs into different OA stages could augment data pools, enhance algorithm training, and offer pre-emptive prognostic insights. In this study, we trained a CycleGAN model to synthesize past and future stages of KOA on any genuine radiograph. The model was validated using a Convolutional Neural Network that was deceived into misclassifying disease stages in transformed images, demonstrating the CycleGAN's ability to effectively transform disease characteristics forward or backward in time. The model was particularly effective in synthesizing future disease states and showed an exceptional ability to retroactively transition late-stage radiographs to earlier stages by eliminating osteophytes and expanding knee joint space, signature characteristics of None or Doubtful KOA. The model's results signify a promising potential for enhancing diagnostic models, data augmentation, and educational and prognostic usage in healthcare. Nevertheless, further refinement, validation, and a broader evaluation process encompassing both CNN-based assessments and expert medical feedback are emphasized for future research and development.
△ Less
Submitted 9 November, 2023;
originally announced November 2023.
-
Improving Performance in Colorectal Cancer Histology Decomposition using Deep and Ensemble Machine Learning
Authors:
Fabi Prezja,
Leevi Annala,
Sampsa Kiiskinen,
Suvi Lahtinen,
Timo Ojala,
Pekka Ruusuvuori,
Teijo Kuopio
Abstract:
In routine colorectal cancer management, histologic samples stained with hematoxylin and eosin are commonly used. Nonetheless, their potential for defining objective biomarkers for patient stratification and treatment selection is still being explored. The current gold standard relies on expensive and time-consuming genetic tests. However, recent research highlights the potential of convolutional…
▽ More
In routine colorectal cancer management, histologic samples stained with hematoxylin and eosin are commonly used. Nonetheless, their potential for defining objective biomarkers for patient stratification and treatment selection is still being explored. The current gold standard relies on expensive and time-consuming genetic tests. However, recent research highlights the potential of convolutional neural networks (CNNs) in facilitating the extraction of clinically relevant biomarkers from these readily available images. These CNN-based biomarkers can predict patient outcomes comparably to golden standards, with the added advantages of speed, automation, and minimal cost. The predictive potential of CNN-based biomarkers fundamentally relies on the ability of convolutional neural networks (CNNs) to classify diverse tissue types from whole slide microscope images accurately. Consequently, enhancing the accuracy of tissue class decomposition is critical to amplifying the prognostic potential of imaging-based biomarkers. This study introduces a hybrid Deep and ensemble machine learning model that surpassed all preceding solutions for this classification task. Our model achieved 96.74% accuracy on the external test set and 99.89% on the internal test set. Recognizing the potential of these models in advancing the task, we have made them publicly available for further research and development.
△ Less
Submitted 25 October, 2023;
originally announced October 2023.
-
Virtual Reality Sickness Reduces Attention During Immersive Experiences
Authors:
Katherine J. Mimnaugh,
Evan G. Center,
Markku Suomalainen,
Israel Becerra,
Eliezer Lozano,
Rafael Murrieta-Cid,
Timo Ojala,
Steven M. LaValle,
Kara D. Federmeier
Abstract:
In this paper, we show that Virtual Reality (VR) sickness is associated with a reduction in attention, which was detected with the P3b Event-Related Potential (ERP) component from electroencephalography (EEG) measurements collected in a dual-task paradigm. We hypothesized that sickness symptoms such as nausea, eyestrain, and fatigue would reduce the users' capacity to pay attention to tasks comple…
▽ More
In this paper, we show that Virtual Reality (VR) sickness is associated with a reduction in attention, which was detected with the P3b Event-Related Potential (ERP) component from electroencephalography (EEG) measurements collected in a dual-task paradigm. We hypothesized that sickness symptoms such as nausea, eyestrain, and fatigue would reduce the users' capacity to pay attention to tasks completed in a virtual environment, and that this reduction in attention would be dynamically reflected in a decrease of the P3b amplitude while VR sickness was experienced. In a user study, participants were taken on a tour through a museum in VR along paths with varying amounts of rotation, shown previously to cause different levels of VR sickness. While paying attention to the virtual museum (the primary task), participants were asked to silently count tones of a different frequency (the secondary task). Control measurements for comparison against the VR sickness conditions were taken when the users were not wearing the Head-Mounted Display (HMD) and while they were immersed in VR but not moving through the environment. This exploratory study shows, across multiple analyses, that the effect mean amplitude of the P3b collected during the task is associated with both sickness severity measured after the task with a questionnaire (SSQ) and with the number of counting errors on the secondary task. Thus, VR sickness may impair attention and task performance, and these changes in attention can be tracked with ERP measures as they happen, without asking participants to assess their sickness symptoms in the moment.
△ Less
Submitted 11 October, 2023; v1 submitted 23 June, 2023;
originally announced June 2023.
-
Leaning-Based Control of an Immersive-Telepresence Robot
Authors:
Joona Halkola,
Markku Suomalainen,
Basak Sakcak,
Katherine J. Mimnaugh,
Juho Kalliokoski,
Alexis P. Chambers,
Timo Ojala,
Steven M. LaValle
Abstract:
In this paper, we present an implementation of a leaning-based control of a differential drive telepresence robot and a user study in simulation, with the goal of bringing the same functionality to a real telepresence robot. The participants used a balance board to control the robot and viewed the virtual environment through a head-mounted display. The main motivation for using a balance board as…
▽ More
In this paper, we present an implementation of a leaning-based control of a differential drive telepresence robot and a user study in simulation, with the goal of bringing the same functionality to a real telepresence robot. The participants used a balance board to control the robot and viewed the virtual environment through a head-mounted display. The main motivation for using a balance board as the control device stems from Virtual Reality (VR) sickness; even small movements of your own body matching the motions seen on the screen decrease the sensory conflict between vision and vestibular organs, which lies at the heart of most theories regarding the onset of VR sickness. To test the hypothesis that the balance board as a control method would be less sickening than using joysticks, we designed a user study (N=32, 15 women) in which the participants drove a simulated differential drive robot in a virtual environment with either a Nintendo Wii Balance Board or joysticks. However, our pre-registered main hypotheses were not supported; the joystick did not cause any more VR sickness on the participants than the balance board, and the board proved to be statistically significantly more difficult to use, both subjectively and objectively. Analyzing the open-ended questions revealed these results to be likely connected, meaning that the difficulty of use seemed to affect sickness; even unlimited training time before the test did not make the use as easy as the familiar joystick. Thus, making the board easier to use is a key to enable its potential; we present a few possibilities towards this goal.
△ Less
Submitted 22 August, 2022;
originally announced August 2022.
-
The Body Scaling Effect and Its Impact on Physics Plausibility
Authors:
Matti Pouke,
Evan G. Center,
Alexis P. Chambers,
Sakaria Pouke,
Timo Ojala,
Steven M. LaValle
Abstract:
In this study we investigated the effect of body ownership illusion-based body scaling on physics plausibility in Virtual Reality (VR). Our interest was in examining whether body ownership illusion-based body scaling could affect the plausibility of rigid body dynamics similarly to altering VR users' scale by manipulating their virtual interpupillary distance and viewpoint height. The procedure in…
▽ More
In this study we investigated the effect of body ownership illusion-based body scaling on physics plausibility in Virtual Reality (VR). Our interest was in examining whether body ownership illusion-based body scaling could affect the plausibility of rigid body dynamics similarly to altering VR users' scale by manipulating their virtual interpupillary distance and viewpoint height. The procedure involved the conceptual replication of two previous studies. We investigated physics plausibility with 40 participants under two conditions. In our synchronous condition, we used visuo-tactile stimuli to elicit a body ownership illusion of inhabiting an invisible doll-sized body on participants reclining on an exam table. Our asynchronous condition was otherwise similar, but the visuo-tactile stimuli were provided asynchronously to prevent the onset of the body ownership illusion. We were interested in whether the correct approximation of physics (true physics) or physics that are incorrect and appearing as if the environment is five times larger instead (movie physics) appear more realistic to participants as a function of body scale. We found that movie physics did appear more realistic to participants under the body ownership illusion condition. However, our hypothesis that true physics would appear more realistic in the asynchronous condition was unsupported. Our exploratory analyses revealed that movie physics were perceived as plausible under both conditions. Moreover, we were not able to replicate previous findings from literature concerning object size estimations while inhabiting a small invisible body. However, we found a significant opposite effect regarding size estimations; the object sizes were on average underestimated during the synchronous visuo-tactile condition when compared to the asynchronous condition.
△ Less
Submitted 30 June, 2022;
originally announced June 2022.
-
HI-DWA: Human-Influenced Dynamic Window Approach for Shared Control of a Telepresence Robot
Authors:
Juho Kalliokoski,
Basak Sakcak,
Markku Suomalainen,
Katherine J. Mimnaugh,
Alexis P. Chambers,
Timo Ojala,
Steven M. LaValle
Abstract:
This paper considers the problem of enabling the user to modify the path of a telepresence robot. The robot is capable of autonomously navigating to a goal predefined by the user, but the user might still want to modify the path, for example, to go further away from other people, or to go closer to landmarks she wants to see on the way. We propose Human-Influenced Dynamic Window Approach (HI-DWA),…
▽ More
This paper considers the problem of enabling the user to modify the path of a telepresence robot. The robot is capable of autonomously navigating to a goal predefined by the user, but the user might still want to modify the path, for example, to go further away from other people, or to go closer to landmarks she wants to see on the way. We propose Human-Influenced Dynamic Window Approach (HI-DWA), a shared control method aimed for telepresence robots based on Dynamic Window Approach (DWA) that allows the user to influence the control input given to the robot. To verify the proposed method, we performed a user study (N=32) in Virtual Reality (VR) to compare HI-DWA with switching between autonomous navigation and manual control for controlling a simulated telepresence robot moving in a virtual environment. Results showed that users reached their goal faster using HI-DWA controller and found it easier to use. Preference between the two methods was split equally. Qualitative analysis revealed that a major reason for the participants that preferred switching between two modes was the feeling of control. We also analyzed the effect of different input methods, joystick and gesture, on the preference and perceived workload.
△ Less
Submitted 25 July, 2022; v1 submitted 5 March, 2022;
originally announced March 2022.
-
A Study of Preference and Comfort for Users Immersed in a Telepresence Robot
Authors:
Adhi Widagdo,
Markku Suomalainen,
Basak Sakcak,
Katherine J. Mimnaugh,
Juho Kalliokoski,
Alexis P. Chambers,
Timo Ojala,
Steven M. LaValle
Abstract:
In this paper, we show that unwinding the rotations of a user immersed in a telepresence robot is preferred and may increase the feeling of presence or "being there". By immersive telepresence, we mean a scenario where a user wearing a head-mounted display embodies a mobile robot equipped with a 360° camera in another location, such that the user can move the robot and communicate with people arou…
▽ More
In this paper, we show that unwinding the rotations of a user immersed in a telepresence robot is preferred and may increase the feeling of presence or "being there". By immersive telepresence, we mean a scenario where a user wearing a head-mounted display embodies a mobile robot equipped with a 360° camera in another location, such that the user can move the robot and communicate with people around it. By unwinding the rotations, the user never perceives rotational motion through the head-mounted display while staying stationary, avoiding sensory mismatch which causes a major part of VR sickness. We performed a user study (N=32) on a Dolly mobile robot platform, mimicking an earlier similar study done in simulation. Unlike the simulated study, in this study there is no significant difference in the VR sickness suffered by the participants, or the condition they find more comfortable (unwinding or automatic rotations). However, participants still prefer the unwinding condition, and they judge it to render a stronger feeling of presence, a major piece in natural communication. We show that participants aboard a real telepresence robot perceive distances similarly suitable as in simulation, presenting further evidence on the applicability of VR as a research platform for robotics and human-robot interaction.
△ Less
Submitted 5 March, 2022;
originally announced March 2022.
-
Unwinding Rotations Improves User Comfort with Immersive Telepresence Robots
Authors:
Markku Suomalainen,
Basak Sakcak,
Adhi Widagdo,
Juho Kalliokoski,
Katherine J. Mimnaugh,
Alexis P. Chambers,
Timo Ojala,
Steven M. LaValle
Abstract:
We propose unwinding the rotations experienced by the user of an immersive telepresence robot to improve comfort and reduce VR sickness of the user. By immersive telepresence we refer to a situation where a 360\textdegree~camera on top of a mobile robot is streaming video and audio into a head-mounted display worn by a remote user possibly far away. Thus, it enables the user to be present at the r…
▽ More
We propose unwinding the rotations experienced by the user of an immersive telepresence robot to improve comfort and reduce VR sickness of the user. By immersive telepresence we refer to a situation where a 360\textdegree~camera on top of a mobile robot is streaming video and audio into a head-mounted display worn by a remote user possibly far away. Thus, it enables the user to be present at the robot's location, look around by turning the head and communicate with people near the robot. By unwinding the rotations of the camera frame, the user's viewpoint is not changed when the robot rotates. The user can change her viewpoint only by physically rotating in her local setting; as visual rotation without the corresponding vestibular stimulation is a major source of VR sickness, physical rotation by the user is expected to reduce VR sickness. We implemented unwinding the rotations for a simulated robot traversing a virtual environment and ran a user study (N=34) comparing unwinding rotations to user's viewpoint turning when the robot turns. Our results show that the users found unwound rotations more preferable and comfortable and that it reduced their level of VR sickness. We also present further results about the users' path integration capabilities, viewing directions, and subjective observations of the robot's speed and distances to simulated people and objects.
△ Less
Submitted 7 January, 2022;
originally announced January 2022.
-
The Plausibility Paradox for Resized Users in Virtual Environments
Authors:
Matti Pouke,
Katherine J. Mimnaugh,
Alexis Chambers,
Timo Ojala,
Steven M. LaValle
Abstract:
This paper identifies and confirms a perceptual phenomenon: when users interact with simulated objects in a virtual environment where the users' scale deviates greatly from normal, there is a mismatch between the object physics they consider realistic and the object physics that would be correct at that scale. We report the findings of two studies investigating the relationship between perceived r…
▽ More
This paper identifies and confirms a perceptual phenomenon: when users interact with simulated objects in a virtual environment where the users' scale deviates greatly from normal, there is a mismatch between the object physics they consider realistic and the object physics that would be correct at that scale. We report the findings of two studies investigating the relationship between perceived realism and a physically accurate approximation of reality in a virtual reality experience in which the user has been scaled by a factor of ten. Study 1 investigated perception of physics when scaled-down by a factor of ten, whereas Study 2 focused on enlargement by a similar amount. Studies were carried out as within-subjects experiments in which a total of 84 subjects performed simple interaction tasks with objects under two different physics simulation conditions. In the true physics condition, the objects, when dropped and thrown, behaved accurately according to the physics that would be correct at that either reduced or enlarged scale in the real world. In the movie physics condition, the objects behaved in a similar manner as they would if no scaling of the user had occurred. We found that a significant majority of the users considered the movie physics condition to be the more realistic one. However, at enlarged scale, many users considered true physics to match their expectations even if they ultimately believed movie physics to be the realistic condition. We argue that our findings have implications for many virtual reality and telepresence applications involving operation with simulated or physical objects in abnormal and especially small scales.
△ Less
Submitted 5 February, 2021;
originally announced February 2021.
-
The Plausibility Paradox for Scaled-Down Users in Virtual Environments
Authors:
Matti Pouke,
Katherine J. Mimnaugh,
Timo Ojala,
Steven M. LaValle
Abstract:
This paper identifies a new phenomenon: when users interact with simulated objects in a virtual environment where the user is much smaller than usual, there is a mismatch between the object physics that they expect and the object physics that would be correct at that scale. We report the findings of our study investigating the relationship between perceived realism and a physically accurate approx…
▽ More
This paper identifies a new phenomenon: when users interact with simulated objects in a virtual environment where the user is much smaller than usual, there is a mismatch between the object physics that they expect and the object physics that would be correct at that scale. We report the findings of our study investigating the relationship between perceived realism and a physically accurate approximation of reality in a virtual reality experience in which the user has been scaled down by a factor of ten. We conducted a within-subjects experiment in which 44 subjects performed a simple interaction task with objects under two different physics simulation conditions. In one condition, the objects, when dropped and thrown, behaved accurately according to the physics that would be correct at that reduced scale in the real world, our true physics condition. In the other condition, the movie physics condition, the objects behaved in a similar manner as they would if no scaling of the user had occurred. We found that a significant majority of the users considered the latter condition to be the more realistic one. We argue that our findings have implications for many virtual reality and telepresence applications involving operation with simulated or physical objects in small scales.
△ Less
Submitted 20 February, 2020; v1 submitted 3 December, 2019;
originally announced December 2019.
