-
PalmGazer: Unimanual Eye-hand Menus in Augmented Reality
Authors:
Ken Pfeuffer,
Jan Obernolte,
Felix Dietz,
Ville Mäkelä,
Ludwig Sidenmark,
Pavel Manakhov,
Minna Pakanen,
Florian Alt
Abstract:
How can we design the user interfaces for augmented reality (AR) so that we can interact as simple, flexible and expressive as we can with smartphones in one hand? To explore this question, we propose PalmGazer as an interaction concept integrating eye-hand interaction to establish a singlehandedly operable menu system. In particular, PalmGazer is designed to support quick and spontaneous digital…
▽ More
How can we design the user interfaces for augmented reality (AR) so that we can interact as simple, flexible and expressive as we can with smartphones in one hand? To explore this question, we propose PalmGazer as an interaction concept integrating eye-hand interaction to establish a singlehandedly operable menu system. In particular, PalmGazer is designed to support quick and spontaneous digital commands -- such as to play a music track, check notifications or browse visual media -- through our devised three-way interaction model: hand opening to summon the menu UI, eye-hand input for selection of items, and dragging gesture for navigation. A key aspect is that it remains always-accessible and movable to the user, as the menu supports meaningful hand and head based reference frames. We demonstrate the concept in practice through a prototypical personal UI with application probes, and describe technique designs specifically-tailored to the application UI. A qualitative evaluation highlights the system's design benefits and drawbacks, e.g., that common 2D scroll and selection tasks are simple to operate, but higher degrees of freedom may be reserved for two hands. Our work contributes interaction techniques and design insights to expand AR's uni-manual capabilities.
△ Less
Submitted 21 June, 2023;
originally announced June 2023.
-
Revealing the Hidden Effects of Phishing Emails: An Analysis of Eye and Mouse Movements in Email Sorting Tasks
Authors:
Yasmeen Abdrabou,
Felix Dietz,
Ahmed Shams,
Pascal Knierim,
Yomna Abdelrahman,
Ken Pfeuffer,
Mariam Hassib,
Florian Alt
Abstract:
Users are the last line of defense as phishing emails pass filter mechanisms. At the same time, phishing emails are designed so that they are challenging to identify by users. To this end, attackers employ techniques, such as eliciting stress, targeting helpfulness, or exercising authority, due to which users often miss being manipulated out of malicious intent. This work builds on the assumption…
▽ More
Users are the last line of defense as phishing emails pass filter mechanisms. At the same time, phishing emails are designed so that they are challenging to identify by users. To this end, attackers employ techniques, such as eliciting stress, targeting helpfulness, or exercising authority, due to which users often miss being manipulated out of malicious intent. This work builds on the assumption that manipulation techniques, even if going unnoticed by users, still lead to changes in their behavior. In this work, we present the outcomes of an online study in which we collected gaze and mouse movement data during an email sorting task. Our findings show that phishing emails lead to significant differences across behavioral features but depend on the nature of the email. We discuss how our findings can be leveraged to build security mechanisms protecting users and companies from phishing.
△ Less
Submitted 26 May, 2023;
originally announced May 2023.
-
Enhancing Reinforcement Learning with discrete interfaces to learn the Dyck Language
Authors:
Florian Dietz,
Dietrich Klakow
Abstract:
Even though most interfaces in the real world are discrete, no efficient way exists to train neural networks to make use of them, yet. We enhance an Interaction Network (a Reinforcement Learning architecture) with discrete interfaces and train it on the generalized Dyck language. This task requires an understanding of hierarchical structures to solve, and has long proven difficult for neural netwo…
▽ More
Even though most interfaces in the real world are discrete, no efficient way exists to train neural networks to make use of them, yet. We enhance an Interaction Network (a Reinforcement Learning architecture) with discrete interfaces and train it on the generalized Dyck language. This task requires an understanding of hierarchical structures to solve, and has long proven difficult for neural networks. We provide the first solution based on learning to use discrete data structures. We encountered unexpected anomalous behavior during training, and utilized pre-training based on execution traces to overcome them. The resulting model is very small and fast, and generalizes to sequences that are an entire order of magnitude longer than the training data.
△ Less
Submitted 27 October, 2021;
originally announced October 2021.
-
Interaction Networks: Using a Reinforcement Learner to train other Machine Learning algorithms
Authors:
Florian Dietz
Abstract:
The wiring of neurons in the brain is more flexible than the wiring of connections in contemporary artificial neural networks. It is possible that this extra flexibility is important for efficient problem solving and learning.
This paper introduces the Interaction Network. Interaction Networks aim to capture some of this extra flexibility.
An Interaction Network consists of a collection of con…
▽ More
The wiring of neurons in the brain is more flexible than the wiring of connections in contemporary artificial neural networks. It is possible that this extra flexibility is important for efficient problem solving and learning.
This paper introduces the Interaction Network. Interaction Networks aim to capture some of this extra flexibility.
An Interaction Network consists of a collection of conventional neural networks, a set of memory locations, and a DQN or other reinforcement learner. The DQN decides when each of the neural networks is executed, and on what memory locations. In this way, the individual neural networks can be trained on different data, for different tasks. At the same time, the results of the individual networks influence the decision process of the reinforcement learner. This results in a feedback loop that allows the DQN to perform actions that improve its own decision-making.
Any existing type of neural network can be reproduced in an Interaction Network in its entirety, with only a constant computational overhead.
Interaction Networks can then introduce additional features to improve performance further. These make the algorithm more flexible and general, but at the expense of being harder to train.
In this paper, thought experiments are used to explore how the additional abilities of Interaction Networks could be used to improve various existing types of neural networks.
Several experiments have been run to prove that the concept is sound. These show that the basic idea works, but they also reveal a number of challenges that do not appear in conventional neural networks, which make Interaction Networks very hard to train.
Further research needs to be done to alleviate these issues. A number of promising avenues of research to achieve this are outlined in this paper.
△ Less
Submitted 15 June, 2020;
originally announced June 2020.
