-
Towards Hypermedia Environments for Adaptive Coordination in Industrial Automation
Authors:
Ganesh Ramanathan,
Simon Mayer,
Andrei Ciortea
Abstract:
Electromechanical systems manage physical processes through a network of inter-connected components. Today, programming the interactions required for coordinating these components is largely a manual process. This process is time-consuming and requires manual adaptation when system features change. To overcome this issue, we use autonomous software agents that process semantic descriptions of the…
▽ More
Electromechanical systems manage physical processes through a network of inter-connected components. Today, programming the interactions required for coordinating these components is largely a manual process. This process is time-consuming and requires manual adaptation when system features change. To overcome this issue, we use autonomous software agents that process semantic descriptions of the system to determine coordination requirements and constraints; on this basis, they then interact with one another to control the system in a decentralized and coordinated manner.Our core insight is that coordination requirements between individual components are, ultimately, largely due to underlying physical interdependencies between the components, which can be (and, in many cases, already are) semantically modeled in automation projects. Agents then use hypermedia to discover, at run time, the plans and protocols required for enacting the coordination. A key novelty of our approach is the use of hypermedia-driven interaction: it reduces coupling in the system and enables its run-time adaptation as features change.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
Learnings from Implementation of a BDI Agent-based Battery-less Wireless Sensor
Authors:
Ganesh Ramanathan,
Andres Gomez,
Simon Mayer
Abstract:
Battery-less embedded devices powered by energy harvesting are increasingly being used in wireless sensing applications. However, their limited and often uncertain energy availability challenges designing application programs. To examine if BDI-based agent programming can address this challenge, we used it for a real-life application involving an environmental sensor that works on energy harvested…
▽ More
Battery-less embedded devices powered by energy harvesting are increasingly being used in wireless sensing applications. However, their limited and often uncertain energy availability challenges designing application programs. To examine if BDI-based agent programming can address this challenge, we used it for a real-life application involving an environmental sensor that works on energy harvested from ambient light. This yielded the first ever implementation of a BDI agent on a low-power battery-less and energy-harvesting embedded system. Furthermore, it uncovered conceptual integration challenges between embedded systems and BDI-based agent programming that, if overcome, will simplify the deployment of more autonomous systems on low-power devices with non-deterministic energy availability. Specifically, we (1) mapped essential device states to default \textit{internal} beliefs, (2) recognized and addressed the need for beliefs in general to be \textit{short-} or \textit{long-term}, and (3) propose dynamic annotation of intentions with their run-time energy impact. We show that incorporating these extensions not only simplified the programming but also improved code readability and understanding of its behavior.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
A Match Made in Semantics: Physics-infused Digital Twins for Smart Building Automation
Authors:
Ganesh Ramanathan,
Simon Mayer
Abstract:
Buildings contain electro-mechanical systems that ensure the occupants' comfort, health, and safety. The functioning of these systems is automated through control programs, which are often available as reusable artifacts in a software library. However, matching these reusable control programs to the installed technical systems requires manual effort and adds engineering cost. In this article, we s…
▽ More
Buildings contain electro-mechanical systems that ensure the occupants' comfort, health, and safety. The functioning of these systems is automated through control programs, which are often available as reusable artifacts in a software library. However, matching these reusable control programs to the installed technical systems requires manual effort and adds engineering cost. In this article, we show that such matching can be accomplished fully automatically through logical rules and based on the creation of semantic relationships between descriptions of \emph{physical processes} and descriptions of technical systems and control programs. For this purpose, we propose a high-level bridging ontology that enables the desired rule-based matching and equips digital twins of the technical systems with the required knowledge about the underlying physical processes in a self-contained manner. We evaluated our approach in a real-life building automation project with a total of 34 deployed air handling units. Our data show that rules based on our bridging ontology enabled the system to infer the suitable choice of control programs automatically in more than 90\% of the cases while avoiding almost an hour of manual work for each such match.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
From Computational to Conversational Notebooks
Authors:
Thomas Weber,
Sven Mayer
Abstract:
Today, we see a drastic increase in LLM-based user interfaces to support users in various tasks. Also, in programming, we witness a productivity boost with features like LLM-supported code completion and conversational agents to generate code. In this work, we look at the future of computational notebooks by enriching them with LLM support. We propose a spectrum of support, from simple inline code…
▽ More
Today, we see a drastic increase in LLM-based user interfaces to support users in various tasks. Also, in programming, we witness a productivity boost with features like LLM-supported code completion and conversational agents to generate code. In this work, we look at the future of computational notebooks by enriching them with LLM support. We propose a spectrum of support, from simple inline code completion to executable code that was the output of a conversation. We showcase five concrete examples for potential user interface designs and discuss their benefits and drawbacks. With this, we hope to inspire the future development of LLM-supported computational notebooks.
△ Less
Submitted 15 June, 2024;
originally announced June 2024.
-
Putting Language into Context Using Smartphone-Based Keyboard Logging
Authors:
Florian Bemmann,
Timo Koch,
Maximilian Bergmann,
Clemens Stachl,
Daniel Buschek,
Ramona Schoedel,
Sven Mayer
Abstract:
While the study of language as typed on smartphones offers valuable insights, existing data collection methods often fall short in providing contextual information and ensuring user privacy. We present a privacy-respectful approach - context-enriched keyboard logging - that allows for the extraction of contextual information on the user's input motive, which is meaningful for linguistics, psycholo…
▽ More
While the study of language as typed on smartphones offers valuable insights, existing data collection methods often fall short in providing contextual information and ensuring user privacy. We present a privacy-respectful approach - context-enriched keyboard logging - that allows for the extraction of contextual information on the user's input motive, which is meaningful for linguistics, psychology, and behavioral sciences. In particular, with our approach, we enable distinguishing language contents by their channel (i.e., comments, messaging, search inputs). Filtering by channel allows for better pre-selection of data, which is in the interest of researchers and improves users' privacy. We demonstrate our approach on a large-scale six-month user study (N=624) of language use in smartphone interactions in the wild. Finally, we highlight the implications for research on language use in human-computer interaction and interdisciplinary contexts.
△ Less
Submitted 8 March, 2024;
originally announced March 2024.
-
PhysioCHI: Towards Best Practices for Integrating Physiological Signals in HCI
Authors:
Francesco Chiossi,
Ekaterina R. Stepanova,
Benjamin Tag,
Monica Perusquia-Hernandez,
Alexandra Kitson,
Arindam Dey,
Sven Mayer,
Abdallah El Ali
Abstract:
Recently, we saw a trend toward using physiological signals in interactive systems. These signals, offering deep insights into users' internal states and health, herald a new era for HCI. However, as this is an interdisciplinary approach, many challenges arise for HCI researchers, such as merging diverse disciplines, from understanding physiological functions to design expertise. Also, isolated re…
▽ More
Recently, we saw a trend toward using physiological signals in interactive systems. These signals, offering deep insights into users' internal states and health, herald a new era for HCI. However, as this is an interdisciplinary approach, many challenges arise for HCI researchers, such as merging diverse disciplines, from understanding physiological functions to design expertise. Also, isolated research endeavors limit the scope and reach of findings. This workshop aims to bridge these gaps, fostering cross-disciplinary discussions on usability, open science, and ethics tied to physiological data in HCI. In this workshop, we will discuss best practices for embedding physiological signals in interactive systems. Through collective efforts, we seek to craft a guiding document for best practices in physiological HCI research, ensuring that it remains grounded in shared principles and methodologies as the field advances.
△ Less
Submitted 11 December, 2023; v1 submitted 7 December, 2023;
originally announced December 2023.
-
Designing and Evaluating an Adaptive Virtual Reality System using EEG Frequencies to Balance Internal and External Attention States
Authors:
Francesco Chiossi,
Changkun Ou,
Carolina Gerhardt,
Felix Putze,
Sven Mayer
Abstract:
Virtual reality finds various applications in productivity, entertainment, and training scenarios requiring working memory and attentional resources. Working memory relies on prioritizing relevant information and suppressing irrelevant information through internal attention, which is fundamental for successful task performance and training. Today, virtual reality systems do not account for the imp…
▽ More
Virtual reality finds various applications in productivity, entertainment, and training scenarios requiring working memory and attentional resources. Working memory relies on prioritizing relevant information and suppressing irrelevant information through internal attention, which is fundamental for successful task performance and training. Today, virtual reality systems do not account for the impact of working memory loads resulting in over or under-stimulation. In this work, we designed an adaptive system based on EEG correlates of external and internal attention to support working memory task performance. Here, participants engaged in a visual working memory N-Back task, and we adapted the visual complexity of distracting surrounding elements. Our study first demonstrated the feasibility of EEG frontal theta and parietal alpha frequency bands for dynamic visual complexity adjustments. Second, our adaptive system showed improved task performance and diminished perceived workload compared to a reverse adaptation. Our results show the effectiveness of the proposed adaptive system, allowing for the optimization of distracting elements in high-demanding conditions. Adaptive systems based on alpha and theta frequency bands allow for the regulation of attentional and executive resources to keep users engaged in a task without resulting in cognitive overload.
△ Less
Submitted 17 November, 2023;
originally announced November 2023.
-
Usability and Adoption of Graphical Data-Driven Development Tools
Authors:
Thomas Weber,
Sven Mayer
Abstract:
Software development of modern, data-driven applications still relies on tools that use interaction paradigms that have remained mostly unchanged for decades. While rich forms of interactions exist as an alternative to textual command input, they find little adoption in professional software creation. In this work, we compare graphical programming using direct manipulation to the traditional, text…
▽ More
Software development of modern, data-driven applications still relies on tools that use interaction paradigms that have remained mostly unchanged for decades. While rich forms of interactions exist as an alternative to textual command input, they find little adoption in professional software creation. In this work, we compare graphical programming using direct manipulation to the traditional, textual way of creating data-driven applications to determine the benefits and drawbacks of each. In a between-subjects user study (N=18), we compared develo** a machine learning architecture with a graphical editor to traditional code-based development. While qualitative and quantitative measures show general benefits of graphical direct manipulation, the user's subjective perception does not always match this. Participants were aware of the possible benefits of such tools but were still biased in their perception. Our findings highlight that alternative software creation tools cannot just rely on good usability but must emphasize the demands of their specific target group, e.g. user control and flexibility, if they want long-term benefits and adoption.
△ Less
Submitted 9 November, 2023;
originally announced November 2023.
-
On the Computational Complexities of Complex-valued Neural Networks
Authors:
Kayol Soares Mayer,
Jonathan Aguiar Soares,
Ariadne Arrais Cruz,
Dalton Soares Arantes
Abstract:
Complex-valued neural networks (CVNNs) are nonlinear filters used in the digital signal processing of complex-domain data. Compared with real-valued neural networks~(RVNNs), CVNNs can directly handle complex-valued input and output signals due to their complex domain parameters and activation functions. With the trend toward low-power systems, computational complexity analysis has become essential…
▽ More
Complex-valued neural networks (CVNNs) are nonlinear filters used in the digital signal processing of complex-domain data. Compared with real-valued neural networks~(RVNNs), CVNNs can directly handle complex-valued input and output signals due to their complex domain parameters and activation functions. With the trend toward low-power systems, computational complexity analysis has become essential for measuring an algorithm's power consumption. Therefore, this paper presents both the quantitative and asymptotic computational complexities of CVNNs. This is a crucial tool in deciding which algorithm to implement. The mathematical operations are described in terms of the number of real-valued multiplications, as these are the most demanding operations. To determine which CVNN can be implemented in a low-power system, quantitative computational complexities can be used to accurately estimate the number of floating-point operations. We have also investigated the computational complexities of CVNNs discussed in some studies presented in the literature.
△ Less
Submitted 19 October, 2023;
originally announced October 2023.
-
CVNN-based Channel Estimation and Equalization in OFDM Systems Without Cyclic Prefix
Authors:
Heitor dos Santos Sousa,
Jonathan Aguiar Soares,
Kayol Soares Mayer,
Dalton Soares Arantes
Abstract:
In modern communication systems operating with Orthogonal Frequency-Division Multiplexing (OFDM), channel estimation requires minimal complexity with one-tap equalizers. However, this depends on cyclic prefixes, which must be sufficiently large to cover the channel impulse response. Conversely, the use of cyclic prefix (CP) decreases the useful information that can be conveyed in an OFDM frame, th…
▽ More
In modern communication systems operating with Orthogonal Frequency-Division Multiplexing (OFDM), channel estimation requires minimal complexity with one-tap equalizers. However, this depends on cyclic prefixes, which must be sufficiently large to cover the channel impulse response. Conversely, the use of cyclic prefix (CP) decreases the useful information that can be conveyed in an OFDM frame, thereby degrading the spectral efficiency of the system. In this context, we study the impact of CPs on channel estimation with complex-valued neural networks (CVNNs). We show that the phase-transmittance radial basis function neural network offers superior results, in terms of required energy per bit, compared to classical minimum mean-squared error and least squares algorithms in scenarios without CP.
△ Less
Submitted 25 August, 2023;
originally announced August 2023.
-
Evolutionary Solution Adaption for Multi-Objective Metal Cutting Process Optimization
Authors:
Leo Francoso Dal Piccol Sotto,
Sebastian Mayer,
Hemanth Janarthanam,
Alexander Butz,
Jochen Garcke
Abstract:
Optimizing manufacturing process parameters is typically a multi-objective problem with often contradictory objectives such as production quality and production time. If production requirements change, process parameters have to be optimized again. Since optimization usually requires costly simulations based on, for example, the Finite Element method, it is of great interest to have means to reduc…
▽ More
Optimizing manufacturing process parameters is typically a multi-objective problem with often contradictory objectives such as production quality and production time. If production requirements change, process parameters have to be optimized again. Since optimization usually requires costly simulations based on, for example, the Finite Element method, it is of great interest to have means to reduce the number of evaluations needed for optimization. To this end, we consider optimizing for different production requirements from the viewpoint of a framework for system flexibility that allows us to study the ability of an algorithm to transfer solutions from previous optimization tasks, which also relates to dynamic evolutionary optimization. Based on the extended Oxley model for orthogonal metal cutting, we introduce a multi-objective optimization benchmark where different materials define related optimization tasks, and use it to study the flexibility of NSGA-II, which we extend by two variants: 1) varying goals, that optimizes solutions for two tasks simultaneously to obtain in-between source solutions expected to be more adaptable, and 2) active-inactive genotype, that accommodates different possibilities that can be activated or deactivated. Results show that adaption with standard NSGA-II greatly reduces the number of evaluations required for optimization to a target goal, while the proposed variants further improve the adaption costs, although further work is needed towards making the methods advantageous for real applications.
△ Less
Submitted 31 May, 2023;
originally announced May 2023.
-
How Can Mixed Reality Benefit From Physiologically-Adaptive Systems? Challenges and Opportunities for Human Factors Applications
Authors:
Francesco Chiossi,
Sven Mayer
Abstract:
Mixed Reality (MR) allows users to interact with digital objects in a physical environment, but several limitations have hampered widespread adoption. Physiologically adaptive systems detecting user's states can drive interaction and address these limitations. Here, we highlight potential usability and interaction limitations in MR and how physiologically adaptive systems can benefit MR experience…
▽ More
Mixed Reality (MR) allows users to interact with digital objects in a physical environment, but several limitations have hampered widespread adoption. Physiologically adaptive systems detecting user's states can drive interaction and address these limitations. Here, we highlight potential usability and interaction limitations in MR and how physiologically adaptive systems can benefit MR experiences and applications. We specifically address potential applications for human factors and operational settings such as healthcare, education, and entertainment. We further discuss benefits and applications in light of ethical and privacy concerns. The use of physiologically adaptive systems in MR has the potential to revolutionize human-computer interactions and provide users with a more personalized and engaging experience.
△ Less
Submitted 31 March, 2023;
originally announced March 2023.
-
Leveraging Mobile Sensing Technology for Societal Change Towards more Sustainable Behavior
Authors:
Florian Bemmann,
Carmen Mayer,
Sven Mayer
Abstract:
A pro-environmental attitude in the general population is essential to combat climate change. Society as a whole has the power to change economic processes through market demands and to exert pressure on policymakers - both are key social factors that currently undermine the goals of decarbonization. Creating long-lasting, sustainable attitudes is challenging and behavior change technologies do ha…
▽ More
A pro-environmental attitude in the general population is essential to combat climate change. Society as a whole has the power to change economic processes through market demands and to exert pressure on policymakers - both are key social factors that currently undermine the goals of decarbonization. Creating long-lasting, sustainable attitudes is challenging and behavior change technologies do hard to overcome their limitations. Environmental psychology proposes social factors to be relevant, a.o. creating a global identity feeling and widening one's view beyond the own bubble. From our experience in the field of mobile sensing and psychometric data inferences, we see strong potential in mobile sensing technologies to implement the aforementioned goals. We present concrete ideas in this paper, aiming to refine and extend them with the workshop and evaluate them afterward.
△ Less
Submitted 22 March, 2023;
originally announced March 2023.
-
Understanding the Uncertainty Loop of Human-Robot Interaction
Authors:
Jan Leusmann,
Chao Wang,
Michael Gienger,
Albrecht Schmidt,
Sven Mayer
Abstract:
Recently the field of Human-Robot Interaction gained popularity, due to the wide range of possibilities of how robots can support humans during daily tasks. One form of supportive robots are socially assistive robots which are specifically built for communicating with humans, e.g., as service robots or personal companions. As they understand humans through artificial intelligence, these robots wil…
▽ More
Recently the field of Human-Robot Interaction gained popularity, due to the wide range of possibilities of how robots can support humans during daily tasks. One form of supportive robots are socially assistive robots which are specifically built for communicating with humans, e.g., as service robots or personal companions. As they understand humans through artificial intelligence, these robots will at some point make wrong assumptions about the humans' current state and give an unexpected response. In human-human conversations, unexpected responses happen frequently. However, it is currently unclear how such robots should act if they understand that the human did not expect their response, or even showing the uncertainty of their response in the first place. For this, we explore the different forms of potential uncertainties during human-robot conversations and how humanoids can, through verbal and non-verbal cues, communicate these uncertainties.
△ Less
Submitted 14 March, 2023;
originally announced March 2023.
-
Signifiers as a First-class Abstraction in Hypermedia Multi-Agent Systems
Authors:
Danai Vachtsevanou,
Andrei Ciortea,
Simon Mayer,
Jérémy Lemée
Abstract:
Hypermedia APIs enable the design of reusable hypermedia clients that discover and exploit affordances on the Web. However, the reusability of such clients remains limited since they cannot plan and reason about interaction. This paper provides a conceptual bridge between hypermedia-driven affordance exploitation on the Web and methods for representing and reasoning about actions that have been ex…
▽ More
Hypermedia APIs enable the design of reusable hypermedia clients that discover and exploit affordances on the Web. However, the reusability of such clients remains limited since they cannot plan and reason about interaction. This paper provides a conceptual bridge between hypermedia-driven affordance exploitation on the Web and methods for representing and reasoning about actions that have been extensively explored for Multi-Agent Systems (MAS) and, more broadly, Artificial Intelligence. We build on concepts and methods from Affordance Theory and Human-Computer Interaction that support interaction efficiency in open and evolvable environments to introduce signifiers as a first-class abstraction in Web-based MAS: Signifiers are designed with respect to the agent-environment context of their usage and enable agents with heterogeneous abilities to act and to reason about action. We define a formal model for the contextual exposure of signifiers in hypermedia environments that aims to drive affordance exploitation. We demonstrate our approach with a prototypical Web-based MAS where two agents with different reasoning abilities proactively discover how to interact with their environment by perceiving only the signifiers that fit their abilities. We show that signifier exposure can be inherently managed based on the dynamic agent-environment context towards facilitating effective and efficient interactions on the Web.
△ Less
Submitted 14 February, 2023;
originally announced February 2023.
-
The Impact of Expertise in the Loop for Exploring Machine Rationality
Authors:
Changkun Ou,
Sven Mayer,
Andreas Butz
Abstract:
Human-in-the-loop optimization utilizes human expertise to guide machine optimizers iteratively and search for an optimal solution in a solution space. While prior empirical studies mainly investigated novices, we analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We conducted a study (N=60) in text, photo, and 3D mesh optimization cont…
▽ More
Human-in-the-loop optimization utilizes human expertise to guide machine optimizers iteratively and search for an optimal solution in a solution space. While prior empirical studies mainly investigated novices, we analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We conducted a study (N=60) in text, photo, and 3D mesh optimization contexts. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization iteration with more explicit preference while kee** satisfaction low. In contrast, novices were more easily satisfied and terminated faster. Therefore, we identified that experts seek more diverse outcomes while the machine reaches optimal results, and the observed behavior can be used as a performance indicator for human-in-the-loop system designers to improve underlying models. We inform future research to be cautious about the impact of user expertise when designing human-in-the-loop systems.
△ Less
Submitted 11 February, 2023;
originally announced February 2023.
-
Investigating Labeler Bias in Face Annotation for Machine Learning
Authors:
Luke Haliburton,
Sinksar Ghebremedhin,
Robin Welsch,
Albrecht Schmidt,
Sven Mayer
Abstract:
In a world increasingly reliant on artificial intelligence, it is more important than ever to consider the ethical implications of artificial intelligence on humanity. One key under-explored challenge is labeler bias, which can create inherently biased datasets for training and subsequently lead to inaccurate or unfair decisions in healthcare, employment, education, and law enforcement. Hence, we…
▽ More
In a world increasingly reliant on artificial intelligence, it is more important than ever to consider the ethical implications of artificial intelligence on humanity. One key under-explored challenge is labeler bias, which can create inherently biased datasets for training and subsequently lead to inaccurate or unfair decisions in healthcare, employment, education, and law enforcement. Hence, we conducted a study to investigate and measure the existence of labeler bias using images of people from different ethnicities and sexes in a labeling task. Our results show that participants possess stereotypes that influence their decision-making process and that labeler demographics impact assigned labels. We also discuss how labeler bias influences datasets and, subsequently, the models trained on them. Overall, a high degree of transparency must be maintained throughout the entire artificial intelligence training process to identify and correct biases in the data as early as possible.
△ Less
Submitted 26 June, 2023; v1 submitted 24 January, 2023;
originally announced January 2023.
-
Standardized Medical Image Classification across Medical Disciplines
Authors:
Simone Mayer,
Dominik Müller,
Frank Kramer
Abstract:
AUCMEDI is a Python-based framework for medical image classification. In this paper, we evaluate the capabilities of AUCMEDI, by applying it to multiple datasets. Datasets were specifically chosen to cover a variety of medical disciplines and imaging modalities. We designed a simple pipeline using Jupyter notebooks and applied it to all datasets. Results show that AUCMEDI was able to train a model…
▽ More
AUCMEDI is a Python-based framework for medical image classification. In this paper, we evaluate the capabilities of AUCMEDI, by applying it to multiple datasets. Datasets were specifically chosen to cover a variety of medical disciplines and imaging modalities. We designed a simple pipeline using Jupyter notebooks and applied it to all datasets. Results show that AUCMEDI was able to train a model with accurate classification capabilities for each dataset: Averaged AUC per dataset range between 0.82 and 1.0, averaged F1 scores range between 0.61 and 1.0. With its high adaptability and strong performance, AUCMEDI proves to be a powerful instrument to build widely applicable neural networks. The notebooks serve as application examples for AUCMEDI.
△ Less
Submitted 20 October, 2022;
originally announced October 2022.
-
PCA-based Channel Estimation for MIMO Communications
Authors:
Jonathan Aguiar Soares,
Kayol Soares Mayer,
Pedro Benevenuto Valadares,
Dalton Soares Arantes
Abstract:
In multiple-input multiple-output communications, channel estimation is paramount to keep base stations and users on track. This paper proposes a novel PCA-based-principal component analysis-channel estimation approach for MIMO orthogonal frequency division multiplexing systems. The channel frequency response is firstly estimated with the least squares method, and then PCA is used to filter only t…
▽ More
In multiple-input multiple-output communications, channel estimation is paramount to keep base stations and users on track. This paper proposes a novel PCA-based-principal component analysis-channel estimation approach for MIMO orthogonal frequency division multiplexing systems. The channel frequency response is firstly estimated with the least squares method, and then PCA is used to filter only the higher singular components of the channel impulse response, which is then converted back to the frequency domain. The proposed approach is compared with the MMSE, the minimum mean square error estimation, in terms of bit error rate versus Eb/N0.
△ Less
Submitted 27 September, 2022;
originally announced September 2022.
-
A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences?
Authors:
Jordan Aiko Deja,
Sven Mayer,
Klen Čopič Pucihar,
Matjaž Kljun
Abstract:
Humans have been develo** and playing musical instruments for millennia. With technological advancements, instruments were becoming ever more sophisticated. In recent decades computer-supported innovations have also been introduced in hardware design, usability, and aesthetics. One of the most commonly digitally augmented instruments is the piano. Besides electronic keyboards, several prototypes…
▽ More
Humans have been develo** and playing musical instruments for millennia. With technological advancements, instruments were becoming ever more sophisticated. In recent decades computer-supported innovations have also been introduced in hardware design, usability, and aesthetics. One of the most commonly digitally augmented instruments is the piano. Besides electronic keyboards, several prototypes augmenting pianos with different projections providing various levels of interactivity on and around the keyboard have been implemented in order to support piano players. However, it is still not understood if these solutions are indeed supporting the learning process. In this paper we present a systematic review of augmented piano prototypes focusing on instrument learning, which is based on the four themes derived from interviews of piano experts to better understand the problems of teaching the piano. These themes are: (i) synchronised movement and body posture, (ii) sight-reading, (iii) ensuring motivation, and (iv) encouraging improvisation. We found that prototypes are saturated on the synchronisation themes, and there are opportunities for sight-reading, motivation, and improvisation themes. We conclude by presenting recommendations on augmenting piano systems towards enriching the piano learning experience as well as on possible directions to expand knowledge in the area.
△ Less
Submitted 3 November, 2022; v1 submitted 21 August, 2022;
originally announced August 2022.
-
Current Challenges of Using Wearable Devices for Online Emotion Sensing
Authors:
Weiwei Jiang,
Kangning Yang,
Maximiliane Windl,
Francesco Chiossi,
Benjamin Tag,
Sven Mayer,
Zhanna Sarsenbayeva
Abstract:
A growing number of wearable devices is becoming increasingly non-invasive, readily available, and versatile for measuring different physiological signals. This renders them ideal for inferring the emotional states of their users. Despite the success of wearable devices in recent emotion studies, there are still several challenges to be addressed. In this position paper, we compare currently avail…
▽ More
A growing number of wearable devices is becoming increasingly non-invasive, readily available, and versatile for measuring different physiological signals. This renders them ideal for inferring the emotional states of their users. Despite the success of wearable devices in recent emotion studies, there are still several challenges to be addressed. In this position paper, we compare currently available wearables that can be used for emotion-sensing and identify the challenges and opportunities for future researchers. Our investigation opens the discussion of what is missing for in-the-wild for emotion-sensing studies.
△ Less
Submitted 10 August, 2022;
originally announced August 2022.
-
The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop Failures
Authors:
Changkun Ou,
Daniel Buschek,
Sven Mayer,
Andreas Butz
Abstract:
Interactive AI systems increasingly employ a human-in-the-loop strategy. This creates new challenges for the HCI community when designing such systems. We reveal and investigate some of these challenges in a case study with an industry partner, and developed a prototype human-in-the-loop system for preference-guided 3D model processing. Two 3D artists used it in their daily work for 3 months. We f…
▽ More
Interactive AI systems increasingly employ a human-in-the-loop strategy. This creates new challenges for the HCI community when designing such systems. We reveal and investigate some of these challenges in a case study with an industry partner, and developed a prototype human-in-the-loop system for preference-guided 3D model processing. Two 3D artists used it in their daily work for 3 months. We found that the human-AI loop often did not converge towards a satisfactory result and designed a lab study (N=20) to investigate this further. We analyze interaction data and user feedback through the lens of theories of human judgment to explain the observed human-in-the-loop failures with two key insights: 1) optimization using preferential choices lacks mechanisms to deal with inconsistent and contradictory human judgments; 2) machine outcomes, in turn, influence future user inputs via heuristic biases and loss aversion. To mitigate these problems, we propose descriptive UI design guidelines. Our case study draws attention to challenging and practically relevant imperfections in human-AI loops that need to be considered when designing human-in-the-loop systems.
△ Less
Submitted 26 July, 2022;
originally announced July 2022.
-
The Vision of a Human-Centered Piano
Authors:
Jordan Aiko Deja,
Sven Mayer,
Klen Čopič Pucihar,
Matjaž Kljun
Abstract:
For around 300 years, humans have been learning to play the modern piano either with a teacher or on their own. In recent years teaching and learning have been enhanced using augmented technologies that support novices. Other technologies have also tried to improve different use cases with the piano, such as composing and performing. Researchers and practitioners have showcased several forms of au…
▽ More
For around 300 years, humans have been learning to play the modern piano either with a teacher or on their own. In recent years teaching and learning have been enhanced using augmented technologies that support novices. Other technologies have also tried to improve different use cases with the piano, such as composing and performing. Researchers and practitioners have showcased several forms of augmentation, from hardware improvements, sound quality, rendering projected visualizations to gesture-based and immersive technologies. Today, the landscape of piano augmentations is very diverse, and it is unclear how to describe the ideal piano and its features. In this work, we discuss how the human-centered piano -- the piano that has been designed with humans in the center of the design process and that effectively supports tasks performed on it -- can support pianists. In detail, we present the three tasks of learning, composing, and improvising in which a human-centered piano would be beneficial for the pianist.
△ Less
Submitted 14 April, 2022;
originally announced April 2022.
-
Neural Photofit: Gaze-based Mental Image Reconstruction
Authors:
Florian Strohm,
Ekta Sood,
Sven Mayer,
Philipp Müller,
Mihai Bâce,
Andreas Bulling
Abstract:
We propose a novel method that leverages human fixations to visually decode the image a person has in mind into a photofit (facial composite). Our method combines three neural networks: An encoder, a scoring network, and a decoder. The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer. A neural scoring network compares the human and ne…
▽ More
We propose a novel method that leverages human fixations to visually decode the image a person has in mind into a photofit (facial composite). Our method combines three neural networks: An encoder, a scoring network, and a decoder. The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer. A neural scoring network compares the human and neural attention and predicts a relevance score for each extracted image feature. Finally, image features are aggregated into a single feature vector as a linear combination of all features weighted by relevance which a decoder decodes into the final photofit. We train the neural scoring network on a novel dataset containing gaze data of 19 participants looking at collages of synthetic faces. We show that our method significantly outperforms a mean baseline predictor and report on a human study that shows that we can decode photofits that are visually plausible and close to the observer's mental image.
△ Less
Submitted 17 August, 2021;
originally announced August 2021.
-
Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
Authors:
Laura von Rueden,
Sebastian Mayer,
Katharina Beckh,
Bogdan Georgiev,
Sven Giesselbach,
Raoul Heese,
Birgit Kirsch,
Julius Pfrommer,
Annika Pick,
Rajkumar Ramamurthy,
Michal Walczak,
Jochen Garcke,
Christian Bauckhage,
Jannis Schuecker
Abstract:
Despite its great success, machine learning can have its limits when dealing with insufficient training data. A potential solution is the additional integration of prior knowledge into the training process which leads to the notion of informed machine learning. In this paper, we present a structured overview of various approaches in this field. We provide a definition and propose a concept for inf…
▽ More
Despite its great success, machine learning can have its limits when dealing with insufficient training data. A potential solution is the additional integration of prior knowledge into the training process which leads to the notion of informed machine learning. In this paper, we present a structured overview of various approaches in this field. We provide a definition and propose a concept for informed machine learning which illustrates its building blocks and distinguishes it from conventional machine learning. We introduce a taxonomy that serves as a classification framework for informed machine learning approaches. It considers the source of knowledge, its representation, and its integration into the machine learning pipeline. Based on this taxonomy, we survey related research and describe how different knowledge representations such as algebraic equations, logic rules, or simulation results can be used in learning systems. This evaluation of numerous papers on the basis of our taxonomy uncovers key methods in the field of informed machine learning.
△ Less
Submitted 28 May, 2021; v1 submitted 29 March, 2019;
originally announced March 2019.
-
Research and Education in Computational Science and Engineering
Authors:
Ulrich Rüde,
Karen Willcox,
Lois Curfman McInnes,
Hans De Sterck,
George Biros,
Hans Bungartz,
James Corones,
Evin Cramer,
James Crowley,
Omar Ghattas,
Max Gunzburger,
Michael Hanke,
Robert Harrison,
Michael Heroux,
Jan Hesthaven,
Peter Jimack,
Chris Johnson,
Kirk E. Jordan,
David E. Keyes,
Rolf Krause,
Vipin Kumar,
Stefan Mayer,
Juan Meza,
Knut Martin Mørken,
J. Tinsley Oden
, et al. (8 additional authors not shown)
Abstract:
Over the past two decades the field of computational science and engineering (CSE) has penetrated both basic and applied research in academia, industry, and laboratories to advance discovery, optimize systems, support decision-makers, and educate the scientific and engineering workforce. Informed by centuries of theory and experiment, CSE performs computational experiments to answer questions that…
▽ More
Over the past two decades the field of computational science and engineering (CSE) has penetrated both basic and applied research in academia, industry, and laboratories to advance discovery, optimize systems, support decision-makers, and educate the scientific and engineering workforce. Informed by centuries of theory and experiment, CSE performs computational experiments to answer questions that neither theory nor experiment alone is equipped to answer. CSE provides scientists and engineers of all persuasions with algorithmic inventions and software systems that transcend disciplines and scales. Carried on a wave of digital technology, CSE brings the power of parallelism to bear on troves of data. Mathematics-based advanced computing has become a prevalent means of discovery and innovation in essentially all areas of science, engineering, technology, and society; and the CSE community is at the core of this transformation. However, a combination of disruptive developments---including the architectural complexity of extreme-scale computing, the data revolution that engulfs the planet, and the specialization required to follow the applications to new frontiers---is redefining the scope and reach of the CSE endeavor. This report describes the rapid expansion of CSE and the challenges to sustaining its bold advances. The report also presents strategies and directions for CSE research and education for the next decade.
△ Less
Submitted 31 December, 2017; v1 submitted 8 October, 2016;
originally announced October 2016.
-
Measuring Visibility using Atmospheric Transmission and Digital Surface Model
Authors:
Jean-Philippe Andreu,
Stefan Mayer,
Karlheinz Gutjahr,
Harald Ganster
Abstract:
Reliable and exact assessment of visibility is essential for safe air traffic. In order to overcome the drawbacks of the currently subjective reports from human observers, we present an approach to automatically derive visibility measures by means of image processing. It first exploits image based estimation of the atmospheric transmission describing the portion of the light that is not scattered…
▽ More
Reliable and exact assessment of visibility is essential for safe air traffic. In order to overcome the drawbacks of the currently subjective reports from human observers, we present an approach to automatically derive visibility measures by means of image processing. It first exploits image based estimation of the atmospheric transmission describing the portion of the light that is not scattered by atmospheric phenomena (e.g., haze, fog, smoke) and reaches the camera. Once the atmospheric transmission is estimated, a 3D representation of the vicinity (digital surface model: DMS) is used to compute depth measurements for the haze-free pixels and then derive a global visibility estimation for the airport. Results on foggy images demonstrate the validity of the proposed method.
△ Less
Submitted 20 May, 2015;
originally announced May 2015.
