Showing 1–50 of 71 results for author: Muller, F

LLMs and Memorization: On Quality and Specificity of Copyright Compliance

Authors: Felix B Mueller, Rebekka Görge, Anna K Bernzen, Janna C Pirk, Maximilian Poretschkin

Abstract: Memorization in large language models (LLMs) is a growing concern. LLMs have been shown to easily reproduce parts of their training data, including copyrighted work. This is an important problem to solve, as it may violate existing copyright laws as well as the European AI Act. In this work, we propose a systematic analysis to quantify the extent of potential copyright infringements in LLMs using… ▽ More Memorization in large language models (LLMs) is a growing concern. LLMs have been shown to easily reproduce parts of their training data, including copyrighted work. This is an important problem to solve, as it may violate existing copyright laws as well as the European AI Act. In this work, we propose a systematic analysis to quantify the extent of potential copyright infringements in LLMs using European law as an example. Unlike previous work, we evaluate instruction-finetuned models in a realistic end-user scenario. Our analysis builds on a proposed threshold of 160 characters, which we borrow from the German Copyright Service Provider Act and a fuzzy text matching algorithm to identify potentially copyright-infringing textual reproductions. The specificity of countermeasures against copyright infringement is analyzed by comparing model behavior on copyrighted and public domain data. We investigate what behaviors models show instead of producing protected text (such as refusal or hallucination) and provide a first legal assessment of these behaviors. We find that there are huge differences in copyright compliance, specificity, and appropriate refusal among popular LLMs. Alpaca, GPT 4, GPT 3.5, and Luminous perform best in our comparison, with OpenGPT-X, Alpaca, and Luminous producing a particularly low absolute number of potential copyright violations. Code will be published soon. △ Less

Submitted 28 June, 2024; v1 submitted 28 May, 2024; originally announced May 2024.

Comments: 10 pages, 3 figures

ACM Class: I.2.7

Develo** trustworthy AI applications with foundation models

Authors: Michael Mock, Sebastian Schmidt, Felix Müller, Rebekka Görge, Anna Schmitz, Elena Haedecke, Angelika Voss, Dirk Hecker, Maximillian Poretschkin

Abstract: The trustworthiness of AI applications has been the subject of recent research and is also addressed in the EU's recently adopted AI Regulation. The currently emerging foundation models in the field of text, speech and image processing offer completely new possibilities for develo** AI applications. This whitepaper shows how the trustworthiness of an AI application developed with foundation mode… ▽ More The trustworthiness of AI applications has been the subject of recent research and is also addressed in the EU's recently adopted AI Regulation. The currently emerging foundation models in the field of text, speech and image processing offer completely new possibilities for develo** AI applications. This whitepaper shows how the trustworthiness of an AI application developed with foundation models can be evaluated and ensured. For this purpose, the application-specific, risk-based approach for testing and ensuring the trustworthiness of AI applications, as developed in the 'AI Assessment Catalog - Guideline for Trustworthy Artificial Intelligence' by Fraunhofer IAIS, is transferred to the context of foundation models. Special consideration is given to the fact that specific risks of foundation models can have an impact on the AI application and must also be taken into account when checking trustworthiness. Chapter 1 of the white paper explains the fundamental relationship between foundation models and AI applications based on them in terms of trustworthiness. Chapter 2 provides an introduction to the technical construction of foundation models and Chapter 3 shows how AI applications can be developed based on them. Chapter 4 provides an overview of the resulting risks regarding trustworthiness. Chapter 5 shows which requirements for AI applications and foundation models are to be expected according to the draft of the European Union's AI Regulation and Chapter 6 finally shows the system and procedure for meeting trustworthiness requirements. △ Less

Submitted 8 May, 2024; originally announced May 2024.

Comments: 24 pages, 11 figures

ACM Class: I.2.0

DiaQ: Efficient State-Vector Quantum Simulation

Authors: Srikar Chundury, Jiajia Li, In-Saeng Suh, Frank Mueller

Abstract: In the current era of Noisy Intermediate Scale Quantum (NISQ) computing, efficient digital simulation of quantum systems holds significant importance for quantum algorithm development, verification and validation. However, analysis of sparsity within these simulations remains largely unexplored. In this paper, we present a novel observation regarding the prevalent sparsity patterns inherent in qua… ▽ More In the current era of Noisy Intermediate Scale Quantum (NISQ) computing, efficient digital simulation of quantum systems holds significant importance for quantum algorithm development, verification and validation. However, analysis of sparsity within these simulations remains largely unexplored. In this paper, we present a novel observation regarding the prevalent sparsity patterns inherent in quantum circuits. We introduce DiaQ, a new sparse matrix format tailored to exploit this quantum-specific sparsity, thereby enhancing simulation performance. Our contribution extends to the development of libdiaq, a numerical library implemented in C++ with OpenMP for multi-core acceleration and SIMD vectorization, featuring essential mathematical kernels for digital quantum simulations. Furthermore, we integrate DiaQ with SV-Sim, a state vector simulator, yielding substantial performance improvements across various quantum circuits (e.g., ~26.67% for GHZ-28 and ~32.72% for QFT-29 with multi-core parallelization and SIMD vectorization on Frontier). Evaluations conducted on benchmarks from SupermarQ and QASMBench demonstrate that DiaQ represents a significant step towards achieving highly efficient quantum simulations. △ Less

Submitted 30 April, 2024; originally announced May 2024.

Comments: 11 pages, 8 figures

Just Undo It: Exploring Undo Mechanics in Multi-User Virtual Reality

Authors: Julian Rasch, Florian Perzl, Yannick Weiss, Florian Müller

Abstract: With the proliferation of VR and a metaverse on the horizon, many multi-user activities are migrating to the VR world, calling for effective collaboration support. As one key feature, traditional collaborative systems provide users with undo mechanics to reverse errors and other unwanted changes. While undo has been extensively researched in this domain and is now considered industry standard, it… ▽ More With the proliferation of VR and a metaverse on the horizon, many multi-user activities are migrating to the VR world, calling for effective collaboration support. As one key feature, traditional collaborative systems provide users with undo mechanics to reverse errors and other unwanted changes. While undo has been extensively researched in this domain and is now considered industry standard, it is strikingly absent for VR systems in research and industry. This work addresses this research gap by exploring different undo techniques for basic object manipulation in different collaboration modes in VR. We conducted a study involving 32 participants organized in teams of two. Here, we studied users' performance and preferences in a tower stacking task, varying the available undo techniques and their mode of collaboration. The results suggest that users desire and use undo in VR and that the choice of the undo technique impacts users' performance and social connection. △ Less

Submitted 18 March, 2024; originally announced March 2024.

Comments: To appear in Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24), May 11-16, 2024, Honolulu, HI, USA

GustosonicSense: Towards understanding the design of playful gustosonic eating experiences

Authors: Yan Wang, Humphrey O. Obie, Zhuying Li, Flora D. Salim, John Grundy, Florian 'Floyd' Mueller

Abstract: The pleasure that often comes with eating can be further enhanced with intelligent technology, as the field of human-food interaction suggests. However, knowledge on how to design such pleasure-supporting eating systems is limited. To begin filling this knowledge gap, we designed "GustosonicSense", a novel gustosonic eating system that utilizes wireless earbuds for sensing different eating and dri… ▽ More The pleasure that often comes with eating can be further enhanced with intelligent technology, as the field of human-food interaction suggests. However, knowledge on how to design such pleasure-supporting eating systems is limited. To begin filling this knowledge gap, we designed "GustosonicSense", a novel gustosonic eating system that utilizes wireless earbuds for sensing different eating and drinking actions with a machine learning algorithm and trigger playful sounds as a way to facilitate pleasurable eating experiences. We present the findings from our design and a study that revealed how we can support the "stimulation", "hedonism", and "reflexivity" for playful human-food interactions. Ultimately, with our work, we aim to support interaction designers in facilitating playful experiences with food. △ Less

Submitted 16 March, 2024; originally announced March 2024.

Comments: To appear at CHI'24: The ACM Conference on Human Factors in Computing Systems (CHI), Honolulu, Hawaii, 2024

MACS: Mass Conditioned 3D Hand and Object Motion Synthesis

Authors: Soshi Shimada, Franziska Mueller, Jan Bednarik, Bardia Doosti, Bernd Bickel, Danhang Tang, Vladislav Golyanik, Jonathan Taylor, Christian Theobalt, Thabo Beeler

Abstract: The physical properties of an object, such as mass, significantly affect how we manipulate it with our hands. Surprisingly, this aspect has so far been neglected in prior work on 3D motion synthesis. To improve the naturalness of the synthesized 3D hand object motions, this work proposes MACS the first MAss Conditioned 3D hand and object motion Synthesis approach. Our approach is based on cascaded… ▽ More The physical properties of an object, such as mass, significantly affect how we manipulate it with our hands. Surprisingly, this aspect has so far been neglected in prior work on 3D motion synthesis. To improve the naturalness of the synthesized 3D hand object motions, this work proposes MACS the first MAss Conditioned 3D hand and object motion Synthesis approach. Our approach is based on cascaded diffusion models and generates interactions that plausibly adjust based on the object mass and interaction type. MACS also accepts a manually drawn 3D object trajectory as input and synthesizes the natural 3D hand motions conditioned by the object mass. This flexibility enables MACS to be used for various downstream applications, such as generating synthetic training data for ML tasks, fast animation of hands for graphics workflows, and generating character interactions for computer games. We show experimentally that a small-scale dataset is sufficient for MACS to reasonably generalize across interpolated and extrapolated object masses unseen during the training. Furthermore, MACS shows moderate generalization to unseen objects, thanks to the mass-conditioned contact labels generated by our surface contact synthesis model ConNet. Our comprehensive user study confirms that the synthesized 3D hand-object interactions are highly plausible and realistic. △ Less

Submitted 22 December, 2023; originally announced December 2023.

Machine Culture

Authors: Levin Brinkmann, Fabian Baumann, Jean-François Bonnefon, Maxime Derex, Thomas F. Müller, Anne-Marie Nussberger, Agnieszka Czaplicka, Alberto Acerbi, Thomas L. Griffiths, Joseph Henrich, Joel Z. Leibo, Richard McElreath, Pierre-Yves Oudeyer, Jonathan Stray, Iyad Rahwan

Abstract: The ability of humans to create and disseminate culture is often credited as the single most important factor of our success as a species. In this Perspective, we explore the notion of machine culture, culture mediated or generated by machines. We argue that intelligent machines simultaneously transform the cultural evolutionary processes of variation, transmission, and selection. Recommender algo… ▽ More The ability of humans to create and disseminate culture is often credited as the single most important factor of our success as a species. In this Perspective, we explore the notion of machine culture, culture mediated or generated by machines. We argue that intelligent machines simultaneously transform the cultural evolutionary processes of variation, transmission, and selection. Recommender algorithms are altering social learning dynamics. Chatbots are forming a new mode of cultural transmission, serving as cultural models. Furthermore, intelligent machines are evolving as contributors in generating cultural traits--from game strategies and visual art to scientific results. We provide a conceptual framework for studying the present and anticipated future impact of machines on cultural evolution, and present a research agenda for the study of machine culture. △ Less

Submitted 22 November, 2023; v1 submitted 19 November, 2023; originally announced November 2023.

Journal ref: Nat Hum Behav 7, 1855-1868 (2023)

A Ferroelectric Compute-in-Memory Annealer for Combinatorial Optimization Problems

Authors: Xunzhao Yin, Yu Qian, Alptekin Vardar, Marcel Gunther, Franz Muller, Nellie Laleni, Zijian Zhao, Zhouhang Jiang, Zhiguo Shi, Yiyu Shi, Xiao Gong, Cheng Zhuo, Thomas Kampfe, Kai Ni

Abstract: Computationally hard combinatorial optimization problems (COPs) are ubiquitous in many applications, including logistical planning, resource allocation, chip design, drug explorations, and more. Due to their critical significance and the inability of conventional hardware in efficiently handling scaled COPs, there is a growing interest in develo** computing hardware tailored specifically for COP… ▽ More Computationally hard combinatorial optimization problems (COPs) are ubiquitous in many applications, including logistical planning, resource allocation, chip design, drug explorations, and more. Due to their critical significance and the inability of conventional hardware in efficiently handling scaled COPs, there is a growing interest in develo** computing hardware tailored specifically for COPs, including digital annealers, dynamical Ising machines, and quantum/photonic systems. However, significant hurdles still remain, such as the memory access issue, the system scalability and restricted applicability to certain types of COPs, and VLSI-incompatibility, respectively. Here, a ferroelectric field effect transistor (FeFET) based compute-in-memory (CiM) annealer is proposed. After converting COPs into quadratic unconstrained binary optimization (QUBO) formulations, a hardware-algorithm co-design is conducted, yielding an energy-efficient, versatile, and scalable hardware for COPs. To accelerate the core vector-matrix-vector (VMV) multiplication of QUBO formulations, a FeFET based CiM array is exploited, which can accelerate the intended operation in-situ due to its unique three-terminal structure. In particular, a lossless compression technique is proposed to prune typically sparse QUBO matrix to reduce hardware cost. Furthermore, a multi-epoch simulated annealing (MESA) algorithm is proposed to replace conventional simulated annealing for its faster convergence and better solution quality. The effectiveness of the proposed techniques is validated through the utilization of developed chip prototypes for successfully solving graph coloring problem, indicating great promise of FeFET CiM annealer in solving general COPs. △ Less

Submitted 24 September, 2023; originally announced September 2023.

Comments: 39 pages, 12 figures

Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images

Authors: Tze Ho Elden Tse, Franziska Mueller, Zhengyang Shen, Danhang Tang, Thabo Beeler, Mingsong Dou, Yinda Zhang, Sasa Petrovic, Hyung ** Chang, Jonathan Taylor, Bardia Doosti

Abstract: We propose a novel transformer-based framework that reconstructs two high fidelity hands from multi-view RGB images. Unlike existing hand pose estimation methods, where one typically trains a deep network to regress hand model parameters from single RGB image, we consider a more challenging problem setting where we directly regress the absolute root poses of two-hands with extended forearm at high… ▽ More We propose a novel transformer-based framework that reconstructs two high fidelity hands from multi-view RGB images. Unlike existing hand pose estimation methods, where one typically trains a deep network to regress hand model parameters from single RGB image, we consider a more challenging problem setting where we directly regress the absolute root poses of two-hands with extended forearm at high resolution from egocentric view. As existing datasets are either infeasible for egocentric viewpoints or lack background variations, we create a large-scale synthetic dataset with diverse scenarios and collect a real dataset from multi-calibrated camera setup to verify our proposed multi-view image feature fusion strategy. To make the reconstruction physically plausible, we propose two strategies: (i) a coarse-to-fine spectral graph convolution decoder to smoothen the meshes during upsampling and (ii) an optimisation-based refinement stage at inference to prevent self-penetrations. Through extensive quantitative and qualitative evaluations, we show that our framework is able to produce realistic two-hand reconstructions and demonstrate the generalisation of synthetic-trained models to real data, as well as real-time AR/VR applications. △ Less

Submitted 21 August, 2023; originally announced August 2023.

Comments: Accepted to ICCV 2023

Fused Spectatorship: Designing Bodily Experiences Where Spectators Become Players

Authors: Rakesh Patibanda, Aryan Saini, Nathalie Overdevest, Maria F. Montoya, Xiang Li, Yuzheng Chen, Shreyas Nisal, Josh Andres, Jarrod Knibbe, Elise van den Hoven, Florian 'Floyd' Mueller

Abstract: Spectating digital games can be exciting. However, due to its vicarious nature, spectators often wish to engage in the gameplay beyond just watching and cheering. To blur the boundaries between spectators and players, we propose a novel approach called "Fused Spectatorship", where spectators watch their hands play games by loaning bodily control to a computational Electrical Muscle Stimulation (EM… ▽ More Spectating digital games can be exciting. However, due to its vicarious nature, spectators often wish to engage in the gameplay beyond just watching and cheering. To blur the boundaries between spectators and players, we propose a novel approach called "Fused Spectatorship", where spectators watch their hands play games by loaning bodily control to a computational Electrical Muscle Stimulation (EMS) system. To showcase this concept, we designed three games where spectators loan control over both their hands to the EMS system and watch them play these competitive and collaborative games. A study with 12 participants suggested that participants could not distinguish if they were watching their hands play, or if they were playing the games themselves. We used our results to articulate four spectator experience themes and four fused spectator types, the behaviours they elicited and offer one design consideration to support each of these behaviours. We also discuss the ethical design considerations of our approach to help game designers create future fused spectatorship experiences. △ Less

Submitted 20 September, 2023; v1 submitted 20 July, 2023; originally announced July 2023.

Comments: This paper is going to be published at Annual Symposium on Computer-Human Interaction in Play (CHI PLAY) 2023

Journal ref: Annual Symposium on Computer-Human Interaction in Play (CHI PLAY) 2023

Auto-Paízo Games: Towards Understanding the Design of Games that Aim to Unify a Player's Physical Body and the Virtual World

Authors: Rakesh Patibanda, Chris Hill, Aryan Saini, Xiang Li, Yuzheng Chen, Andrii Matviienko, Jarrod Knibbe, Elise van den Hoven, Florian 'Floyd' Mueller

Abstract: Most digital bodily games focus on the body as they use movement as input. However, they also draw the player's focus away from the body as the output occurs on visual displays, creating a divide between the physical body and the virtual world. We propose a novel approach - the "Body as a Play Material" - where a player uses their body as both input and output to unify the physical body and the vi… ▽ More Most digital bodily games focus on the body as they use movement as input. However, they also draw the player's focus away from the body as the output occurs on visual displays, creating a divide between the physical body and the virtual world. We propose a novel approach - the "Body as a Play Material" - where a player uses their body as both input and output to unify the physical body and the virtual world. To showcase this approach, we designed three games where a player uses one of their hands (input) to play against the other hand (output) by loaning control over its movements to an Electrical Muscle Stimulation (EMS) system. We conducted a thematic analysis on the data obtained from a field study with 12 participants to articulate four player experience themes. We discuss our results about how participants appreciated the engagement with the variety of bodily movements for play and the ambiguity of using their body as a play material. Ultimately, our work aims to unify the physical body and the virtual world. △ Less

Submitted 20 September, 2023; v1 submitted 20 July, 2023; originally announced July 2023.

Comments: This paper is published at the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY) 2023

Journal ref: Annual Symposium on Computer-Human Interaction in Play (CHI PLAY) 2023

Image Denoising and the Generative Accumulation of Photons

Authors: Alexander Krull, Hector Basevi, Benjamin Salmon, Andre Zeug, Franziska Müller, Samuel Tonks, Leela Muppala, Ales Leonardis

Abstract: We present a fresh perspective on shot noise corrupted images and noise removal. By viewing image formation as the sequential accumulation of photons on a detector grid, we show that a network trained to predict where the next photon could arrive is in fact solving the minimum mean square error (MMSE) denoising task. This new perspective allows us to make three contributions: We present a new stra… ▽ More We present a fresh perspective on shot noise corrupted images and noise removal. By viewing image formation as the sequential accumulation of photons on a detector grid, we show that a network trained to predict where the next photon could arrive is in fact solving the minimum mean square error (MMSE) denoising task. This new perspective allows us to make three contributions: We present a new strategy for self-supervised denoising, We present a new method for sampling from the posterior of possible solutions by iteratively sampling and adding small numbers of photons to the image. We derive a full generative model by starting this process from an empty canvas. We call this approach generative accumulation of photons (GAP). We evaluate our method quantitatively and qualitatively on 4 new fluorescence microscopy datasets, which will be made available to the community. We find that it outperforms supervised, self-supervised and unsupervised baselines or performs on-par. △ Less

Submitted 1 August, 2023; v1 submitted 13 July, 2023; originally announced July 2023.

Comments: Paper with supplement. Typos corrected

MSC Class: 68

Digital Modeling for Everyone: Exploring How Novices Approach Voice-Based 3D Modeling

Authors: Giuseppe Desolda, Andrea Esposito, Florian Müller, Sebastian Feger

Abstract: Manufacturing tools like 3D printers have become accessible to the wider society, making the promise of digital fabrication for everyone seemingly reachable. While the actual manufacturing process is largely automated today, users still require knowledge of complex design applications to produce ready-designed objects and adapt them to their needs or design new objects from scratch. To lower the b… ▽ More Manufacturing tools like 3D printers have become accessible to the wider society, making the promise of digital fabrication for everyone seemingly reachable. While the actual manufacturing process is largely automated today, users still require knowledge of complex design applications to produce ready-designed objects and adapt them to their needs or design new objects from scratch. To lower the barrier to the design and customization of personalized 3D models, we explored novice mental models in voice-based 3D modeling by conducting a high-fidelity Wizard of Oz study with 22 participants. We performed a thematic analysis of the collected data to understand how the mental model of novices translates into voice-based 3D modeling. We conclude with design implications for voice assistants. For example, they have to: deal with vague, incomplete and wrong commands; provide a set of straightforward commands to shape simple and composite objects; and offer different strategies to select 3D objects. △ Less

Submitted 30 August, 2023; v1 submitted 10 July, 2023; originally announced July 2023.

Comments: Presented at INTERACT 2023. This is a self-archival version: the version of record is available via Springer at https://doi.org/10.1007/978-3-031-42293-5_11

ACM Class: H.5.2; I.2.1

Journal ref: J. Abdelnour Nocera et al. (Eds.): INTERACT 2023, LNCS 14145, pp. 133-155, 2023

Embedding Security into Ferroelectric FET Array via In-Situ Memory Operation

Authors: Yixin Xu, Yi Xiao, Zijian Zhao, Franz Müller, Alptekin Vardar, Xiao Gong, Sumitha George, Thomas Kämpfe, Vijaykrishnan Narayanan, Kai Ni

Abstract: Non-volatile memories (NVMs) have the potential to reshape next-generation memory systems because of their promising properties of near-zero leakage power consumption, high density and non-volatility. However, NVMs also face critical security threats that exploit the non-volatile property. Compared to volatile memory, the capability of retaining data even after power down makes NVM more vulnerable… ▽ More Non-volatile memories (NVMs) have the potential to reshape next-generation memory systems because of their promising properties of near-zero leakage power consumption, high density and non-volatility. However, NVMs also face critical security threats that exploit the non-volatile property. Compared to volatile memory, the capability of retaining data even after power down makes NVM more vulnerable. Existing solutions to address the security issues of NVMs are mainly based on Advanced Encryption Standard (AES), which incurs significant performance and power overhead. In this paper, we propose a lightweight memory encryption/decryption scheme by exploiting in-situ memory operations with negligible overhead. To validate the feasibility of the encryption/decryption scheme, device-level and array-level experiments are performed using ferroelectric field effect transistor (FeFET) as an example NVM without loss of generality. Besides, a comprehensive evaluation is performed on a 128x128 FeFET AND-type memory array in terms of area, latency, power and throughput. Compared with the AES-based scheme, our scheme shows around 22.6x/14.1x increase in encryption/decryption throughput with negligible power penalty. Furthermore, we evaluate the performance of our scheme over the AES-based scheme when deploying different neural network workloads. Our scheme yields significant latency reduction by 90% on average for encryption and decryption processes. △ Less

Submitted 2 June, 2023; originally announced June 2023.

In Sync: Exploring Synchronization to Increase Trust Between Humans and Non-humanoid Robots

Authors: Wieslaw Bartkowski, Andrzej Nowak, Filip Ignacy Czajkowski, Albrecht Schmidt, Florian Müller

Abstract: When we go for a walk with friends, we can observe an interesting effect: From step lengths to arm movements - our movements unconsciously align; they synchronize. Prior research found that this synchronization is a crucial aspect of human relations that strengthens social cohesion and trust. Generalizing from these findings in synchronization theory, we propose a dynamical approach that can be ap… ▽ More When we go for a walk with friends, we can observe an interesting effect: From step lengths to arm movements - our movements unconsciously align; they synchronize. Prior research found that this synchronization is a crucial aspect of human relations that strengthens social cohesion and trust. Generalizing from these findings in synchronization theory, we propose a dynamical approach that can be applied in the design of non-humanoid robots to increase trust. We contribute the results of a controlled experiment with 51 participants exploring our concept in a between-subjects design. For this, we built a prototype of a simple non-humanoid robot that can bend to follow human movements and vary the movement synchronization patterns. We found that synchronized movements lead to significantly higher ratings in an established questionnaire on trust between people and automation but did not influence the willingness to spend money in a trust game. △ Less

Submitted 6 April, 2023; v1 submitted 28 March, 2023; originally announced March 2023.

Comments: To appear in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI 23), April 23-28, 2023, Hamburg, Germany. ACM, New York, NY, USA, 14 pages

Journal ref: In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 367, 1-14

Around-Body Interaction: Leveraging Limb Movements for Interacting in a Digitally Augmented Physical World

Authors: Florian Müller

Abstract: Recent technological advances have made head-mounted displays (HMDs) smaller and untethered, fostering the vision of ubiquitous interaction with information in a digitally augmented physical world. For interacting with such devices, three main types of input - besides not very intuitive finger gestures - have emerged so far: 1) Touch input on the frame of the devices or 2) on accessories (controll… ▽ More Recent technological advances have made head-mounted displays (HMDs) smaller and untethered, fostering the vision of ubiquitous interaction with information in a digitally augmented physical world. For interacting with such devices, three main types of input - besides not very intuitive finger gestures - have emerged so far: 1) Touch input on the frame of the devices or 2) on accessories (controller) as well as 3) voice input. While these techniques have both advantages and disadvantages depending on the current situation of the user, they largely ignore the skills and dexterity that we show when interacting with the real world: Throughout our lives, we have trained extensively to use our limbs to interact with and manipulate the physical world around us. This thesis explores how the skills and dexterity of our upper and lower limbs, acquired and trained in interacting with the real world, can be transferred to the interaction with HMDs. Thus, this thesis develops the vision of around-body interaction, in which we use the space around our body, defined by the reach of our limbs, for fast, accurate, and enjoyable interaction with such devices. This work contributes four interaction techniques, two for the upper limbs and two for the lower limbs: The first contribution shows how the proximity between our head and hand can be used to interact with HMDs. The second contribution extends the interaction with the upper limbs to multiple users and illustrates how the registration of augmented information in the real world can support cooperative use cases. The third contribution shifts the focus to the lower limbs and discusses how foot taps can be leveraged as an input modality for HMDs. The fourth contribution presents how lateral shifts of the walking path can be exploited for mobile and hands-free interaction with HMDs while walking. △ Less

Submitted 28 March, 2023; originally announced March 2023.

Comments: thesis

TicTacToes: Assessing Toe Movements as an Input Modality

Authors: Florian Müller, Daniel Schmitt, Andrii Matviienko, Dominik Schön, Sebastian Günther, Thomas Kosch, Martin Schmitz

Abstract: From carrying grocery bags to holding onto handles on the bus, there are a variety of situations where one or both hands are busy, hindering the vision of ubiquitous interaction with technology. Voice commands, as a popular hands-free alternative, struggle with ambient noise and privacy issues. As an alternative approach, research explored movements of various body parts (e.g., head, arms) as inpu… ▽ More From carrying grocery bags to holding onto handles on the bus, there are a variety of situations where one or both hands are busy, hindering the vision of ubiquitous interaction with technology. Voice commands, as a popular hands-free alternative, struggle with ambient noise and privacy issues. As an alternative approach, research explored movements of various body parts (e.g., head, arms) as input modalities, with foot-based techniques proving particularly suitable for hands-free interaction. Whereas previous research only considered the movement of the foot as a whole, in this work, we argue that our toes offer further degrees of freedom that can be leveraged for interaction. To explore the viability of toe-based interaction, we contribute the results of a controlled experiment with 18 participants assessing the impact of five factors on the accuracy, efficiency and user experience of such interfaces. Based on the findings, we provide design recommendations for future toe-based interfaces. △ Less

Submitted 6 April, 2023; v1 submitted 28 March, 2023; originally announced March 2023.

Comments: To appear in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI 23), April 23-28, 2023, Hamburg, Germany. ACM, New York, NY, USA, 17 pages

Journal ref: In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 520, 1-17

UndoPort: Exploring the Influence of Undo-Actions for Locomotion in Virtual Reality on the Efficiency, Spatial Understanding and User Experience

Authors: Florian Müller, Arantxa Ye, Dominik Schön, Julian Rasch

Abstract: When we get lost in Virtual Reality (VR) or want to return to a previous location, we use the same methods of locomotion for the way back as for the way forward. This is time-consuming and requires additional physical orientation changes, increasing the risk of getting tangled in the headsets' cables. In this paper, we propose the use of undo actions to revert locomotion steps in VR. We explore ei… ▽ More When we get lost in Virtual Reality (VR) or want to return to a previous location, we use the same methods of locomotion for the way back as for the way forward. This is time-consuming and requires additional physical orientation changes, increasing the risk of getting tangled in the headsets' cables. In this paper, we propose the use of undo actions to revert locomotion steps in VR. We explore eight different variations of undo actions as extensions of point\&teleport, based on the possibility to undo position and orientation changes together with two different visualizations of the undo step (discrete and continuous). We contribute the results of a controlled experiment with 24 participants investigating the efficiency and orientation of the undo techniques in a radial maze task. We found that the combination of position and orientation undo together with a discrete visualization resulted in the highest efficiency without increasing orientation errors. △ Less

Submitted 6 April, 2023; v1 submitted 28 March, 2023; originally announced March 2023.

Comments: To appear in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI 23), April 23-28, 2023, Hamburg, Germany. ACM, New York, NY, USA, 15 pages

Journal ref: In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 234, 1-15

Data management and execution systems for the Rubin Observatory Science Pipelines

Authors: Nate B. Lust, Tim Jenness, James F. Bosch, Andrei Salnikov, Nathan M. Pease, Michelle Gower, Mikolaj Kowalik, Gregory P. Dubois-Felsmann, Fritz Mueller, Pim Schellart

Abstract: We present the Rubin Observatory system for data storage/retrieval and pipelined code execution. The layer for data storage and retrieval is named the Butler. It consists of a relational database, known as the registry, to keep track of metadata and relations, and a system to manage where the data is located, named the datastore. Together these systems create an abstraction layer that science algo… ▽ More We present the Rubin Observatory system for data storage/retrieval and pipelined code execution. The layer for data storage and retrieval is named the Butler. It consists of a relational database, known as the registry, to keep track of metadata and relations, and a system to manage where the data is located, named the datastore. Together these systems create an abstraction layer that science algorithms can be written against. This abstraction layer manages the complexities of the large data volumes expected and allows algorithms to be written independently, yet be tied together automatically into a coherent processing pipeline. This system consists of tools which execute these pipelines by transforming them into execution graphs which contain concrete data stored in the Butler. The pipeline infrastructure is designed to be scalable in nature, allowing execution on environments ranging from a laptop all the way up to multi-facility data centers. This presentation will focus on the data management aspects as well as an overview on the creation of pipelines and the corresponding execution graphs. △ Less

Submitted 6 March, 2023; originally announced March 2023.

Comments: 4 pages, submitted to Astronomical Data Analysis Software and Systems XXXII, October 2022

PointFISH -- learning point cloud representations for RNA localization patterns

Authors: Arthur Imbert, Florian Mueller, Thomas Walter

Abstract: Subcellular RNA localization is a critical mechanism for the spatial control of gene expression. Its mechanism and precise functional role is not yet very well understood. Single Molecule Fluorescence in Situ Hybridization (smFISH) images allow for the detection of individual RNA molecules with subcellular accuracy. In return, smFISH requires robust methods to quantify and classify RNA spatial dis… ▽ More Subcellular RNA localization is a critical mechanism for the spatial control of gene expression. Its mechanism and precise functional role is not yet very well understood. Single Molecule Fluorescence in Situ Hybridization (smFISH) images allow for the detection of individual RNA molecules with subcellular accuracy. In return, smFISH requires robust methods to quantify and classify RNA spatial distribution. Here, we present PointFISH, a novel computational approach for the recognition of RNA localization patterns. PointFISH is an attention-based network for computing continuous vector representations of RNA point clouds. Trained on simulations only, it can directly process extracted coordinates from experimental smFISH images. The resulting embedding allows scalable and flexible spatial transcriptomics analysis and matches performance of hand-crafted pipelines. △ Less

Submitted 21 February, 2023; originally announced February 2023.

Understanding Stationary and Moving Direct Skin Vibrotactile Stimulation on the Palm

Authors: Hesham Elsayed, Martin Weigel, Florian Müller, George Ibrahim, Jan Gugenheimer, Martin Schmitz, Sebastian Günther, Max Mühlhäuser

Abstract: Palm-based tactile displays have the potential to evolve from single motor interfaces (e.g., smartphones) to high-resolution tactile displays (e.g., back-of-device haptic interfaces) enabling richer multi-modal experiences with more information. However, we lack a systematic understanding of vibrotactile perception on the palm and the influence of various factors on the core design decisions of ta… ▽ More Palm-based tactile displays have the potential to evolve from single motor interfaces (e.g., smartphones) to high-resolution tactile displays (e.g., back-of-device haptic interfaces) enabling richer multi-modal experiences with more information. However, we lack a systematic understanding of vibrotactile perception on the palm and the influence of various factors on the core design decisions of tactile displays (number of actuators, resolution, and intensity). In a first experiment (N=16), we investigated the effect of these factors on the users' ability to localize stationary sensations. In a second experiment (N=20), we explored the influence of resolution on recognition rate for moving tactile sensations.Findings show that for stationary sensations a 9 actuator display offers a good trade-off and a $3\times3$ resolution can be accurately localized. For moving sensations, a $2\times4$ resolution led to the highest recognition accuracy, while $5\times10$ enables higher resolution output with a reasonable accuracy. △ Less

Submitted 17 February, 2023; originally announced February 2023.

Improving the Timing Resolution of Positron Emission Tomography Detectors Using Boosted Learning -- A Residual Physics Approach

Authors: Stephan Naunheim, Yannick Kuhl, David Schug, Volkmar Schulz, Florian Mueller

Abstract: Artificial intelligence (AI) is entering medical imaging, mainly enhancing image reconstruction. Nevertheless, improvements throughout the entire processing, from signal detection to computation, potentially offer significant benefits. This work presents a novel and versatile approach to detector optimization using machine learning (ML) and residual physics. We apply the concept to positron emissi… ▽ More Artificial intelligence (AI) is entering medical imaging, mainly enhancing image reconstruction. Nevertheless, improvements throughout the entire processing, from signal detection to computation, potentially offer significant benefits. This work presents a novel and versatile approach to detector optimization using machine learning (ML) and residual physics. We apply the concept to positron emission tomography (PET), intending to improve the coincidence time resolution (CTR). PET visualizes metabolic processes in the body by detecting photons with scintillation detectors. Improved CTR performance offers the advantage of reducing radioactive dose exposure for patients. Modern PET detectors with sophisticated concepts and read-out topologies represent complex physical and electronic systems requiring dedicated calibration techniques. Traditional methods primarily depend on analytical formulations successfully describing the main detector characteristics. However, when accounting for higher-order effects, additional complexities arise matching theoretical models to experimental reality. Our work addresses this challenge by combining traditional calibration with AI and residual physics, presenting a highly promising approach. We present a residual physics-based strategy using gradient tree boosting and physics-guided data generation. The explainable AI framework SHapley Additive exPlanations (SHAP) was used to identify known physical effects with learned patterns. In addition, the models were tested against basic physical laws. We were able to improve the CTR significantly (more than 20%) for clinically relevant detectors of 19 mm height, reaching CTRs of 185 ps (450-550 keV). △ Less

Submitted 26 October, 2023; v1 submitted 3 February, 2023; originally announced February 2023.

Software Architecture and System Design of Rubin Observatory

Authors: William O'Mullane, Frossie Economou, Kian-Tat Lim, Fritz Mueller, Tim Jenness, Gregory P. Dubois-Felsmann, Leanne P. Guy, Ian S. Sullivan, Yusra AlSayyad, John D. Swinbank, K. Simon Krughoff

Abstract: Starting from a description of the Rubin Observatory Data Management System Architecture, and drawing on our experience with and involvement in a range of other projects including Gaia, SDSS, UKIRT, and JCMT, we derive a series of generic design patterns and lessons learned. Starting from a description of the Rubin Observatory Data Management System Architecture, and drawing on our experience with and involvement in a range of other projects including Gaia, SDSS, UKIRT, and JCMT, we derive a series of generic design patterns and lessons learned. △ Less

Submitted 24 November, 2022; originally announced November 2022.

Comments: 10 pages ADASS XXXII submission

Report number: DMTN-240

Quadrupole Magnet Design based on Genetic Multi-Objective Optimization

Authors: Eric Diehl, Moritz von Tresckow, Lou Scholtissek, Dimitrios Loukrezis, Nicolas Marsic, Wolfgang F. O. Müller, Herbert De Gersem

Abstract: This work suggests to optimize the geometry of a quadrupole magnet by means of a genetic algorithm adapted to solve multi-objective optimization problems. To that end, a non-domination sorting genetic algorithm known as NSGA-III is used. The optimization objectives are chosen such that a high magnetic field quality in the aperture of the magnet is guaranteed, while simultaneously the magnet design… ▽ More This work suggests to optimize the geometry of a quadrupole magnet by means of a genetic algorithm adapted to solve multi-objective optimization problems. To that end, a non-domination sorting genetic algorithm known as NSGA-III is used. The optimization objectives are chosen such that a high magnetic field quality in the aperture of the magnet is guaranteed, while simultaneously the magnet design remains cost-efficient. The field quality is computed using a magnetostatic finite element model of the quadrupole, the results of which are post-processed and integrated into the optimization algorithm. An extensive analysis of the optimization results is performed, including Pareto front movements and identification of best designs. △ Less

Submitted 3 November, 2023; v1 submitted 17 November, 2022; originally announced November 2022.

Comments: 22 pages, 7 figures

HandFlow: Quantifying View-Dependent 3D Ambiguity in Two-Hand Reconstruction with Normalizing Flow

Authors: Jiayi Wang, Diogo Luvizon, Franziska Mueller, Florian Bernard, Adam Kortylewski, Dan Casas, Christian Theobalt

Abstract: Reconstructing two-hand interactions from a single image is a challenging problem due to ambiguities that stem from projective geometry and heavy occlusions. Existing methods are designed to estimate only a single pose, despite the fact that there exist other valid reconstructions that fit the image evidence equally well. In this paper we propose to address this issue by explicitly modeling the di… ▽ More Reconstructing two-hand interactions from a single image is a challenging problem due to ambiguities that stem from projective geometry and heavy occlusions. Existing methods are designed to estimate only a single pose, despite the fact that there exist other valid reconstructions that fit the image evidence equally well. In this paper we propose to address this issue by explicitly modeling the distribution of plausible reconstructions in a conditional normalizing flow framework. This allows us to directly supervise the posterior distribution through a novel determinant magnitude regularization, which is key to varied 3D hand pose samples that project well into the input image. We also demonstrate that metrics commonly used to assess reconstruction quality are insufficient to evaluate pose predictions under such severe ambiguity. To address this, we release the first dataset with multiple plausible annotations per image called MultiHands. The additional annotations enable us to evaluate the estimated distribution using the maximum mean discrepancy metric. Through this, we demonstrate the quality of our probabilistic reconstruction and show that explicit ambiguity modeling is better-suited for this challenging problem. △ Less

Submitted 4 October, 2022; originally announced October 2022.

Comments: VMV 2022 - Symposium on Vision, Modeling, and Visualization

A Homogeneous Processing Fabric for Matrix-Vector Multiplication and Associative Search Using Ferroelectric Time-Domain Compute-in-Memory

Authors: Xunzhao Yin, Qingrong Huang, Franz Müller, Shan Deng, Alptekin Vardar, Sourav De, Zhouhang Jiang, Mohsen Imani, Cheng Zhuo, Thomas Kämpfe, Kai Ni

Abstract: In this work, we propose a ferroelectric FET(FeFET) time-domain compute-in-memory (TD-CiM) array as a homogeneous processing fabric for binary multiplication-accumulation (MAC) and content addressable memory (CAM). We demonstrate that: i) the XOR(XNOR)/AND logic function can be realized using a single cell composed of 2FeFETs connected in series; ii) a two-phase computation in an inverter chain wi… ▽ More In this work, we propose a ferroelectric FET(FeFET) time-domain compute-in-memory (TD-CiM) array as a homogeneous processing fabric for binary multiplication-accumulation (MAC) and content addressable memory (CAM). We demonstrate that: i) the XOR(XNOR)/AND logic function can be realized using a single cell composed of 2FeFETs connected in series; ii) a two-phase computation in an inverter chain with each stage featuring the XOR/AND cell to control the associated capacitor loading and the computation results of binary MAC and CAM are reflected in the chain output signal delay, illustrating full digital compatibility; iii) comprehensive theoretical and experimental validation of the proposed 2FeFET cell and inverter delay chains and their robustness against FeFET variation; iv) the homogeneous processing fabric is applied in hyperdimensional computing to show dynamic and fine-grain resource allocation to accommodate different tasks requiring varying demands over the binary MAC and CAM resources. △ Less

Submitted 24 September, 2022; originally announced September 2022.

Comments: 8 pages, 8 figures

Ferroelectric FET-based strong physical unclonable function: a low-power, high-reliable and reconfigurable solution for Internet-of-Things security

Authors: Xinrui Guo, Xiaoyang Ma, Franz Muller, Kai Ni, Thomas Kampfe, Yongpan Liu, Vijaykrishnan Narayanan, Xueqing Li

Abstract: Hardware security has been a key concern in modern information technologies. Especially, as the number of Internet-of-Things (IoT) devices grows rapidly, to protect the device security with low-cost security primitives becomes essential, among which Physical Unclonable Function (PUF) is a widely-used solution. In this paper, we propose the first FeFET-based strong PUF exploiting the cycle-to-cycle… ▽ More Hardware security has been a key concern in modern information technologies. Especially, as the number of Internet-of-Things (IoT) devices grows rapidly, to protect the device security with low-cost security primitives becomes essential, among which Physical Unclonable Function (PUF) is a widely-used solution. In this paper, we propose the first FeFET-based strong PUF exploiting the cycle-to-cycle (C2C) variation of FeFETs as the entropy source. Based on the experimental measurements, the proposed PUF shows satisfying performance including high uniformity, uniqueness, reconfigurability and reliability. To resist machine-learning attack, XOR structure was introduced, and simulations show that our proposed PUF has similar resistance to existing attack models with traditional arbiter PUFs. Furthermore, our design is shown to be power-efficient, and highly robust to write voltage, temperature and device size, which makes it a competitive security solution for Internet-of-Things edge devices. △ Less

Submitted 31 August, 2022; originally announced August 2022.

The Vera C. Rubin Observatory Data Butler and Pipeline Execution System

Authors: Tim Jenness, James F. Bosch, Nate B. Lust, Nathan M. Pease, Michelle Gower, Mikolaj Kowalik, Gregory P. Dubois-Felsmann, Fritz Mueller, Pim Schellart

Abstract: The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a lapto… ▽ More The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a laptop using a local file system. The Butler and pipeline system are now in daily use during Rubin construction and early operations. △ Less

Submitted 29 June, 2022; originally announced June 2022.

Comments: 14 pages, 3 figures, submitted to Proc SPIE 12189, "Software and Cyberinfrastructure for Astronomy VII", Montreal, CA, July 2022

EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes

Authors: Gengyan Li, Abhimitra Meka, Franziska Müller, Marcel C. Bühler, Otmar Hilliges, Thabo Beeler

Abstract: A unique challenge in creating high-quality animatable and relightable 3D avatars of people is modeling human eyes. The challenge of synthesizing eyes is multifold as it requires 1) appropriate representations for the various components of the eye and the periocular region for coherent viewpoint synthesis, capable of representing diffuse, refractive and highly reflective surfaces, 2) disentangling… ▽ More A unique challenge in creating high-quality animatable and relightable 3D avatars of people is modeling human eyes. The challenge of synthesizing eyes is multifold as it requires 1) appropriate representations for the various components of the eye and the periocular region for coherent viewpoint synthesis, capable of representing diffuse, refractive and highly reflective surfaces, 2) disentangling skin and eye appearance from environmental illumination such that it may be rendered under novel lighting conditions, and 3) capturing eyeball motion and the deformation of the surrounding skin to enable re-gazing. These challenges have traditionally necessitated the use of expensive and cumbersome capture setups to obtain high-quality results, and even then, modeling of the eye region holistically has remained elusive. We present a novel geometry and appearance representation that enables high-fidelity capture and photorealistic animation, view synthesis and relighting of the eye region using only a sparse set of lights and cameras. Our hybrid representation combines an explicit parametric surface model for the eyeball with implicit deformable volumetric representations for the periocular region and the interior of the eye. This novel hybrid model has been designed to address the various parts of that challenging facial area - the explicit eyeball surface allows modeling refraction and high-frequency specular reflection at the cornea, whereas the implicit representation is well suited to model lower-frequency skin reflection via spherical harmonics and can represent non-surface structures such as hair or diffuse volumetric bodies, both of which are a challenge for explicit surface models. We show that for high-resolution close-ups of the eye, our model can synthesize high-fidelity animated gaze from novel views under unseen illumination conditions. △ Less

Submitted 12 July, 2022; v1 submitted 16 June, 2022; originally announced June 2022.

Comments: 16 pages, 16 figures, 1 table, to be published in ACM Transactions on Graphics (TOG) (Volume: 41, Issue: 4), 2022

ACM Class: I.4.5; I.3

Proper Posture: Designing Posture Feedback Across Musical Instruments

Authors: Bettina Eska, Jasmin Niess, Florian Müller

Abstract: There is a recommended body posture and hand position for playing every musical instrument, allowing efficient and quick movements without blockage. Due to humans' limited cognitive capabilities, they struggle to concentrate on several things simultaneously and thus sometimes lose the correct position while playing their instrument. Incorrect positions when playing an instrument can lead to injuri… ▽ More There is a recommended body posture and hand position for playing every musical instrument, allowing efficient and quick movements without blockage. Due to humans' limited cognitive capabilities, they struggle to concentrate on several things simultaneously and thus sometimes lose the correct position while playing their instrument. Incorrect positions when playing an instrument can lead to injuries and movement disorders in the long run. Previous work in HCI mainly focused on develo** systems to assist in learning an instrument. However, the design space for posture correction when playing a musical instrument has not yet been explored. In this position paper, we present our vision of providing subtle vibrotactile or thermal feedback to guide the focus of attention back to the correct posture when playing a musical instrument. We discuss our concept with a focus on motion recognition and feedback modalities. Finally, we outline the next steps for future research. △ Less

Submitted 30 May, 2022; originally announced May 2022.

Comments: CHI 22, Workshop "Intelligent Music Interfaces: When Interactive Assistance and Augmentation Meet Musical Instruments"

Simulation of machine learning-based 6G systems in virtual worlds

Authors: Ailton Oliveira, Felipe Bastos, Isabela Trindade, Walter Frazao, Arthur Nascimento, Diego Gomes, Francisco Muller, Aldebaro Klautau

Abstract: Digital representations of the real world are being used in many applications, such as augmented reality. 6G systems will not only support use cases that rely on virtual worlds but also benefit from their rich contextual information to improve performance and reduce communication overhead. This paper focuses on the simulation of 6G systems that rely on a 3D representation of the environment, as ca… ▽ More Digital representations of the real world are being used in many applications, such as augmented reality. 6G systems will not only support use cases that rely on virtual worlds but also benefit from their rich contextual information to improve performance and reduce communication overhead. This paper focuses on the simulation of 6G systems that rely on a 3D representation of the environment, as captured by cameras and other sensors. We present new strategies for obtaining paired MIMO channels and multimodal data. We also discuss trade-offs between speed and accuracy when generating channels via ray tracing. We finally provide beam selection simulation results to assess the proposed methodology. △ Less

Submitted 15 April, 2022; originally announced April 2022.

Journal ref: ITU Journal on Future and Evolving Technologies, Volume 2 (2021), Issue 4 - AI and machine learning solutions in 5G and future networks, Pages 113-123

Lateral Ego-Vehicle Control without Supervision using Point Clouds

Authors: Florian Müller, Qadeer Khan, Daniel Cremers

Abstract: Existing vision based supervised approaches to lateral vehicle control are capable of directly map** RGB images to the appropriate steering commands. However, they are prone to suffering from inadequate robustness in real world scenarios due to a lack of failure cases in the training data. In this paper, a framework for training a more robust and scalable model for lateral vehicle control is pro… ▽ More Existing vision based supervised approaches to lateral vehicle control are capable of directly map** RGB images to the appropriate steering commands. However, they are prone to suffering from inadequate robustness in real world scenarios due to a lack of failure cases in the training data. In this paper, a framework for training a more robust and scalable model for lateral vehicle control is proposed. The framework only requires an unlabeled sequence of RGB images. The trained model takes a point cloud as input and predicts the lateral offset to a subsequent frame from which the steering angle is inferred. The frame poses are in turn obtained from visual odometry. The point cloud is conceived by projecting dense depth maps into 3D. An arbitrary number of additional trajectories from this point cloud can be generated during training. This is to increase the robustness of the model. Online experiments show that the performance of our method is superior to that of the supervised model. △ Less

Submitted 20 March, 2022; originally announced March 2022.

An Ultra-Compact Single FeFET Binary and Multi-Bit Associative Search Engine

Authors: Xunzhao Yin, Franz Müller, Qingrong Huang, Chao Li, Mohsen Imani, Zeyu Yang, Jiahao Cai, Maximilian Lederer, Ricardo Olivo, Nellie Laleni, Shan Deng, Zijian Zhao, Cheng Zhuo, Thomas Kämpfe, Kai Ni

Abstract: Content addressable memory (CAM) is widely used in associative search tasks for its highly parallel pattern matching capability. To accommodate the increasingly complex and data-intensive pattern matching tasks, it is critical to keep improving the CAM density to enhance the performance and area efficiency. In this work, we demonstrate: i) a novel ultra-compact 1FeFET CAM design that enables paral… ▽ More Content addressable memory (CAM) is widely used in associative search tasks for its highly parallel pattern matching capability. To accommodate the increasingly complex and data-intensive pattern matching tasks, it is critical to keep improving the CAM density to enhance the performance and area efficiency. In this work, we demonstrate: i) a novel ultra-compact 1FeFET CAM design that enables parallel associative search and in-memory hamming distance calculation; ii) a multi-bit CAM for exact search using the same CAM cell; iii) compact device designs that integrate the series resistor current limiter into the intrinsic FeFET structure to turn the 1FeFET1R into an effective 1FeFET cell; iv) a successful 2-step search operation and a sufficient sensing margin of the proposed binary and multi-bit 1FeFET1R CAM array with sizes of practical interests in both experiments and simulations, given the existing unoptimized FeFET device variation; v) 89.9x speedup and 66.5x energy efficiency improvement over the state-of-the art alignment tools on GPU in accelerating genome pattern matching applications through the hyperdimensional computing paradigm. △ Less

Submitted 15 March, 2022; originally announced March 2022.

Comments: 20 pages, 14 figures

Recognition-free Question Answering on Handwritten Document Collections

Authors: Oliver Tüselmann, Friedrich Müller, Fabian Wolf, Gernot A. Fink

Abstract: In recent years, considerable progress has been made in the research area of Question Answering (QA) on document images. Current QA approaches from the Document Image Analysis community are mainly focusing on machine-printed documents and perform rather limited on handwriting. This is mainly due to the reduced recognition performance on handwritten documents. To tackle this problem, we propose a r… ▽ More In recent years, considerable progress has been made in the research area of Question Answering (QA) on document images. Current QA approaches from the Document Image Analysis community are mainly focusing on machine-printed documents and perform rather limited on handwriting. This is mainly due to the reduced recognition performance on handwritten documents. To tackle this problem, we propose a recognition-free QA approach, especially designed for handwritten document image collections. We present a robust document retrieval method, as well as two QA models. Our approaches outperform the state-of-the-art recognition-free models on the challenging BenthamQA and HW-SQuAD datasets. △ Less

Submitted 12 February, 2022; originally announced February 2022.

Deep Random Forest with Ferroelectric Analog Content Addressable Memory

Authors: Xunzhao Yin, Franz Müller, Ann Franchesca Laguna, Chao Li, Wenwen Ye, Qingrong Huang, Qinming Zhang, Zhiguo Shi, Maximilian Lederer, Nellie Laleni, Shan Deng, Zijian Zhao, Michael Niemier, Xiaobo Sharon Hu, Cheng Zhuo, Thomas Kämpfe, Kai Ni

Abstract: Deep random forest (DRF), which incorporates the core features of deep learning and random forest (RF), exhibits comparable classification accuracy, interpretability, and low memory and computational overhead when compared with deep neural networks (DNNs) in various information processing tasks for edge intelligence. However, the development of efficient hardware to accelerate DRF is lagging behin… ▽ More Deep random forest (DRF), which incorporates the core features of deep learning and random forest (RF), exhibits comparable classification accuracy, interpretability, and low memory and computational overhead when compared with deep neural networks (DNNs) in various information processing tasks for edge intelligence. However, the development of efficient hardware to accelerate DRF is lagging behind its DNN counterparts. The key for hardware acceleration of DRF lies in efficiently realizing the branch-split operation at decision nodes when traversing a decision tree. In this work, we propose to implement DRF through simple associative searches realized with ferroelectric analog content addressable memory (ACAM). Utilizing only two ferroelectric field effect transistors (FeFETs), the ultra-compact ACAM cell can perform a branch-split operation with an energy-efficient associative search by storing the decision boundaries as the analog polarization states in an FeFET. The DRF accelerator architecture and the corresponding map** of the DRF model to the ACAM arrays are presented. The functionality, characteristics, and scalability of the FeFET ACAM based DRF and its robustness against FeFET device non-idealities are validated both in experiments and simulations. Evaluation results show that the FeFET ACAM DRF accelerator exhibits 10^6x/16x and 10^6x/2.5x improvements in terms of energy and latency when compared with other deep random forest hardware implementations on the state-of-the-art CPU/ReRAM, respectively. △ Less

Submitted 6 October, 2021; originally announced October 2021.

Comments: 44 pages, 16 figures

Myopic Bike and Say Hi: Games for Empathizing with The Myopic

Authors: Xiang Li, Xiaohang Tang, Xin Tong, Rakesh Patibanda, Florian 'Floyd' Mueller, Hai-Ning Liang

Abstract: Myopia is an eye condition that makes it difficult for people to focus on faraway objects. It has become one of the most serious eye conditions worldwide and negatively impacts the quality of life of those who suffer from it. Although myopia is prevalent, many non-myopic people have misconceptions about it and encounter challenges empathizing with myopia situations and those who suffer from it. In… ▽ More Myopia is an eye condition that makes it difficult for people to focus on faraway objects. It has become one of the most serious eye conditions worldwide and negatively impacts the quality of life of those who suffer from it. Although myopia is prevalent, many non-myopic people have misconceptions about it and encounter challenges empathizing with myopia situations and those who suffer from it. In this research, we developed two virtual reality (VR) games, (1) Myopic Bike and (2) Say Hi, to provide a means for the non-myopic population to experience the frustration and difficulties of myopic people. Our two games simulate two inconvenient daily life scenarios (riding a bicycle and greeting someone on the street) that myopic people encounter when not wearing glasses. We evaluated four participants' game experiences through questionnaires and semi-structured interviews. Overall, our two VR games can create an engaging and non-judgmental experience for the non-myopic population to better understand and empathize with those who suffer from myopia. △ Less

Submitted 28 September, 2021; v1 submitted 11 September, 2021; originally announced September 2021.

Comments: 7 pages, CHI PLAY 2021

Neural Predictive Control for the Optimization of Smart Grid Flexibility Schedules

Authors: Steven de Jongh, Sina Steinle, Anna Hlawatsch, Felicitas Mueller, Michael Suriyah, Thomas Leibfried

Abstract: Model predictive control (MPC) is a method to formulate the optimal scheduling problem for grid flexibilities in a mathematical manner. The resulting time-constrained optimization problem can be re-solved in each optimization time step using classical optimization methods such as Second Order Cone Programming (SOCP) or Interior Point Methods (IPOPT). When applying MPC in a rolling horizon scheme,… ▽ More Model predictive control (MPC) is a method to formulate the optimal scheduling problem for grid flexibilities in a mathematical manner. The resulting time-constrained optimization problem can be re-solved in each optimization time step using classical optimization methods such as Second Order Cone Programming (SOCP) or Interior Point Methods (IPOPT). When applying MPC in a rolling horizon scheme, the impact of uncertainty in forecasts on the optimal schedule is reduced. While MPC methods promise accurate results for time-constrained grid optimization they are inherently limited by the calculation time needed for large and complex power system models. Learning the optimal control behaviour using function approximation offers the possibility to determine near-optimal control actions with short calculation time. A Neural Predictive Control (NPC) scheme is proposed to learn optimal control policies for linear and nonlinear power systems through imitation. It is demonstrated that this procedure can find near-optimal solutions, while reducing the calculation time by an order of magnitude. The learned controllers are validated using a benchmark smart grid. △ Less

Submitted 19 August, 2021; originally announced August 2021.

Comments: 978-1-6654-4389-0/21/$31.00 C 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective

Empirical Evaluation of Circuit Approximations on Noisy Quantum Devices

Authors: Ellis Wilson, Frank Mueller, Lindsay Bassman, Constin Iancu

Abstract: Noisy Intermediate-Scale Quantum (NISQ) devices fail to produce outputs with sufficient fidelity for deep circuits with many gates today. Such devices suffer from read-out, multi-qubit gate and crosstalk noise combined with short decoherence times limiting circuit depth. This work develops a methodology to generate shorter circuits with fewer multi-qubit gates whose unitary transformations approxi… ▽ More Noisy Intermediate-Scale Quantum (NISQ) devices fail to produce outputs with sufficient fidelity for deep circuits with many gates today. Such devices suffer from read-out, multi-qubit gate and crosstalk noise combined with short decoherence times limiting circuit depth. This work develops a methodology to generate shorter circuits with fewer multi-qubit gates whose unitary transformations approximate the original reference one. It explores the benefit of such generated approximations under NISQ devices. Experimental results with Grover's algorithm, multiple-control Toffoli gates, and the Transverse Field Ising Model show that such approximate circuits produce higher fidelity results than longer, theoretically precise circuits on NISQ devices, especially when the reference circuits have many CNOT gates to begin with. With this ability to fine-tune circuits, it is demonstrated that quantum computations can be performed for more complex problems on today's devices than was feasible before, sometimes even with a gain in overall precision by up to 60%. △ Less

Submitted 14 July, 2021; originally announced July 2021.

RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

Authors: Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A. Otaduy, Dan Casas, Christian Theobalt

Abstract: Tracking and reconstructing the 3D pose and geometry of two hands in interaction is a challenging problem that has a high relevance for several human-computer interaction applications, including AR/VR, robotics, or sign language recognition. Existing works are either limited to simpler tracking settings (e.g., considering only a single hand or two spatially separated hands), or rely on less ubiqui… ▽ More Tracking and reconstructing the 3D pose and geometry of two hands in interaction is a challenging problem that has a high relevance for several human-computer interaction applications, including AR/VR, robotics, or sign language recognition. Existing works are either limited to simpler tracking settings (e.g., considering only a single hand or two spatially separated hands), or rely on less ubiquitous sensors, such as depth cameras. In contrast, in this work we present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera that explicitly considers close interactions. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN that regresses multiple complementary pieces of information, including segmentation, dense matchings to a 3D hand model, and 2D keypoint positions, together with newly proposed intra-hand relative depth and inter-hand distance maps. These predictions are subsequently used in a generative model fitting framework in order to estimate pose and shape parameters of a 3D hand model for both hands. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline through an extensive ablation study. Moreover, we demonstrate that our approach offers previously unseen two-hand tracking performance from RGB, and quantitatively and qualitatively outperforms existing RGB-based methods that were not explicitly designed for two-hand interactions. Moreover, our method even performs on-par with depth-based real-time methods. △ Less

Submitted 22 June, 2021; originally announced June 2021.

Comments: SIGGRAPH Asia 2020

Journal ref: ACM Transactions on Graphics (TOG) 39 (6), 1-16, 2020

Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera

Authors: Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A. Otaduy, Dan Casas, Christian Theobalt

Abstract: We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach is the first two-hand tracking solution that combines an extensive list of favorable properties, namely it is marker-less, uses a single consumer-level depth camera, runs in real time, handles inter- and intra-hand collisions, and automatically adjusts to the user's hand shape. In… ▽ More We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach is the first two-hand tracking solution that combines an extensive list of favorable properties, namely it is marker-less, uses a single consumer-level depth camera, runs in real time, handles inter- and intra-hand collisions, and automatically adjusts to the user's hand shape. In order to achieve this, we embed a recent parametric hand pose and shape model and a dense correspondence predictor based on a deep neural network into a suitable energy minimization framework. For training the correspondence prediction network, we synthesize a two-hand dataset based on physical simulations that includes both hand pose and shape annotations while at the same time avoiding inter-hand penetrations. To achieve real-time rates, we phrase the model fitting in terms of a nonlinear least-squares problem so that the energy can be optimized based on a highly efficient GPU-based Gauss-Newton optimizer. We show state-of-the-art results in scenes that exceed the complexity level demonstrated by previous work, including tight two-hand grasps, significant inter-hand occlusions, and gesture interaction. △ Less

Submitted 15 June, 2021; originally announced June 2021.

Comments: ACM Transactions on Graphics (Proceedings SIGGRAPH 2019)

zbMATH Open: API Solutions and Research Challenges

Authors: Matteo Petrera, Dennis Trautwein, Isabel Beckenbach, Dariush Ehsani, Fabian Mueller, Olaf Teschke, Bela Gipp, Moritz Schubotz

Abstract: We present zbMATH Open, the most comprehensive collection of reviews and bibliographic metadata of scholarly literature in mathematics. Besides our website https://zbMATH.org which is openly accessible since the beginning of this year, we provide API endpoints to offer our data. The API improves interoperability with others, i.e., digital libraries, and allows using our data for research purposes.… ▽ More We present zbMATH Open, the most comprehensive collection of reviews and bibliographic metadata of scholarly literature in mathematics. Besides our website https://zbMATH.org which is openly accessible since the beginning of this year, we provide API endpoints to offer our data. The API improves interoperability with others, i.e., digital libraries, and allows using our data for research purposes. In this article, we (1) illustrate the current and future overview of the services offered by zbMATH; (2) present the initial version of the zbMATH links API; (3) analyze potentials and limitations of the links API based on the example of the NIST Digital Library of Mathematical Functions; (4) and finally, present the zbMATH Open dataset as a research resource and discuss connected open research problems. △ Less

Submitted 23 June, 2021; v1 submitted 8 June, 2021; originally announced June 2021.

Differentiable Event Stream Simulator for Non-Rigid 3D Tracking

Authors: Jalees Nehvi, Vladislav Golyanik, Franziska Mueller, Hans-Peter Seidel, Mohamed Elgharib, Christian Theobalt

Abstract: This paper introduces the first differentiable simulator of event streams, i.e., streams of asynchronous brightness change signals recorded by event cameras. Our differentiable simulator enables non-rigid 3D tracking of deformable objects (such as human hands, isometric surfaces and general watertight meshes) from event streams by leveraging an analysis-by-synthesis principle. So far, event-based… ▽ More This paper introduces the first differentiable simulator of event streams, i.e., streams of asynchronous brightness change signals recorded by event cameras. Our differentiable simulator enables non-rigid 3D tracking of deformable objects (such as human hands, isometric surfaces and general watertight meshes) from event streams by leveraging an analysis-by-synthesis principle. So far, event-based tracking and reconstruction of non-rigid objects in 3D, like hands and body, has been either tackled using explicit event trajectories or large-scale datasets. In contrast, our method does not require any such processing or data, and can be readily applied to incoming event streams. We show the effectiveness of our approach for various types of non-rigid objects and compare to existing methods for non-rigid 3D tracking. In our experiments, the proposed energy-based formulations outperform competing RGB-based methods in terms of 3D errors. The source code and the new data are publicly available. △ Less

Submitted 30 April, 2021; originally announced April 2021.

Comments: In CVPR 2021 Workshop on Event-based Vision. Project page: http://gvv.mpi-inf.mpg.de/projects/Event-based_Non-rigid_3D_Tracking

vrCAPTCHA: Exploring CAPTCHA Designs in Virtual Reality

Authors: Xiang Li, Yuzheng Chen, Rakesh Patibanda, Florian 'Floyd' Mueller

Abstract: With the popularity of online access in virtual reality (VR) devices, it will become important to investigate exclusive and interactive CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) designs for VR devices. In this paper, we first present four traditional two-dimensional (2D) CAPTCHAs (i.e., text-based, image-rotated, image-puzzled, and image-selected CAPTCHAs… ▽ More With the popularity of online access in virtual reality (VR) devices, it will become important to investigate exclusive and interactive CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) designs for VR devices. In this paper, we first present four traditional two-dimensional (2D) CAPTCHAs (i.e., text-based, image-rotated, image-puzzled, and image-selected CAPTCHAs) in VR. Then, based on the three-dimensional (3D) interaction characteristics of VR devices, we propose two vrCAPTCHA design prototypes (i.e., task-driven and bodily motion-based CAPTCHAs). We conducted a user study with six participants for exploring the feasibility of our two vrCAPTCHAs and traditional CAPTCHAs in VR. We believe that our two vrCAPTCHAs can be an inspiration for the further design of CAPTCHAs in VR. △ Less

Submitted 25 February, 2021; v1 submitted 24 February, 2021; originally announced February 2021.

Comments: 4 pages, 3 figures, CHI EA' 2021

EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream

Authors: Viktor Rudnev, Vladislav Golyanik, Jiayi Wang, Hans-Peter Seidel, Franziska Mueller, Mohamed Elgharib, Christian Theobalt

Abstract: 3D hand pose estimation from monocular videos is a long-standing and challenging problem, which is now seeing a strong upturn. In this work, we address it for the first time using a single event camera, i.e., an asynchronous vision sensor reacting on brightness changes. Our EventHands approach has characteristics previously not demonstrated with a single RGB or depth camera such as high temporal r… ▽ More 3D hand pose estimation from monocular videos is a long-standing and challenging problem, which is now seeing a strong upturn. In this work, we address it for the first time using a single event camera, i.e., an asynchronous vision sensor reacting on brightness changes. Our EventHands approach has characteristics previously not demonstrated with a single RGB or depth camera such as high temporal resolution at low data throughputs and real-time performance at 1000 Hz. Due to the different data modality of event cameras compared to classical cameras, existing methods cannot be directly applied to and re-trained for event streams. We thus design a new neural approach which accepts a new event stream representation suitable for learning, which is trained on newly-generated synthetic event streams and can generalise to real data. Experiments show that EventHands outperforms recent monocular methods using a colour (or depth) camera in terms of accuracy and its ability to capture hand motions of unprecedented speed. Our method, the event stream simulator and the dataset are publicly available; see https://4dqv.mpi-inf.mpg.de/EventHands/ △ Less

Submitted 11 October, 2021; v1 submitted 11 December, 2020; originally announced December 2020.

Comments: 16 pages, 10 figures, 1 table; project page: https://4dqv.mpi-inf.mpg.de/EventHands/

Journal ref: International Conference on Computer Vision (ICCV) 2021

In-Memory Nearest Neighbor Search with FeFET Multi-Bit Content-Addressable Memories

Authors: Arman Kazemi, Mohammad Mehdi Sharifi, Ann Franchesca Laguna, Franz Müller, Ramin Rajaei, Ricardo Olivo, Thomas Kämpfe, Michael Niemier, X. Sharon Hu

Abstract: Nearest neighbor (NN) search is an essential operation in many applications, such as one/few-shot learning and image classification. As such, fast and low-energy hardware support for accurate NN search is highly desirable. Ternary content-addressable memories (TCAMs) have been proposed to accelerate NN search for few-shot learning tasks by implementing $L_\infty$ and Hamming distance metrics, but… ▽ More Nearest neighbor (NN) search is an essential operation in many applications, such as one/few-shot learning and image classification. As such, fast and low-energy hardware support for accurate NN search is highly desirable. Ternary content-addressable memories (TCAMs) have been proposed to accelerate NN search for few-shot learning tasks by implementing $L_\infty$ and Hamming distance metrics, but they cannot achieve software-comparable accuracies. This paper proposes a novel distance function that can be natively evaluated with multi-bit content-addressable memories (MCAMs) based on ferroelectric FETs (FeFETs) to perform a single-step, in-memory NN search. Moreover, this approach achieves accuracies comparable to floating-point precision implementations in software for NN classification and one/few-shot learning tasks. As an example, the proposed method achieves a 98.34% accuracy for a 5-way, 5-shot classification task for the Omniglot dataset (only 0.8% lower than software-based implementations) with a 3-bit MCAM. This represents a 13% accuracy improvement over state-of-the-art TCAM-based implementations at iso-energy and iso-delay. The presented distance function is resilient to the effects of FeFET device-to-device variations. Furthermore, this work experimentally demonstrates a 2-bit implementation of FeFET MCAM using AND arrays from GLOBALFOUNDRIES to further validate proof of concept. △ Less

Submitted 13 November, 2020; originally announced November 2020.

Comments: To be published in DATE'21

Systematic Analysis of the Sensor Coverage of Automated Vehicles Using Phenomenological Sensor Models

Authors: Thomas Ponn, Fabian Müller, Frank Diermeyer

Abstract: The objective of this paper is to propose a systematic analysis of the sensor coverage of automated vehicles. Due to an unlimited number of possible traffic situations, a selection of scenarios to be tested must be applied in the safety assessment of automated vehicles. This paper describes how phenomenological sensor models can be used to identify system-specific relevant scenarios. In automated… ▽ More The objective of this paper is to propose a systematic analysis of the sensor coverage of automated vehicles. Due to an unlimited number of possible traffic situations, a selection of scenarios to be tested must be applied in the safety assessment of automated vehicles. This paper describes how phenomenological sensor models can be used to identify system-specific relevant scenarios. In automated driving, the following sensors are predominantly used: camera, ultrasonic, \radar and \lidarohne. Based on the literature, phenomenological models have been developed for the four sensor types, which take into account phenomena such as environmental influences, sensor properties and the type of object to be detected. These phenomenological models have a significantly higher reliability than simple ideal sensor models and require lower computing costs than realistic physical sensor models, which represents an optimal compromise for systematic investigations of sensor coverage. The simulations showed significant differences between different system configurations and thus support the system-specific selection of relevant scenarios for the safety assessment of automated vehicles. △ Less

Submitted 27 August, 2020; v1 submitted 26 August, 2020; originally announced August 2020.

Comments: Published at 2019 IEEE Intelligent Vehicles Symposium (IV19), June 2019

SeismoFlow -- Data augmentation for the class imbalance problem

Authors: Ruy Luiz Milidiú, Luis Felipe Müller

Abstract: In several application areas, such as medical diagnosis, spam filtering, fraud detection, and seismic data analysis, it is very usual to find relevant classification tasks where some class occurrences are rare. This is the so called class imbalance problem, which is a challenge in machine learning. In this work, we propose the SeismoFlow a flow-based generative model to create synthetic samples, a… ▽ More In several application areas, such as medical diagnosis, spam filtering, fraud detection, and seismic data analysis, it is very usual to find relevant classification tasks where some class occurrences are rare. This is the so called class imbalance problem, which is a challenge in machine learning. In this work, we propose the SeismoFlow a flow-based generative model to create synthetic samples, aiming to address the class imbalance. Inspired by the Glow model, it uses interpolation on the learned latent space to produce synthetic samples for one rare class. We apply our approach to the development of a seismogram signal quality classifier. We introduce a dataset composed of5.223seismograms that are distributed between the good, medium, and bad classes and with their respective frequencies of 66.68%,31.54%, and 1.76%. Our methodology is evaluated on a stratified 10-fold cross-validation setting, using the Miniceptionmodel as a baseline, and assessing the effects of adding the generated samples on the training set of each iteration. In our experiments, we achieve an improvement of 13.9% on the rare class F1-score, while not hurting the metric value for the other classes and thus observing the overall accuracy improvement. Our empirical findings indicate that our method can generate high-quality synthetic seismograms with realistic looking and sufficient plurality to help the Miniception model to overcome the class imbalance problem. We believe that our results are a step forward in solving both the task of seismogram signal quality classification and class imbalance. △ Less

Submitted 2 September, 2020; v1 submitted 23 July, 2020; originally announced July 2020.

Comments: 10 pages

Generative Model-Based Loss to the Rescue: A Method to Overcome Annotation Errors for Depth-Based Hand Pose Estimation

Authors: Jiayi Wang, Franziska Mueller, Florian Bernard, Christian Theobalt

Abstract: We propose to use a model-based generative loss for training hand pose estimators on depth images based on a volumetric hand model. This additional loss allows training of a hand pose estimator that accurately infers the entire set of 21 hand keypoints while only using supervision for 6 easy-to-annotate keypoints (fingertips and wrist). We show that our partially-supervised method achieves results… ▽ More We propose to use a model-based generative loss for training hand pose estimators on depth images based on a volumetric hand model. This additional loss allows training of a hand pose estimator that accurately infers the entire set of 21 hand keypoints while only using supervision for 6 easy-to-annotate keypoints (fingertips and wrist). We show that our partially-supervised method achieves results that are comparable to those of fully-supervised methods which enforce articulation consistency. Moreover, for the first time we demonstrate that such an approach can be used to train on datasets that have erroneous annotations, i.e. "ground truth" with notable measurement errors, while obtaining predictions that explain the depth images better than the given "ground truth". △ Less

Submitted 30 May, 2021; v1 submitted 6 July, 2020; originally announced July 2020.

Enumerating Teams in First-Order Team Logics

Authors: Anselm Haak, Arne Meier, Fabian Müller, Heribert Vollmer

Abstract: We start the study of the enumeration complexity of different satisfiability problems in first-order team logics. Since many of our problems go beyond DelP, we use a framework for hard enumeration analogous to the polynomial hierarchy, which was recently introduced by Creignou et al. (Discret. Appl. Math. 2019). We show that the problem to enumerate all satisfying teams of a fixed formula in a giv… ▽ More We start the study of the enumeration complexity of different satisfiability problems in first-order team logics. Since many of our problems go beyond DelP, we use a framework for hard enumeration analogous to the polynomial hierarchy, which was recently introduced by Creignou et al. (Discret. Appl. Math. 2019). We show that the problem to enumerate all satisfying teams of a fixed formula in a given first-order structure is DelNP-complete for certain formulas of dependence logic and independence logic. For inclusion logic formulas, this problem is even in DelP. Furthermore, we study the variants of this problems where only maximal, minimal, maximum and minimum solutions, respectively, are considered. For the most part these share the same complexity as the original problem. An exception is the minimum-variant for inclusion logic, which is DelNP-complete. △ Less

Submitted 29 December, 2020; v1 submitted 12 June, 2020; originally announced June 2020.

Just-in-time Quantum Circuit Transpilation Reduces Noise

Authors: Ellis Wilson, Sudhakar Singh, Frank Mueller

Abstract: Running quantum programs is fraught with challenges on on today's noisy intermediate scale quantum (NISQ) devices. Many of these challenges originate from the error characteristics that stem from rapid decoherence and noise during measurement, qubit connections, crosstalk, the qubits themselves, and transformations of qubit state via gates. Not only are qubits not "created equal", but their noise… ▽ More Running quantum programs is fraught with challenges on on today's noisy intermediate scale quantum (NISQ) devices. Many of these challenges originate from the error characteristics that stem from rapid decoherence and noise during measurement, qubit connections, crosstalk, the qubits themselves, and transformations of qubit state via gates. Not only are qubits not "created equal", but their noise level also changes over time. IBM is said to calibrate their quantum systems once per day and reports noise levels (errors) at the time of such calibration. This information is subsequently used to map circuits to higher quality qubits and connections up to the next calibration point. This work provides evidence that there is room for improvement over this daily calibration cycle. It contributes a technique to measure noise levels (errors) related to qubits immediately before executing one or more sensitive circuits and shows that just-in-time noise measurements benefit late physical qubit map**s. With this just-in-time recalibrated transpilation, the fidelity of results is improved over IBM's default map**s, which only uses their daily calibrations. The framework assess two major sources of noise, namely readout errors (measurement errors) and two-qubit gate/connection errors. Experiments indicate that the accuracy of circuit results improves by 3-304% on average and up to 400% with on-the-fly circuit map**s based on error measurements just prior to application execution. △ Less

Submitted 26 May, 2020; originally announced May 2020.

Comments: under submission