Showing 1–24 of 24 results for author: Hein, B

6-DoF Grasp Pose Evaluation and Optimization via Transfer Learning from NeRFs

Authors: Gergely Sóti, Xi Huang, Christian Wurll, Björn Hein

Abstract: We address the problem of robotic gras** of known and unknown objects using implicit behavior cloning. We train a grasp evaluation model from a small number of demonstrations that outputs higher values for grasp candidates that are more likely to succeed in gras**. This evaluation model serves as an objective function, that we maximize to identify successful grasps. Key to our approach is the… ▽ More We address the problem of robotic gras** of known and unknown objects using implicit behavior cloning. We train a grasp evaluation model from a small number of demonstrations that outputs higher values for grasp candidates that are more likely to succeed in gras**. This evaluation model serves as an objective function, that we maximize to identify successful grasps. Key to our approach is the utilization of learned implicit representations of visual and geometric features derived from a pre-trained NeRF. Though trained exclusively in a simulated environment with simplified objects and 4-DoF top-down grasps, our evaluation model and optimization procedure demonstrate generalization to 6-DoF grasps and novel objects both in simulation and in real-world settings, without the need for additional data. Supplementary material is available at: https://gergely-soti.github.io/grasp △ Less

Submitted 15 January, 2024; originally announced January 2024.

Gradient based Grasp Pose Optimization on a NeRF that Approximates Grasp Success

Authors: Gergely Sóti, Björn Hein, Christian Wurll

Abstract: Current robotic gras** methods often rely on estimating the pose of the target object, explicitly predicting grasp poses, or implicitly estimating grasp success probabilities. In this work, we propose a novel approach that directly maps gripper poses to their corresponding grasp success values, without considering objectness. Specifically, we leverage a Neural Radiance Field (NeRF) architecture… ▽ More Current robotic gras** methods often rely on estimating the pose of the target object, explicitly predicting grasp poses, or implicitly estimating grasp success probabilities. In this work, we propose a novel approach that directly maps gripper poses to their corresponding grasp success values, without considering objectness. Specifically, we leverage a Neural Radiance Field (NeRF) architecture to learn a scene representation and use it to train a grasp success estimator that maps each pose in the robot's task space to a grasp success value. We employ this learned estimator to tune its inputs, i.e., grasp poses, by gradient-based optimization to obtain successful grasp poses. Contrary to other NeRF-based methods which enhance existing grasp pose estimation approaches by relying on NeRF's rendering capabilities or directly estimate grasp poses in a discretized space using NeRF's scene representation capabilities, our approach uniquely sidesteps both the need for rendering and the limitation of discretization. We demonstrate the effectiveness of our approach on four simulated 3DoF (Degree of Freedom) robotic gras** tasks and show that it can generalize to novel objects. Our best model achieves an average translation error of 3mm from valid grasp poses. This work opens the door for future research to apply our approach to higher DoF grasps and real-world scenarios. △ Less

Submitted 14 September, 2023; originally announced September 2023.

Human Emergency Detection during Autonomous Hospital Transports

Authors: Andreas Zachariae, Julia Widera, Frederik Plahl, Björn Hein, Christian Wurll

Abstract: Human transports in hospitals are labor-intensive and primarily performed in beds to save time. This transfer method does not promote the mobility or autonomy of the patient. To relieve the caregivers from this time-consuming task, a mobile robot is developed to autonomously transport humans around the hospital. It provides different transfer modes including walking and sitting in a wheelchair. Th… ▽ More Human transports in hospitals are labor-intensive and primarily performed in beds to save time. This transfer method does not promote the mobility or autonomy of the patient. To relieve the caregivers from this time-consuming task, a mobile robot is developed to autonomously transport humans around the hospital. It provides different transfer modes including walking and sitting in a wheelchair. The problem that this paper focuses on is to detect emergencies and ensure the well-being of the patient during the transport. For this purpose, the patient is tracked and monitored with a camera system. OpenPose is used for Human Pose Estimation and a trained classifier for emergency detection. We collected and published a dataset of 18,000 images in lab and hospital environments. It differs from related work because we have a moving robot with different transfer modes in a highly dynamic environment with multiple people in the scene using only RGB-D data. To improve the critical recall metric, we apply threshold moving and a time delay. We compare different models with an AutoML approach. This paper shows that emergencies while walking are best detected by a SVM with a recall of 95.8% on single frames. In the case of sitting transport, the best model achieves a recall of 62.2%. The contribution is to establish a baseline on this new dataset and to provide a proof of concept for the human emergency detection in this use case. △ Less

Submitted 17 July, 2023; originally announced July 2023.

Comments: Preprint of the corresponding IAS18-2023 conference publication (Proceedings of the 18th International Conference IAS-18)

Inside-out Infrared Marker Tracking via Head Mounted Displays for Smart Robot Programming

Authors: David Puljiz, Alexandru-George Vasilache, Michael Mende, Björn Hein

Abstract: Intuitive robot programming through use of tracked smart input devices relies on fixed, external tracking systems, most often employing infra-red markers. Such an approach is frequently combined with projector-based augmented reality for better visualisation and interface. The combined system, although providing an intuitive programming platform with short cycle times even for inexperienced users,… ▽ More Intuitive robot programming through use of tracked smart input devices relies on fixed, external tracking systems, most often employing infra-red markers. Such an approach is frequently combined with projector-based augmented reality for better visualisation and interface. The combined system, although providing an intuitive programming platform with short cycle times even for inexperienced users, is immobile, expensive and requires extensive calibration. When faced with a changing environment and large number of robots it becomes sorely impractical. Here we present our work on infra-red marker tracking using the Microsoft HoloLens head-mounted display. The HoloLens can map the environment, register the robot on-line, and track smart devices equipped with infra-red markers in the robot coordinate system. We envision our work to provide the basis to transfer many of the paradigms developed over the years for systems requiring a projector and a tracked input device into a highly-portable system that does not require any calibration or special set-up. We test the quality of the marker-tracking in an industrial robot cell and compare our tracking with a ground truth obtained via an ART-3 tracking system. △ Less

Submitted 28 March, 2023; originally announced March 2023.

Comments: As accepted to the 5th International Workshop on Virtual, Augmented, and Mixed Reality for HRI

Train What You Know -- Precise Pick-and-Place with Transporter Networks

Authors: Gergely Sóti, Xi Huang, Christian Wurll, Björn Hein

Abstract: Precise pick-and-place is essential in robotic applications. To this end, we define a novel exact training method and an iterative inference method that improve pick-and-place precision with Transporter Networks. We conduct a large scale experiment on 8 simulated tasks. A systematic analysis shows, that the proposed modifications have a significant positive effect on model performance. Considering… ▽ More Precise pick-and-place is essential in robotic applications. To this end, we define a novel exact training method and an iterative inference method that improve pick-and-place precision with Transporter Networks. We conduct a large scale experiment on 8 simulated tasks. A systematic analysis shows, that the proposed modifications have a significant positive effect on model performance. Considering picking and placing independently, our methods achieve up to 60% lower rotation and translation errors than baselines. For the whole pick-and-place process we observe 50% lower rotation errors for most tasks with slight improvements in terms of translation errors. Furthermore, we propose architectural changes that retain model performance and reduce computational costs and time. We validate our methods with an interactive teaching procedure on real hardware. Supplementary material will be made available at: https://gergely-soti.github.io/p △ Less

Submitted 17 February, 2023; originally announced February 2023.

Updating Industrial Robots for Emerging Technologies

Authors: David Puljiz, Björn Hein

Abstract: Industrial arms need to evolve beyond their standard shape to embrace new and emerging technologies. In this paper, we shall first perform an analysis of four popular but different modern industrial robot arms. By seeing the common trends we will try to extrapolate and expand these trends for the future. Here, particular focus will be on interaction based on augmented reality (AR) through head-mou… ▽ More Industrial arms need to evolve beyond their standard shape to embrace new and emerging technologies. In this paper, we shall first perform an analysis of four popular but different modern industrial robot arms. By seeing the common trends we will try to extrapolate and expand these trends for the future. Here, particular focus will be on interaction based on augmented reality (AR) through head-mounted displays (HMD), but also through smartphones. Long-term human-robot interaction and personalization of said interaction will also be considered. The use of AR in human-robot interaction has proven to enhance communication and information exchange. A basic addition to industrial arm design would be the integration of QR markers on the robot, both for accessing information and adding tracking capabilities to more easily display AR overlays. In a recent example of information access, Mercedes Benz added QR markers on their cars to help rescue workers estimate the best places to cut and evacuate people after car crashes. One has also to deal with safety in an environment that will be more and more about collaboration. The QR markers can therefore be combined with RF-based ranging modules, developed in the EU-project SafeLog, that can be used both for safety as well as for tracking of human positions while in close proximity interactions with the industrial arms. The industrial arms of the future should also be intuitive to program and interact with. This would be achieved through AR and head mounted displays as well as the already mentioned RF-based person tracking. Finally, a more personalized interaction between the robots and humans can be achieved through life-long learning AI and disembodied, personalized agents. We propose a design that not only exists in the physical world, but also partly in the digital world of mixed reality. △ Less

Submitted 28 March, 2023; v1 submitted 7 April, 2022; originally announced April 2022.

Comments: As accepted to the 2nd International Workshop on Designerly HRI; HRI 2022

Using Deep Reinforcement Learning with Automatic Curriculum Learning for Mapless Navigation in Intralogistics

Authors: Honghu Xue, Benedikt Hein, Mohamed Bakr, Georg Schildbach, Bengt Abel, Elmar Rueckert

Abstract: We propose a deep reinforcement learning approach for solving a mapless navigation problem in warehouse scenarios. In our approach, an automation guided vehicle is equipped with LiDAR and frontal RGB sensors and learns to perform a targeted navigation task. The challenges reside in the sparseness of positive samples for learning, multi-modal sensor perception with partial observability, the demand… ▽ More We propose a deep reinforcement learning approach for solving a mapless navigation problem in warehouse scenarios. In our approach, an automation guided vehicle is equipped with LiDAR and frontal RGB sensors and learns to perform a targeted navigation task. The challenges reside in the sparseness of positive samples for learning, multi-modal sensor perception with partial observability, the demand for accurate steering maneuvers together with long training cycles. To address these points, we propose NavACL-Q as a method for automatic curriculum learning in combination with a distributed version of the soft actor-critic algorithm. The performance of the learning algorithm is evaluated exhaustively in an unseen warehouse environment to validate both robustness and generalizability of the learned policy. Results in NVIDIA Isaac Sim demonstrates that our trained agent significantly outperforms a map-based navigation pipeline provided by NVIDIA Isaac Sim with an increased agent-goal distance of 3m and wider initial relative agent-goal rotations of 45 degree. The ablation studies also suggests that NavACL-Q greatly facilitates the learning process with a performance gain of roughly 40% compared to training with random starts and that the utilization of a pre-trained feature extractor manifestly boosts the performance by approximately 60%. △ Less

Submitted 16 March, 2022; v1 submitted 23 February, 2022; originally announced February 2022.

Mechanisms for spontaneous symmetry breaking in develo** visual cortex

Authors: Francesco Fumarola, Bettina Hein, Kenneth D. Miller

Abstract: For the brain to recognize local orientations within images, neurons must spontaneously break the translation and rotation symmetry of their response functions -- an archetypal example of unsupervised learning. The dominant framework for unsupervised learning in biology is Hebb's principle, but how Hebbian learning could break such symmetries is a longstanding biophysical riddle. Theoretical studi… ▽ More For the brain to recognize local orientations within images, neurons must spontaneously break the translation and rotation symmetry of their response functions -- an archetypal example of unsupervised learning. The dominant framework for unsupervised learning in biology is Hebb's principle, but how Hebbian learning could break such symmetries is a longstanding biophysical riddle. Theoretical studies agree that this should require inputs to visual cortex to invert the relative magnitude of their correlations at long distances. Empirical measurements have searched in vain for such an inversion, and report the opposite to be true. We formally approach the question through the hermitianization of a multi-layer model, which maps it into a problem of zero-temperature phase transitions. In the emerging phase diagram, both symmetries break spontaneously as long as (1) recurrent interactions are sufficiently long-range and (2) Hebbian competition is duly accounted for. The relevant mechanism for symmetry breaking is competition among connections sprouting from the same afferent cell. Such competition, and not the structure of the inputs, is capable of triggering the broken-symmetry phase required by image processing. We provide analytic predictions on the relative magnitudes of the relevant length-scales needed for this novel mechanism to occur. These results reconcile experimental observations to the Hebbian paradigm, shed light on a new mechanism for visual cortex development, and contribute to our growing understanding of the relationship between learning and symmetry breaking. △ Less

Submitted 5 September, 2021; originally announced September 2021.

Comments: 33 pages, 11 figures

Proximity Perception in Human-Centered Robotics: A Survey on Sensing Systems and Applications

Authors: Stefan Escaida Navarro, Stephan Mühlbacher-Karrer, Hosam Alagi, Hubert Zangl, Keisuke Koyama, Björn Hein, Christian Duriez, Joshua R. Smith

Abstract: Proximity perception is a technology that has the potential to play an essential role in the future of robotics. It can fulfill the promise of safe, robust, and autonomous systems in industry and everyday life, alongside humans, as well as in remote locations in space and underwater. In this survey paper, we cover the developments of this field from the early days up to the present, with a focus o… ▽ More Proximity perception is a technology that has the potential to play an essential role in the future of robotics. It can fulfill the promise of safe, robust, and autonomous systems in industry and everyday life, alongside humans, as well as in remote locations in space and underwater. In this survey paper, we cover the developments of this field from the early days up to the present, with a focus on human-centered robotics. Here, proximity sensors are typically deployed in two scenarios: first, on the exterior of manipulator arms to support safety and interaction functionality, and second, on the inside of grippers or hands to support gras** and exploration. Starting from this observation, we propose a categorization for the approaches found in the literature. To provide a basis for understanding these approaches, we devote effort to present the technologies and different measuring principles that were developed over the years, also providing a summary in form of a table. Then, we show the diversity of applications that have been presented in the literature. Finally, we give an overview of the most important trends that will shape the future of this domain. △ Less

Submitted 17 August, 2021; v1 submitted 16 August, 2021; originally announced August 2021.

Journal ref: IEEE Transactions on Robotics 2021

HAIR: Head-mounted AR Intention Recognition

Authors: David Puljiz, Bowen Zhou, Ke Ma, Björn Hein

Abstract: Human teams exhibit both implicit and explicit intention sharing. To further development of human-robot collaboration, intention recognition is crucial on both sides. Present approaches rely on a vast sensor suite on and around the robot to achieve intention recognition. This relegates intuitive human-robot collaboration purely to such bulky systems, which are inadequate for large-scale, real-worl… ▽ More Human teams exhibit both implicit and explicit intention sharing. To further development of human-robot collaboration, intention recognition is crucial on both sides. Present approaches rely on a vast sensor suite on and around the robot to achieve intention recognition. This relegates intuitive human-robot collaboration purely to such bulky systems, which are inadequate for large-scale, real-world scenarios due to their complexity and cost. In this paper we propose an intention recognition system that is based purely on a portable head-mounted display. In addition robot intention visualisation is also supported. We present experiments to show the quality of our human goal estimation component and some basic interactions with an industrial robot. HAIR should raise the quality of interaction between robots and humans, instead of such interactions raising the hair on the necks of the human coworkers. △ Less

Submitted 22 February, 2021; originally announced February 2021.

Comments: As accepted to the 4th Virtual, Augmented and Mixed reality Human-robot interaction workshop (VAM-HRI), HRI 2021

Adapting the Human: Leveraging Wearable Technology in HRI

Authors: David Puljiz, Björn Hein

Abstract: Adhering to current HRI paradigms, all of the sensors, visualisation and legibility of actions and motions are borne by the robot or its working cell. This necessarily makes robots more complex or confines them into specialised, structured environments. We propose leveraging the state of the art of wearable technologies, such as augmented reality head mounted displays, smart watches, sensor tags a… ▽ More Adhering to current HRI paradigms, all of the sensors, visualisation and legibility of actions and motions are borne by the robot or its working cell. This necessarily makes robots more complex or confines them into specialised, structured environments. We propose leveraging the state of the art of wearable technologies, such as augmented reality head mounted displays, smart watches, sensor tags and radio-frequency ranging, to "adapt" the human and reduce the requirements and complexity of robots. △ Less

Submitted 10 December, 2020; originally announced December 2020.

Comments: As submitted to the Robot Metaphors Workshop, ICSR 2020

What the HoloLens Maps Is Your Workspace: Fast Map** and Set-up of Robot Cells via Head Mounted Displays and Augmented Reality

Authors: David Puljiz, Franziska Krebs, Fabian Bösing, Björn Hein

Abstract: Classical methods of modelling and map** robot work cells are time consuming, expensive and involve expert knowledge. We present a novel approach to map** and cell setup using modern Head Mounted Displays (HMDs) that possess self-localisation and map** capabilities. We leveraged these capabilities to create a point cloud of the environment and build an OctoMap - a voxel occupancy grid repres… ▽ More Classical methods of modelling and map** robot work cells are time consuming, expensive and involve expert knowledge. We present a novel approach to map** and cell setup using modern Head Mounted Displays (HMDs) that possess self-localisation and map** capabilities. We leveraged these capabilities to create a point cloud of the environment and build an OctoMap - a voxel occupancy grid representation of the robot's workspace for path planning. Through the use of Augmented Reality (AR) interactions, the user can edit the created Octomap and add security zones. We perform comprehensive tests of the HoloLens' depth sensing capabilities and the quality of the resultant point cloud. A high-end laser scanner is used to provide the ground truth for the evaluation of the point cloud quality. The amount of false-positive and false-negative voxels in the OctoMap are also tested. △ Less

Submitted 26 May, 2020; originally announced May 2020.

Comments: As submited to IROS 2020

Concepts for End-to-end Augmented Reality based Human-Robot Interaction Systems

Authors: David Puljiz, Björn Hein

Abstract: The field of Augmented Reality (AR) based Human Robot Interaction (HRI) has progressed significantly since its inception more than two decades ago. With more advanced devices, particularly head-mounted displays (HMD), freely available programming environments and better connectivity, the possible application space expanded significantly. Here we present concepts and systems currently being develop… ▽ More The field of Augmented Reality (AR) based Human Robot Interaction (HRI) has progressed significantly since its inception more than two decades ago. With more advanced devices, particularly head-mounted displays (HMD), freely available programming environments and better connectivity, the possible application space expanded significantly. Here we present concepts and systems currently being developed at our lab to enable a truly end-to-end application of AR in HRI, from setting up the working environment of the robot, through programming and finally interaction with the programmed robot. Relevant papers by other authors will also be overviewed. We demonstrate the use of such technologies with systems not inherently designed to be collaborative, namely industrial manipulators. By trying to make such industrial systems easily-installable, collaborative and interactive, the vision of universal robot co-workers can be pushed one step closer to reality. The main goal of the paper is to provide a short overview of the capabilities of HMD-based HRI to researchers unfamiliar with the concepts. For researchers already using such techniques, the hope is to perhaps introduce some new ideas and to broaden the field of research. △ Less

Submitted 10 October, 2019; originally announced October 2019.

Comments: As accepted to the "Factory of the Future" workshop, IROS 2019

General Hand Guidance Framework using Microsoft HoloLens

Authors: David Puljiz, Erik Stöhr, Katharina S. Riesterer, Björn Hein, Torsten Kröger

Abstract: Hand guidance emerged from the safety requirements for collaborative robots, namely possessing joint-torque sensors. Since then it has proven to be a powerful tool for easy trajectory programming, allowing lay-users to reprogram robots intuitively. Going beyond, a robot can learn tasks by user demonstrations through kinesthetic teaching, enabling robots to generalise tasks and further reducing the… ▽ More Hand guidance emerged from the safety requirements for collaborative robots, namely possessing joint-torque sensors. Since then it has proven to be a powerful tool for easy trajectory programming, allowing lay-users to reprogram robots intuitively. Going beyond, a robot can learn tasks by user demonstrations through kinesthetic teaching, enabling robots to generalise tasks and further reducing the need for reprogramming. However, hand guidance is still mostly relegated to collaborative robots. Here we propose a method that doesn't require any sensors on the robot or in the robot cell, by using a Microsoft HoloLens augmented reality head mounted display. We reference the robot using a registration algorithm to match the robot model to the spatial mesh. The in-built hand tracking and localisation capabilities are then used to calculate the position of the hands relative to the robot. By decomposing the hand movements into orthogonal rotations and propagating it down through the kinematic chain, we achieve a generalised hand guidance without the need to build a dynamic model of the robot itself. We tested our approach on a commonly used industrial manipulator, the KUKA KR-5. △ Less

Submitted 13 August, 2019; originally announced August 2019.

Comments: As accepted to IROS 2019

Interpreting OWL Complex Classes in AutomationML based on Bidirectional Translation

Authors: Yingbing Hua, Björn Hein

Abstract: The World Wide Web Consortium (W3C) has published several recommendations for building and storing ontologies, including the most recent OWL 2 Web Ontology Language (OWL). These initiatives have been followed by practical implementations that popularize OWL in various domains. For example, OWL has been used for conceptual modeling in industrial engineering, and its reasoning facilities are used to… ▽ More The World Wide Web Consortium (W3C) has published several recommendations for building and storing ontologies, including the most recent OWL 2 Web Ontology Language (OWL). These initiatives have been followed by practical implementations that popularize OWL in various domains. For example, OWL has been used for conceptual modeling in industrial engineering, and its reasoning facilities are used to provide a wealth of services, e.g. model diagnosis, automated code generation, and semantic integration. More specifically, recent studies have shown that OWL is well suited for harmonizing information of engineering tools stored as AutomationML (AML) files. However, OWL and its tools can be cumbersome for direct use by engineers such that an ontology expert is often required in practice. Although much attention has been paid in the literature to overcome this issue by transforming OWL ontologies from/to AML models automatically, dealing with OWL complex classes remains an open research question. In this paper, we introduce the AML concept models for representing OWL complex classes in AutomationML, and present algorithms for the bidirectional translation between OWL complex classes and their corresponding AML concept models. We show that this approach provides an efficient and intuitive interface for nonexperts to visualize, modify, and create OWL complex classes. △ Less

Submitted 4 June, 2019; originally announced June 2019.

Comments: As accepted to IEEE 24th International Conference on Emerging Technologies and Factory Automation (ETFA 2019)

Referencing between a Head-Mounted Device and Robotic Manipulators

Authors: David Puljiz, Katharina S. Riesterer, Björn Hein, Torsten Kröger

Abstract: Having a precise and robust transformation between the robot coordinate system and the AR-device coordinate system is paramount during human-robot interaction (HRI) based on augmented reality using Head mounted displays (HMD), both for intuitive information display and for the tracking of human motions. Most current solutions in this area rely either on the tracking of visual markers, e.g. QR code… ▽ More Having a precise and robust transformation between the robot coordinate system and the AR-device coordinate system is paramount during human-robot interaction (HRI) based on augmented reality using Head mounted displays (HMD), both for intuitive information display and for the tracking of human motions. Most current solutions in this area rely either on the tracking of visual markers, e.g. QR codes, or on manual referencing, both of which provide unsatisfying results. Meanwhile a plethora of object detection and referencing methods exist in the wider robotic and machine vision communities. The precision of the referencing is likewise almost never measured. Here we would like to address this issue by firstly presenting an overview of currently used referencing methods between robots and HMDs. This is followed by a brief overview of object detection and referencing methods used in the field of robotics. Based on these methods we suggest three classes of referencing algorithms we intend to pursue - semi-automatic, on-shot; automatic, one-shot; and automatic continuous. We describe the general workflows of these three classes as well as describing our proposed algorithms in each of these classes. Finally we present the first experimental results of a semi-automatic referencing algorithm, tested on an industrial KUKA KR-5 manipulator. △ Less

Submitted 4 April, 2019; originally announced April 2019.

Comments: As submitted to the 2nd VAM-HRI workshop, HRI 2019

Sensorless Hand Guidance using Microsoft Hololens

Authors: David Puljiz, Erik Stöhr, Katharina S. Riesterer, Björn Hein, Torsten Kröger

Abstract: Hand guidance of robots has proven to be a useful tool both for programming trajectories and in kinesthetic teaching. However hand guidance is usually relegated to robots possessing joint-torque sensors (JTS). Here we propose to extend hand guidance to robots lacking those sensors through the use of an Augmented Reality (AR) device, namely Microsoft's Hololens. Augmented reality devices have been… ▽ More Hand guidance of robots has proven to be a useful tool both for programming trajectories and in kinesthetic teaching. However hand guidance is usually relegated to robots possessing joint-torque sensors (JTS). Here we propose to extend hand guidance to robots lacking those sensors through the use of an Augmented Reality (AR) device, namely Microsoft's Hololens. Augmented reality devices have been envisioned as a helpful addition to ease both robot programming and increase situational awareness of humans working in close proximity to robots. We reference the robot by using a registration algorithm to match a robot model to the spatial mesh. The in-built hand tracking capabilities are then used to calculate the position of the hands relative to the robot. By decomposing the hand movements into orthogonal rotations we achieve a completely sensorless hand guidance without any need to build a dynamic model of the robot itself. We did the first tests our approach on a commonly used industrial manipulator, the KUKA KR-5. △ Less

Submitted 15 January, 2019; originally announced January 2019.

Comments: Accepted to HRI2019 - Late Breaking reports

Human Intention Estimation based on Hidden Markov Model Motion Validation for Safe Flexible Robotized Warehouses

Authors: Tomislav Petković, David Puljiz, Ivan Marković, Björn Hein

Abstract: With the substantial growth of logistics businesses the need for larger warehouses and their automation arises, thus using robots as assistants to human workers is becoming a priority. In order to operate efficiently and safely, robot assistants or the supervising system should recognize human intentions in real-time. Theory of mind (ToM) is an intuitive human conception of other humans' mental st… ▽ More With the substantial growth of logistics businesses the need for larger warehouses and their automation arises, thus using robots as assistants to human workers is becoming a priority. In order to operate efficiently and safely, robot assistants or the supervising system should recognize human intentions in real-time. Theory of mind (ToM) is an intuitive human conception of other humans' mental state, i.e., beliefs and desires, and how they cause behavior. In this paper we propose a ToM based human intention estimation algorithm for flexible robotized warehouses. We observe human's, i.e., worker's motion and validate it with respect to the goal locations using generalized Voronoi diagram based path planning. These observations are then processed by the proposed hidden Markov model framework which estimates worker intentions in an online manner, capable of handling changing environments. To test the proposed intention estimation we ran experiments in a real-world laboratory warehouse with a worker wearing Microsoft Hololens augmented reality glasses. Furthermore, in order to demonstrate the scalability of the approach to larger warehouses, we propose to use virtual reality digital warehouse twins in order to realistically simulate worker behavior. We conducted intention estimation experiments in the larger warehouse digital twin with up to 24 running robots. We demonstrate that the proposed framework estimates warehouse worker intentions precisely and in the end we discuss the experimental results. △ Less

Submitted 20 November, 2018; originally announced November 2018.

Journal ref: Robotics and Computer-Integrated Manufacturing 57 (2019): 182-196

Implementation of Augmented Reality in Autonomous Warehouses: Challenges and Opportunities

Authors: David Puljiz, Gleb Gorbachev, Björn Hein

Abstract: Autonomous warehouses with mobile, rack-carrying robots are starting to become commonplace, with systems such as Amazon's Kiva and Swisslog's CarryPick already implemented in functional warehouses. Such warehouses however still require human intervention for object picking and maintenance. In the European project SafeLog we are develo** a safety-vest, used for safety-critical ranging and stoppin… ▽ More Autonomous warehouses with mobile, rack-carrying robots are starting to become commonplace, with systems such as Amazon's Kiva and Swisslog's CarryPick already implemented in functional warehouses. Such warehouses however still require human intervention for object picking and maintenance. In the European project SafeLog we are develo** a safety-vest, used for safety-critical ranging and stop** of mobile robots, an improved planner that can handle large fleets of heterogeneous agents as well as an AR interaction system to navigate and support human workers in such automated environments. Here we present the AR interaction modalities, namely navigation, pick-by-AR and general system interactions that were developed at the moment of writing, as well as the overall system concept and planned future work. △ Less

Submitted 1 June, 2018; originally announced June 2018.

Comments: As submitted to the VAM-HRI workshop, HRI 2018

Modelling of Oscillations in Two-Dimensional Echo-Spectra of the Fenna-Matthews-Olson Complex

Authors: Birgit Hein, Christoph Kreisbeck, Tobias Kramer, Mirta Rodríguez

Abstract: Recent experimental observations of time-dependent beatings in the two-dimensional echo-spectra of light-harvesting complexes at ambient temperatures have opened up the question whether coherence and wave-like behaviour plays a significant role in photosynthesis. We perform a numerical study of the absorption and echo-spectra of the Fenna-Matthews-Olson (FMO) complex in chlorobium tepidum and anal… ▽ More Recent experimental observations of time-dependent beatings in the two-dimensional echo-spectra of light-harvesting complexes at ambient temperatures have opened up the question whether coherence and wave-like behaviour plays a significant role in photosynthesis. We perform a numerical study of the absorption and echo-spectra of the Fenna-Matthews-Olson (FMO) complex in chlorobium tepidum and analyse the requirements in the theoretical model needed to reproduce beatings in the calculated spectra. The energy transfer in the FMO pigment-protein complex is theoretically described by an exciton Hamiltonian coupled to a phonon bath which account for the pigments electronic and vibrational excitations respectively. We use the hierarchical equations of motions method to treat the strong couplings in a non-perturbative way. We show that the oscillations in the two-dimensional echo-spectra persist in the presence of thermal noise and static disorder. △ Less

Submitted 8 December, 2011; v1 submitted 7 October, 2011; originally announced October 2011.

Comments: updated text and references, corrected figures 4-7, 20 pages, 7 figures

Journal ref: New J. Phys. 14 023018 (2012)

High-performance solution of hierarchical equations of motions for studying energy-transfer in light-harvesting complexes

Authors: Christoph Kreisbeck, Tobias Kramer, Mirta Rodriguez, Birgit Hein

Abstract: Excitonic models of light-harvesting complexes, where the vibrational degrees of freedom are treated as a bath, are commonly used to describe the motion of the electronic excitation through a molecule. Recent experiments point toward the possibility of memory effects in this process and require to consider time non-local propagation techniques. The hierarchical equations of motion (HEOM) were prop… ▽ More Excitonic models of light-harvesting complexes, where the vibrational degrees of freedom are treated as a bath, are commonly used to describe the motion of the electronic excitation through a molecule. Recent experiments point toward the possibility of memory effects in this process and require to consider time non-local propagation techniques. The hierarchical equations of motion (HEOM) were proposed by Ishizaki and Fleming to describe the site-dependent reorganization dynamics of protein environments (J. Chem. Phys., 130, p. 234111, 2009), which plays a significant role in photosynthetic electronic energy transfer. HEOM are often used as a reference for other approximate methods, but have been implemented only for small systems due to their adverse computational scaling with the system size. Here, we show that HEOM are also solvable for larger systems, since the underlying algorithm is ideally suited for the usage of graphics processing units (GPU). The tremendous reduction in computational time due to the GPU allows us to perform a systematic study of the energy-transfer efficiency in the Fenna-Matthews-Olson (FMO) light-harvesting complex at physiological temperature under full consideration of memory-effects. We find that approximative methods differ qualitatively and quantitatively from the HEOM results and discuss the importance of finite temperature to achieve high energy-transfer efficiencies. △ Less

Submitted 25 May, 2011; v1 submitted 20 December, 2010; originally announced December 2010.

Comments: 14 pages; Journal of Chemical Theory and Computation (2011)

Journal ref: Journal of Chemical Theory and Computation, volume 7, page 2166 (2011)

Quantum search algorithms on a regular lattice

Authors: Birgit Hein, Gregor Tanner

Abstract: Quantum algorithms for searching one or more marked items on a d-dimensional lattice provide an extension of Grover's search algorithm including a spatial component. We demonstrate that these lattice search algorithms can be viewed in terms of the level dynamics near an avoided crossing of a one-parameter family of quantum random walks. We give approximations for both the level-splitting at the av… ▽ More Quantum algorithms for searching one or more marked items on a d-dimensional lattice provide an extension of Grover's search algorithm including a spatial component. We demonstrate that these lattice search algorithms can be viewed in terms of the level dynamics near an avoided crossing of a one-parameter family of quantum random walks. We give approximations for both the level-splitting at the avoided crossing and the effectively two-dimensional subspace of the full Hilbert space spanning the level crossing. This makes it possible to give the leading order behaviour for the search time and the localisation probability in the limit of large lattice size including the leading order coefficients. For d=2 and d=3, these coefficients are calculated explicitly. Closed form expressions are given for higher dimensions. △ Less

Submitted 20 May, 2010; originally announced May 2010.

Wave communication across regular lattices

Authors: Birgit Hein, Gregor Tanner

Abstract: We propose a novel way to communicate signals in the form of waves across a d - dimensional lattice. The mechanism is based on quantum search algorithms and makes it possible to both search for marked positions in a regular grid and to communicate between two (or more) points on the lattice. Remarkably, neither the sender nor the receiver needs to know the position of each other despite the fact… ▽ More We propose a novel way to communicate signals in the form of waves across a d - dimensional lattice. The mechanism is based on quantum search algorithms and makes it possible to both search for marked positions in a regular grid and to communicate between two (or more) points on the lattice. Remarkably, neither the sender nor the receiver needs to know the position of each other despite the fact that the signal is only exchanged between the contributing parties. This is an example of using wave interference as a resource by controlling localisation phenomena effectively. Possible experimental realisations will be discussed. △ Less

Submitted 2 January, 2010; originally announced January 2010.

Journal ref: Physical Review Letters 103, 260501 (2009)

Quantum search algorithms on the hypercube

Authors: Birgit Hein, Gregor Tanner

Abstract: We investigate a set of discrete-time quantum search algorithms on the n-dimensional hypercube following a proposal by Shenvi, Kempe and Whaley. We show that there exists a whole class of quantum search algorithms in the symmetry reduced space which perform a search of a marked vertex in time of order $\sqrt{N}$ where $N = 2^n$, the number of vertices. In analogy to Grover's algorithm, the spati… ▽ More We investigate a set of discrete-time quantum search algorithms on the n-dimensional hypercube following a proposal by Shenvi, Kempe and Whaley. We show that there exists a whole class of quantum search algorithms in the symmetry reduced space which perform a search of a marked vertex in time of order $\sqrt{N}$ where $N = 2^n$, the number of vertices. In analogy to Grover's algorithm, the spatial search is effectively facilitated through a rotation in a two-level sub-space of the full Hilbert space. In the hypercube, these two-level systems are introduced through avoided crossings. We give estimates on the quantum states forming the 2-level sub-spaces at the avoided crossings and derive improved estimates on the search times. △ Less

Submitted 17 June, 2009; originally announced June 2009.

Journal ref: J. Phys. A: Math. Theor. 42 (2009) 085303