-
HARMONIOUS -- Human-like reactive motion control and multimodal perception for humanoid robots
Authors:
Jakub Rozlivek,
Alessandro Roncone,
Ugo Pattacini,
Matej Hoffmann
Abstract:
For safe and effective operation of humanoid robots in human-populated environments, the problem of commanding a large number of Degrees of Freedom (DoF) while simultaneously considering dynamic obstacles and human proximity has still not been solved. We present a new reactive motion controller that commands two arms of a humanoid robot and three torso joints (17 DoF in total). We formulate a quad…
▽ More
For safe and effective operation of humanoid robots in human-populated environments, the problem of commanding a large number of Degrees of Freedom (DoF) while simultaneously considering dynamic obstacles and human proximity has still not been solved. We present a new reactive motion controller that commands two arms of a humanoid robot and three torso joints (17 DoF in total). We formulate a quadratic program that seeks joint velocity commands respecting multiple constraints while minimizing the magnitude of the velocities. We introduce a new unified treatment of obstacles that dynamically maps visual and proximity (pre-collision) and tactile (post-collision) obstacles as additional constraints to the motion controller, in a distributed fashion over surface of the upper-body of the iCub robot (with 2000 pressure-sensitive receptors). The bio-inspired controller: (i) produces human-like minimum jerk movement profiles; (ii) gives rise to a robot with whole-body visuo-tactile awareness, resembling peripersonal space representations. The controller was extensively experimentally validated, including a physical human-robot interaction scenario.
△ Less
Submitted 5 December, 2023;
originally announced December 2023.
-
iCub3 Avatar System: Enabling Remote Fully-Immersive Embodiment of Humanoid Robots
Authors:
Stefano Dafarra,
Ugo Pattacini,
Giulio Romualdi,
Lorenzo Rapetti,
Riccardo Grieco,
Kourosh Darvish,
Gianluca Milani,
Enrico Valli,
Ines Sorrentino,
Paolo Maria Viceconte,
Alessandro Scalzo,
Silvio Traversaro,
Carlotta Sartore,
Mohamed Elobaid,
Nuno Guedelha,
Connor Herron,
Alexander Leonessa,
Francesco Draicchio,
Giorgio Metta,
Marco Maggiali,
Daniele Pucci
Abstract:
We present an avatar system designed to facilitate the embodiment of humanoid robots by human operators, validated through iCub3, a humanoid developed at the Istituto Italiano di Tecnologia (IIT). More precisely, the contribution of the paper is twofold: first, we present the humanoid iCub3 as a robotic avatar which integrates the latest significant improvements after about fifteen years of develo…
▽ More
We present an avatar system designed to facilitate the embodiment of humanoid robots by human operators, validated through iCub3, a humanoid developed at the Istituto Italiano di Tecnologia (IIT). More precisely, the contribution of the paper is twofold: first, we present the humanoid iCub3 as a robotic avatar which integrates the latest significant improvements after about fifteen years of development of the iCub series; second, we present a versatile avatar system enabling humans to embody humanoid robots encompassing aspects such as locomotion, manipulation, voice, and face expressions with comprehensive sensory feedback including visual, auditory, haptic, weight, and touch modalities. We validate the system by implementing several avatar architecture instances, each tailored to specific requirements. First, we evaluated the optimized architecture for verbal, non-verbal, and physical interactions with a remote recipient. This testing involved the operator in Genoa and the avatar in the Biennale di Venezia, Venice - about 290 Km away - thus allowing the operator to visit remotely the Italian art exhibition. Second, we evaluated the optimised architecture for recipient physical collaboration and public engagement on-stage, live, at the We Make Future show, a prominent world digital innovation festival. In this instance, the operator was situated in Genoa while the avatar operates in Rimini - about 300 Km away - interacting with a recipient who entrusted the avatar a payload to carry on stage before an audience of approximately 2000 spectators. Third, we present the architecture implemented by the iCub Team for the ANA Avatar XPrize competition.
△ Less
Submitted 25 January, 2024; v1 submitted 14 March, 2022;
originally announced March 2022.
-
ROFT: Real-Time Optical Flow-Aided 6D Object Pose and Velocity Tracking
Authors:
Nicola A. Piga,
Yuriy Onyshchuk,
Giulia Pasquale,
Ugo Pattacini,
Lorenzo Natale
Abstract:
6D object pose tracking has been extensively studied in the robotics and computer vision communities. The most promising solutions, leveraging on deep neural networks and/or filtering and optimization, exhibit notable performance on standard benchmarks. However, to our best knowledge, these have not been tested thoroughly against fast object motions. Tracking performance in this scenario degrades…
▽ More
6D object pose tracking has been extensively studied in the robotics and computer vision communities. The most promising solutions, leveraging on deep neural networks and/or filtering and optimization, exhibit notable performance on standard benchmarks. However, to our best knowledge, these have not been tested thoroughly against fast object motions. Tracking performance in this scenario degrades significantly, especially for methods that do not achieve real-time performance and introduce non negligible delays. In this work, we introduce ROFT, a Kalman filtering approach for 6D object pose and velocity tracking from a stream of RGB-D images. By leveraging real-time optical flow, ROFT synchronizes delayed outputs of low frame rate Convolutional Neural Networks for instance segmentation and 6D object pose estimation with the RGB-D input stream to achieve fast and precise 6D object pose and velocity tracking. We test our method on a newly introduced photorealistic dataset, Fast-YCB, which comprises fast moving objects from the YCB model set, and on the dataset for object and hand pose estimation HO-3D. Results demonstrate that our approach outperforms state-of-the-art methods for 6D object pose tracking, while also providing 6D object velocity tracking. A video showing the experiments is provided as supplementary material.
△ Less
Submitted 6 November, 2021;
originally announced November 2021.
-
GRASPA 1.0: GRASPA is a Robot Arm graS** Performance benchmArk
Authors:
Fabrizio Bottarel,
Giulia Vezzani,
Ugo Pattacini,
Lorenzo Natale
Abstract:
The use of benchmarks is a widespread and scientifically meaningful practice to validate performance of different approaches to the same task. In the context of robot gras** the use of common object sets has emerged in recent years, however no dominant protocols and metrics to test gras** pipelines have taken root yet. In this paper, we present version 1.0 of GRASPA, a benchmark to test effect…
▽ More
The use of benchmarks is a widespread and scientifically meaningful practice to validate performance of different approaches to the same task. In the context of robot gras** the use of common object sets has emerged in recent years, however no dominant protocols and metrics to test gras** pipelines have taken root yet. In this paper, we present version 1.0 of GRASPA, a benchmark to test effectiveness of gras** pipelines on physical robot setups. This approach tackles the complexity of such pipelines by proposing different metrics that account for the features and limits of the test platform. As an example application, we deploy GRASPA on the iCub humanoid robot and use it to benchmark our gras** pipeline. As closing remarks, we discuss how the GRASPA indicators we obtained as outcome can provide insight into how different steps of the pipeline affect the overall gras** performance.
△ Less
Submitted 12 February, 2020;
originally announced February 2020.
-
Sequence-to-Sequence Natural Language to Humanoid Robot Sign Language
Authors:
Jennifer J. Gago,
Valentina Vasco,
Bartek Łukawski,
Ugo Pattacini,
Vadim Tikhanoff,
Juan G. Victores,
Carlos Balaguer
Abstract:
This paper presents a study on natural language to sign language translation with human-robot interaction application purposes. By means of the presented methodology, the humanoid robot TEO is expected to represent Spanish sign language automatically by converting text into movements, thanks to the performance of neural networks. Natural language to sign language translation presents several chall…
▽ More
This paper presents a study on natural language to sign language translation with human-robot interaction application purposes. By means of the presented methodology, the humanoid robot TEO is expected to represent Spanish sign language automatically by converting text into movements, thanks to the performance of neural networks. Natural language to sign language translation presents several challenges to developers, such as the discordance between the length of input and output data and the use of non-manual markers. Therefore, neural networks and, consequently, sequence-to-sequence models, are selected as a data-driven system to avoid traditional expert system approaches or temporal dependencies limitations that lead to limited or too complex translation systems. To achieve these objectives, it is necessary to find a way to perform human skeleton acquisition in order to collect the signing input data. OpenPose and skeletonRetriever are proposed for this purpose and a 3D sensor specification study is developed to select the best acquisition hardware.
△ Less
Submitted 9 July, 2019;
originally announced July 2019.
-
Compact Real-time avoidance on a Humanoid Robot for Human-robot Interaction
Authors:
Dong Hai Phuong Nguyen,
Matej Hoffmann,
Alessandro Roncone,
Ugo Pattacini,
Giorgio Metta
Abstract:
With robots leaving factories and entering less controlled domains, possibly sharing the space with humans, safety is paramount and multimodal awareness of the body surface and the surrounding environment is fundamental. Taking inspiration from peripersonal space representations in humans, we present a framework on a humanoid robot that dynamically maintains such a protective safety zone, composed…
▽ More
With robots leaving factories and entering less controlled domains, possibly sharing the space with humans, safety is paramount and multimodal awareness of the body surface and the surrounding environment is fundamental. Taking inspiration from peripersonal space representations in humans, we present a framework on a humanoid robot that dynamically maintains such a protective safety zone, composed of the following main components: (i) a human 2D keypoints estimation pipeline employing a deep learning based algorithm, extended here into 3D using disparity; (ii) a distributed peripersonal space representation around the robot's body parts; (iii) a reaching controller that incorporates all obstacles entering the robot's safety zone on the fly into the task. Pilot experiments demonstrate that an effective safety margin between the robot's and the human's body parts is kept. The proposed solution is flexible and versatile since the safety zone around individual robot and human body parts can be selectively modulated---here we demonstrate stronger avoidance of the human head compared to rest of the body. Our system works in real time and is self-contained, with no external sensory equipment and use of onboard cameras only.
△ Less
Submitted 17 January, 2018;
originally announced January 2018.
-
Markerless visual servoing on unknown objects for humanoid robot platforms
Authors:
Claudio Fantacci,
Giulia Vezzani,
Ugo Pattacini,
Vadim Tikhanoff,
Lorenzo Natale
Abstract:
To precisely reach for an object with a humanoid robot, it is of central importance to have good knowledge of both end-effector, object pose and shape. In this work we propose a framework for markerless visual servoing on unknown objects, which is divided in four main parts: I) a least-squares minimization problem is formulated to find the volume of the object graspable by the robot's hand using i…
▽ More
To precisely reach for an object with a humanoid robot, it is of central importance to have good knowledge of both end-effector, object pose and shape. In this work we propose a framework for markerless visual servoing on unknown objects, which is divided in four main parts: I) a least-squares minimization problem is formulated to find the volume of the object graspable by the robot's hand using its stereo vision; II) a recursive Bayesian filtering technique, based on Sequential Monte Carlo (SMC) filtering, estimates the 6D pose (position and orientation) of the robot's end-effector without the use of markers; III) a nonlinear constrained optimization problem is formulated to compute the desired graspable pose about the object; IV) an image-based visual servo control commands the robot's end-effector toward the desired pose. We demonstrate effectiveness and robustness of our approach with extensive experiments on the iCub humanoid robot platform, achieving real-time computation, smooth trajectories and sub-pixel precisions.
△ Less
Submitted 12 October, 2017;
originally announced October 2017.
-
DAC-h3: A Proactive Robot Cognitive Architecture to Acquire and Express Knowledge About the World and the Self
Authors:
Clément Moulin-Frier,
Tobias Fischer,
Maxime Petit,
Grégoire Pointeau,
Jordi-Ysard Puigbo,
Ugo Pattacini,
Sock Ching Low,
Daniel Camilleri,
Phuong Nguyen,
Matej Hoffmann,
Hyung ** Chang,
Martina Zambelli,
Anne-Laure Mealier,
Andreas Damianou,
Giorgio Metta,
Tony J. Prescott,
Yiannis Demiris,
Peter Ford Dominey,
Paul F. M. J. Verschure
Abstract:
This paper introduces a cognitive architecture for a humanoid robot to engage in a proactive, mixed-initiative exploration and manipulation of its environment, where the initiative can originate from both the human and the robot. The framework, based on a biologically-grounded theory of the brain and mind, integrates a reactive interaction engine, a number of state-of-the-art perceptual and motor…
▽ More
This paper introduces a cognitive architecture for a humanoid robot to engage in a proactive, mixed-initiative exploration and manipulation of its environment, where the initiative can originate from both the human and the robot. The framework, based on a biologically-grounded theory of the brain and mind, integrates a reactive interaction engine, a number of state-of-the-art perceptual and motor learning algorithms, as well as planning abilities and an autobiographical memory. The architecture as a whole drives the robot behavior to solve the symbol grounding problem, acquire language capabilities, execute goal-oriented behavior, and express a verbal narrative of its own experience in the world. We validate our approach in human-robot interaction experiments with the iCub humanoid robot, showing that the proposed cognitive architecture can be applied in real time within a realistic scenario and that it can be used with naive users.
△ Less
Submitted 18 September, 2017; v1 submitted 12 June, 2017;
originally announced June 2017.
-
Visual end-effector tracking using a 3D model-aided particle filter for humanoid robot platforms
Authors:
Claudio Fantacci,
Ugo Pattacini,
Vadim Tikhanoff,
Lorenzo Natale
Abstract:
This paper addresses recursive markerless estimation of a robot's end-effector using visual observations from its cameras. The problem is formulated into the Bayesian framework and addressed using Sequential Monte Carlo (SMC) filtering. We use a 3D rendering engine and Computer Aided Design (CAD) schematics of the robot to virtually create images from the robot's camera viewpoints. These images ar…
▽ More
This paper addresses recursive markerless estimation of a robot's end-effector using visual observations from its cameras. The problem is formulated into the Bayesian framework and addressed using Sequential Monte Carlo (SMC) filtering. We use a 3D rendering engine and Computer Aided Design (CAD) schematics of the robot to virtually create images from the robot's camera viewpoints. These images are then used to extract information and estimate the pose of the end-effector. To this aim, we developed a particle filter for estimating the position and orientation of the robot's end-effector using the Histogram of Oriented Gradient (HOG) descriptors to capture robust characteristic features of shapes in both cameras and rendered images. We implemented the algorithm on the iCub humanoid robot and employed it in a closed-loop reaching scenario. We demonstrate that the tracking is robust to clutter, allows compensating for errors in the robot kinematics and servoing the arm in closed loop using vision.
△ Less
Submitted 4 August, 2017; v1 submitted 14 March, 2017;
originally announced March 2017.
-
Memory Unscented Particle Filter for 6-DOF Tactile Localization
Authors:
Giulia Vezzani,
Ugo Pattacini,
Giorgio Battistelli,
Luigi Chisci,
Lorenzo Natale
Abstract:
This paper addresses 6-DOF (degree-of-freedom) tactile localization, i.e. the pose estimation of tridimensional objects given tactile measurements. This estimation problem is fundamental for the operation of autonomous robots that are often required to manipulate and grasp objects whose pose is a-priori unknown. The nature of tactile measurements, the strict time requirements for real-time operati…
▽ More
This paper addresses 6-DOF (degree-of-freedom) tactile localization, i.e. the pose estimation of tridimensional objects given tactile measurements. This estimation problem is fundamental for the operation of autonomous robots that are often required to manipulate and grasp objects whose pose is a-priori unknown. The nature of tactile measurements, the strict time requirements for real-time operation and the multimodality of the involved probability distributions pose remarkable challenges and call for advanced nonlinear filtering techniques. Following a Bayesian approach, this paper proposes a novel and effective algorithm, named Memory Unscented Particle Filter (MUPF), which solves the 6-DOF localization problem recursively in real-time by only exploiting contact point measurements. MUPF combines a modified particle filter that incorporates a sliding memory of past measurements to better handle multimodal distributions, along with the unscented Kalman filter that moves the particles towards regions of the search space that are more likely with the measurements. The performance of the proposed MUPF algorithm has been assessed both in simulation and on a real robotic system equipped with tactile sensors (i.e., the iCub humanoid robot). The experiments show that the algorithm provides accurate and reliable localization even with a low number of particles and, hence, is compatible with real-time requirements.
△ Less
Submitted 10 November, 2016; v1 submitted 10 July, 2016;
originally announced July 2016.
-
Gaze Stabilization for Humanoid Robots: a Comprehensive Framework
Authors:
Alessandro Roncone,
Ugo Pattacini,
Giorgio Metta,
Lorenzo Natale
Abstract:
Gaze stabilization is an important requisite for humanoid robots. Previous work on this topic has focused on the integration of inertial and visual information. Little attention has been given to a third component, which is the knowledge that the robot has about its own movement. In this work we propose a comprehensive framework for gaze stabilization in a humanoid robot. We focus on the problem o…
▽ More
Gaze stabilization is an important requisite for humanoid robots. Previous work on this topic has focused on the integration of inertial and visual information. Little attention has been given to a third component, which is the knowledge that the robot has about its own movement. In this work we propose a comprehensive framework for gaze stabilization in a humanoid robot. We focus on the problem of compensating for disturbances induced in the cameras due to self-generated movements of the robot. In this work we employ two separate signals for stabilization: (1) an anticipatory term obtained from the velocity commands sent to the joints while the robot moves autonomously; (2) a feedback term from the on board gyroscope, which compensates unpredicted external disturbances. We first provide the mathematical formulation to derive the forward and the differential kinematics of the fixation point of the stereo system. We finally test our method on the iCub robot. We show that the stabilization consistently reduces the residual optical flow during the movement of the robot and in presence of external disturbances. We also demonstrate that proper integration of the neck DoF is crucial to achieve correct stabilization.
△ Less
Submitted 13 November, 2014;
originally announced November 2014.
