Showing 1–48 of 48 results for author: Caldwell, D

Kinematically-Decoupled Impedance Control for Fast Object Visual Servoing and Gras** on Quadruped Manipulators

Authors: Riccardo Parosi, Mattia Risiglione, Darwin G. Caldwell, Claudio Semini, Victor Barasuol

Abstract: We propose a control pipeline for SAG (Searching, Approaching, and Gras**) of objects, based on a decoupled arm kinematic chain and impedance control, which integrates image-based visual servoing (IBVS). The kinematic decoupling allows for fast end-effector motions and recovery that leads to robust visual servoing. The whole approach and pipeline can be generalized for any mobile platform (wheel… ▽ More We propose a control pipeline for SAG (Searching, Approaching, and Gras**) of objects, based on a decoupled arm kinematic chain and impedance control, which integrates image-based visual servoing (IBVS). The kinematic decoupling allows for fast end-effector motions and recovery that leads to robust visual servoing. The whole approach and pipeline can be generalized for any mobile platform (wheeled or tracked vehicles), but is most suitable for dynamically moving quadruped manipulators thanks to their reactivity against disturbances. The compliance of the impedance controller makes the robot safer for interactions with humans and the environment. We demonstrate the performance and robustness of the proposed approach with various experiments on our 140 kg HyQReal quadruped robot equipped with a 7-DoF manipulator arm. The experiments consider dynamic locomotion, tracking under external disturbances, and fast motions of the target object. △ Less

Submitted 10 July, 2023; originally announced July 2023.

Comments: Accepted as contributed paper at 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023)

Reactive Landing Controller for Quadruped Robots

Authors: Francesco Roscia, Michele Focchi, Andrea Del Prete, Darwin G. Caldwell, Claudio Semini

Abstract: Quadruped robots are machines intended for challenging and harsh environments. Despite the progress in locomotion strategy, safely recovering from unexpected falls or planned drops is still an open problem. It is further made more difficult when high horizontal velocities are involved. In this work, we propose an optimization-based reactive Landing Controller that uses only proprioceptive measures… ▽ More Quadruped robots are machines intended for challenging and harsh environments. Despite the progress in locomotion strategy, safely recovering from unexpected falls or planned drops is still an open problem. It is further made more difficult when high horizontal velocities are involved. In this work, we propose an optimization-based reactive Landing Controller that uses only proprioceptive measures for torque-controlled quadruped robots that free-fall on a flat horizontal ground, knowing neither the distance to the landing surface nor the flight time. Based on an estimate of the Center of Mass horizontal velocity, the method uses the Variable Height Springy Inverted Pendulum model for continuously recomputing the feet position while the robot is falling. In this way, the quadruped is ready to attain a successful landing in all directions, even in the presence of significant horizontal velocities. The method is demonstrated to dramatically enlarge the region of horizontal velocities that can be dealt with by a naive approach that keeps the feet still during the airborne stage. To the best of our knowledge, this is the first time that a quadruped robot can successfully recover from falls with horizontal velocities up to 3 m/s in simulation. Experiments prove that the used platform, Go1, can successfully attain a stable standing configuration from falls with various horizontal velocity and different angular perturbations. △ Less

Submitted 12 September, 2023; v1 submitted 12 May, 2023; originally announced May 2023.

Comments: 8 pages, 5 figures, 2 tables, submitted to ral, accompanying video at https://youtu.be/KnmNbhkOKWI

Report number: 10.1109/LRA.2023.3313919

Journal ref: IEEE Robotics and Automation Letters (RA-L), 2023

Control of a Back-Support Exoskeleton to Assist Carrying Activities

Authors: Maria Lazzaroni, Giorgia Chini, Francesco Draicchio, Christian Di Natali, Darwin G. Caldwell, Jesús Ortiz

Abstract: Back-support exoskeletons are commonly used in the workplace to reduce low back pain risk for workers performing demanding activities. However, for the assistance of tasks differing from lifting, back-support exoskeletons potential has not been exploited extensively. This work focuses on the use of an active back-support exoskeleton to assist carrying. Two control strategies are designed that modu… ▽ More Back-support exoskeletons are commonly used in the workplace to reduce low back pain risk for workers performing demanding activities. However, for the assistance of tasks differing from lifting, back-support exoskeletons potential has not been exploited extensively. This work focuses on the use of an active back-support exoskeleton to assist carrying. Two control strategies are designed that modulate the exoskeleton torques to comply with the task assistance requirements. In particular, two gait phase detection frameworks are exploited to adapt the assistance according to the legs' motion. The two strategies are assessed through an experimental analysis on ten subjects. Carrying task is performed without and with the exoskeleton assistance. Results prove the potential of the presented controls in assisting the task without hindering the gait movement and improving the usability experienced by users. Moreover, the exoskeleton assistance significantly reduces the lumbar load associated with the task, demonstrating its promising use for risk mitigation in the workplace. △ Less

Submitted 3 July, 2023; v1 submitted 11 May, 2023; originally announced May 2023.

Comments: in-press article, accepted for 2023 IEEE International Conference on Rehabilitation Robotics (ICORR)

Multiplicity Boost Of Transit Signal Classifiers: Validation of 69 New Exoplanets Using The Multiplicity Boost of ExoMiner

Authors: Hamed Valizadegan, Miguel J. S. Martinho, Jon M. Jenkins, Douglas A. Caldwell, Joseph D. Twicken, Stephen T. Bryson

Abstract: Most existing exoplanets are discovered using validation techniques rather than being confirmed by complementary observations. These techniques generate a score that is typically the probability of the transit signal being an exoplanet (y(x)=exoplanet) given some information related to that signal (represented by x). Except for the validation technique in Rowe et al. (2014) that uses multiplicity… ▽ More Most existing exoplanets are discovered using validation techniques rather than being confirmed by complementary observations. These techniques generate a score that is typically the probability of the transit signal being an exoplanet (y(x)=exoplanet) given some information related to that signal (represented by x). Except for the validation technique in Rowe et al. (2014) that uses multiplicity information to generate these probability scores, the existing validation techniques ignore the multiplicity boost information. In this work, we introduce a framework with the following premise: given an existing transit signal vetter (classifier), improve its performance using multiplicity information. We apply this framework to several existing classifiers, which include vespa (Morton et al. 2016), Robovetter (Coughlin et al. 2017), AstroNet (Shallue & Vanderburg 2018), ExoNet (Ansdel et al. 2018), GPC and RFC (Armstrong et al. 2020), and ExoMiner (Valizadegan et al. 2022), to support our claim that this framework is able to improve the performance of a given classifier. We then use the proposed multiplicity boost framework for ExoMiner V1.2, which addresses some of the shortcomings of the original ExoMiner classifier (Valizadegan et al. 2022), and validate 69 new exoplanets for systems with multiple KOIs from the Kepler catalog. △ Less

Submitted 5 May, 2023; v1 submitted 3 May, 2023; originally announced May 2023.

Comments: The paper is accepted for publication in the Astronomical Journal in April 27th, 2023

Learning Skills from Demonstrations: A Trend from Motion Primitives to Experience Abstraction

Authors: Mehrdad Tavassoli, Sunny Katyara, Maria Pozzi, Nikhil Deshpande, Darwin G. Caldwell, Domenico Prattichizzo

Abstract: The uses of robots are changing from static environments in factories to encompass novel concepts such as Human-Robot Collaboration in unstructured settings. Pre-programming all the functionalities for robots becomes impractical, and hence, robots need to learn how to react to new events autonomously, just like humans. However, humans, unlike machines, are naturally skilled in responding to unexpe… ▽ More The uses of robots are changing from static environments in factories to encompass novel concepts such as Human-Robot Collaboration in unstructured settings. Pre-programming all the functionalities for robots becomes impractical, and hence, robots need to learn how to react to new events autonomously, just like humans. However, humans, unlike machines, are naturally skilled in responding to unexpected circumstances based on either experiences or observations. Hence, embedding such anthropoid behaviours into robots entails the development of neuro-cognitive models that emulate motor skills under a robot learning paradigm. Effective encoding of these skills is bound to the proper choice of tools and techniques. This paper studies different motion and behaviour learning methods ranging from Movement Primitives (MP) to Experience Abstraction (EA), applied to different robotic tasks. These methods are scrutinized and then experimentally benchmarked by reconstructing a standard pick-n-place task. Apart from providing a standard guideline for the selection of strategies and algorithms, this paper aims to draw a perspectives on their possible extensions and improvements △ Less

Submitted 14 October, 2022; originally announced October 2022.

Comments: Under Review at IEEE TCDS for future Publication

A Perception-Driven Approach To Immersive Remote Telerobotics

Authors: Y. T. Tefera, D. Mazzanti, S. Anastasi, D. G. Caldwell, P. Fiorini, N. Deshpande

Abstract: Virtual Reality (VR) interfaces are increasingly used as remote visualization media in telerobotics. Remote environments captured through RGB-D cameras and visualized using VR interfaces can enhance operators' situational awareness and sense of presence. However, this approach has strict requirements for the speed, throughput, and quality of the visualized 3D data.Further, telerobotics requires op… ▽ More Virtual Reality (VR) interfaces are increasingly used as remote visualization media in telerobotics. Remote environments captured through RGB-D cameras and visualized using VR interfaces can enhance operators' situational awareness and sense of presence. However, this approach has strict requirements for the speed, throughput, and quality of the visualized 3D data.Further, telerobotics requires operators to focus on their tasks fully, requiring high perceptual and cognitive skills. This paper shows a work-in-progress framework to address these challenges by taking the human visual system (HVS) as an inspiration. Human eyes use attentional mechanisms to select and draw user engagement to a specific place from the dynamic environment. Inspired by this, the framework implements functionalities to draw users's engagement to a specific place while simultaneously reducing latency and bandwidth requirements. △ Less

Submitted 11 October, 2022; originally announced October 2022.

A Whole-Body Controller Based on a Simplified Template for Rendering Impedances in Quadruped Manipulators

Authors: Mattia Risiglione, Victor Barasuol, Darwin G. Caldwell, Claudio Semini

Abstract: Quadrupedal manipulators require to be compliant when dealing with external forces during autonomous manipulation, tele-operation or physical human-robot interaction. This paper presents a whole-body controller that allows for the implementation of a Cartesian impedance control to coordinate tracking performance and desired compliance for the robot base and manipulator arm. The controller is formu… ▽ More Quadrupedal manipulators require to be compliant when dealing with external forces during autonomous manipulation, tele-operation or physical human-robot interaction. This paper presents a whole-body controller that allows for the implementation of a Cartesian impedance control to coordinate tracking performance and desired compliance for the robot base and manipulator arm. The controller is formulated through an optimization problem using Quadratic Programming (QP) to impose a desired behavior for the system while satisfying friction cone constraints, unilateral force constraints, joint and torque limits. The presented strategy decouples the arm and the base of the platform, enforcing the behavior of a linear double-mass spring damper system, and allows to independently tune their inertia, stiffness and dam** properties. The control architecture is validated through an extensive simulation study using the 90kg HyQ robot equipped with a 7-DoF manipulator arm. Simulation results show the impedance rendering performance when external forces are applied at the arm's end-effector. The paper presents results for full stance condition (all legs on the ground) and, for the first time, also shows how the impedance rendering is affected by the contact conditions during a dynamic gait. △ Less

Submitted 1 August, 2022; originally announced August 2022.

Comments: Accepted as contributed paper at the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2022)

ExoMiner: A Highly Accurate and Explainable Deep Learning Classifier that Validates 301 New Exoplanets

Authors: Hamed Valizadegan, Miguel Martinho, Laurent S. Wilkens, Jon M. Jenkins, Jeffrey Smith, Douglas A. Caldwell, Joseph D. Twicken, Pedro C. Gerum, Nikash Walia, Kaylie Hausknecht, Noa Y. Lubin, Stephen T. Bryson, Nikunj C. Oza

Abstract: The kepler and TESS missions have generated over 100,000 potential transit signals that must be processed in order to create a catalog of planet candidates. During the last few years, there has been a growing interest in using machine learning to analyze these data in search of new exoplanets. Different from the existing machine learning works, ExoMiner, the proposed deep learning classifier in th… ▽ More The kepler and TESS missions have generated over 100,000 potential transit signals that must be processed in order to create a catalog of planet candidates. During the last few years, there has been a growing interest in using machine learning to analyze these data in search of new exoplanets. Different from the existing machine learning works, ExoMiner, the proposed deep learning classifier in this work, mimics how domain experts examine diagnostic tests to vet a transit signal. ExoMiner is a highly accurate, explainable, and robust classifier that 1) allows us to validate 301 new exoplanets from the MAST Kepler Archive and 2) is general enough to be applied across missions such as the on-going TESS mission. We perform an extensive experimental study to verify that ExoMiner is more reliable and accurate than the existing transit signal classifiers in terms of different classification and ranking metrics. For example, for a fixed precision value of 99%, ExoMiner retrieves 93.6% of all exoplanets in the test set (i.e., recall=0.936) while this rate is 76.3% for the best existing classifier. Furthermore, the modular design of ExoMiner favors its explainability. We introduce a simple explainability framework that provides experts with feedback on why ExoMiner classifies a transit signal into a specific class label (e.g., planet candidate or not planet candidate). △ Less

Submitted 8 December, 2021; v1 submitted 18 November, 2021; originally announced November 2021.

Comments: Accepted for Publication in Astrophysical Journals, November 12, 2021

MSC Class: J.2; I.2.6

Towards Precise Pruning Points Detection using Semantic-Instance-Aware Plant Models for Grapevine Winter Pruning Automation

Authors: Miguel Fernandes, Antonello Scaldaferri, Paolo Guadagna, Giuseppe Fiameni, Tao Teng, Matteo Gatti, Stefano Poni, Claudio Semini, Darwin Caldwell, Fei Chen

Abstract: Grapevine winter pruning is a complex task, that requires skilled workers to execute it correctly. The complexity makes it time consuming. It is an operation that requires about 80-120 hours per hectare annually, making an automated robotic system that helps in speeding up the process a crucial tool in large-size vineyards. We will describe (a) a novel expert annotated dataset for grapevine segmen… ▽ More Grapevine winter pruning is a complex task, that requires skilled workers to execute it correctly. The complexity makes it time consuming. It is an operation that requires about 80-120 hours per hectare annually, making an automated robotic system that helps in speeding up the process a crucial tool in large-size vineyards. We will describe (a) a novel expert annotated dataset for grapevine segmentation, (b) a state of the art neural network implementation and (c) generation of pruning points following agronomic rules, leveraging the simplified structure of the plant. With this approach, we are able to generate a set of pruning points on the canes, paving the way towards a correct automation of grapevine winter pruning. △ Less

Submitted 15 September, 2021; originally announced September 2021.

Comments: arXiv admin note: text overlap with arXiv:2106.04208

Fusing Visuo-Tactile Perception into Kernelized Synergies for Robust Gras** and Fine Manipulation of Non-rigid Objects

Authors: Sunny Katyara, Nikhil Deshpande, Fanny Ficuciello, Fei Chen, Bruno Siciliano, Darwin G. Caldwell

Abstract: Handling non-rigid objects using robot hands necessities a framework that does not only incorporate human-level dexterity and cognition but also the multi-sensory information and system dynamics for robust and fine interactions. In this research, our previously developed kernelized synergies framework, inspired from human behaviour on reusing same subspace for gras** and manipulation, is augment… ▽ More Handling non-rigid objects using robot hands necessities a framework that does not only incorporate human-level dexterity and cognition but also the multi-sensory information and system dynamics for robust and fine interactions. In this research, our previously developed kernelized synergies framework, inspired from human behaviour on reusing same subspace for gras** and manipulation, is augmented with visuo-tactile perception for autonomous and flexible adaptation to unknown objects. To detect objects and estimate their poses, a simplified visual pipeline using RANSAC algorithm with Euclidean clustering and SVM classifier is exploited. To modulate interaction efforts while gras** and manipulating non-rigid objects, the tactile feedback using T40S shokac chip sensor, generating 3D force information, is incorporated. Moreover, different kernel functions are examined in the kernelized synergies framework, to evaluate its performance and potential against task reproducibility, execution, generalization and synergistic re-usability. Experiments performed with robot arm-hand system validates the capability and usability of upgraded framework on stably gras** and dexterously manipulating the non-rigid objects. △ Less

Submitted 15 September, 2021; originally announced September 2021.

Comments: IEEE ICRA 2022 (under review)

Grapevine Winter Pruning Automation: On Potential Pruning Points Detection through 2D Plant Modeling using Grapevine Segmentation

Authors: Miguel Fernandes, Antonello Scaldaferri, Giuseppe Fiameni, Tao Teng, Matteo Gatti, Stefano Poni, Claudio Semini, Darwin Caldwell, Fei Chen

Abstract: Grapevine winter pruning is a complex task, that requires skilled workers to execute it correctly. The complexity of this task is also the reason why it is time consuming. Considering that this operation takes about 80-120 hours/ha to be completed, and therefore is even more crucial in large-size vineyards, an automated system can help to speed up the process. To this end, this paper presents a no… ▽ More Grapevine winter pruning is a complex task, that requires skilled workers to execute it correctly. The complexity of this task is also the reason why it is time consuming. Considering that this operation takes about 80-120 hours/ha to be completed, and therefore is even more crucial in large-size vineyards, an automated system can help to speed up the process. To this end, this paper presents a novel multidisciplinary approach that tackles this challenging task by performing object segmentation on grapevine images, used to create a representative model of the grapevine plants. Second, a set of potential pruning points is generated from this plant representation. We will describe (a) a methodology for data acquisition and annotation, (b) a neural network fine-tuning for grapevine segmentation, (c) an image processing based method for creating the representative model of grapevines, starting from the inferred segmentation and (d) potential pruning points detection and localization, based on the plant model which is a simplification of the grapevine structure. With this approach, we are able to identify a significant set of potential pruning points on the canes, that can be used, with further selection, to derive the final set of the real pruning points. △ Less

Submitted 8 June, 2021; originally announced June 2021.

Image-Based Plant Wilting Estimation

Authors: Changye Yang, Sriram Baireddy, Enyu Cai, Valerian Meline, Denise Caldwell, Anjali S. Iyer-Pascuzzi, Edward J. Delp

Abstract: Many plants become limp or droop through heat, loss of water, or disease. This is also known as wilting. In this paper, we examine plant wilting caused by bacterial infection. In particular, we want to design a metric for wilting based on images acquired of the plant. A quantifiable wilting metric will be useful in studying bacterial wilt and identifying resistance genes. Since there is no standar… ▽ More Many plants become limp or droop through heat, loss of water, or disease. This is also known as wilting. In this paper, we examine plant wilting caused by bacterial infection. In particular, we want to design a metric for wilting based on images acquired of the plant. A quantifiable wilting metric will be useful in studying bacterial wilt and identifying resistance genes. Since there is no standard way to estimate wilting, it is common to use ad hoc visual scores. This is very subjective and requires expert knowledge of the plants and the disease mechanism. Our solution consists of using various wilting metrics acquired from RGB images of the plants. We also designed several experiments to demonstrate that our metrics are effective at estimating wilting in plants. △ Less

Submitted 26 May, 2021; originally announced May 2021.

Whole-Body Control on Non-holonomic Mobile Manipulation for Grapevine Winter Pruning Automation

Authors: Tao Teng, Miguel Fernandes, Matteo Gatti, Stefano Poni, Claudio Semini, Darwin Caldwell, Fei Chen

Abstract: Mobile manipulators that combine mobility and manipulability, are increasingly being used for various unstructured application scenarios in the field, e.g. vineyards. Therefore, the coordinated motion of the mobile base and manipulator is an essential feature of the overall performance. In this paper, we explore a whole-body motion controller of a robot which is composed of a 2-DoFs non-holonomic… ▽ More Mobile manipulators that combine mobility and manipulability, are increasingly being used for various unstructured application scenarios in the field, e.g. vineyards. Therefore, the coordinated motion of the mobile base and manipulator is an essential feature of the overall performance. In this paper, we explore a whole-body motion controller of a robot which is composed of a 2-DoFs non-holonomic wheeled mobile base with a 7-DoFs manipulator (non-holonomic wheeled mobile manipulator, NWMM) This robotic platform is designed to efficiently undertake complex grapevine pruning tasks. In the control framework, a task priority coordinated motion of the NWMM is guaranteed. Lower-priority tasks are projected into the null space of the top-priority tasks so that higher-priority tasks are completed without interruption from lower-priority tasks. The proposed controller was evaluated in a grapevine spur pruning experiment scenario. △ Less

Submitted 22 May, 2021; originally announced May 2021.

Formulating Intuitive Stack-of-Tasks using Visuo-Tactile Perception for Collaborative Human-Robot Fine Manipulation

Authors: Sunny Katyara, Nikhil Deshpande, Fanny Ficuciello, Tao Teng, Bruno Siciliano, Darwin G. Caldwell, Fei Chen

Abstract: Enabling robots to work in close proximity to humans necessitates a control framework that does not only incorporate multi-sensory information for autonomous and coordinated interactions but also has perceptive task planning to ensure an adaptable and flexible collaborative behaviour. In this research, an intuitive stack-of-tasks (iSoT) formulation is proposed, that defines the robot's actions by… ▽ More Enabling robots to work in close proximity to humans necessitates a control framework that does not only incorporate multi-sensory information for autonomous and coordinated interactions but also has perceptive task planning to ensure an adaptable and flexible collaborative behaviour. In this research, an intuitive stack-of-tasks (iSoT) formulation is proposed, that defines the robot's actions by considering the human-arm postures and the task progression. The framework is augmented with visuo-tactile information to effectively perceive the collaborative environment and intuitively switch between the planned sub-tasks. The visual feedback from depth cameras monitors and estimates the objects' poses and human-arm postures, while the tactile data provides the exploration skills to detect and maintain the desired contacts to avoid object slippage. To evaluate the performance, effectiveness and usability of the proposed framework, assembly and disassembly tasks, performed by the human-human and human-robot partners, are considered and analyzed using distinct evaluation metrics i.e, approach adaptation, grasp correction, task coordination latency, cumulative posture deviation, and task repeatability. △ Less

Submitted 4 January, 2022; v1 submitted 9 March, 2021; originally announced March 2021.

Vision Based Adaptation to Kernelized Synergies for Human Inspired Robotic Manipulation

Authors: Sunny Katyara, Fanny Ficuciello, Fei Chen, Bruno Siciliano, Darwin G. Caldwell

Abstract: Humans in contrast to robots are excellent in performing fine manipulation tasks owing to their remarkable dexterity and sensorimotor organization. Enabling robots to acquire such capabilities, necessitates a framework that not only replicates the human behaviour but also integrates the multi-sensory information for autonomous object interaction. To address such limitations, this research proposes… ▽ More Humans in contrast to robots are excellent in performing fine manipulation tasks owing to their remarkable dexterity and sensorimotor organization. Enabling robots to acquire such capabilities, necessitates a framework that not only replicates the human behaviour but also integrates the multi-sensory information for autonomous object interaction. To address such limitations, this research proposes to augment the previously developed kernelized synergies framework with visual perception to automatically adapt to the unknown objects. The kernelized synergies, inspired from humans, retain the same reduced subspace for object gras** and manipulation. To detect object in the scene, a simplified perception pipeline is used that leverages the RANSAC algorithm with Euclidean clustering and SVM for object segmentation and recognition respectively. Further, the comparative analysis of kernelized synergies with other state of art approaches is made to confirm their flexibility and effectiveness on the robotic manipulation tasks. The experiments conducted on the robot hand confirm the robustness of modified kernelized synergies framework against the uncertainties related to the perception of environment. △ Less

Submitted 13 December, 2020; originally announced December 2020.

Reproducible Pruning System on Dynamic Natural Plants for Field Agricultural Robots

Authors: Sunny Katyara, Fanny Ficuciello, Darwin G. Caldwell, Fei Chen, Bruno Siciliano

Abstract: Pruning is the art of cutting unwanted and unhealthy plant branches and is one of the difficult tasks in the field robotics. It becomes even more complex when the plant branches are moving. Moreover, the reproducibility of robot pruning skills is another challenge to deal with due to the heterogeneous nature of vines in the vineyard. This research proposes a multi-modal framework to deal with the… ▽ More Pruning is the art of cutting unwanted and unhealthy plant branches and is one of the difficult tasks in the field robotics. It becomes even more complex when the plant branches are moving. Moreover, the reproducibility of robot pruning skills is another challenge to deal with due to the heterogeneous nature of vines in the vineyard. This research proposes a multi-modal framework to deal with the dynamic vines with the aim of sim2real skill transfer. The 3D models of vines are constructed in blender engine and rendered in simulated environment as a need for training the robot. The Natural Admittance Controller (NAC) is applied to deal with the dynamics of vines. It uses force feedback and compensates the friction effects while maintaining the passivity of system. The faster R-CNN is used to detect the spurs on the vines and then statistical pattern recognition algorithm using K-means clustering is applied to find the effective pruning points. The proposed framework is tested in simulated and real environments. △ Less

Submitted 13 December, 2020; v1 submitted 26 August, 2020; originally announced August 2020.

Comments: Under Review at SPAR

Leveraging Kernelized Synergies on Shared Subspace for Precision Grasp and Dexterous Manipulation

Authors: Sunny Katyara, Fanny Ficuciello, Darwin Caldwell, Bruno Siciliano, Fei Chen

Abstract: Manipulation in contrast to gras** is a trajectorial task that needs to use dexterous hands. Improving the dexterity of robot hands, increases the controller complexity and thus requires to use the concept of postural synergies. Inspired from postural synergies, this research proposes a new framework called kernelized synergies that focuses on the re-usability of the same subspace for precision… ▽ More Manipulation in contrast to gras** is a trajectorial task that needs to use dexterous hands. Improving the dexterity of robot hands, increases the controller complexity and thus requires to use the concept of postural synergies. Inspired from postural synergies, this research proposes a new framework called kernelized synergies that focuses on the re-usability of the same subspace for precision gras** and dexterous manipulation. In this work, the computed subspace of postural synergies; parameterized by probabilistic movement primitives, is treated with kernel to preserve its gras** and manipulation characteristics and allows its reuse for new objects. The grasp stability of the proposed framework is assessed with a force closure quality index. For performance evaluation, the proposed framework is tested on two different simulated robot hand models using the Syngrasp toolbox and experimentally, four complex gras** and manipulation tasks are performed and reported. The results confirm the hand agnostic approach of the proposed framework and its generalization to distinct objects irrespective of their shape and size. △ Less

Submitted 3 September, 2020; v1 submitted 26 August, 2020; originally announced August 2020.

Line Walking and Balancing for Legged Robots with Point Feet

Authors: Carlos Gonzalez, Victor Barasuol, Marco Frigerio, Roy Featherstone, Darwin G. Caldwell, Claudio Semini

Abstract: The ability of legged systems to traverse highly-constrained environments depends by and large on the performance of their motion and balance controllers. This paper presents a controller that excels in a scenario that most state-of-the-art balance controllers have not yet addressed: line walking, or walking on nearly null support regions. Our approach uses a low-dimensional virtual model (2-DoF)… ▽ More The ability of legged systems to traverse highly-constrained environments depends by and large on the performance of their motion and balance controllers. This paper presents a controller that excels in a scenario that most state-of-the-art balance controllers have not yet addressed: line walking, or walking on nearly null support regions. Our approach uses a low-dimensional virtual model (2-DoF) to generate balancing actions through a previously derived four-term balance controller and transforms them to the robot through a derived kinematic map**. The capabilities of this controller are tested in simulation, where we show the 90kg quadruped robot HyQ crossing a bridge of only 6 cm width (compared to its 4 cm diameter spherical foot), by balancing on two feet at any time while moving along a line. Lastly, we present our preliminary experimental results showing HyQ balancing on two legs while being disturbed. △ Less

Submitted 21 August, 2020; v1 submitted 2 July, 2020; originally announced July 2020.

Motion Planning for Quadrupedal Locomotion: Coupled Planning, Terrain Map** and Whole-Body Control

Authors: Carlos Mastalli, Ioannis Havoutis, Michele Focchi, Darwin G. Caldwell, Claudio Semini

Abstract: Planning whole-body motions while taking into account the terrain conditions is a challenging problem for legged robots since the terrain model might produce many local minima. Our coupled planning method uses stochastic and derivatives-free search to plan both foothold locations and horizontal motions due to the local minima produced by the terrain model. It jointly optimizes body motion, step du… ▽ More Planning whole-body motions while taking into account the terrain conditions is a challenging problem for legged robots since the terrain model might produce many local minima. Our coupled planning method uses stochastic and derivatives-free search to plan both foothold locations and horizontal motions due to the local minima produced by the terrain model. It jointly optimizes body motion, step duration and foothold selection, and it models the terrain as a cost-map. Due to the novel attitude planning method, the horizontal motion plans can be applied to various terrain conditions. The attitude planner ensures the robot stability by imposing limits to the angular acceleration. Our whole-body controller tracks compliantly trunk motions while avoiding slippage, as well as kinematic and torque limits. Despite the use of a simplified model, which is restricted to flat terrain, our approach shows remarkable capability to deal with a wide range of non-coplanar terrains. The results are validated by experimental trials and comparative evaluations in a series of terrains of progressively increasing complexity. △ Less

Submitted 27 June, 2020; v1 submitted 11 March, 2020; originally announced March 2020.

Comments: 15 pages, pre-print, journal

Journal ref: IEEE Transactions on Robotics, 2020

Plant Stem Segmentation Using Fast Ground Truth Generation

Authors: Changye Yang, Sriram Baireddy, Yuhao Chen, Enyu Cai, Denise Caldwell, Valérian Méline, Anjali S. Iyer-Pascuzzi, Edward J. Delp

Abstract: Accurately phenoty** plant wilting is important for understanding responses to environmental stress. Analysis of the shape of plants can potentially be used to accurately quantify the degree of wilting. Plant shape analysis can be enhanced by locating the stem, which serves as a consistent reference point during wilting. In this paper, we show that deep learning methods can accurately segment to… ▽ More Accurately phenoty** plant wilting is important for understanding responses to environmental stress. Analysis of the shape of plants can potentially be used to accurately quantify the degree of wilting. Plant shape analysis can be enhanced by locating the stem, which serves as a consistent reference point during wilting. In this paper, we show that deep learning methods can accurately segment tomato plant stems. We also propose a control-point-based ground truth method that drastically reduces the resources needed to create a training dataset for a deep learning approach. Experimental results show the viability of both our proposed ground truth approach and deep learning based stem segmentation. △ Less

Submitted 23 January, 2020; originally announced January 2020.

Algorithms and Statistical Models for Scientific Discovery in the Petabyte Era

Authors: Brian Nord, Andrew J. Connolly, Jamie Kinney, Jeremy Kubica, Gautaum Narayan, Joshua E. G. Peek, Chad Schafer, Erik J. Tollerud, Camille Avestruz, G. Jogesh Babu, Simon Birrer, Douglas Burke, João Caldeira, Douglas A. Caldwell, Joleen K. Carlberg, Yen-Chi Chen, Chuanfei Dong, Eric D. Feigelson, V. Zach Golkhou, Vinay Kashyap, T. S. Li, Thomas Loredo, Luisa Lucie-Smith, Kaisey S. Mandel, J. R. Martínez-Galarza , et al. (13 additional authors not shown)

Abstract: The field of astronomy has arrived at a turning point in terms of size and complexity of both datasets and scientific collaboration. Commensurately, algorithms and statistical models have begun to adapt --- e.g., via the onset of artificial intelligence --- which itself presents new challenges and opportunities for growth. This white paper aims to offer guidance and ideas for how we can evolve our… ▽ More The field of astronomy has arrived at a turning point in terms of size and complexity of both datasets and scientific collaboration. Commensurately, algorithms and statistical models have begun to adapt --- e.g., via the onset of artificial intelligence --- which itself presents new challenges and opportunities for growth. This white paper aims to offer guidance and ideas for how we can evolve our technical and collaborative frameworks to promote efficient algorithmic development and take advantage of opportunities for scientific discovery in the petabyte era. We discuss challenges for discovery in large and complex data sets; challenges and requirements for the next stage of development of statistical methodologies and algorithmic tool sets; how we might change our paradigms of collaboration and education; and the ethical implications of scientists' contributions to widely applicable algorithms and computational modeling. We start with six distinct recommendations that are supported by the commentary following them. This white paper is related to a larger corpus of effort that has taken place within and around the Petabytes to Science Workshops (https://petabytestoscience.github.io/). △ Less

Submitted 4 November, 2019; originally announced November 2019.

Comments: arXiv admin note: substantial text overlap with arXiv:1905.05116

Report number: FERMILAB-FN-1093-A-AE-SCD

MPC-based Controller with Terrain Insight for Dynamic Legged Locomotion

Authors: Octavio Villarreal, Victor Barasuol, Patrick M. Wensing, Darwin G. Caldwell, Claudio Semini

Abstract: We present a novel control strategy for dynamic legged locomotion in complex scenarios, that considers information about the morphology of the terrain in contexts when only on-board map** and computation are available. The strategy is built on top of two main elements: first a contact sequence task that provides safe foothold locations based on a convolutional neural network to perform fast and… ▽ More We present a novel control strategy for dynamic legged locomotion in complex scenarios, that considers information about the morphology of the terrain in contexts when only on-board map** and computation are available. The strategy is built on top of two main elements: first a contact sequence task that provides safe foothold locations based on a convolutional neural network to perform fast and continuous evaluation of the terrain in search of safe foothold locations; then a model predictive controller that considers the foothold locations given by the contact sequence task to optimize target ground reaction forces. We assess the performance of our strategy through simulations of the hydraulically actuated quadruped robot HyQReal traversing rough terrain under realistic on-board sensing and computing conditions. △ Less

Submitted 31 March, 2020; v1 submitted 30 September, 2019; originally announced September 2019.

Comments: 8 pages, 4 figures, accepted to International Conference on Robotics and Automation (ICRA) 2020

A Linearly Constrained Nonparametric Framework for Imitation Learning

Authors: Yanlong Huang, Darwin G. Caldwell

Abstract: In recent years, a myriad of advanced results have been reported in the community of imitation learning, ranging from parametric to non-parametric, probabilistic to non-probabilistic and Bayesian to frequentist approaches. Meanwhile, ample applications (e.g., gras** tasks and human-robot collaborations) further show the applicability of imitation learning in a wide range of domains. While numero… ▽ More In recent years, a myriad of advanced results have been reported in the community of imitation learning, ranging from parametric to non-parametric, probabilistic to non-probabilistic and Bayesian to frequentist approaches. Meanwhile, ample applications (e.g., gras** tasks and human-robot collaborations) further show the applicability of imitation learning in a wide range of domains. While numerous literature is dedicated to the learning of human skills in unconstrained environment, the problem of learning constrained motor skills, however, has not received equal attention yet. In fact, constrained skills exist widely in robotic systems. For instance, when a robot is demanded to write letters on a board, its end-effector trajectory must comply with the plane constraint from the board. In this paper, we aim to tackle the problem of imitation learning with linear constraints. Specifically, we propose to exploit the probabilistic properties of multiple demonstrations, and subsequently incorporate them into a linearly constrained optimization problem, which finally leads to a non-parametric solution. In addition, a connection between our framework and the classical model predictive control is provided. Several examples including simulated writing and locomotion tasks are presented to show the effectiveness of our framework. △ Less

Submitted 15 September, 2019; originally announced September 2019.

Comments: 7 pages

Towards Orientation Learning and Adaptation in Cartesian Space

Authors: Yanlong Huang, Fares J. Abu-Dakka, João Silvério, Darwin G. Caldwell

Abstract: As a promising branch of robotics, imitation learning emerges as an important way to transfer human skills to robots, where human demonstrations represented in Cartesian or joint spaces are utilized to estimate task/skill models that can be subsequently generalized to new situations. While learning Cartesian positions suffices for many applications, the end-effector orientation is required in many… ▽ More As a promising branch of robotics, imitation learning emerges as an important way to transfer human skills to robots, where human demonstrations represented in Cartesian or joint spaces are utilized to estimate task/skill models that can be subsequently generalized to new situations. While learning Cartesian positions suffices for many applications, the end-effector orientation is required in many others. Despite recent advances in learning orientations from demonstrations, several crucial issues have not been adequately addressed yet. For instance, how can demonstrated orientations be adapted to pass through arbitrary desired points that comprise orientations and angular velocities? In this paper, we propose an approach that is capable of learning multiple orientation trajectories and adapting learned orientation skills to new situations (e.g., via-points and end-points), where both orientation and angular velocity are considered. Specifically, we introduce a kernelized treatment to alleviate explicit basis functions when learning orientations, which allows for learning orientation trajectories associated with high-dimensional inputs. In addition, we extend our approach to the learning of quaternions with angular acceleration or jerk constraints, which allows for generating smoother orientation profiles for robots. Several examples including experiments with real 7-DoF robot arms are provided to verify the effectiveness of our method. △ Less

Submitted 15 August, 2020; v1 submitted 8 July, 2019; originally announced July 2019.

Comments: Accepted for publication at IEEE Transactions on Robotics

High-Throughput Machine Learning from Electronic Health Records

Authors: Ross S. Kleiman, Paul S. Bennett, Peggy L. Peissig, Richard L. Berg, Zhaobin Kuang, Scott J. Hebbring, Michael D. Caldwell, David Page

Abstract: The widespread digitization of patient data via electronic health records (EHRs) has created an unprecedented opportunity to use machine learning algorithms to better predict disease risk at the patient level. Although predictive models have previously been constructed for a few important diseases, such as breast cancer and myocardial infarction, we currently know very little about how accurately… ▽ More The widespread digitization of patient data via electronic health records (EHRs) has created an unprecedented opportunity to use machine learning algorithms to better predict disease risk at the patient level. Although predictive models have previously been constructed for a few important diseases, such as breast cancer and myocardial infarction, we currently know very little about how accurately the risk for most diseases or events can be predicted, and how far in advance. Machine learning algorithms use training data rather than preprogrammed rules to make predictions and are well suited for the complex task of disease prediction. Although there are thousands of conditions and illnesses patients can encounter, no prior research simultaneously predicts risks for thousands of diagnosis codes and thereby establishes a comprehensive patient risk profile. Here we show that such pandiagnostic prediction is possible with a high level of performance across diagnosis codes. For the tasks of predicting diagnosis risks both 1 and 6 months in advance, we achieve average areas under the receiver operating characteristic curve (AUCs) of 0.803 and 0.758, respectively, across thousands of prediction tasks. Finally, our research contributes a new clinical prediction dataset in which researchers can explore how well a diagnosis can be predicted and what health factors are most useful for prediction. For the first time, we can get a much more complete picture of how well risks for thousands of different diagnosis codes can be predicted. △ Less

Submitted 3 July, 2019; originally announced July 2019.

Hierarchical Reinforcement Learning for Concurrent Discovery of Compound and Composable Policies

Authors: Domingo Esteban, Leonel Rozo, Darwin G. Caldwell

Abstract: A common strategy to deal with the expensive reinforcement learning (RL) of complex tasks is to decompose them into a collection of subtasks that are usually simpler to learn as well as reusable for new problems. However, when a robot learns the policies for these subtasks, common approaches treat every policy learning process separately. Therefore, all these individual (composable) policies need… ▽ More A common strategy to deal with the expensive reinforcement learning (RL) of complex tasks is to decompose them into a collection of subtasks that are usually simpler to learn as well as reusable for new problems. However, when a robot learns the policies for these subtasks, common approaches treat every policy learning process separately. Therefore, all these individual (composable) policies need to be learned before tackling the learning process of the complex task through policies composition. Moreover, such composition of individual policies is usually performed sequentially, which is not suitable for tasks that require to perform the subtasks concurrently. In this paper, we propose to combine a set of composable Gaussian policies corresponding to these subtasks using a set of activation vectors, resulting in a complex Gaussian policy that is a function of the means and covariances matrices of the composable policies. Moreover, we propose an algorithm for learning both compound and composable policies within the same learning process by exploiting the off-policy data generated from the compound policy. The algorithm is built on a maximum entropy RL approach to favor exploration during the learning process. The results of the experiments show that the experience collected with the compound policy permits not only to solve the complex task but also to obtain useful composable policies that successfully perform in their corresponding subtasks. △ Less

Submitted 1 August, 2019; v1 submitted 23 May, 2019; originally announced May 2019.

Comments: Accepted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019)

Hierarchical Planning of Dynamic Movements without Scheduled Contact Sequences

Authors: Carlos Mastalli, Ioannis Havoutis, Michele Focchi, Darwin G. Caldwell, Claudio Semini

Abstract: Most animal and human locomotion behaviors for solving complex tasks involve dynamic motions and rich contact interaction. In fact, complex maneuvers need to consider dynamic movement and contact events at the same time. We present a hierarchical trajectory optimization approach for planning dynamic movements with unscheduled contact sequences. We compute whole-body motions that achieve goals that… ▽ More Most animal and human locomotion behaviors for solving complex tasks involve dynamic motions and rich contact interaction. In fact, complex maneuvers need to consider dynamic movement and contact events at the same time. We present a hierarchical trajectory optimization approach for planning dynamic movements with unscheduled contact sequences. We compute whole-body motions that achieve goals that cannot be reached in a kinematic fashion. First, we find a feasible CoM motion according to the centroidal dynamics of the robot. Then, we refine the solution by applying the robot's full-dynamics model, where the feasible CoM trajectory is used as a warm-start point. To accomplish the unscheduled contact behavior, we use complementarity constraints to describe the contact model, i.e. environment geometry and non-sliding active contacts. Both optimization phases are posed as Mathematical Program with Complementarity Constraints (MPCC). Experimental trials demonstrate the performance of our planning approach in a set of challenging tasks. △ Less

Submitted 9 April, 2019; originally announced April 2019.

Comments: 6 pages, IEEE International Conference on Robotics and Automation (ICRA)

Simultaneous Contact, Gait and Motion Planning for Robust Multi-Legged Locomotion via Mixed-Integer Convex Optimization

Authors: Bernardo Aceituno-Cabezas, Carlos Mastalli, Hongkai Dai, Michele Focchi, Andreea Radulescu, Darwin G. Caldwell, Jose Cappelletto, Juan C. Grieco, Gerardo Fernandez-Lopez, Claudio Semini

Abstract: Traditional motion planning approaches for multi-legged locomotion divide the problem into several stages, such as contact search and trajectory generation. However, reasoning about contacts and motions simultaneously is crucial for the generation of complex whole-body behaviors. Currently, coupling theses problems has required either the assumption of a fixed gait sequence and flat terrain condit… ▽ More Traditional motion planning approaches for multi-legged locomotion divide the problem into several stages, such as contact search and trajectory generation. However, reasoning about contacts and motions simultaneously is crucial for the generation of complex whole-body behaviors. Currently, coupling theses problems has required either the assumption of a fixed gait sequence and flat terrain condition, or non-convex optimization with intractable computation time. In this paper, we propose a mixed-integer convex formulation to plan simultaneously contact locations, gait transitions and motion, in a computationally efficient fashion. In contrast to previous works, our approach is not limited to flat terrain nor to a pre-specified gait sequence. Instead, we incorporate the friction cone stability margin, approximate the robot's torque limits, and plan the gait using mixed-integer convex constraints. We experimentally validated our approach on the HyQ robot by traversing different challenging terrains, where non-convexity and flat terrain assumptions might lead to sub-optimal or unstable plans. Our method increases the motion generality while kee** a low computation time. △ Less

Submitted 9 April, 2019; originally announced April 2019.

Comments: 8 pages, IEEE Robotics and Automation Letters

Planning and Execution of Dynamic Whole-Body Locomotion for a Hydraulic Quadruped on Challenging Terrain

Authors: Alexander W. Winkler, Carlos Mastalli, Ioannis Havoutis, Michele Focchi, Darwin G. Caldwell, Claudio Semini

Abstract: We present a framework for dynamic quadrupedal locomotion over challenging terrain, where the choice of appropriate footholds is crucial for the success of the behaviour. We build a model of the environment on-line and on-board using an efficient occupancy grid representation. We use Any-time-Repairing A* (ARA*) to search over a tree of possible actions, choose a rough body path and select the loc… ▽ More We present a framework for dynamic quadrupedal locomotion over challenging terrain, where the choice of appropriate footholds is crucial for the success of the behaviour. We build a model of the environment on-line and on-board using an efficient occupancy grid representation. We use Any-time-Repairing A* (ARA*) to search over a tree of possible actions, choose a rough body path and select the locally-best footholds accordingly. We run a n-step lookahead optimization of the body trajectory using a dynamic stability metric, the Zero Moment Point (ZMP), that generates natural dynamic whole-body motions. A combination of floating-base inverse dynamics and virtual model control accurately executes the desired motions on an actively compliant system. Experimental trials show that this framework allows us to traverse terrains at nearly 6 times the speed of our previous work, evaluated over the same set of trials. △ Less

Submitted 7 April, 2019; originally announced April 2019.

Comments: 7 pages, IEEE International Conference on Robotics and Automation

Journal ref: 2015

On-line and on-board planning and perception for quadrupedal locomotion

Authors: Carlos Mastalli, Ioannis Havoutis, Alexander W. Winkler, Darwin G. Caldwell, Claudio Semini

Abstract: We present a legged motion planning approach for quadrupedal locomotion over challenging terrain. We decompose the problem into body action planning and footstep planning. We use a lattice representation together with a set of defined body movement primitives for computing a body action plan. The lattice representation allows us to plan versatile movements that ensure feasibility for every possibl… ▽ More We present a legged motion planning approach for quadrupedal locomotion over challenging terrain. We decompose the problem into body action planning and footstep planning. We use a lattice representation together with a set of defined body movement primitives for computing a body action plan. The lattice representation allows us to plan versatile movements that ensure feasibility for every possible plan. To this end, we propose a set of rules that define the footstep search regions and footstep sequence given a body action. We use Anytime Repairing A* (ARA*) search that guarantees bounded suboptimal plans. Our main contribution is a planning approach that generates on-line versatile movements. Experimental trials demonstrate the performance of our planning approach in a set of challenging terrain conditions. The terrain information and plans are computed on-line and on-board. △ Less

Submitted 7 April, 2019; originally announced April 2019.

Comments: 7 pages, International Conference on Technologies for Practical Robot Applications

Journal ref: published 2015

V2CNet: A Deep Learning Framework to Translate Videos to Commands for Robotic Manipulation

Authors: Anh Nguyen, Thanh-Toan Do, Ian Reid, Darwin G. Caldwell, Nikos G. Tsagarakis

Abstract: We propose V2CNet, a new deep learning framework to automatically translate the demonstration videos to commands that can be directly used in robotic applications. Our V2CNet has two branches and aims at understanding the demonstration video in a fine-grained manner. The first branch has the encoder-decoder architecture to encode the visual features and sequentially generate the output words as a… ▽ More We propose V2CNet, a new deep learning framework to automatically translate the demonstration videos to commands that can be directly used in robotic applications. Our V2CNet has two branches and aims at understanding the demonstration video in a fine-grained manner. The first branch has the encoder-decoder architecture to encode the visual features and sequentially generate the output words as a command, while the second branch uses a Temporal Convolutional Network (TCN) to learn the fine-grained actions. By jointly training both branches, the network is able to model the sequential information of the command, while effectively encodes the fine-grained actions. The experimental results on our new large-scale dataset show that V2CNet outperforms recent state-of-the-art methods by a substantial margin, while its output can be applied in real robotic applications. The source code and trained models will be made available. △ Less

Submitted 23 March, 2019; originally announced March 2019.

Comments: 15 pages. arXiv admin note: substantial text overlap with arXiv:1710.00290

Uncertainty-Aware Imitation Learning using Kernelized Movement Primitives

Authors: João Silvério, Yanlong Huang, Fares J. Abu-Dakka, Leonel Rozo, Darwin G. Caldwell

Abstract: During the past few years, probabilistic approaches to imitation learning have earned a relevant place in the literature. One of their most prominent features, in addition to extracting a mean trajectory from task demonstrations, is that they provide a variance estimation. The intuitive meaning of this variance, however, changes across different techniques, indicating either variability or uncerta… ▽ More During the past few years, probabilistic approaches to imitation learning have earned a relevant place in the literature. One of their most prominent features, in addition to extracting a mean trajectory from task demonstrations, is that they provide a variance estimation. The intuitive meaning of this variance, however, changes across different techniques, indicating either variability or uncertainty. In this paper we leverage kernelized movement primitives (KMP) to provide a new perspective on imitation learning by predicting variability, correlations and uncertainty about robot actions. This rich set of information is used in combination with optimal controller fusion to learn actions from data, with two main advantages: i) robots become safe when uncertain about their actions and ii) they are able to leverage partial demonstrations, given as elementary sub-tasks, to optimally perform a higher level, more complex task. We showcase our approach in a painting task, where a human user and a KUKA robot collaborate to paint a wooden board. The task is divided into two sub-tasks and we show that using our approach the robot becomes compliant (hence safe) outside the training regions and executes the two sub-tasks with optimal gains. △ Less

Submitted 30 September, 2019; v1 submitted 5 March, 2019; originally announced March 2019.

Comments: Published in the proceedings of IROS 2019

Geometry-aware Manipulability Learning, Tracking and Transfer

Authors: Noémie Jaquier, Leonel Rozo, Darwin G. Caldwell, Sylvain Calinon

Abstract: Body posture influences human and robots performance in manipulation tasks, as appropriate poses facilitate motion or force exertion along different axes. In robotics, manipulability ellipsoids arise as a powerful descriptor to analyze, control and design the robot dexterity as a function of the articulatory joint configuration. This descriptor can be designed according to different task requireme… ▽ More Body posture influences human and robots performance in manipulation tasks, as appropriate poses facilitate motion or force exertion along different axes. In robotics, manipulability ellipsoids arise as a powerful descriptor to analyze, control and design the robot dexterity as a function of the articulatory joint configuration. This descriptor can be designed according to different task requirements, such as tracking a desired position or apply a specific force. In this context, this paper presents a novel \emph{manipulability transfer} framework, a method that allows robots to learn and reproduce manipulability ellipsoids from expert demonstrations. The proposed learning scheme is built on a tensor-based formulation of a Gaussian mixture model that takes into account that manipulability ellipsoids lie on the manifold of symmetric positive definite matrices. Learning is coupled with a geometry-aware tracking controller allowing robots to follow a desired profile of manipulability ellipsoids. Extensive evaluations in simulation with redundant manipulators, a robotic hand and humanoids agents, as well as an experiment with two real dual-arm systems validate the feasibility of the approach. △ Less

Submitted 1 March, 2021; v1 submitted 27 November, 2018; originally announced November 2018.

Comments: In the Intl. Journal of Robotics Research (IJRR). Website: https://sites.google.com/view/manipulability. Code: https://github.com/NoemieJaquier/Manipulability. 24 pages, 20 figures, 3 tables, 4 appendices

Fast and Continuous Foothold Adaptation for Dynamic Locomotion through CNNs

Authors: Octavio Villarreal, Victor Barasuol, Marco Camurri, Luca Franceschi, Michele Focchi, Massimiliano Pontil, Darwin G. Caldwell, Claudio Semini

Abstract: Legged robots can outperform wheeled machines for most navigation tasks across unknown and rough terrains. For such tasks, visual feedback is a fundamental asset to provide robots with terrain-awareness. However, robust dynamic locomotion on difficult terrains with real-time performance guarantees remains a challenge. We present here a real-time, dynamic foothold adaptation strategy based on visua… ▽ More Legged robots can outperform wheeled machines for most navigation tasks across unknown and rough terrains. For such tasks, visual feedback is a fundamental asset to provide robots with terrain-awareness. However, robust dynamic locomotion on difficult terrains with real-time performance guarantees remains a challenge. We present here a real-time, dynamic foothold adaptation strategy based on visual feedback. Our method adjusts the landing position of the feet in a fully reactive manner, using only on-board computers and sensors. The correction is computed and executed continuously along the swing phase trajectory of each leg. To efficiently adapt the landing position, we implement a self-supervised foothold classifier based on a Convolutional Neural Network (CNN). Our method results in an up to 200 times faster computation with respect to the full-blown heuristics. Our goal is to react to visual stimuli from the environment, bridging the gap between blind reactive locomotion and purely vision-based planning strategies. We assess the performance of our method on the dynamic quadruped robot HyQ, executing static and dynamic gaits (at speeds up to 0.5 m/s) in both simulated and real scenarios; the benefit of safe foothold adaptation is clearly demonstrated by the overall robot behavior. △ Less

Submitted 15 February, 2019; v1 submitted 25 September, 2018; originally announced September 2018.

Comments: 9 pages, 11 figures. Accepted to RA-L + ICRA 2019, January 2019

Heuristic Planning for Rough Terrain Locomotion in Presence of External Disturbances and Variable Perception Quality

Authors: Michele Focchi, Romeo Orsolino, Marco Camurri, Victor Barasuol, Carlos Mastalli, Darwin G. Caldwell, Claudio Semini

Abstract: The quality of the visual feedback can vary significantly on a legged robot that is meant to traverse unknown and unstructured terrains. The map of the environment, acquired with online state-of-the-art algorithms, often degrades after a few steps, due to sensing inaccuracies, slippage and unexpected disturbances. When designing locomotion algorithms, this degradation can result in planned traject… ▽ More The quality of the visual feedback can vary significantly on a legged robot that is meant to traverse unknown and unstructured terrains. The map of the environment, acquired with online state-of-the-art algorithms, often degrades after a few steps, due to sensing inaccuracies, slippage and unexpected disturbances. When designing locomotion algorithms, this degradation can result in planned trajectories that are not consistent with the reality, if not dealt properly. In this work, we propose a heuristic-based planning approach that enables a quadruped robot to successfully traverse a significantly rough terrain (e.g., stones up to 10 cm of diameter), in absence of visual feedback. When available, the approach allows also to exploit the visual feedback (e.g., to enhance the step** strategy) in multiple ways, according to the quality of the 3D map. The proposed framework also includes reflexes, triggered in specific situations, and the possibility to estimate online an unknown time-varying disturbance and compensate for it. We demonstrate the effectiveness of the approach with experiments performed on our quadruped robot HyQ (85 kg), traversing different terrains, such as: ramps, rocks, bricks, pallets and stairs. We also demonstrate the capability to estimate and compensate for disturbances, showing the robot walking up a ramp while pulling a cart attached to its back. △ Less

Submitted 7 November, 2018; v1 submitted 25 May, 2018; originally announced May 2018.

Comments: 20 pages, 24 figures

Object Captioning and Retrieval with Natural Language

Authors: Anh Nguyen, Thanh-Toan Do, Ian Reid, Darwin G. Caldwell, Nikos G. Tsagarakis

Abstract: We address the problem of jointly learning vision and language to understand the object in a fine-grained manner. The key idea of our approach is the use of object descriptions to provide the detailed understanding of an object. Based on this idea, we propose two new architectures to solve two related problems: object captioning and natural language-based object retrieval. The goal of the object c… ▽ More We address the problem of jointly learning vision and language to understand the object in a fine-grained manner. The key idea of our approach is the use of object descriptions to provide the detailed understanding of an object. Based on this idea, we propose two new architectures to solve two related problems: object captioning and natural language-based object retrieval. The goal of the object captioning task is to simultaneously detect the object and generate its associated description, while in the object retrieval task, the goal is to localize an object given an input query. We demonstrate that both problems can be solved effectively using hybrid end-to-end CNN-LSTM networks. The experimental results on our new challenging dataset show that our methods outperform recent methods by a fair margin, while providing a detailed understanding of the object and having fast inference time. The source code will be made available. △ Less

Submitted 16 March, 2018; originally announced March 2018.

Comments: 8 pages, 8 figures

Center-of-Mass-Based Grasp Pose Adaptation Using 3D Range and Force/Torque Sensing

Authors: Dimitrios Kanoulas, **oh Lee, Darwin G. Caldwell, Nikos G. Tsagarakis

Abstract: Lifting objects, whose mass may produce high wrist torques that exceed the hardware strength limits, could lead to unstable grasps or serious robot damage. This work introduces a new Center-of-Mass (CoM)-based grasp pose adaptation method, for picking up objects using a combination of exteroceptive 3D perception and proprioceptive force/torque sensor feedback. The method works in two iterative sta… ▽ More Lifting objects, whose mass may produce high wrist torques that exceed the hardware strength limits, could lead to unstable grasps or serious robot damage. This work introduces a new Center-of-Mass (CoM)-based grasp pose adaptation method, for picking up objects using a combination of exteroceptive 3D perception and proprioceptive force/torque sensor feedback. The method works in two iterative stages to provide reliable and wrist torque efficient grasps. Initially, a geometric object CoM is estimated from the input range data. In the first stage, a set of hand-size handle grasps are localized on the object and the closest to its CoM is selected for gras**. In the second stage, the object is lifted using a single arm, while the force and torque readings from the sensor on the wrist are monitored. Based on these readings, a displacement to the new CoM estimation is calculated. The object is released and the process is repeated until the wrist torque effort is minimized. The advantage of our method is the blending of both exteroceptive (3D range) and proprioceptive (force/torque) sensing for finding the grasp location that minimizes the wrist effort, potentially improving the reliability of the gras** and the subsequent manipulation task. We experimentally validate the proposed method by executing a number of tests on a set of objects that include handles, using the humanoid robot WALK-MAN. △ Less

Submitted 18 February, 2018; originally announced February 2018.

Comments: 25 pages, 10 figures, International Journal of Humanoid Robotics (IJHR)

Journal ref: International Journal of Humanoid Robotics Vol. 15 (2018) 1850013 (25 pages), World Scientific Publishing Company

Probabilistic Learning of Torque Controllers from Kinematic and Force Constraints

Authors: João Silvério, Yanlong Huang, Leonel Rozo, Sylvain Calinon, Darwin G. Caldwell

Abstract: When learning skills from demonstrations, one is often required to think in advance about the appropriate task representation (usually in either operational or configuration space). We here propose a probabilistic approach for simultaneously learning and synthesizing torque control commands which take into account task space, joint space and force constraints. We treat the problem by considering d… ▽ More When learning skills from demonstrations, one is often required to think in advance about the appropriate task representation (usually in either operational or configuration space). We here propose a probabilistic approach for simultaneously learning and synthesizing torque control commands which take into account task space, joint space and force constraints. We treat the problem by considering different torque controllers acting on the robot, whose relevance is learned probabilistically from demonstrations. This information is used to combine the controllers by exploiting the properties of Gaussian distributions, generating new torque commands that satisfy the important features of the task. We validate the approach in two experimental scenarios using 7-DoF torquecontrolled manipulators, with tasks that require the consideration of different controllers to be properly executed. △ Less

Submitted 3 August, 2018; v1 submitted 19 December, 2017; originally announced December 2017.

Comments: Accepted for publication at 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Application of Wrench based Feasibility Analysis to the Online Trajectory Optimization of Legged Robots

Authors: Romeo Orsolino, Michele Focchi, Carlos Mastalli, Hongkai Dai, Darwin G. Caldwell, Claudio Semini

Abstract: Motion planning in multi-contact scenarios has recently gathered interest within the legged robotics community, however actuator force/torque limits are rarely considered. We believe that these limits gain paramount importance when the complexity of the terrains to be traversed increases. We build on previous research from the field of robotic gras** to propose two new six-dimensional bounded po… ▽ More Motion planning in multi-contact scenarios has recently gathered interest within the legged robotics community, however actuator force/torque limits are rarely considered. We believe that these limits gain paramount importance when the complexity of the terrains to be traversed increases. We build on previous research from the field of robotic gras** to propose two new six-dimensional bounded polytopes named the Actuation Wrench Polytope (AWP) and the Feasible Wrench Polytope (FWP). We define the AWP as the set of all the wrenches that a robot can generate while considering its actuation limits. This considers the admissible contact forces that the robot can generate given its current configuration and actuation capabilities. The Contact Wrench Cone (CWC), instead, includes features of the environment such as the contact normal or the friction coefficient. The intersection of the AWP and of the CWC results in a convex polytope, the FWP, which turns out to be more descriptive of the real robot capabilities than existing simplified models, while maintaining the same compact representation. We explain how to efficiently compute the vertex-description of the FWP that is then used to evaluate a feasibility factor that we adapted from the field of robotic gras**. This allows us to optimize for robustness to external disturbance wrenches. Based on this, we present an implementation of a motion planner for our quadruped robot HyQ that provides online Center of Mass (CoM) trajectories that are guaranteed to be statically stable and actuation consistent. △ Less

Submitted 18 June, 2018; v1 submitted 19 December, 2017; originally announced December 2017.

Journal ref: IEEE Robotics and Automation Letters (RA-L) 2018

The Actuation-consistent Wrench Polytope (AWP) and the Feasible Wrench Polytope (FWP)

Authors: Romeo Orsolino, Michele Focchi, Carlos Mastalli, Hongkai Dai, Darwin G. Caldwell, Claudio Semini

Abstract: The motivation of our current research is to devise motion planners for legged locomotion that are able to exploit the robot's actuation capabilities. This means, when possible, to minimize joint torques or to propel as much as admissible when required. For this reason we define two new 6 dimensional bounded polytopes that we name Actuation-consistent Wrench Polytope (AWP) and Feasible Wrench Poly… ▽ More The motivation of our current research is to devise motion planners for legged locomotion that are able to exploit the robot's actuation capabilities. This means, when possible, to minimize joint torques or to propel as much as admissible when required. For this reason we define two new 6 dimensional bounded polytopes that we name Actuation-consistent Wrench Polytope (AWP) and Feasible Wrench Polytope (FWP). These objects turn out to be very useful in motion planning for the definition of constraints on the accelerations of the Center of Mass of the robot that respect the friction cones and the actuation limits. The AWP and the FWP could be used also in the robot design phase to size the actuators of the system based on some predefined reference motion. △ Less

Submitted 7 December, 2017; originally announced December 2017.

Translating Videos to Commands for Robotic Manipulation with Deep Recurrent Neural Networks

Authors: Anh Nguyen, Dimitrios Kanoulas, Luca Muratore, Darwin G. Caldwell, Nikos G. Tsagarakis

Abstract: We present a new method to translate videos to commands for robotic manipulation using Deep Recurrent Neural Networks (RNN). Our framework first extracts deep features from the input video frames with a deep Convolutional Neural Networks (CNN). Two RNN layers with an encoder-decoder architecture are then used to encode the visual features and sequentially generate the output words as the command.… ▽ More We present a new method to translate videos to commands for robotic manipulation using Deep Recurrent Neural Networks (RNN). Our framework first extracts deep features from the input video frames with a deep Convolutional Neural Networks (CNN). Two RNN layers with an encoder-decoder architecture are then used to encode the visual features and sequentially generate the output words as the command. We demonstrate that the translation accuracy can be improved by allowing a smooth transaction between two RNN layers and using the state-of-the-art feature extractor. The experimental results on our new challenging dataset show that our approach outperforms recent methods by a fair margin. Furthermore, we combine the proposed translation module with the vision and planning system to let a robot perform various manipulation tasks. Finally, we demonstrate the effectiveness of our framework on a full-size humanoid robot WALK-MAN. △ Less

Submitted 1 October, 2017; originally announced October 2017.

Real-Time 6DOF Pose Relocalization for Event Cameras with Stacked Spatial LSTM Networks

Authors: Anh Nguyen, Thanh-Toan Do, Darwin G. Caldwell, Nikos G. Tsagarakis

Abstract: We present a new method to relocalize the 6DOF pose of an event camera solely based on the event stream. Our method first creates the event image from a list of events that occurs in a very short time interval, then a Stacked Spatial LSTM Network (SP-LSTM) is used to learn the camera pose. Our SP-LSTM is composed of a CNN to learn deep features from the event images and a stack of LSTM to learn sp… ▽ More We present a new method to relocalize the 6DOF pose of an event camera solely based on the event stream. Our method first creates the event image from a list of events that occurs in a very short time interval, then a Stacked Spatial LSTM Network (SP-LSTM) is used to learn the camera pose. Our SP-LSTM is composed of a CNN to learn deep features from the event images and a stack of LSTM to learn spatial dependencies in the image feature space. We show that the spatial dependency plays an important role in the relocalization task and the SP-LSTM can effectively learn this information. The experimental results on a publicly available dataset show that our approach generalizes well and outperforms recent methods by a substantial margin. Overall, our proposed method reduces by approx. 6 times the position error and 3 times the orientation error compared to the current state of the art. The source code and trained models will be released. △ Less

Submitted 13 April, 2018; v1 submitted 22 August, 2017; originally announced August 2017.

Comments: 7 pages, 5 figures

Journal ref: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019

Kernelized Movement Primitives

Authors: Yanlong Huang, Leonel Rozo, João Silvério, Darwin G. Caldwell

Abstract: Imitation learning has been studied widely as a convenient way to transfer human skills to robots. This learning approach is aimed at extracting relevant motion patterns from human demonstrations and subsequently applying these patterns to different situations. Despite many advancements have been achieved, the solutions for co** with unpredicted situations (e.g., obstacles and external perturbat… ▽ More Imitation learning has been studied widely as a convenient way to transfer human skills to robots. This learning approach is aimed at extracting relevant motion patterns from human demonstrations and subsequently applying these patterns to different situations. Despite many advancements have been achieved, the solutions for co** with unpredicted situations (e.g., obstacles and external perturbations) and high-dimensional inputs are still largely open. In this paper, we propose a novel kernelized movement primitive (KMP), which allows the robot to adapt the learned motor skills and fulfill a variety of additional constraints arising over the course of a task. Specifically, KMP is capable of learning trajectories associated with high-dimensional inputs due to the kernel treatment, which in turn renders a model with fewer open parameters in contrast to methods that rely on basis functions. Moreover, we extend our approach by exploiting local trajectory representations in different coordinate systems that describe the task at hand, endowing KMP with reliable extrapolation capabilities in broader domains. We apply KMP to the learning of time-driven trajectories as a special case, where a compact parametric representation describing a trajectory and its first-order derivative is utilized. In order to verify the effectiveness of our method, several examples of trajectory modulations and extrapolations associated with time inputs, as well as trajectory adaptations with high-dimensional inputs are provided. △ Less

Submitted 6 March, 2018; v1 submitted 29 August, 2017; originally announced August 2017.

Comments: 17 pages

Learning Task Priorities from Demonstrations

Authors: João Silvério, Sylvain Calinon, Leonel Rozo, Darwin G. Caldwell

Abstract: Bimanual operations in humanoids offer the possibility to carry out more than one manipulation task at the same time, which in turn introduces the problem of task prioritization. We address this problem from a learning from demonstration perspective, by extending the Task-Parameterized Gaussian Mixture Model (TP-GMM) to Jacobian and null space structures. The proposed approach is tested on bimanua… ▽ More Bimanual operations in humanoids offer the possibility to carry out more than one manipulation task at the same time, which in turn introduces the problem of task prioritization. We address this problem from a learning from demonstration perspective, by extending the Task-Parameterized Gaussian Mixture Model (TP-GMM) to Jacobian and null space structures. The proposed approach is tested on bimanual skills but can be applied in any scenario where the prioritization between potentially conflicting tasks needs to be learned. We evaluate the proposed framework in: two different tasks with humanoids requiring the learning of priorities and a loco-manipulation scenario, showing that the approach can be exploited to learn the prioritization of multiple tasks in parallel. △ Less

Submitted 20 November, 2018; v1 submitted 21 July, 2017; originally announced July 2017.

Comments: Accepted for publication at the IEEE Transactions on Robotics

Generalized Task-Parameterized Skill Learning

Authors: Yanlong Huang, João Silvério, Leonel Rozo, Darwin G. Caldwell

Abstract: Programming by demonstration has recently gained much attention due to its user-friendly and natural way to transfer human skills to robots. In order to facilitate the learning of multiple demonstrations and meanwhile generalize to new situations, a task-parameterized Gaussian mixture model (TP-GMM) has been recently developed. This model has achieved reliable performance in areas such as human-ro… ▽ More Programming by demonstration has recently gained much attention due to its user-friendly and natural way to transfer human skills to robots. In order to facilitate the learning of multiple demonstrations and meanwhile generalize to new situations, a task-parameterized Gaussian mixture model (TP-GMM) has been recently developed. This model has achieved reliable performance in areas such as human-robot collaboration and dual-arm manipulation. However, the crucial task frames and associated parameters in this learning framework are often set by the human teacher, which renders three problems that have not been addressed yet: (i) task frames are treated equally, without considering their individual importance, (ii) task parameters are defined without taking into account additional task constraints, such as robot joint limits and motion smoothness, and (iii) a fixed number of task frames are pre-defined regardless of whether some of them may be redundant or even irrelevant for the task at hand. In this paper, we generalize the task-parameterized learning by addressing the aforementioned problems. Moreover, we provide a novel learning perspective which allows the robot to refine and adapt previously learned skills in a low dimensional space. Several examples are studied in both simulated and real robotic systems, showing the applicability of our approach. △ Less

Submitted 5 March, 2018; v1 submitted 6 July, 2017; originally announced July 2017.

Comments: 8 pages

Electrocorticographic Dynamics Predict Visually Guided Motor Imagery of Grasp Sha**

Authors: **g Wu, Kaitlyn Casimo, David J. Caldwell, Rajesh P. N. Rao, Jeffrey G. Ojemann

Abstract: Identification of intended movement type and movement phase of hand grasp sha** are critical features for the control of volitional neuroprosthetics. We demonstrate that neural dynamics during visually-guided imagined grasp sha** can encode intended movement. We apply Procrustes analysis and LASSO regression to achieve 72% accuracy (chance = 25%) in distinguishing between visually-guided imagi… ▽ More Identification of intended movement type and movement phase of hand grasp sha** are critical features for the control of volitional neuroprosthetics. We demonstrate that neural dynamics during visually-guided imagined grasp sha** can encode intended movement. We apply Procrustes analysis and LASSO regression to achieve 72% accuracy (chance = 25%) in distinguishing between visually-guided imagined grasp trajectories. Further, we can predict the stage of grasp sha** in the form of elapsed time from start of trial (R2=0.4). Our approach contributes to more accurate single-trial decoding of higher-level movement goals and the phase of gras** movements in individuals not trained with brain-computer interfaces. We also find that the overall time-varying trajectory structure of imagined movements tend to be consistent within individuals, and that transient trajectory deviations within trials return to the task-dependent trajectory mean. These overall findings may contribute to the further understanding of the cortical dynamics of human motor imagery. △ Less

Submitted 20 February, 2017; originally announced February 2017.

Comments: 4 pages, 6 figures, accepted to IEEE NER 2017 (8th International IEEE EMBS Conference on Neural Engineering)

Robot Impedance Control and Passivity Analysis with Inner Torque and Velocity Feedback Loops

Authors: Michele Focchi, Gustavo A. Medrano-Cerda, Thiago Boaventura, Marco Frigerio, Jonas Buchli, Darwin G. Caldwell, Claudio Semini

Abstract: Impedance control is a well-established technique to control interaction forces in robotics. However, real implementations of impedance control with an inner loop may suffer from several limitations. Although common practice in designing nested control systems is to maximize the bandwidth of the inner loop to improve tracking performance, it may not be the most suitable approach when a certain ran… ▽ More Impedance control is a well-established technique to control interaction forces in robotics. However, real implementations of impedance control with an inner loop may suffer from several limitations. Although common practice in designing nested control systems is to maximize the bandwidth of the inner loop to improve tracking performance, it may not be the most suitable approach when a certain range of impedance parameters has to be rendered. In particular, it turns out that the viable range of stable stiffness and dam** values can be strongly affected by the bandwidth of the inner control loops (e.g. a torque loop) as well as by the filtering and sampling frequency. This paper provides an extensive analysis on how these aspects influence the stability region of impedance parameters as well as the passivity of the system. This will be supported by both simulations and experimental data. Moreover, a methodology for designing joint impedance controllers based on an inner torque loop and a positive velocity feedback loop will be presented. The goal of the velocity feedback is to increase (given the constraints to preserve stability) the bandwidth of the torque loop without the need of a complex controller. △ Less

Submitted 23 May, 2016; v1 submitted 16 June, 2014; originally announced June 2014.

Comments: 14 pages in Control Theory and Technology (2016)

A Domain Specific Language for kinematic models and fast implementations of robot dynamics algorithms

Authors: Marco Frigerio, Jonas Buchli, Darwin G. Caldwell

Abstract: Rigid body dynamics algorithms play a crucial role in several components of a robot controller and simulations. Real time constraints in high frequency control loops and time requirements of specific applications demand these functions to be very efficient. Despite the availability of established algorithms, their efficient implementation for a specific robot still is a tedious and error-prone tas… ▽ More Rigid body dynamics algorithms play a crucial role in several components of a robot controller and simulations. Real time constraints in high frequency control loops and time requirements of specific applications demand these functions to be very efficient. Despite the availability of established algorithms, their efficient implementation for a specific robot still is a tedious and error-prone task. However, these components are simply necessary to get high performance controllers. To achieve efficient yet well maintainable implementations of dynamics algorithms we propose to use a domain specific language to describe the kinematics/dynamics model of a robot. Since the algorithms are parameterized on this model, executable code tailored for a specific robot can be generated, thanks to the facilities available for \dsls. This approach allows the users to deal only with the high level description of their robot and relieves them from problematic hand-crafted development; resources and efforts can then be focused on open research questions. Preliminary results about the generation of efficient code for inverse dynamics will be presented as a proof of concept of this approach. △ Less

Submitted 30 January, 2013; originally announced January 2013.

Comments: Presented at DSLRob 2011 (arXiv:1212.3308)

Report number: DSLRob/2011/02