-
RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation
Authors:
Konstantinos Bousmalis,
Giulia Vezzani,
Dushyant Rao,
Coline Devin,
Alex X. Lee,
Maria Bauza,
Todor Davchev,
Yuxiang Zhou,
Agrim Gupta,
Akhil Raju,
Antoine Laurens,
Claudio Fantacci,
Valentin Dalibard,
Martina Zambelli,
Murilo Martins,
Rugile Pevceviciute,
Michiel Blokzijl,
Misha Denil,
Nathan Batchelor,
Thomas Lampe,
Emilio Parisotto,
Konrad Żołna,
Scott Reed,
Sergio Gómez Colmenarejo,
Jon Scholz
, et al. (14 additional authors not shown)
Abstract:
The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learning. Inspired by recent advances in foundation models for vision and language, we propose a multi-embodiment, multi-task generalist agent for robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned de…
▽ More
The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learning. Inspired by recent advances in foundation models for vision and language, we propose a multi-embodiment, multi-task generalist agent for robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned decision transformer capable of consuming action-labelled visual experience. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With RoboCat, we demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100-1000 examples for the target task. We also show how a trained model itself can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop. We investigate the agent's capabilities, with large-scale evaluations both in simulation and on three different real robot embodiments. We find that as we grow and diversify its training data, RoboCat not only shows signs of cross-task transfer, but also becomes more efficient at adapting to new tasks.
△ Less
Submitted 22 December, 2023; v1 submitted 20 June, 2023;
originally announced June 2023.
-
Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies
Authors:
Dushyant Rao,
Fereshteh Sadeghi,
Leonard Hasenclever,
Markus Wulfmeier,
Martina Zambelli,
Giulia Vezzani,
Dhruva Tirumala,
Yusuf Aytar,
Josh Merel,
Nicolas Heess,
Raia Hadsell
Abstract:
For robots operating in the real world, it is desirable to learn reusable behaviours that can effectively be transferred and adapted to numerous tasks and scenarios. We propose an approach to learn abstract motor skills from data using a hierarchical mixture latent variable model. In contrast to existing work, our method exploits a three-level hierarchy of both discrete and continuous latent varia…
▽ More
For robots operating in the real world, it is desirable to learn reusable behaviours that can effectively be transferred and adapted to numerous tasks and scenarios. We propose an approach to learn abstract motor skills from data using a hierarchical mixture latent variable model. In contrast to existing work, our method exploits a three-level hierarchy of both discrete and continuous latent variables, to capture a set of high-level behaviours while allowing for variance in how they are executed. We demonstrate in manipulation domains that the method can effectively cluster offline data into distinct, executable behaviours, while retaining the flexibility of a continuous latent variable model. The resulting skills can be transferred and fine-tuned on new tasks, unseen objects, and from state to vision-based policies, yielding better sample efficiency and asymptotic performance compared to existing skill- and imitation-based methods. We further analyse how and when the skills are most beneficial: they encourage directed exploration to cover large regions of the state space relevant to the task, making them most effective in challenging sparse-reward settings.
△ Less
Submitted 14 March, 2022; v1 submitted 9 December, 2021;
originally announced December 2021.
-
Learning rich touch representations through cross-modal self-supervision
Authors:
Martina Zambelli,
Yusuf Aytar,
Francesco Visin,
Yuxiang Zhou,
Raia Hadsell
Abstract:
The sense of touch is fundamental in several manipulation tasks, but rarely used in robot manipulation. In this work we tackle the problem of learning rich touch features from cross-modal self-supervision. We evaluate them identifying objects and their properties in a few-shot classification setting. Two new datasets are introduced using a simulated anthropomorphic robotic hand equipped with tacti…
▽ More
The sense of touch is fundamental in several manipulation tasks, but rarely used in robot manipulation. In this work we tackle the problem of learning rich touch features from cross-modal self-supervision. We evaluate them identifying objects and their properties in a few-shot classification setting. Two new datasets are introduced using a simulated anthropomorphic robotic hand equipped with tactile sensors on both synthetic and daily life objects. Several self-supervised learning methods are benchmarked on these datasets, by evaluating few-shot classification on unseen objects and poses. Our experiments indicate that cross-modal self-supervision effectively improves touch representation, and in turn has great potential to enhance robot manipulation skills.
△ Less
Submitted 21 January, 2021;
originally announced January 2021.
-
A Distributional View on Multi-Objective Policy Optimization
Authors:
Abbas Abdolmaleki,
Sandy H. Huang,
Leonard Hasenclever,
Michael Neunert,
H. Francis Song,
Martina Zambelli,
Murilo F. Martins,
Nicolas Heess,
Raia Hadsell,
Martin Riedmiller
Abstract:
Many real-world problems require trading off multiple competing objectives. However, these objectives are often in different units and/or scales, which can make it challenging for practitioners to express numerical preferences over objectives in their native units. In this paper we propose a novel algorithm for multi-objective reinforcement learning that enables setting desired preferences for obj…
▽ More
Many real-world problems require trading off multiple competing objectives. However, these objectives are often in different units and/or scales, which can make it challenging for practitioners to express numerical preferences over objectives in their native units. In this paper we propose a novel algorithm for multi-objective reinforcement learning that enables setting desired preferences for objectives in a scale-invariant way. We propose to learn an action distribution for each objective, and we use supervised learning to fit a parametric policy to a combination of these distributions. We demonstrate the effectiveness of our approach on challenging high-dimensional real and simulated robotics tasks, and show that setting different preferences in our framework allows us to trace out the space of nondominated solutions.
△ Less
Submitted 15 May, 2020;
originally announced May 2020.
-
Multimodal representation models for prediction and control from partial information
Authors:
Martina Zambelli,
Antoine Cully,
Yiannis Demiris
Abstract:
Similar to humans, robots benefit from interacting with their environment through a number of different sensor modalities, such as vision, touch, sound. However, learning from different sensor modalities is difficult, because the learning model must be able to handle diverse types of signals, and learn a coherent representation even when parts of the sensor inputs are missing. In this paper, a mul…
▽ More
Similar to humans, robots benefit from interacting with their environment through a number of different sensor modalities, such as vision, touch, sound. However, learning from different sensor modalities is difficult, because the learning model must be able to handle diverse types of signals, and learn a coherent representation even when parts of the sensor inputs are missing. In this paper, a multimodal variational autoencoder is proposed to enable an iCub humanoid robot to learn representations of its sensorimotor capabilities from different sensor modalities. The proposed model is able to (1) reconstruct missing sensory modalities, (2) predict the sensorimotor state of self and the visual trajectories of other agents actions, and (3) control the agent to imitate an observed visual trajectory. Also, the proposed multimodal variational autoencoder can capture the kinematic redundancy of the robot motion through the learned probability distribution. Training multimodal models is not trivial due to the combinatorial complexity given by the possibility of missing modalities. We propose a strategy to train multimodal models, which successfully achieves improved performance of different reconstruction models. Finally, extensive experiments have been carried out using an iCub humanoid robot, showing high performance in multiple reconstruction, prediction and imitation tasks.
△ Less
Submitted 9 October, 2019;
originally announced October 2019.
-
Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
Authors:
Sandy H. Huang,
Martina Zambelli,
Jackie Kay,
Murilo F. Martins,
Yuval Tassa,
Patrick M. Pilarski,
Raia Hadsell
Abstract:
Robots must know how to be gentle when they need to interact with fragile objects, or when the robot itself is prone to wear and tear. We propose an approach that enables deep reinforcement learning to train policies that are gentle, both during exploration and task execution. In a reward-based learning environment, a natural approach involves augmenting the (task) reward with a penalty for non-ge…
▽ More
Robots must know how to be gentle when they need to interact with fragile objects, or when the robot itself is prone to wear and tear. We propose an approach that enables deep reinforcement learning to train policies that are gentle, both during exploration and task execution. In a reward-based learning environment, a natural approach involves augmenting the (task) reward with a penalty for non-gentleness, which can be defined as excessive impact force. However, augmenting with only this penalty impairs learning: policies get stuck in a local optimum which avoids all contact with the environment. Prior research has shown that combining auxiliary tasks or intrinsic rewards can be beneficial for stabilizing and accelerating learning in sparse-reward domains, and indeed we find that introducing a surprise-based intrinsic reward does avoid the no-contact failure case. However, we show that a simple dynamics-based surprise is not as effective as penalty-based surprise. Penalty-based surprise, based on predicting forceful contacts, has a further benefit: it encourages exploration which is contact-rich yet gentle. We demonstrate the effectiveness of the approach using a complex, tendon-powered robot hand with tactile sensors. Videos are available at http://sites.google.com/view/gentlemanipulation.
△ Less
Submitted 20 March, 2019;
originally announced March 2019.
-
Balancing-Aware Charging Strategy For Series-Connected Lithium-Ion Cells: A Nonlinear Model Predictive Control Approach
Authors:
Andrea Pozzi,
Massimo Zambelli,
Antonella Ferrara,
Davide Martino Raimondo
Abstract:
Charge unbalance is one of the key issues for series-connected Lithium-ion cells. Within this context, model-based optimization strategies have proven to be the most effective. In the present paper, an ad-hoc electrochemical model, tailored to control purposes, is firstly presented. Relying on this latter, a general nonlinear MPC for balancing-aware optimal charging is then proposed. In view of th…
▽ More
Charge unbalance is one of the key issues for series-connected Lithium-ion cells. Within this context, model-based optimization strategies have proven to be the most effective. In the present paper, an ad-hoc electrochemical model, tailored to control purposes, is firstly presented. Relying on this latter, a general nonlinear MPC for balancing-aware optimal charging is then proposed. In view of the possibility of a practical implementation, the concepts are subsequently specialized for an easily implementable power supply scheme. Finally, the nonlinear MPC approach is validated on commercial cells using a detailed battery simulator, with sound evidence of its effectiveness.
△ Less
Submitted 6 February, 2019;
originally announced February 2019.
-
DAC-h3: A Proactive Robot Cognitive Architecture to Acquire and Express Knowledge About the World and the Self
Authors:
Clément Moulin-Frier,
Tobias Fischer,
Maxime Petit,
Grégoire Pointeau,
Jordi-Ysard Puigbo,
Ugo Pattacini,
Sock Ching Low,
Daniel Camilleri,
Phuong Nguyen,
Matej Hoffmann,
Hyung ** Chang,
Martina Zambelli,
Anne-Laure Mealier,
Andreas Damianou,
Giorgio Metta,
Tony J. Prescott,
Yiannis Demiris,
Peter Ford Dominey,
Paul F. M. J. Verschure
Abstract:
This paper introduces a cognitive architecture for a humanoid robot to engage in a proactive, mixed-initiative exploration and manipulation of its environment, where the initiative can originate from both the human and the robot. The framework, based on a biologically-grounded theory of the brain and mind, integrates a reactive interaction engine, a number of state-of-the-art perceptual and motor…
▽ More
This paper introduces a cognitive architecture for a humanoid robot to engage in a proactive, mixed-initiative exploration and manipulation of its environment, where the initiative can originate from both the human and the robot. The framework, based on a biologically-grounded theory of the brain and mind, integrates a reactive interaction engine, a number of state-of-the-art perceptual and motor learning algorithms, as well as planning abilities and an autobiographical memory. The architecture as a whole drives the robot behavior to solve the symbol grounding problem, acquire language capabilities, execute goal-oriented behavior, and express a verbal narrative of its own experience in the world. We validate our approach in human-robot interaction experiments with the iCub humanoid robot, showing that the proposed cognitive architecture can be applied in real time within a realistic scenario and that it can be used with naive users.
△ Less
Submitted 18 September, 2017; v1 submitted 12 June, 2017;
originally announced June 2017.
