-
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Authors:
Tianhe Yu,
Deirdre Quillen,
Zhanpeng He,
Ryan Julian,
Avnish Narayan,
Hayden Shively,
Adithya Bellathur,
Karol Hausman,
Chelsea Finn,
Sergey Levine
Abstract:
Meta-reinforcement learning algorithms can enable robots to acquire new skills much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as d…
▽ More
Meta-reinforcement learning algorithms can enable robots to acquire new skills much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as different tasks. When policies are meta-trained on such narrow task distributions, they cannot possibly generalize to more quickly acquire entirely new tasks. Therefore, if the aim of these methods is to enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation tasks. Our aim is to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new, held-out tasks. We evaluate 7 state-of-the-art meta-reinforcement learning and multi-task learning algorithms on these tasks. Surprisingly, while each task and its variations (e.g., with different object positions) can be learned with reasonable success, these algorithms struggle to learn with multiple tasks at the same time, even with as few as ten distinct training tasks. Our analysis and open-source environments pave the way for future research in multi-task learning and meta-learning that can enable meaningful generalization, thereby unlocking the full potential of these methods.
△ Less
Submitted 14 June, 2021; v1 submitted 23 October, 2019;
originally announced October 2019.
-
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
Authors:
Kate Rakelly,
Aurick Zhou,
Deirdre Quillen,
Chelsea Finn,
Sergey Levine
Abstract:
Deep reinforcement learning algorithms require large amounts of experience to learn an individual task. While in principle meta-reinforcement learning (meta-RL) algorithms enable agents to learn new skills from small amounts of experience, several major challenges preclude their practicality. Current methods rely heavily on on-policy experience, limiting their sample efficiency. The also lack mech…
▽ More
Deep reinforcement learning algorithms require large amounts of experience to learn an individual task. While in principle meta-reinforcement learning (meta-RL) algorithms enable agents to learn new skills from small amounts of experience, several major challenges preclude their practicality. Current methods rely heavily on on-policy experience, limiting their sample efficiency. The also lack mechanisms to reason about task uncertainty when adapting to new tasks, limiting their effectiveness in sparse reward problems. In this paper, we address these challenges by develo** an off-policy meta-RL algorithm that disentangles task inference and control. In our approach, we perform online probabilistic filtering of latent task variables to infer how to solve a new task from small amounts of experience. This probabilistic interpretation enables posterior sampling for structured and efficient exploration. We demonstrate how to integrate these task variables with off-policy RL algorithms to achieve both meta-training and adaptation efficiency. Our method outperforms prior algorithms in sample efficiency by 20-100X as well as in asymptotic performance on several meta-RL benchmarks.
△ Less
Submitted 19 March, 2019;
originally announced March 2019.
-
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Authors:
Dmitry Kalashnikov,
Alex Irpan,
Peter Pastor,
Julian Ibarz,
Alexander Herzog,
Eric Jang,
Deirdre Quillen,
Ethan Holly,
Mrinal Kalakrishnan,
Vincent Vanhoucke,
Sergey Levine
Abstract:
In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of gras**, a longstanding challenge in robotic manipulation. In contrast to static learning behaviors that choose a grasp point and then execute the desired grasp, our method enables closed-loop vision-based control, where…
▽ More
In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of gras**, a longstanding challenge in robotic manipulation. In contrast to static learning behaviors that choose a grasp point and then execute the desired grasp, our method enables closed-loop vision-based control, whereby the robot continuously updates its grasp strategy based on the most recent observations to optimize long-horizon grasp success. To that end, we introduce QT-Opt, a scalable self-supervised vision-based reinforcement learning framework that can leverage over 580k real-world grasp attempts to train a deep neural network Q-function with over 1.2M parameters to perform closed-loop, real-world gras** that generalizes to 96% grasp success on unseen objects. Aside from attaining a very high success rate, our method exhibits behaviors that are quite distinct from more standard gras** systems: using only RGB vision-based perception from an over-the-shoulder camera, our method automatically learns regras** strategies, probes objects to find the most effective grasps, learns to reposition objects and perform other non-prehensile pre-grasp manipulations, and responds dynamically to disturbances and perturbations.
△ Less
Submitted 27 November, 2018; v1 submitted 27 June, 2018;
originally announced June 2018.
-
Deep Reinforcement Learning for Vision-Based Robotic Gras**: A Simulated Comparative Evaluation of Off-Policy Methods
Authors:
Deirdre Quillen,
Eric Jang,
Ofir Nachum,
Chelsea Finn,
Julian Ibarz,
Sergey Levine
Abstract:
In this paper, we explore deep reinforcement learning algorithms for vision-based robotic gras**. Model-free deep reinforcement learning (RL) has been successfully applied to a range of challenging environments, but the proliferation of algorithms makes it difficult to discern which particular approach would be best suited for a rich, diverse task like gras**. To answer this question, we propo…
▽ More
In this paper, we explore deep reinforcement learning algorithms for vision-based robotic gras**. Model-free deep reinforcement learning (RL) has been successfully applied to a range of challenging environments, but the proliferation of algorithms makes it difficult to discern which particular approach would be best suited for a rich, diverse task like gras**. To answer this question, we propose a simulated benchmark for robotic gras** that emphasizes off-policy learning and generalization to unseen objects. Off-policy learning enables utilization of gras** data over a wide variety of objects, and diversity is important to enable the method to generalize to new objects that were not seen during training. We evaluate the benchmark tasks against a variety of Q-function estimation methods, a method previously proposed for robotic gras** with deep neural network models, and a novel approach based on a combination of Monte Carlo return estimation and an off-policy correction. Our results indicate that several simple methods provide a surprisingly strong competitor to popular algorithms such as double Q-learning, and our analysis of stability sheds light on the relative tradeoffs between the algorithms.
△ Less
Submitted 28 March, 2018; v1 submitted 28 February, 2018;
originally announced February 2018.
-
Learning Hand-Eye Coordination for Robotic Gras** with Deep Learning and Large-Scale Data Collection
Authors:
Sergey Levine,
Peter Pastor,
Alex Krizhevsky,
Deirdre Quillen
Abstract:
We describe a learning-based approach to hand-eye coordination for robotic gras** from monocular images. To learn hand-eye coordination for gras**, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose.…
▽ More
We describe a learning-based approach to hand-eye coordination for robotic gras** from monocular images. To learn hand-eye coordination for gras**, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. To train our network, we collected over 800,000 grasp attempts over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and hardware. Our experimental evaluation demonstrates that our method achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing.
△ Less
Submitted 28 August, 2016; v1 submitted 7 March, 2016;
originally announced March 2016.
