-
Yell At Your Robot: Improving On-the-Fly from Language Corrections
Authors:
Lucy Xiaoyang Shi,
Zheyuan Hu,
Tony Z. Zhao,
Archit Sharma,
Karl Pertsch,
Jianlan Luo,
Sergey Levine,
Chelsea Finn
Abstract:
Hierarchical policies that combine language and low-level control have been shown to perform impressively long-horizon robotic tasks, by leveraging either zero-shot high-level planners like pretrained language and vision-language models (LLMs/VLMs) or models trained on annotated robotic demonstrations. However, for complex and dexterous skills, attaining high success rates on long-horizon tasks st…
▽ More
Hierarchical policies that combine language and low-level control have been shown to perform impressively long-horizon robotic tasks, by leveraging either zero-shot high-level planners like pretrained language and vision-language models (LLMs/VLMs) or models trained on annotated robotic demonstrations. However, for complex and dexterous skills, attaining high success rates on long-horizon tasks still represents a major challenge -- the longer the task is, the more likely it is that some stage will fail. Can humans help the robot to continuously improve its long-horizon task performance through intuitive and natural feedback? In this paper, we make the following observation: high-level policies that index into sufficiently rich and expressive low-level language-conditioned skills can be readily supervised with human feedback in the form of language corrections. We show that even fine-grained corrections, such as small movements ("move a bit to the left"), can be effectively incorporated into high-level policies, and that such corrections can be readily obtained from humans observing the robot and making occasional suggestions. This framework enables robots not only to rapidly adapt to real-time language feedback, but also incorporate this feedback into an iterative training scheme that improves the high-level policy's ability to correct errors in both low-level execution and high-level decision-making purely from verbal feedback. Our evaluation on real hardware shows that this leads to significant performance improvement in long-horizon, dexterous manipulation tasks without the need for any additional teleoperation. Videos and code are available at https://yay-robot.github.io/.
△ Less
Submitted 19 March, 2024;
originally announced March 2024.
-
Cross-Episodic Curriculum for Transformer Agents
Authors:
Lucy Xiaoyang Shi,
Yunfan Jiang,
Jake Grigsby,
Linxi "Jim" Fan,
Yuke Zhu
Abstract:
We present a new algorithm, Cross-Episodic Curriculum (CEC), to boost the learning efficiency and generalization of Transformer agents. Central to CEC is the placement of cross-episodic experiences into a Transformer's context, which forms the basis of a curriculum. By sequentially structuring online learning trials and mixed-quality demonstrations, CEC constructs curricula that encapsulate learni…
▽ More
We present a new algorithm, Cross-Episodic Curriculum (CEC), to boost the learning efficiency and generalization of Transformer agents. Central to CEC is the placement of cross-episodic experiences into a Transformer's context, which forms the basis of a curriculum. By sequentially structuring online learning trials and mixed-quality demonstrations, CEC constructs curricula that encapsulate learning progression and proficiency increase across episodes. Such synergy combined with the potent pattern recognition capabilities of Transformer models delivers a powerful cross-episodic attention mechanism. The effectiveness of CEC is demonstrated under two representative scenarios: one involving multi-task reinforcement learning with discrete control, such as in DeepMind Lab, where the curriculum captures the learning progression in both individual and progressively complex settings; and the other involving imitation learning with mixed-quality data for continuous control, as seen in RoboMimic, where the curriculum captures the improvement in demonstrators' expertise. In all instances, policies resulting from CEC exhibit superior performance and strong generalization. Code is open-sourced at https://cec-agent.github.io/ to facilitate research on Transformer agent learning.
△ Less
Submitted 12 October, 2023;
originally announced October 2023.
-
Waypoint-Based Imitation Learning for Robotic Manipulation
Authors:
Lucy Xiaoyang Shi,
Archit Sharma,
Tony Z. Zhao,
Chelsea Finn
Abstract:
While imitation learning methods have seen a resurgent interest for robotic manipulation, the well-known problem of compounding errors continues to afflict behavioral cloning (BC). Waypoints can help address this problem by reducing the horizon of the learning problem for BC, and thus, the errors compounded over time. However, waypoint labeling is underspecified, and requires additional human supe…
▽ More
While imitation learning methods have seen a resurgent interest for robotic manipulation, the well-known problem of compounding errors continues to afflict behavioral cloning (BC). Waypoints can help address this problem by reducing the horizon of the learning problem for BC, and thus, the errors compounded over time. However, waypoint labeling is underspecified, and requires additional human supervision. Can we generate waypoints automatically without any additional human supervision? Our key insight is that if a trajectory segment can be approximated by linear motion, the endpoints can be used as waypoints. We propose Automatic Waypoint Extraction (AWE) for imitation learning, a preprocessing module to decompose a demonstration into a minimal set of waypoints which when interpolated linearly can approximate the trajectory up to a specified error threshold. AWE can be combined with any BC algorithm, and we find that AWE can increase the success rate of state-of-the-art algorithms by up to 25% in simulation and by 4-28% on real-world bimanual manipulation tasks, reducing the decision making horizon by up to a factor of 10. Videos and code are available at https://lucys0.github.io/awe/
△ Less
Submitted 26 July, 2023;
originally announced July 2023.
-
Skill-based Model-based Reinforcement Learning
Authors:
Lucy Xiaoyang Shi,
Joseph J. Lim,
Youngwoon Lee
Abstract:
Model-based reinforcement learning (RL) is a sample-efficient way of learning complex behaviors by leveraging a learned single-step dynamics model to plan actions in imagination. However, planning every action for long-horizon tasks is not practical, akin to a human planning out every muscle movement. Instead, humans efficiently plan with high-level skills to solve complex tasks. From this intuiti…
▽ More
Model-based reinforcement learning (RL) is a sample-efficient way of learning complex behaviors by leveraging a learned single-step dynamics model to plan actions in imagination. However, planning every action for long-horizon tasks is not practical, akin to a human planning out every muscle movement. Instead, humans efficiently plan with high-level skills to solve complex tasks. From this intuition, we propose a Skill-based Model-based RL framework (SkiMo) that enables planning in the skill space using a skill dynamics model, which directly predicts the skill outcomes, rather than predicting all small details in the intermediate states, step by step. For accurate and efficient long-term planning, we jointly learn the skill dynamics model and a skill repertoire from prior experience. We then harness the learned skill dynamics model to accurately simulate and plan over long horizons in the skill space, which enables efficient downstream learning of long-horizon, sparse reward tasks. Experimental results in navigation and manipulation domains show that SkiMo extends the temporal horizon of model-based approaches and improves the sample efficiency for both model-based RL and skill-based RL. Code and videos are available at https://clvrai.com/skimo
△ Less
Submitted 11 December, 2022; v1 submitted 15 July, 2022;
originally announced July 2022.
-
The Effects of Interfacial Recombination and Injection Barrier on the Electrical Characteristics of Perovskite Solar Cells
Authors:
Lin Xing Shi,
Zi Shuai Wang,
Zengguang Huang,
Wei E. I. Sha,
Haoran Wang,
Zhen Zhou
Abstract:
Charge carrier recombination in the perovskite solar cells (PSCs) has a deep influence on the electrical performance, such as open circuit voltage, short circuit current, fill factor and ultimately power conversion efficiency. The impacts of injection barrier, recombination channels, do** properties of carrier transport layers and light intensity on the performance of PSCs are theoretically inve…
▽ More
Charge carrier recombination in the perovskite solar cells (PSCs) has a deep influence on the electrical performance, such as open circuit voltage, short circuit current, fill factor and ultimately power conversion efficiency. The impacts of injection barrier, recombination channels, do** properties of carrier transport layers and light intensity on the performance of PSCs are theoretically investigated by drift-diffusion model in this work. The results indicate that due to the injection barrier at the interfaces of perovskite and carrier transport layer, the accumulated carriers modify the electric field distribution throughout the PSCs. Thus, a zero electric field is generated at a specific applied voltage, with greatly increases the interfacial recombination, resulting in a local kink of current density-voltage (J-V) curve. This work provides an effective strategy to improve the efficiency of PSCs by pertinently reducing both the injection barrier and interfacial recombination.
△ Less
Submitted 26 March, 2018;
originally announced March 2018.
