-
Replay across Experiments: A Natural Extension of Off-Policy RL
Authors:
Dhruva Tirumala,
Thomas Lampe,
Jose Enrique Chen,
Tuomas Haarnoja,
Sandy Huang,
Guy Lever,
Ben Moran,
Tim Hertweck,
Leonard Hasenclever,
Martin Riedmiller,
Nicolas Heess,
Markus Wulfmeier
Abstract:
Replaying data is a principal mechanism underlying the stability and data efficiency of off-policy reinforcement learning (RL). We present an effective yet simple framework to extend the use of replays across multiple experiments, minimally adapting the RL workflow for sizeable improvements in controller performance and research iteration times. At its core, Replay Across Experiments (RaE) involve…
▽ More
Replaying data is a principal mechanism underlying the stability and data efficiency of off-policy reinforcement learning (RL). We present an effective yet simple framework to extend the use of replays across multiple experiments, minimally adapting the RL workflow for sizeable improvements in controller performance and research iteration times. At its core, Replay Across Experiments (RaE) involves reusing experience from previous experiments to improve exploration and bootstrap learning while reducing required changes to a minimum in comparison to prior work. We empirically show benefits across a number of RL algorithms and challenging control domains spanning both locomotion and manipulation, including hard exploration tasks from egocentric vision. Through comprehensive ablations, we demonstrate robustness to the quality and amount of data available and various hyperparameter choices. Finally, we discuss how our approach can be applied more broadly across research life cycles and can increase resilience by reloading data across random seeds or hyperparameter variations.
△ Less
Submitted 28 November, 2023; v1 submitted 27 November, 2023;
originally announced November 2023.
-
RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation
Authors:
Konstantinos Bousmalis,
Giulia Vezzani,
Dushyant Rao,
Coline Devin,
Alex X. Lee,
Maria Bauza,
Todor Davchev,
Yuxiang Zhou,
Agrim Gupta,
Akhil Raju,
Antoine Laurens,
Claudio Fantacci,
Valentin Dalibard,
Martina Zambelli,
Murilo Martins,
Rugile Pevceviciute,
Michiel Blokzijl,
Misha Denil,
Nathan Batchelor,
Thomas Lampe,
Emilio Parisotto,
Konrad Żołna,
Scott Reed,
Sergio Gómez Colmenarejo,
Jon Scholz
, et al. (14 additional authors not shown)
Abstract:
The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learning. Inspired by recent advances in foundation models for vision and language, we propose a multi-embodiment, multi-task generalist agent for robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned de…
▽ More
The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learning. Inspired by recent advances in foundation models for vision and language, we propose a multi-embodiment, multi-task generalist agent for robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned decision transformer capable of consuming action-labelled visual experience. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With RoboCat, we demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100-1000 examples for the target task. We also show how a trained model itself can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop. We investigate the agent's capabilities, with large-scale evaluations both in simulation and on three different real robot embodiments. We find that as we grow and diversify its training data, RoboCat not only shows signs of cross-task transfer, but also becomes more efficient at adapting to new tasks.
△ Less
Submitted 22 December, 2023; v1 submitted 20 June, 2023;
originally announced June 2023.
-
Barkour: Benchmarking Animal-level Agility with Quadruped Robots
Authors:
Ken Caluwaerts,
Atil Iscen,
J. Chase Kew,
Wenhao Yu,
Tingnan Zhang,
Daniel Freeman,
Kuang-Huei Lee,
Lisa Lee,
Stefano Saliceti,
Vincent Zhuang,
Nathan Batchelor,
Steven Bohez,
Federico Casarini,
Jose Enrique Chen,
Omar Cortes,
Erwin Coumans,
Adil Dostmohamed,
Gabriel Dulac-Arnold,
Alejandro Escontrela,
Erik Frey,
Roland Hafner,
Deepali Jain,
Bauyrjan Jyenis,
Yuheng Kuang,
Edward Lee
, et al. (19 additional authors not shown)
Abstract:
Animals have evolved various agile locomotion strategies, such as sprinting, lea**, and jum**. There is a growing interest in develo** legged robots that move like their biological counterparts and show various agile skills to navigate complex environments quickly. Despite the interest, the field lacks systematic benchmarks to measure the performance of control policies and hardware in agili…
▽ More
Animals have evolved various agile locomotion strategies, such as sprinting, lea**, and jum**. There is a growing interest in develo** legged robots that move like their biological counterparts and show various agile skills to navigate complex environments quickly. Despite the interest, the field lacks systematic benchmarks to measure the performance of control policies and hardware in agility. We introduce the Barkour benchmark, an obstacle course to quantify agility for legged robots. Inspired by dog agility competitions, it consists of diverse obstacles and a time based scoring mechanism. This encourages researchers to develop controllers that not only move fast, but do so in a controllable and versatile way. To set strong baselines, we present two methods for tackling the benchmark. In the first approach, we train specialist locomotion skills using on-policy reinforcement learning methods and combine them with a high-level navigation controller. In the second approach, we distill the specialist skills into a Transformer-based generalist locomotion policy, named Locomotion-Transformer, that can handle various terrains and adjust the robot's gait based on the perceived environment and robot states. Using a custom-built quadruped robot, we demonstrate that our method can complete the course at half the speed of a dog. We hope that our work represents a step towards creating controllers that enable robots to reach animal-level agility.
△ Less
Submitted 23 May, 2023;
originally announced May 2023.
-
Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes
Authors:
Alex X. Lee,
Coline Devin,
Yuxiang Zhou,
Thomas Lampe,
Konstantinos Bousmalis,
Jost Tobias Springenberg,
Arunkumar Byravan,
Abbas Abdolmaleki,
Nimrod Gileadi,
David Khosid,
Claudio Fantacci,
Jose Enrique Chen,
Akhil Raju,
Rae Jeong,
Michael Neunert,
Antoine Laurens,
Stefano Saliceti,
Federico Casarini,
Martin Riedmiller,
Raia Hadsell,
Francesco Nori
Abstract:
We study the problem of robotic stacking with objects of complex geometry. We propose a challenging and diverse set of such objects that was carefully designed to require strategies beyond a simple "pick-and-place" solution. Our method is a reinforcement learning (RL) approach combined with vision-based interactive policy distillation and simulation-to-reality transfer. Our learned policies can ef…
▽ More
We study the problem of robotic stacking with objects of complex geometry. We propose a challenging and diverse set of such objects that was carefully designed to require strategies beyond a simple "pick-and-place" solution. Our method is a reinforcement learning (RL) approach combined with vision-based interactive policy distillation and simulation-to-reality transfer. Our learned policies can efficiently handle multiple object combinations in the real world and exhibit a large variety of stacking skills. In a large experimental study, we investigate what choices matter for learning such general vision-based agents in simulation, and what affects optimal transfer to the real robot. We then leverage data collected by such policies and improve upon them with offline RL. A video and a blog post of our work are provided as supplementary material.
△ Less
Submitted 3 November, 2021; v1 submitted 12 October, 2021;
originally announced October 2021.
-
Cascaded Generation of a Sub-10-Attosecond Half-Cycle Pulse
Authors:
Yinren Shou,
Ronghao Hu,
Zheng Gong,
**qing Yu,
Jia erh Chen,
Gerard Mourou,
Xueqing Yan,
Wenjun Ma
Abstract:
Sub-10-attosecond pulses with half-cycle electric fields provide exceptional options to detect and manipulate electrons in the atomic timescale. However, the availability of such pulses is still challenging. Here, we propose a method to generate isolated sub-10-attosecond half-cycle pulses based on a cascade process naturally happening in plasma. A 100s-attosecond pulse is first generated by shoot…
▽ More
Sub-10-attosecond pulses with half-cycle electric fields provide exceptional options to detect and manipulate electrons in the atomic timescale. However, the availability of such pulses is still challenging. Here, we propose a method to generate isolated sub-10-attosecond half-cycle pulses based on a cascade process naturally happening in plasma. A 100s-attosecond pulse is first generated by shooting a moderate overdense plasma with a one-cycle femtosecond pulse. After that, the generated attosecond pulse cascadedly produce a sub-10-attosecond half-cycle pulse in the transmission direction by unipolarly perturbing a nanometer-thin relativistic electron sheet naturally form in the plasma. Two-dimensional particle-in-cell simulations indicate that an isolated half-cycle pulse with the duration of 8.3 attoseconds can be produced. Apart from one-cycle driving pulse, such a scheme also can be realized with a commercial 100-TW 25-fs driving laser by sha** the pulse with a relativistic plasma lens in advance.
△ Less
Submitted 14 October, 2020; v1 submitted 12 October, 2020;
originally announced October 2020.
-
Computational Modeling of the Effects of Inflammatory Response and Granulation Tissue Properties on Human Bone Fracture Healing
Authors:
Mohammad S. Ghiasi,
Jason E. Chen,
Edward K. Rodriguez,
Ashkan Vaziri,
Ara Nazarian
Abstract:
Bone healing process includes four phases: inflammatory response, soft callus formation, hard callus development, and remodeling. Mechanobiological models have been used to investigate the role of various mechanical and biological factors on the bone healing. However, the initial phase of healing, which includes the inflammatory response, the granulation tissue formation and the initial callus for…
▽ More
Bone healing process includes four phases: inflammatory response, soft callus formation, hard callus development, and remodeling. Mechanobiological models have been used to investigate the role of various mechanical and biological factors on the bone healing. However, the initial phase of healing, which includes the inflammatory response, the granulation tissue formation and the initial callus formation during the first few days post-fracture, are generally neglected in such studies. In this study, we developed a finite-element-based model to simulate different levels of diffusion coefficient for mesenchymal stem cell (MSC) migration, Young's modulus of granulation tissue, callus thickness and interfragmentary gap size to understand the modulatory effects of these initial phase parameters on bone healing. The results showed that faster MSC migration, stiffer granulation tissue, thicker callus and smaller interfragmentary gap enhanced healing to some extent. After a certain threshold, a state of saturation was reached for MSC migration rate, granulation tissue stiffness and callus thickness. Therefore, a parametric study was performed to verify that the callus formed at the initial phase, in agreement with experimental observations, has an ideal range of geometry and material properties to have the most efficient healing time. Findings from this paper quantified the effects of the healing initial phase on healing outcome to better understand the biological and mechanobiological mechanisms and their utilization in the design and optimization of treatment strategies. Simulation outcomes also demonstrated that for fractures, where bone segments are in close proximity, callus development is not required. This finding is consistent with the concepts of primary and secondary bone healing.
△ Less
Submitted 13 August, 2018;
originally announced August 2018.
-
Laser acceleration of highly energetic carbon ions using a double-layer target composed of slightly underdense plasma and ultrathin foil
Authors:
W. J. Ma,
I Jong Kim,
J. Q. Yu,
Il Woo Choi,
P. K. Singh,
Hwang Woon Lee,
Jae Hee Sung,
Seong Ku Lee,
C. Lin,
Q. Liao,
J. G. Zhu,
H. Y. Lu,
B. Liu,
H. Y. Wang,
R. F. Xu,
X. T. He,
J. E. Chen,
M. Zepf,
J. Schreiber,
X. Q. Yan,
Chang Hee Nam
Abstract:
We report the experimental generation of highly energetic carbon ions up to 48 MeV per nucleon by shooting double-layer targets composed of well-controlled slightly underdense plasma (SUP) and ultrathin foils with ultra-intense femtosecond laser pulses. Particle-in-cell simulations reveal that carbon ions residing in the ultrathin foils undergo radiation pressure acceleration and long-time sheath…
▽ More
We report the experimental generation of highly energetic carbon ions up to 48 MeV per nucleon by shooting double-layer targets composed of well-controlled slightly underdense plasma (SUP) and ultrathin foils with ultra-intense femtosecond laser pulses. Particle-in-cell simulations reveal that carbon ions residing in the ultrathin foils undergo radiation pressure acceleration and long-time sheath field acceleration in sequence due to the existence of the SUP in front of the foils. Such an acceleration scheme is especially suited for heavy ion acceleration with femtosecond laser pulses. The breakthrough of heavy ion energy up to multi-tens of MeV/u at high-repetition-rate would be able to trigger significant advances in nuclear physics, high energy density physics, and medical physics.
△ Less
Submitted 31 January, 2018;
originally announced January 2018.
-
Distribution uniformity of laser-accelerated proton beams
Authors:
J. G. Zhu,
K. Zhu,
L. Tao,
X. H. Xu,
C. Lin,
W. J. Ma,
H. Y. Lu,
Y. Y. Zhao,
Y. R. Lu,
J. E. Chen,
X. Q. Yan
Abstract:
Compared with conventional accelerators, laser plasma accelerators can generate high energy ions at a greatly reduced scale, due to their TV/m acceleration gradient. A compact laser plasma accelerator (CLAPA) has been built at the Institute of Heavy Ion Physics at Peking University. It will be used for applied research like biological irradiation, astrophysics simulations, etc. A beamline system w…
▽ More
Compared with conventional accelerators, laser plasma accelerators can generate high energy ions at a greatly reduced scale, due to their TV/m acceleration gradient. A compact laser plasma accelerator (CLAPA) has been built at the Institute of Heavy Ion Physics at Peking University. It will be used for applied research like biological irradiation, astrophysics simulations, etc. A beamline system with multiple quadrupoles and an analyzing magnet for laser-accelerated ions is proposed here. Since laser-accelerated ion beams have broad energy spectra and large angular divergence, the parameters (beam waist position in the Y direction, beam line layout, drift distance, magnet angles etc.) of the beamline system are carefully designed and optimised to obtain a radially symmetric proton distribution at the irradiation platform. Requirements of energy selection and differences in focusing or defocusing in application systems greatly influence the evolution of proton distributions. With optimal parameters, radially symmetric proton distributions can be achieved and protons with different energy spread within 5% have similar transverse areas at the experiment target.
△ Less
Submitted 14 October, 2017; v1 submitted 10 February, 2017;
originally announced February 2017.
-
Preliminary design of laser accelerator beam line
Authors:
Y. Shang,
K. Zhu,
C. Cao,
J. G. Zhu,
Y. R. Lu,
Z. Y. Guo,
J. E. Chen,
X. Q Yan
Abstract:
A Compact laser plasma accelerator (CLAPA) is being built in Peking University, which is based on RPA-PSA mechanism or other acceleration mechanisms. According to the beam parameters from preparatory experiments and theoretical simulations, the beam line is preliminarily designed. The beam line is mainly constituted by common transport elements to deliver proton beam with the energy of 1~50MeV, en…
▽ More
A Compact laser plasma accelerator (CLAPA) is being built in Peking University, which is based on RPA-PSA mechanism or other acceleration mechanisms. According to the beam parameters from preparatory experiments and theoretical simulations, the beam line is preliminarily designed. The beam line is mainly constituted by common transport elements to deliver proton beam with the energy of 1~50MeV, energy spread of 0~1% and current of 0~108 proton per pulse to satisfy the requirement of different experiments. The simulation result of 15MeV proton beam with an energy spread of 1%, current of 1x108 proton per pulse and final spot radius of 9mm is presented in this paper.
△ Less
Submitted 28 November, 2013; v1 submitted 4 November, 2013;
originally announced November 2013.
-
High-quality proton bunch from laser interaction with a gas-filled cone target
Authors:
H. Y. Wang,
F. L. Zheng,
Y. R. Lu,
Z. Y. Guo,
X. T. He,
J. E. Chen,
X. Q. Yan
Abstract:
Generation of high-energy proton bunch from interaction of an intense short circularly polarized(CP) laser pulse with a gas-filled cone target(GCT) is investigated using two-dimensional particle-in-cell simulation. The GCT target consists of a hollow cone filled with near-critical gas-plasma and a thin foil attached to the tip of the cone. It is observed that as the laser pulse propagates in the g…
▽ More
Generation of high-energy proton bunch from interaction of an intense short circularly polarized(CP) laser pulse with a gas-filled cone target(GCT) is investigated using two-dimensional particle-in-cell simulation. The GCT target consists of a hollow cone filled with near-critical gas-plasma and a thin foil attached to the tip of the cone. It is observed that as the laser pulse propagates in the gas-plasma, the nonlinear focusing will result in an enhancement of the laser pulse intensity. It is shown that a large number of energetic electrons are generated from the gas-plasma and accelerated by the self-focused laser pulse. The energetic electrons then transports through the foil, forming a backside sheath field which is stronger than that produced by a simple planar target. A quasi-monoenergetic proton beam with maximum energy of 181 MeV is produced from this GCT target irradiated by a CP laser pulse at an intensity of $2.6\times10^{20}W/cm^2$, which is nearly three times higher compared to simple planar target(67MeV).
△ Less
Submitted 20 June, 2011;
originally announced June 2011.
-
Determination of Carrier-Envelope Phase of Relativistic Few-Cycle Laser Pulses by Thomson Backscattering Spectroscopy
Authors:
M. Wen,
L. L. **,
H. Y. Wang,
Z. Wang,
Y. R. Lu,
J. E. Chen,
X. Q. Yan
Abstract:
A novel method is proposed to determine the carrier-envelope phase (CEP) of a relativistic few-cycle laser pulse via the central frequency of the isolated light generated from Thomson backscattering (TBS). We theoretically investigate the generation of a uniform flying mirror when a few-cycle drive pulse with relativistic intensity (…
▽ More
A novel method is proposed to determine the carrier-envelope phase (CEP) of a relativistic few-cycle laser pulse via the central frequency of the isolated light generated from Thomson backscattering (TBS). We theoretically investigate the generation of a uniform flying mirror when a few-cycle drive pulse with relativistic intensity ($I > 10^{18} {\rm{W} \mathord{/
{\vphantom {\rm{W} {\rm{cm}^{\rm{2}}}}}.
\kern-\nulldelimiterspace} {\rm{cm}^{\rm{2}}}}$) interacts with a target combined with a thin and a thick foil. The central frequency of the isolated TBS light generated from the flying mirror shows a sensitive dependence on the CEP of the drive pulse. The obtained results are verified by one dimensional particle in cell (1D-PIC) simulations.
△ Less
Submitted 20 June, 2011;
originally announced June 2011.
-
Generating sub-TeV quasi-monoenergetic proton beam by an ultra-relativistically intense laser in the snowplow regime
Authors:
F. L. Zheng,
H. Y. Wang,
X. Q. Yan,
J. E. Chen,
Y. R. Lu,
Z. Y. Guo,
T. Tajima,
X. T. He
Abstract:
Snowplow ion acceleration is presented, using an ultra-relativistically intense laser pulse irradi- ating on a combination target, where the relativistic proton beam generated by radiation pressure acceleration can be trapped and accelerated by the laser plasma wakefield. The theory suggests that sub-TeV quasi-monoenergetic proton bunches can be generated by a centimeter-scale laser wakefield acce…
▽ More
Snowplow ion acceleration is presented, using an ultra-relativistically intense laser pulse irradi- ating on a combination target, where the relativistic proton beam generated by radiation pressure acceleration can be trapped and accelerated by the laser plasma wakefield. The theory suggests that sub-TeV quasi-monoenergetic proton bunches can be generated by a centimeter-scale laser wakefield accelerator, driven by a circularly polarized (CP) laser pulse with the peak intensity of 10^23W/cm^2 and duration of 116fs.
△ Less
Submitted 16 January, 2011; v1 submitted 12 January, 2011;
originally announced January 2011.
-
Self-organizing GeV, nano-Coulomb, collimated proton beam from laser foil interaction at 7 * 10^21 W/cm2
Authors:
X. Q. Yan,
H. C. Wu,
Z. M. Sheng,
J. E. Chen,
J. Meyer-ter-Vehn
Abstract:
We report on a self-organizing, quasi-stable regime of laser proton acceleration, producing 1 GeV nano-Coulomb proton bunches from laser foil interaction at an intensity of 7*10^21 W/cm2. The results are obtained from 2D PIC simulations, using circular polarized light normally incident on a planar, 500 nm thick hydrogen foil with Gaussian transverse profile. While foil plasma driven in the wings…
▽ More
We report on a self-organizing, quasi-stable regime of laser proton acceleration, producing 1 GeV nano-Coulomb proton bunches from laser foil interaction at an intensity of 7*10^21 W/cm2. The results are obtained from 2D PIC simulations, using circular polarized light normally incident on a planar, 500 nm thick hydrogen foil with Gaussian transverse profile. While foil plasma driven in the wings of the driving pulse is dispersed, a stable central clump with 1 - 2 lamda diameter is forming on the axis. The stabilisation is related to laser light having passed the transparent parts of the foil in the wing region and encompassing the still opaque central clump. This feature is observed consistently in 2D and 3D simulations. It depends on a laser pulse shape with high contrast ratio.
△ Less
Submitted 29 March, 2009; v1 submitted 26 March, 2009;
originally announced March 2009.
-
Monoenergetic proton beams accelerated by circularly polarized laser with thin solid foils
Authors:
X. Q. Yan,
C. Lin,
Z. M. Sheng,
Z. Y. Guo,
B. C. Liu,
Y. R. Lu,
J. X. Fang,
J. E. Chen
Abstract:
The acceleration of ions in the interaction of circular polarized laser pulses with overdense plasmas is investigated. For circular polarization laser pulses, the quasi-equilibrium for electrons is established due to the light pressure and the electrostatic field built up at the interacting front of the laser pulse. The ions located within the skin-depth of the laser pulse can be synchronously a…
▽ More
The acceleration of ions in the interaction of circular polarized laser pulses with overdense plasmas is investigated. For circular polarization laser pulses, the quasi-equilibrium for electrons is established due to the light pressure and the electrostatic field built up at the interacting front of the laser pulse. The ions located within the skin-depth of the laser pulse can be synchronously accelerated and bunched in the charge couple processes by the electrostatic field, and thereby monoenergetic and high intensity proton beam can be generated. The dynamics equations for accelerated ions are deduced and proved by particle-in-cell simulations.
△ Less
Submitted 22 December, 2007; v1 submitted 22 November, 2007;
originally announced November 2007.
