-
AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch
Authors:
Max Yang,
Chenghua Lu,
Alex Church,
Yijiong Lin,
Chris Ford,
Haoran Li,
Efi Psomopoulou,
David A. W. Barton,
Nathan F. Lepora
Abstract:
Human hands are capable of in-hand manipulation in the presence of different hand motions. For a robot hand, harnessing rich tactile information to achieve this level of dexterity still remains a significant challenge. In this paper, we present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. We tackle this problem by training a d…
▽ More
Human hands are capable of in-hand manipulation in the presence of different hand motions. For a robot hand, harnessing rich tactile information to achieve this level of dexterity still remains a significant challenge. In this paper, we present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. We tackle this problem by training a dense tactile policy in simulation and present a sim-to-real method for rich tactile sensing to achieve zero-shot policy transfer. Our formulation allows the training of a unified policy to rotate unseen objects about arbitrary rotation axes in any hand direction. In our experiments, we highlight the benefit of capturing detailed contact information when handling objects with varying properties. Interestingly, despite not having explicit slip detection, we found rich multi-fingered tactile sensing can implicitly detect object movement within grasp and provide a reactive behavior that improves the robustness of the policy. The project website can be found at https://maxyang27896.github.io/anyrotate/.
△ Less
Submitted 11 June, 2024; v1 submitted 12 May, 2024;
originally announced May 2024.
-
Neural Context Flows for Learning Generalizable Dynamical Systems
Authors:
Roussel Desmond Nzoyem,
David A. W. Barton,
Tom Deakin
Abstract:
Neural Ordinary Differential Equations typically struggle to generalize to new dynamical behaviors created by parameter changes in the underlying system, even when the dynamics are close to previously seen behaviors. The issue gets worse when the changing parameters are unobserved, i.e., their value or influence is not directly measurable when collecting data. We introduce Neural Context Flow (NCF…
▽ More
Neural Ordinary Differential Equations typically struggle to generalize to new dynamical behaviors created by parameter changes in the underlying system, even when the dynamics are close to previously seen behaviors. The issue gets worse when the changing parameters are unobserved, i.e., their value or influence is not directly measurable when collecting data. We introduce Neural Context Flow (NCF), a framework that encodes said unobserved parameters in a latent context vector as input to a vector field. NCFs leverage differentiability of the vector field with respect to the parameters, along with first-order Taylor expansion to allow any context vector to influence trajectories from other parameters. We validate our method and compare it to established Multi-Task and Meta-Learning alternatives, showing competitive performance in mean squared error for in-domain and out-of-distribution evaluation on the Lotka-Volterra, Glycolytic Oscillator, and Gray-Scott problems. This study holds practical implications for foundational models in science and related areas that benefit from conditional neural ODEs. Our code is openly available at https://github.com/ddrous/ncflow.
△ Less
Submitted 8 July, 2024; v1 submitted 3 May, 2024;
originally announced May 2024.
-
A Comparison of Mesh-Free Differentiable Programming and Data-Driven Strategies for Optimal Control under PDE Constraints
Authors:
Roussel Desmond Nzoyem,
David A. W. Barton,
Tom Deakin
Abstract:
The field of Optimal Control under Partial Differential Equations (PDE) constraints is rapidly changing under the influence of Deep Learning and the accompanying automatic differentiation libraries. Novel techniques like Physics-Informed Neural Networks (PINNs) and Differentiable Programming (DP) are to be contrasted with established numerical schemes like Direct-Adjoint Loo** (DAL). We present…
▽ More
The field of Optimal Control under Partial Differential Equations (PDE) constraints is rapidly changing under the influence of Deep Learning and the accompanying automatic differentiation libraries. Novel techniques like Physics-Informed Neural Networks (PINNs) and Differentiable Programming (DP) are to be contrasted with established numerical schemes like Direct-Adjoint Loo** (DAL). We present a comprehensive comparison of DAL, PINN, and DP using a general-purpose mesh-free differentiable PDE solver based on Radial Basis Functions. Under Laplace and Navier-Stokes equations, we found DP to be extremely effective as it produces the most accurate gradients; thriving even when DAL fails and PINNs struggle. Additionally, we provide a detailed benchmark highlighting the limited conditions under which any of those methods can be efficiently used. Our work provides a guide to Optimal Control practitioners and connects them further to the Deep Learning community.
△ Less
Submitted 2 October, 2023;
originally announced October 2023.
-
Sim-to-Real Model-Based and Model-Free Deep Reinforcement Learning for Tactile Pushing
Authors:
Max Yang,
Yijiong Lin,
Alex Church,
John Lloyd,
Dandan Zhang,
David A. W. Barton,
Nathan F. Lepora
Abstract:
Object pushing presents a key non-prehensile manipulation problem that is illustrative of more complex robotic manipulation tasks. While deep reinforcement learning (RL) methods have demonstrated impressive learning capabilities using visual input, a lack of tactile sensing limits their capability for fine and reliable control during manipulation. Here we propose a deep RL approach to object pushi…
▽ More
Object pushing presents a key non-prehensile manipulation problem that is illustrative of more complex robotic manipulation tasks. While deep reinforcement learning (RL) methods have demonstrated impressive learning capabilities using visual input, a lack of tactile sensing limits their capability for fine and reliable control during manipulation. Here we propose a deep RL approach to object pushing using tactile sensing without visual input, namely tactile pushing. We present a goal-conditioned formulation that allows both model-free and model-based RL to obtain accurate policies for pushing an object to a goal. To achieve real-world performance, we adopt a sim-to-real approach. Our results demonstrate that it is possible to train on a single object and a limited sample of goals to produce precise and reliable policies that can generalize to a variety of unseen objects and pushing scenarios without domain randomization. We experiment with the trained agents in harsh pushing conditions, and show that with significantly more training samples, a model-free policy can outperform a model-based planner, generating shorter and more reliable pushing trajectories despite large disturbances. The simplicity of our training environment and effective real-world performance highlights the value of rich tactile information for fine manipulation. Code and videos are available at https://sites.google.com/view/tactile-rl-pushing/.
△ Less
Submitted 26 July, 2023;
originally announced July 2023.
-
Modelling of physical systems with a Hopf bifurcation using mechanistic models and machine learning
Authors:
K. H. Lee,
D. A. W. Barton,
L. Renson
Abstract:
We propose a new hybrid modelling approach that combines a mechanistic model with a machine-learnt model to predict the limit cycle oscillations of physical systems with a Hopf bifurcation. The mechanistic model is an ordinary differential equation normal-form model capturing the bifurcation structure of the system. A data-driven map** from this model to the experimental observations is then ide…
▽ More
We propose a new hybrid modelling approach that combines a mechanistic model with a machine-learnt model to predict the limit cycle oscillations of physical systems with a Hopf bifurcation. The mechanistic model is an ordinary differential equation normal-form model capturing the bifurcation structure of the system. A data-driven map** from this model to the experimental observations is then identified based on experimental data using machine learning techniques. The proposed method is first demonstrated numerically on a Van der Pol oscillator and a three-degree-of-freedom aeroelastic model. It is then applied to model the behaviour of a physical aeroelastic structure exhibiting limit cycle oscillations during wind tunnel tests. The method is shown to be general, data-efficient and to offer good accuracy without any prior knowledge about the system other than its bifurcation structure.
△ Less
Submitted 7 September, 2022;
originally announced September 2022.
-
Uncertainty quantification in mechanistic epidemic models via cross-entropy approximate Bayesian computation
Authors:
Americo Cunha Jr,
David A. W. Barton,
Thiago G. Ritto
Abstract:
This paper proposes a data-driven approximate Bayesian computation framework for parameter estimation and uncertainty quantification of epidemic models, which incorporates two novelties: (i) the identification of the initial conditions by using plausible dynamic states that are compatible with observational data; (ii) learning of an informative prior distribution for the model parameters via the c…
▽ More
This paper proposes a data-driven approximate Bayesian computation framework for parameter estimation and uncertainty quantification of epidemic models, which incorporates two novelties: (i) the identification of the initial conditions by using plausible dynamic states that are compatible with observational data; (ii) learning of an informative prior distribution for the model parameters via the cross-entropy method. The new methodology's effectiveness is illustrated with the aid of actual data from the COVID-19 epidemic in Rio de Janeiro city in Brazil, employing an ordinary differential equation-based model with a generalized SEIR mechanistic structure that includes time-dependent transmission rate, asymptomatics, and hospitalizations. A minimization problem with two cost terms (number of hospitalizations and deaths) is formulated, and twelve parameters are identified. The calibrated model provides a consistent description of the available data, able to extrapolate forecasts over a few weeks, making the proposed methodology very appealing for real-time epidemic modeling.
△ Less
Submitted 2 February, 2023; v1 submitted 13 July, 2022;
originally announced July 2022.
-
Using scientific machine learning for experimental bifurcation analysis of dynamic systems
Authors:
Sandor Beregi,
David A. W. Barton,
Djamel Rezgui,
Simon A. Neild
Abstract:
Augmenting mechanistic ordinary differential equation (ODE) models with machine-learnable structures is an novel approach to create highly accurate, low-dimensional models of engineering systems incorporating both expert knowledge and reality through measurement data. Our exploratory study focuses on training universal differential equation (UDE) models for physical nonlinear dynamical systems wit…
▽ More
Augmenting mechanistic ordinary differential equation (ODE) models with machine-learnable structures is an novel approach to create highly accurate, low-dimensional models of engineering systems incorporating both expert knowledge and reality through measurement data. Our exploratory study focuses on training universal differential equation (UDE) models for physical nonlinear dynamical systems with limit cycles: an aerofoil undergoing flutter oscillations and an electrodynamic nonlinear oscillator. We consider examples where training data is generated by numerical simulations, whereas we also employ the proposed modelling concept to physical experiments allowing us to investigate problems with a wide range of complexity. To collect the training data, the method of control-based continuation is used as it captures not just the stable but also the unstable limit cycles of the observed system. This feature makes it possible to extract more information about the observed system than the open-loop approach (surveying the steady state response by parameter sweeps without using control) would allow. We use both neural networks and Gaussian processes as universal approximators alongside the mechanistic models to give a critical assessment of the accuracy and robustness of the UDE modelling approach. We also highlight the potential issues one may run into during the training procedure indicating the limits of the current modelling framework.
△ Less
Submitted 18 June, 2022; v1 submitted 22 October, 2021;
originally announced October 2021.
-
Walking on TacTip toes: A tactile sensing foot for walking robots
Authors:
Elizabeth A. Stone,
Nathan F. Lepora,
David A. W. Barton
Abstract:
Little research into tactile feet has been done for walking robots despite the benefits such feedback could give when walking on uneven terrain. This paper describes the development of a simple, robust and inexpensive tactile foot for legged robots based on a high-resolution biomimetic TacTip tactile sensor. Several design improvements were made to facilitate tactile sensing while walking, includi…
▽ More
Little research into tactile feet has been done for walking robots despite the benefits such feedback could give when walking on uneven terrain. This paper describes the development of a simple, robust and inexpensive tactile foot for legged robots based on a high-resolution biomimetic TacTip tactile sensor. Several design improvements were made to facilitate tactile sensing while walking, including the use of phosphorescent markers to remove the need for internal LED lighting. The usefulness of the foot is verified on a quadrupedal robot performing a beam walking task and it is found the sensor prevents the robot falling off the beam. Further, this capability also enables the robot to walk along the edge of a curved table. This tactile foot design can be easily modified for use with any legged robot, including much larger walking robots, enabling stable walking in challenging terrain.
△ Less
Submitted 12 August, 2020;
originally announced August 2020.
-
Learning to Live Life on the Edge: Online Learning for Data-Efficient Tactile Contour Following
Authors:
Elizabeth A. Stone,
Nathan F. Lepora,
David A. W. Barton
Abstract:
Tactile sensing has been used for a variety of robotic exploration and manipulation tasks but a common constraint is a requirement for a large amount of training data. This paper addresses the issue of data-efficiency by proposing a novel method for online learning based on a Gaussian Process Latent Variable Model (GP-LVM), whereby the robot learns from tactile data whilst performing a contour fol…
▽ More
Tactile sensing has been used for a variety of robotic exploration and manipulation tasks but a common constraint is a requirement for a large amount of training data. This paper addresses the issue of data-efficiency by proposing a novel method for online learning based on a Gaussian Process Latent Variable Model (GP-LVM), whereby the robot learns from tactile data whilst performing a contour following task thus enabling generalisation to a wide variety of tactile stimuli. The results show that contour following is successful with comparatively little data and is robust to novel stimuli. This work highlights that even with a simple learning architecture there are significant advantages to be gained in efficient and robust task performance by using latent variable models and online learning for tactile sensing tasks. This paves the way for a new generation of robust, fast, and data-efficient tactile systems.
△ Less
Submitted 12 August, 2020; v1 submitted 12 September, 2019;
originally announced September 2019.
-
Shear-invariant Sliding Contact Perception with a Soft Tactile Sensor
Authors:
Kirsty Aquilina,
David A. W. Barton,
Nathan F. Lepora
Abstract:
Manipulation tasks often require robots to be continuously in contact with an object. Therefore tactile perception systems need to handle continuous contact data. Shear deformation causes the tactile sensor to output path-dependent readings in contrast to discrete contact readings. As such, in some continuous-contact tasks, sliding can be regarded as a disturbance over the sensor signal. Here we p…
▽ More
Manipulation tasks often require robots to be continuously in contact with an object. Therefore tactile perception systems need to handle continuous contact data. Shear deformation causes the tactile sensor to output path-dependent readings in contrast to discrete contact readings. As such, in some continuous-contact tasks, sliding can be regarded as a disturbance over the sensor signal. Here we present a shear-invariant perception method based on principal component analysis (PCA) which outputs the required information about the environment despite sliding motion. A compliant tactile sensor (the TacTip) is used to investigate continuous tactile contact. First, we evaluate the method offline using test data collected whilst the sensor slides over an edge. Then, the method is used within a contour-following task applied to 6 objects with varying curvatures; all contours are successfully traced. The method demonstrates generalisation capabilities and could underlie a more sophisticated controller for challenging manipulation or exploration tasks in unstructured environments. A video showing the work described in the paper can be found at https://youtu.be/wrTM61-pieU
△ Less
Submitted 2 May, 2019;
originally announced May 2019.
-
Principal Components of Touch
Authors:
Kirsty Aquilina,
David A. W. Barton,
Nathan F. Lepora
Abstract:
Our human sense of touch enables us to manipulate our surroundings; therefore, complex robotic manipulation will require artificial tactile sensing. Typically tactile sensor arrays are used in robotics, implying that a straightforward way of interpreting multidimensional data is required. In this paper we present a simple visualisation approach based on applying principal component analysis (PCA)…
▽ More
Our human sense of touch enables us to manipulate our surroundings; therefore, complex robotic manipulation will require artificial tactile sensing. Typically tactile sensor arrays are used in robotics, implying that a straightforward way of interpreting multidimensional data is required. In this paper we present a simple visualisation approach based on applying principal component analysis (PCA) to systematically collected sets of tactile data. We apply the visualisation approach to 4 different types of tactile sensor, encompassing fingertips and vibrissal arrays. The results show that PCA can reveal structure and regularities in the tactile data, which also permits the use of simple classifiers such as $k$-NN to achieve good inference. Additionally, the Euclidean distance in principal component space gives a measure of sensitivity, which can aid visualisation and also be used to find regions in the tactile input space where the sensor is able to perceive with higher accuracy. We expect that these observations will generalise, and thus offer the potential for novel control methods based on touch.
△ Less
Submitted 8 June, 2018; v1 submitted 21 May, 2018;
originally announced May 2018.
