Showing 1–50 of 62 results for author: Hermans, T

DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Gras** with Geometric Fabrics

Authors: Tyler Ga Wei Lum, Martin Matak, Viktor Makoviychuk, Ankur Handa, Arthur Allshire, Tucker Hermans, Nathan D. Ratliff, Karl Van Wyk

Abstract: A pivotal challenge in robotics is achieving fast, safe, and robust dexterous gras** across a diverse range of objects, an important goal within industrial applications. However, existing methods often have very limited speed, dexterity, and generality, along with limited or no hardware safety guarantees. In this work, we introduce DextrAH-G, a depth-based dexterous gras** policy trained entir… ▽ More A pivotal challenge in robotics is achieving fast, safe, and robust dexterous gras** across a diverse range of objects, an important goal within industrial applications. However, existing methods often have very limited speed, dexterity, and generality, along with limited or no hardware safety guarantees. In this work, we introduce DextrAH-G, a depth-based dexterous gras** policy trained entirely in simulation that combines reinforcement learning, geometric fabrics, and teacher-student distillation. We address key challenges in joint arm-hand policy learning, such as high-dimensional observation and action spaces, the sim2real gap, collision avoidance, and hardware constraints. DextrAH-G enables a 23 motor arm-hand robot to safely and continuously grasp and transport a large variety of objects at high speed using multi-modal inputs including depth images, allowing generalization across object geometry. Videos at https://sites.google.com/view/dextrah-g. △ Less

Submitted 2 July, 2024; originally announced July 2024.

Composable Part-Based Manipulation

Authors: Weiyu Liu, Jiayuan Mao, Joy Hsu, Tucker Hermans, Animesh Garg, Jiajun Wu

Abstract: In this paper, we propose composable part-based manipulation (CPM), a novel approach that leverages object-part decomposition and part-part correspondences to improve learning and generalization of robotic manipulation skills. By considering the functional correspondences between object parts, we conceptualize functional actions, such as pouring and constrained placing, as combinations of differen… ▽ More In this paper, we propose composable part-based manipulation (CPM), a novel approach that leverages object-part decomposition and part-part correspondences to improve learning and generalization of robotic manipulation skills. By considering the functional correspondences between object parts, we conceptualize functional actions, such as pouring and constrained placing, as combinations of different correspondence constraints. CPM comprises a collection of composable diffusion models, where each model captures a different inter-object correspondence. These diffusion models can generate parameters for manipulation skills based on the specific object parts. Leveraging part-based correspondences coupled with the task decomposition into distinct constraints enables strong generalization to novel objects and object categories. We validate our approach in both simulated and real-world scenarios, demonstrating its effectiveness in achieving robust and generalized manipulation capabilities. △ Less

Submitted 9 May, 2024; originally announced May 2024.

Comments: Presented at CoRL 2023. For videos and additional results, see our website: https://cpmcorl2023.github.io/

Point Cloud Models Improve Visual Robustness in Robotic Learners

Authors: Skand Peri, Iain Lee, Chanho Kim, Li Fuxin, Tucker Hermans, Stefan Lee

Abstract: Visual control policies can encounter significant performance degradation when visual conditions like lighting or camera position differ from those seen during training -- often exhibiting sharp declines in capability even for minor differences. In this work, we examine robustness to a suite of these types of visual changes for RGB-D and point cloud based visual control policies. To perform these… ▽ More Visual control policies can encounter significant performance degradation when visual conditions like lighting or camera position differ from those seen during training -- often exhibiting sharp declines in capability even for minor differences. In this work, we examine robustness to a suite of these types of visual changes for RGB-D and point cloud based visual control policies. To perform these experiments on both model-free and model-based reinforcement learners, we introduce a novel Point Cloud World Model (PCWM) and point cloud based control policies. Our experiments show that policies that explicitly encode point clouds are significantly more robust than their RGB-D counterparts. Further, we find our proposed PCWM significantly outperforms prior works in terms of sample efficiency during training. Taken together, these results suggest reasoning about the 3D scene through point clouds can improve performance, reduce learning time, and increase robustness for robotic learners. Project Webpage: https://pvskand.github.io/projects/PCWM △ Less

Submitted 29 April, 2024; originally announced April 2024.

Comments: Accepted at International Conference on Robotics and Automation, 2024

V-PRISM: Probabilistic Map** of Unknown Tabletop Scenes

Authors: Herbert Wright, Weiming Zhi, Matthew Johnson-Roberson, Tucker Hermans

Abstract: The ability to construct concise scene representations from sensor input is central to the field of robotics. This paper addresses the problem of robustly creating a 3D representation of a tabletop scene from a segmented RGB-D image. These representations are then critical for a range of downstream manipulation tasks. Many previous attempts to tackle this problem do not capture accurate uncertaint… ▽ More The ability to construct concise scene representations from sensor input is central to the field of robotics. This paper addresses the problem of robustly creating a 3D representation of a tabletop scene from a segmented RGB-D image. These representations are then critical for a range of downstream manipulation tasks. Many previous attempts to tackle this problem do not capture accurate uncertainty, which is required to subsequently produce safe motion plans. In this paper, we cast the representation of 3D tabletop scenes as a multi-class classification problem. To tackle this, we introduce V-PRISM, a framework and method for robustly creating probabilistic 3D segmentation maps of tabletop scenes. Our maps contain both occupancy estimates, segmentation information, and principled uncertainty measures. We evaluate the robustness of our method in (1) procedurally generated scenes using open-source object datasets, and (2) real-world tabletop data collected from a depth camera. Our experiments show that our approach outperforms alternative continuous reconstruction approaches that do not explicitly reason about objects in a multi-class formulation. △ Less

Submitted 13 March, 2024; v1 submitted 12 March, 2024; originally announced March 2024.

Scaling and flow profiles in magnetically confined liquid-in-liquid channels

Authors: Arvind Arun Dev, Florencia Sacarelli, G Bagheri, Aleena Joseph, Anna Oleshkevych, E Bodenschatz, Peter Dunne, Thomas Hermans, Bernard Doudin

Abstract: Ferrofluids kept in place by permanent magnet quadrupoles can act as liquid walls to surround a second non-magnetic inside, resulting in a liquid fluidic channel with diameter size ranging from mm down to less than 10 micrometer. Micro particle tracking velocimetry (micro PTV) experiments and modeling show that near ideal plug flow is possible in such liquid-in-liquid channels due to the reduced f… ▽ More Ferrofluids kept in place by permanent magnet quadrupoles can act as liquid walls to surround a second non-magnetic inside, resulting in a liquid fluidic channel with diameter size ranging from mm down to less than 10 micrometer. Micro particle tracking velocimetry (micro PTV) experiments and modeling show that near ideal plug flow is possible in such liquid-in-liquid channels due to the reduced friction at the walls. The measured fluids velocity profiles agree with the predictions of a hydrodynamic model of cylindrical symmetry with a minimal set of hypotheses. By introducing symmetry breaking elements in the system, we show how unique velocity and flow properties can be obtained. Our liquid-in-liquid confinement opens new possibilities for < 10 micrometer-sized microfluidics with low pressures and low shear, with flow characteristics not attainable in comparable solid-wall devices. △ Less

Submitted 26 February, 2024; originally announced February 2024.

Comments: 12 pages 7 figures

Pick and Place Planning is Better than Pick Planning then Place Planning

Authors: Mohanraj Devendran Shanthi, Tucker Hermans

Abstract: Robotic pick and place stands at the heart of autonomous manipulation. When conducted in cluttered or complex environments robots must jointly reason about the selected grasp and desired placement locations to ensure success. While several works have examined this joint pick-and-place problem, none have fully leveraged recent learning-based approaches for multi-fingered grasp planning. We present… ▽ More Robotic pick and place stands at the heart of autonomous manipulation. When conducted in cluttered or complex environments robots must jointly reason about the selected grasp and desired placement locations to ensure success. While several works have examined this joint pick-and-place problem, none have fully leveraged recent learning-based approaches for multi-fingered grasp planning. We present a modular algorithm for joint pick and place planning that can make use of state of the art grasp classifiers for planning multi-fingered grasps for novel objects from partial view point clouds. We demonstrate our joint pick and place formulation with several costs associated with different placement tasks. Experiments on pick and place tasks with cluttered scenes using a physical robot show that our joint inference method is more successful than a sequential pick then place approach, while also achieving better placement configurations. △ Less

Submitted 29 January, 2024; originally announced January 2024.

Comments: 8 pages, 14 figures, IEEE RA-L

Journal ref: IEEE RA-L, Volume 9 Issue 3, 2024, 2790 - 2797

Multidimensional surrogate modelling for Airborne TDEM data

Authors: Wouter Deleersnyder, David Dudal, Thomas Hermans

Abstract: The computational resources required to solve the full 3D inversion of time-domain electromagnetic data are immense. To overcome the time-consuming 3D simulations, we construct a surrogate model, more precisely, a data-driven statistical model. It is trained on 3D simulation data and predicts the approximate output much faster. Given the computational cost related to the simulations, there are lim… ▽ More The computational resources required to solve the full 3D inversion of time-domain electromagnetic data are immense. To overcome the time-consuming 3D simulations, we construct a surrogate model, more precisely, a data-driven statistical model. It is trained on 3D simulation data and predicts the approximate output much faster. Given the computational cost related to the simulations, there are limitations in the number of training samples that can be generated. In addition, certain applications require a wide range of parameters to be sampled, such as the electrical conductivity parameters in a saltwater intrusion case. This chapter is therefore limited to a two-layer model. We construct a surrogate model that predicts the discrepancy between a 1D two-layered subsurface model and a deviation of the 1D assumption. The latter response is quickly computed with a semi-analytical 1D forward model. The results are encouraging even with few training samples, but obtaining a high accuracy is difficult with relatively simple data fit models. We propose to view the performance in terms of learning gain, representing the gain from the surrogate model whilst still acknowledging a residual discrepancy. △ Less

Submitted 23 November, 2023; originally announced November 2023.

Suppressing Rayleigh-Plateau Instability with a Magnetic Force Field for Deformable Interfaces Engineering

Authors: Arvind Arun Dev, Thomas Hermans, Bernard Doudin

Abstract: The Rayleigh-Plateau instability (RPI) is a classical hydrodynamics phenomenon that prevents a jet of liquid to flow indefinitely within air or another liquid. Here, we show how adding a magnetic force field makes possible its suppression. Enclosing the jet in a ferrofluid held by magnetic forces allows flow focusing without sheath flow, which completely avoids drip** failure at small flow rates… ▽ More The Rayleigh-Plateau instability (RPI) is a classical hydrodynamics phenomenon that prevents a jet of liquid to flow indefinitely within air or another liquid. Here, we show how adding a magnetic force field makes possible its suppression. Enclosing the jet in a ferrofluid held by magnetic forces allows flow focusing without sheath flow, which completely avoids drip** failure at small flow rates and provides conditional stability for a continuous fluid jet. Highly deformable liquid interfaces withstanding spatial and time varying flow conditions within a large parameter space can be realized. △ Less

Submitted 22 October, 2023; originally announced October 2023.

Comments: 14 pages , 4 figures

Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Models

Authors: Yixuan Huang, Jialin Yuan, Chanho Kim, Pupul Pradhan, Bryan Chen, Li Fuxin, Tucker Hermans

Abstract: Robots need to have a memory of previously observed, but currently occluded objects to work reliably in realistic environments. We investigate the problem of encoding object-oriented memory into a multi-object manipulation reasoning and planning framework. We propose DOOM and LOOM, which leverage transformer relational dynamics to encode the history of trajectories given partial-view point clouds… ▽ More Robots need to have a memory of previously observed, but currently occluded objects to work reliably in realistic environments. We investigate the problem of encoding object-oriented memory into a multi-object manipulation reasoning and planning framework. We propose DOOM and LOOM, which leverage transformer relational dynamics to encode the history of trajectories given partial-view point clouds and an object discovery and tracking engine. Our approaches can perform multiple challenging tasks including reasoning with occluded objects, novel objects appearance, and object reappearance. Throughout our extensive simulation and real-world experiments, we find that our approaches perform well in terms of different numbers of objects and different numbers of distractor actions. Furthermore, we show our approaches outperform an implicit memory baseline. △ Less

Submitted 24 May, 2024; v1 submitted 26 September, 2023; originally announced September 2023.

Comments: Presented at IEEE Conference on Robotics and Automation (ICRA) 2024. Website: https://sites.google.com/view/rdmemory

DefGoalNet: Contextual Goal Learning from Demonstrations For Deformable Object Manipulation

Authors: Bao Thach, Tanner Watts, Shing-Hei Ho, Tucker Hermans, Alan Kuntz

Abstract: Shape servoing, a robotic task dedicated to controlling objects to desired goal shapes, is a promising approach to deformable object manipulation. An issue arises, however, with the reliance on the specification of a goal shape. This goal has been obtained either by a laborious domain knowledge engineering process or by manually manipulating the object into the desired shape and capturing the goal… ▽ More Shape servoing, a robotic task dedicated to controlling objects to desired goal shapes, is a promising approach to deformable object manipulation. An issue arises, however, with the reliance on the specification of a goal shape. This goal has been obtained either by a laborious domain knowledge engineering process or by manually manipulating the object into the desired shape and capturing the goal shape at that specific moment, both of which are impractical in various robotic applications. In this paper, we solve this problem by develo** a novel neural network DefGoalNet, which learns deformable object goal shapes directly from a small number of human demonstrations. We demonstrate our method's effectiveness on various robotic tasks, both in simulation and on a physical robot. Notably, in the surgical retraction task, even when trained with as few as 10 demonstrations, our method achieves a median success percentage of nearly 90%. These results mark a substantial advancement in enabling shape servoing methods to bring deformable object manipulation closer to practical, real-world applications. △ Less

Submitted 25 September, 2023; originally announced September 2023.

Comments: Submitted to IEEE Conference on Robotics and Automation (ICRA) 2024. 8 pages, 11 figures

Latent Space Planning for Multi-Object Manipulation with Environment-Aware Relational Classifiers

Authors: Yixuan Huang, Nichols Crawford Taylor, Adam Conkey, Weiyu Liu, Tucker Hermans

Abstract: Objects rarely sit in isolation in everyday human environments. If we want robots to operate and perform tasks in our human environments, they must understand how the objects they manipulate will interact with structural elements of the environment for all but the simplest of tasks. As such, we'd like our robots to reason about how multiple objects and environmental elements relate to one another… ▽ More Objects rarely sit in isolation in everyday human environments. If we want robots to operate and perform tasks in our human environments, they must understand how the objects they manipulate will interact with structural elements of the environment for all but the simplest of tasks. As such, we'd like our robots to reason about how multiple objects and environmental elements relate to one another and how those relations may change as the robot interacts with the world. We examine the problem of predicting inter-object and object-environment relations between previously unseen objects and novel environments purely from partial-view point clouds. Our approach enables robots to plan and execute sequences to complete multi-object manipulation tasks defined from logical relations. This removes the burden of providing explicit, continuous object states as goals to the robot. We explore several different neural network architectures for this task. We find the best performing model to be a novel transformer-based neural network that both predicts object-environment relations and learns a latent-space dynamics function. We achieve reliable sim-to-real transfer without any fine-tuning. Our experiments show that our model understands how changes in observed environmental geometry relate to semantic relations between objects. We show more videos on our website: https://sites.google.com/view/erelationaldynamics. △ Less

Submitted 28 January, 2024; v1 submitted 18 May, 2023; originally announced May 2023.

Comments: Accepted at IEEE Transactions on Robotics (T-RO). arXiv admin note: text overlap with arXiv:2209.11943

DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects

Authors: Bao Thach, Brian Y. Cho, Shing-Hei Ho, Tucker Hermans, Alan Kuntz

Abstract: Applications in fields ranging from home care to warehouse fulfillment to surgical assistance require robots to reliably manipulate the shape of 3D deformable objects. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely o… ▽ More Applications in fields ranging from home care to warehouse fulfillment to surgical assistance require robots to reliably manipulate the shape of 3D deformable objects. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the manipulated object and a point cloud of the goal shape to learn a low-dimensional representation of the object shape. This shape embedding enables the robot to learn a visual servo controller that computes the desired robot end-effector action to iteratively deform the object toward the target shape. We demonstrate both in simulation and on a physical robot that DeformerNet reliably generalizes to object shapes and material stiffness not seen during training, including ex vivo chicken muscle tissue. Crucially, using DeformerNet, the robot successfully accomplishes three surgical sub-tasks: retraction (moving tissue aside to access a site underneath it), tissue wrap** (a sub-task in procedures like aortic stent placements), and connecting two tubular pieces of tissue (a sub-task in anastomosis). △ Less

Submitted 19 February, 2024; v1 submitted 8 May, 2023; originally announced May 2023.

Comments: Submitted to IEEE Transactions on Robotics (T-RO). 20 pages, 27 figures. arXiv admin note: text overlap with arXiv:2110.04685

DefGraspNets: Grasp Planning on 3D Fields with Graph Neural Nets

Authors: Isabella Huang, Yashraj Narang, Ruzena Bajcsy, Fabio Ramos, Tucker Hermans, Dieter Fox

Abstract: Robotic gras** of 3D deformable objects is critical for real-world applications such as food handling and robotic surgery. Unlike rigid and articulated objects, 3D deformable objects have infinite degrees of freedom. Fully defining their state requires 3D deformation and stress fields, which are exceptionally difficult to analytically compute or experimentally measure. Thus, evaluating grasp can… ▽ More Robotic gras** of 3D deformable objects is critical for real-world applications such as food handling and robotic surgery. Unlike rigid and articulated objects, 3D deformable objects have infinite degrees of freedom. Fully defining their state requires 3D deformation and stress fields, which are exceptionally difficult to analytically compute or experimentally measure. Thus, evaluating grasp candidates for grasp planning typically requires accurate, but slow 3D finite element method (FEM) simulation. Sampling-based grasp planning is often impractical, as it requires evaluation of a large number of grasp candidates. Gradient-based grasp planning can be more efficient, but requires a differentiable model to synthesize optimal grasps from initial candidates. Differentiable FEM simulators may fill this role, but are typically no faster than standard FEM. In this work, we propose learning a predictive graph neural network (GNN), DefGraspNets, to act as our differentiable model. We train DefGraspNets to predict 3D stress and deformation fields based on FEM-based grasp simulations. DefGraspNets not only runs up to 1500 times faster than the FEM simulator, but also enables fast gradient-based grasp optimization over 3D stress and deformation metrics. We design DefGraspNets to align with real-world grasp planning practices and demonstrate generalization across multiple test sets, including real-world experiments. △ Less

Submitted 28 March, 2023; originally announced March 2023.

Comments: To be published in the IEEE Conference on Robotics and Automation (ICRA), 2023

Planning Visual-Tactile Precision Grasps via Complementary Use of Vision and Touch

Authors: Martin Matak, Tucker Hermans

Abstract: Reliably planning fingertip grasps for multi-fingered hands lies as a key challenge for many tasks including tool use, insertion, and dexterous in-hand manipulation. This task becomes even more difficult when the robot lacks an accurate model of the object to be grasped. Tactile sensing offers a promising approach to account for uncertainties in object shape. However, current robotic hands tend to… ▽ More Reliably planning fingertip grasps for multi-fingered hands lies as a key challenge for many tasks including tool use, insertion, and dexterous in-hand manipulation. This task becomes even more difficult when the robot lacks an accurate model of the object to be grasped. Tactile sensing offers a promising approach to account for uncertainties in object shape. However, current robotic hands tend to lack full tactile coverage. As such, a problem arises of how to plan and execute grasps for multi-fingered hands such that contact is made with the area covered by the tactile sensors. To address this issue, we propose an approach to grasp planning that explicitly reasons about where the fingertips should contact the estimated object surface while maximizing the probability of grasp success. Key to our method's success is the use of visual surface estimation for initial planning to encode the contact constraint. The robot then executes this plan using a tactile-feedback controller that enables the robot to adapt to online estimates of the object's surface to correct for errors in the initial plan. Importantly, the robot never explicitly integrates object pose or surface estimates between visual and tactile sensing, instead it uses the two modalities in complementary ways. Vision guides the robots motion prior to contact; touch updates the plan when contact occurs differently than predicted from vision. We show that our method successfully synthesises and executes precision grasps for previously unseen objects using surface estimates from a single camera view. Further, our approach outperforms a state of the art multi-fingered grasp planner, while also beating several baselines we propose. △ Less

Submitted 16 December, 2022; originally announced December 2022.

StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects

Authors: Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, Chris Paxton

Abstract: Robots operating in human environments must be able to rearrange objects into semantically-meaningful configurations, even if these objects are previously unseen. In this work, we focus on the problem of building physically-valid structures without step-by-step instructions. We propose StructDiffusion, which combines a diffusion model and an object-centric transformer to construct structures given… ▽ More Robots operating in human environments must be able to rearrange objects into semantically-meaningful configurations, even if these objects are previously unseen. In this work, we focus on the problem of building physically-valid structures without step-by-step instructions. We propose StructDiffusion, which combines a diffusion model and an object-centric transformer to construct structures given partial-view point clouds and high-level language goals, such as "set the table". Our method can perform multiple challenging language-conditioned multi-step 3D planning tasks using one model. StructDiffusion even improves the success rate of assembling physically-valid structures out of unseen objects by on average 16% over an existing multi-modal transformer model trained on specific structures. We show experiments on held-out objects in both simulation and on real-world rearrangement tasks. Importantly, we show how integrating both a diffusion model and a collision-discriminator model allows for improved generalization over other methods when rearranging previously-unseen objects. For videos and additional results, see our website: https://structdiffusion.github.io/. △ Less

Submitted 25 April, 2023; v1 submitted 8 November, 2022; originally announced November 2022.

Comments: Accepted to Robotics: Science and Systems (RSS) 2023. The previous version appeared in CoRL Workshop on Language and Robot Learning 2022

Novel Airborne EM Image Appraisal Tool for Imperfect Forward Modelling

Authors: Wouter Deleersnyder, David Dudal, Thomas Hermans

Abstract: Full 3D inversion of time-domain Airborne ElectroMagnetic (AEM) data requires specialists' expertise and a tremendous amount of computational resources, not readily available to everyone. Consequently, quasi-2D/3D inversion methods are prevailing, using a much faster but approximate (1D) forward model. We propose an appraisal tool that indicates zones in the inversion model that are not in agreeme… ▽ More Full 3D inversion of time-domain Airborne ElectroMagnetic (AEM) data requires specialists' expertise and a tremendous amount of computational resources, not readily available to everyone. Consequently, quasi-2D/3D inversion methods are prevailing, using a much faster but approximate (1D) forward model. We propose an appraisal tool that indicates zones in the inversion model that are not in agreement with the multidimensional data and therefore, should not be interpreted quantitatively. The image appraisal relies on multidimensional forward modelling to compute a so-called normalized gradient. Large values in that gradient indicate model parameters that do not fit the true multidimensionality of the observed data well and should not be interpreted quantitatively. An alternative approach is proposed to account for imperfect forward modelling, such that the appraisal tool is computationally inexpensive. The method is demonstrated on an AEM survey in a salinization context, revealing possible problematic zones in the estimated fresh-saltwater interface. △ Less

Submitted 12 October, 2022; originally announced October 2022.

Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers

Authors: Yixuan Huang, Adam Conkey, Tucker Hermans

Abstract: Objects rarely sit in isolation in human environments. As such, we'd like our robots to reason about how multiple objects relate to one another and how those relations may change as the robot interacts with the world. To this end, we propose a novel graph neural network framework for multi-object manipulation to predict how inter-object relations change given robot actions. Our model operates on p… ▽ More Objects rarely sit in isolation in human environments. As such, we'd like our robots to reason about how multiple objects relate to one another and how those relations may change as the robot interacts with the world. To this end, we propose a novel graph neural network framework for multi-object manipulation to predict how inter-object relations change given robot actions. Our model operates on partial-view point clouds and can reason about multiple objects dynamically interacting during the manipulation. By learning a dynamics model in a learned latent graph embedding space, our model enables multi-step planning to reach target goal relations. We show our model trained purely in simulation transfers well to the real world. Our planner enables the robot to rearrange a variable number of objects with a range of shapes and sizes using both push and pick and place skills. △ Less

Submitted 16 March, 2023; v1 submitted 24 September, 2022; originally announced September 2022.

Comments: 10 pages, 7 figures, to be published in the proceedings of the IEEE Conference on Robotics and Automation (ICRA) 2023. Robot Demos: https://robot-learning.cs.utah.edu/project/graph_nets

Occlusion-Robust Multi-Sensory Posture Estimation in Physical Human-Robot Interaction

Authors: Amir Yazdani, Roya Sabbagh Novin, Andrew Merryweather, Tucker Hermans

Abstract: 3D posture estimation is important in analyzing and improving ergonomics in physical human-robot interaction and reducing the risk of musculoskeletal disorders. Vision-based posture estimation approaches are prone to sensor and model errors, as well as occlusion, while posture estimation solely from the interacting robot's trajectory suffers from ambiguous solutions. To benefit from the advantages… ▽ More 3D posture estimation is important in analyzing and improving ergonomics in physical human-robot interaction and reducing the risk of musculoskeletal disorders. Vision-based posture estimation approaches are prone to sensor and model errors, as well as occlusion, while posture estimation solely from the interacting robot's trajectory suffers from ambiguous solutions. To benefit from the advantages of both approaches and improve upon their drawbacks, we introduce a low-cost, non-intrusive, and occlusion-robust multi-sensory 3D postural estimation algorithm in physical human-robot interaction. We use 2D postures from OpenPose over a single camera, and the trajectory of the interacting robot while the human performs a task. We model the problem as a partially-observable dynamical system and we infer the 3D posture via a particle filter. We present our work in teleoperation, but it can be generalized to other applications of physical human-robot interaction. We show that our multi-sensory system resolves human kinematic redundancy better than posture estimation solely using OpenPose or posture estimation solely using the robot's trajectory. This will increase the accuracy of estimated postures compared to the gold-standard motion capture postures. Moreover, our approach also performs better than other single sensory methods when postural assessment using RULA assessment tool. △ Less

Submitted 12 August, 2022; originally announced August 2022.

Comments: Submitted to ACM Transaction on Human-Robot ItneractionL: Special Issue on AI-HRI

Flexible quasi-2D inversion of time-domain AEM data, using a wavelet-based complexity measure

Authors: Wouter Deleersnyder, Benjamin Maveau, David Dudal, Thomas Hermans

Abstract: Regularization methods improve the stability of ill-posed inverse problems by introducing some a priori characteristics for the solution such as smoothness or sharpness. In this contribution, we propose a multidimensional, scale-dependent wavelet-based L1-regularization term to cure the ill-posedness of the airborne (time-domain) electromagnetic induction inverse problem. The regularization term i… ▽ More Regularization methods improve the stability of ill-posed inverse problems by introducing some a priori characteristics for the solution such as smoothness or sharpness. In this contribution, we propose a multidimensional, scale-dependent wavelet-based L1-regularization term to cure the ill-posedness of the airborne (time-domain) electromagnetic induction inverse problem. The regularization term is flexible, as it can recover blocky, smooth and tunable in-between inversion models, based on a suitable wavelet basis function. For each orientation, a different wavelet basis function can be used, introducing an additional relative regularization parameter. We propose a calibration method to determine (an educated initial guess for) this relative regularization parameter, which reduces the need to optimize for this parameter, and, consequently, the overall computation time is under control. We apply our novel scheme to a time-domain airborne electromagnetic data set in Belgian saltwater intrusion context, but the scheme could equally apply to any other 2D or 3D geophysical inverse problem. △ Less

Submitted 25 January, 2023; v1 submitted 13 May, 2022; originally announced May 2022.

DULA and DEBA: Differentiable Ergonomic Risk Models for Postural Assessment and Optimization in Ergonomically Intelligent pHRI

Authors: Amir Yazdani, Roya Sabbagh Novin, Andrew Merryweather, Tucker Hermans

Abstract: Ergonomics and human comfort are essential concerns in physical human-robot interaction applications. Defining an accurate and easy-to-use ergonomic assessment model stands as an important step in providing feedback for postural correction to improve operator health and comfort. Common practical methods in the area suffer from inaccurate ergonomics models in performing postural optimization. In or… ▽ More Ergonomics and human comfort are essential concerns in physical human-robot interaction applications. Defining an accurate and easy-to-use ergonomic assessment model stands as an important step in providing feedback for postural correction to improve operator health and comfort. Common practical methods in the area suffer from inaccurate ergonomics models in performing postural optimization. In order to retain assessment quality, while improving computational considerations, we propose a novel framework for postural assessment and optimization for ergonomically intelligent physical human-robot interaction. We introduce DULA and DEBA, differentiable and continuous ergonomics models learned to replicate the popular and scientifically validated RULA and REBA assessments with more than 99% accuracy. We show that DULA and DEBA provide assessment comparable to RULA and REBA while providing computational benefits when being used in postural optimization. We evaluate our framework through human and simulation experiments. We highlight DULA and DEBA's strength in a demonstration of postural optimization for a simulated pHRI task. △ Less

Submitted 6 May, 2022; originally announced May 2022.

Comments: Submitted to IROS 2022. arXiv admin note: substantial text overlap with arXiv:2108.05971

Correcting Robot Plans with Natural Language Feedback

Authors: Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, Dieter Fox

Abstract: When humans design cost or goal specifications for robots, they often produce specifications that are ambiguous, underspecified, or beyond planners' ability to solve. In these cases, corrections provide a valuable tool for human-in-the-loop robot control. Corrections might take the form of new goal specifications, new constraints (e.g. to avoid specific objects), or hints for planning algorithms (… ▽ More When humans design cost or goal specifications for robots, they often produce specifications that are ambiguous, underspecified, or beyond planners' ability to solve. In these cases, corrections provide a valuable tool for human-in-the-loop robot control. Corrections might take the form of new goal specifications, new constraints (e.g. to avoid specific objects), or hints for planning algorithms (e.g. to visit specific waypoints). Existing correction methods (e.g. using a joystick or direct manipulation of an end effector) require full teleoperation or real-time interaction. In this paper, we explore natural language as an expressive and flexible tool for robot correction. We describe how to map from natural language sentences to transformations of cost functions. We show that these transformations enable users to correct goals, update robot motions to accommodate additional user preferences, and recover from planning errors. These corrections can be leveraged to get 81% and 93% success rates on tasks where the original planner failed, with either one or two language corrections. Our method makes it possible to compose multiple constraints and generalizes to unseen scenes, objects, and sentences in simulated environments and real-world environments. △ Less

Submitted 11 April, 2022; originally announced April 2022.

Comments: 10 pages, 13 figures

DefGraspSim: Physics-based simulation of grasp outcomes for 3D deformable objects

Authors: Isabella Huang, Yashraj Narang, Clemens Eppner, Balakumar Sundaralingam, Miles Macklin, Ruzena Bajcsy, Tucker Hermans, Dieter Fox

Abstract: Robotic gras** of 3D deformable objects (e.g., fruits/vegetables, internal organs, bottles/boxes) is critical for real-world applications such as food processing, robotic surgery, and household automation. However, develo** grasp strategies for such objects is uniquely challenging. Unlike rigid objects, deformable objects have infinite degrees of freedom and require field quantities (e.g., def… ▽ More Robotic gras** of 3D deformable objects (e.g., fruits/vegetables, internal organs, bottles/boxes) is critical for real-world applications such as food processing, robotic surgery, and household automation. However, develo** grasp strategies for such objects is uniquely challenging. Unlike rigid objects, deformable objects have infinite degrees of freedom and require field quantities (e.g., deformation, stress) to fully define their state. As these quantities are not easily accessible in the real world, we propose studying interaction with deformable objects through physics-based simulation. As such, we simulate grasps on a wide range of 3D deformable objects using a GPU-based implementation of the corotational finite element method (FEM). To facilitate future research, we open-source our simulated dataset (34 objects, 1e5 Pa elasticity range, 6800 grasp evaluations, 1.1M grasp measurements), as well as a code repository that allows researchers to run our full FEM-based grasp evaluation pipeline on arbitrary 3D object models of their choice. Finally, we demonstrate good correspondence between grasp outcomes on simulated objects and their real counterparts. △ Less

Submitted 21 March, 2022; originally announced March 2022.

Comments: For associated web page, see \url{https://sites.google.com/nvidia.com/defgraspsim}. To be published in the IEEE Robotics and Automation Letters (RA-L) special issue on Robotic Handling of Deformable Objects, 2022. arXiv admin note: substantial text overlap with arXiv:2107.05778

L4KDE: Learning for KinoDynamic Tree Expansion

Authors: Tin Lai, Weiming Zhi, Tucker Hermans, Fabio Ramos

Abstract: We present the Learning for KinoDynamic Tree Expansion (L4KDE) method for kinodynamic planning. Tree-based planning approaches, such as rapidly exploring random tree (RRT), are the dominant approach to finding globally optimal plans in continuous state-space motion planning. Central to these approaches is tree-expansion, the procedure in which new nodes are added into an ever-expanding tree. We st… ▽ More We present the Learning for KinoDynamic Tree Expansion (L4KDE) method for kinodynamic planning. Tree-based planning approaches, such as rapidly exploring random tree (RRT), are the dominant approach to finding globally optimal plans in continuous state-space motion planning. Central to these approaches is tree-expansion, the procedure in which new nodes are added into an ever-expanding tree. We study the kinodynamic variants of tree-based planning, where we have known system dynamics and kinematic constraints. In the interest of quickly selecting nodes to connect newly sampled coordinates, existing methods typically cannot optimise to find nodes that have low cost to transition to sampled coordinates. Instead, they use metrics like Euclidean distance between coordinates as a heuristic for selecting candidate nodes to connect to the search tree. We propose L4KDE to address this issue. L4KDE uses a neural network to predict transition costs between queried states, which can be efficiently computed in batch, providing much higher quality estimates of transition cost compared to commonly used heuristics while maintaining almost-surely asymptotic optimality guarantee. We empirically demonstrate the significant performance improvement provided by L4KDE on a variety of challenging system dynamics, with the ability to generalise across different instances of the same model class, and in conjunction with a suite of modern tree-based motion planners. △ Less

Submitted 17 September, 2023; v1 submitted 2 March, 2022; originally announced March 2022.

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

Authors: Weiyu Liu, Chris Paxton, Tucker Hermans, Dieter Fox

Abstract: Geometric organization of objects into semantically meaningful arrangements pervades the built world. As such, assistive robots operating in warehouses, offices, and homes would greatly benefit from the ability to recognize and rearrange objects into these semantically meaningful structures. To be useful, these robots must contend with previously unseen objects and receive instructions without sig… ▽ More Geometric organization of objects into semantically meaningful arrangements pervades the built world. As such, assistive robots operating in warehouses, offices, and homes would greatly benefit from the ability to recognize and rearrange objects into these semantically meaningful structures. To be useful, these robots must contend with previously unseen objects and receive instructions without significant programming. While previous works have examined recognizing pairwise semantic relations and sequential manipulation to change these simple relations none have shown the ability to arrange objects into complex structures such as circles or table settings. To address this problem we propose a novel transformer-based neural network, StructFormer, which takes as input a partial-view point cloud of the current object arrangement and a structured language command encoding the desired object configuration. We show through rigorous experiments that StructFormer enables a physical robot to rearrange novel objects into semantically meaningful structures with multi-object relational constraints inferred from the language command. △ Less

Submitted 19 October, 2021; originally announced October 2021.

Toward Learning Context-Dependent Tasks from Demonstration for Tendon-Driven Surgical Robots

Authors: Yixuan Huang, Michael Bentley, Tucker Hermans, Alan Kuntz

Abstract: Tendon-driven robots, a type of continuum robot, have the potential to reduce the invasiveness of surgery by enabling access to difficult-to-reach anatomical targets. In the future, the automation of surgical tasks for these robots may help reduce surgeon strain in the face of a rapidly growing population. However, directly encoding surgical tasks and their associated context for these robots is i… ▽ More Tendon-driven robots, a type of continuum robot, have the potential to reduce the invasiveness of surgery by enabling access to difficult-to-reach anatomical targets. In the future, the automation of surgical tasks for these robots may help reduce surgeon strain in the face of a rapidly growing population. However, directly encoding surgical tasks and their associated context for these robots is infeasible. In this work we take steps toward a system that is able to learn to successfully perform context-dependent surgical tasks by learning directly from a set of expert demonstrations. We present three models trained on the demonstrations conditioned on a vector encoding the context of the demonstration. We then use these models to plan and execute motions for the tendon-driven robot similar to the demonstrations for novel context not seen in the training set. We demonstrate the efficacy of our method on three surgery-inspired tasks. △ Less

Submitted 14 October, 2021; originally announced October 2021.

Comments: 7 pages, 6 figures, to be published in the proceedings of the 2021 International Symposium on Medical Robotics (ISMR)

Planning Sensing Sequences for Subsurface 3D Tumor Map**

Authors: Brian Y. Cho, Tucker Hermans, Alan Kuntz

Abstract: Surgical automation has the potential to enable increased precision and reduce the per-patient workload of overburdened human surgeons. An effective automation system must be able to sense and map subsurface anatomy, such as tumors, efficiently and accurately. In this work, we present a method that plans a sequence of sensing actions to map the 3D geometry of subsurface tumors. We leverage a seque… ▽ More Surgical automation has the potential to enable increased precision and reduce the per-patient workload of overburdened human surgeons. An effective automation system must be able to sense and map subsurface anatomy, such as tumors, efficiently and accurately. In this work, we present a method that plans a sequence of sensing actions to map the 3D geometry of subsurface tumors. We leverage a sequential Bayesian Hilbert map to create a 3D probabilistic occupancy model that represents the likelihood that any given point in the anatomy is occupied by a tumor, conditioned on sensor readings. We iteratively update the map, utilizing Bayesian optimization to determine sensing poses that explore unsensed regions of anatomy and exploit the knowledge gained by previous sensing actions. We demonstrate our method's efficiency and accuracy in three anatomical scenarios including a liver tumor scenario generated from a real patient's CT scan. The results show that our proposed method significantly outperforms comparison methods in terms of efficiency while detecting subsurface tumors with high accuracy. △ Less

Submitted 12 October, 2021; originally announced October 2021.

Comments: 7 pages, 9 figures, to be published in the proceedings of the 2021 International Symposium on Medical Robotics (ISMR)

Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds

Authors: Bao Thach, Brian Y. Cho, Alan Kuntz, Tucker Hermans

Abstract: If robots could reliably manipulate the shape of 3D deformable objects, they could find applications in fields ranging from home care to warehouse fulfillment to surgical assistance. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape… ▽ More If robots could reliably manipulate the shape of 3D deformable objects, they could find applications in fields ranging from home care to warehouse fulfillment to surgical assistance. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the object being manipulated and a point cloud of the goal shape to learn a low-dimensional representation of the object shape. This shape embedding enables the robot to learn to define a visual servo controller that provides Cartesian pose changes to the robot end-effector causing the object to deform towards its target shape. Crucially, we demonstrate both in simulation and on a physical robot that DeformerNet reliably generalizes to object shapes and material stiffness not seen during training and outperforms comparison methods for both the generic shape control and the surgical task of retraction. △ Less

Submitted 18 April, 2022; v1 submitted 9 October, 2021; originally announced October 2021.

Comments: Published in the proceedings of the IEEE Conference on Robotics and Automation (ICRA) 2022. 8 pages, 10 figures

Predicting Stable Configurations for Semantic Placement of Novel Objects

Authors: Chris Paxton, Chris Xie, Tucker Hermans, Dieter Fox

Abstract: Human environments contain numerous objects configured in a variety of arrangements. Our goal is to enable robots to repose previously unseen objects according to learned semantic relationships in novel environments. We break this problem down into two parts: (1) finding physically valid locations for the objects and (2) determining if those poses satisfy learned, high-level semantic relationships… ▽ More Human environments contain numerous objects configured in a variety of arrangements. Our goal is to enable robots to repose previously unseen objects according to learned semantic relationships in novel environments. We break this problem down into two parts: (1) finding physically valid locations for the objects and (2) determining if those poses satisfy learned, high-level semantic relationships. We build our models and training from the ground up to be tightly integrated with our proposed planning algorithm for semantic placement of unknown objects. We train our models purely in simulation, with no fine-tuning needed for use in the real world. Our approach enables motion planning for semantic rearrangement of unknown objects in scenes with varying geometry from only RGB-D sensing. Our experiments through a set of simulated ablations demonstrate that using a relational classifier alone is not sufficient for reliable planning. We further demonstrate the ability of our planner to generate and execute diverse manipulation plans through a set of real-world experiments with a variety of objects. △ Less

Submitted 26 August, 2021; originally announced August 2021.

Parallelised Diffeomorphic Sampling-based Motion Planning

Authors: Tin Lai, Weiming Zhi, Tucker Hermans, Fabio Ramos

Abstract: We propose Parallelised Diffeomorphic Sampling-based Motion Planning (PDMP). PDMP is a novel parallelised framework that uses bijective and differentiable map**s, or diffeomorphisms, to transform sampling distributions of sampling-based motion planners, in a manner akin to normalising flows. Unlike normalising flow models which use invertible neural network structures to represent these diffeomo… ▽ More We propose Parallelised Diffeomorphic Sampling-based Motion Planning (PDMP). PDMP is a novel parallelised framework that uses bijective and differentiable map**s, or diffeomorphisms, to transform sampling distributions of sampling-based motion planners, in a manner akin to normalising flows. Unlike normalising flow models which use invertible neural network structures to represent these diffeomorphisms, we develop them from gradient information of desired costs, and encode desirable behaviour, such as obstacle avoidance. These transformed sampling distributions can then be used for sampling-based motion planning. A particular example is when we wish to imbue the sampling distribution with knowledge of the environment geometry, such that drawn samples are less prone to be in collisions. To this end, we propose to learn a continuous occupancy representation from environment occupancy data, such that gradients of the representation defines a valid diffeomorphism and is amenable to fast parallel evaluation. We use this to "morph" the sampling distribution to draw far fewer collision-prone samples. PDMP is able to leverage gradient information of costs, to inject specifications, in a manner similar to optimisation-based motion planning methods, but relies on drawing from a sampling distribution, retaining the tendency to find more global solutions, thereby bridging the gap between trajectory optimisation and sampling-based planning methods. △ Less

Submitted 22 September, 2021; v1 submitted 26 August, 2021; originally announced August 2021.

Ergonomically Intelligent Physical Human-Robot Interaction: Postural Estimation, Assessment, and Optimization

Authors: Amir Yazdani, Roya Sabbagh Novin, Andrew Merryweather, Tucker Hermans

Abstract: Ergonomics and human comfort are essential concerns in physical human-robot interaction. Common practical methods in the area either fail in estimating the correct posture due to occlusion or suffer from inaccurate ergonomics models in performing postural optimization. We propose a novel alternative framework for posture estimation, assessment, and optimization for ergonomically intelligent physic… ▽ More Ergonomics and human comfort are essential concerns in physical human-robot interaction. Common practical methods in the area either fail in estimating the correct posture due to occlusion or suffer from inaccurate ergonomics models in performing postural optimization. We propose a novel alternative framework for posture estimation, assessment, and optimization for ergonomically intelligent physical human-robot interaction. We show that we can estimate human posture solely from the trajectory of the interacting robot with median deviation of 5 deg from motion capture. We propose DULA, a differentiable ergonomics assessment tool with 99.73% accuracy comparing to RULA. We use DULA in postural optimization for physical human-robot interaction tasks such as co-manipulation and teleoperation. We evaluate our framework through human and simulation experiments. △ Less

Submitted 7 October, 2021; v1 submitted 12 August, 2021; originally announced August 2021.

Comments: Presented at AI-HRI symposium as part of AAAI-FSS 2021 (arXiv:2109.10836)

Report number: AIHRI/2021/31

DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation

Authors: Bao Thach, Alan Kuntz, Tucker Hermans

Abstract: In this paper, we propose a novel approach to 3D deformable object manipulation leveraging a deep neural network called DeformerNet. Controlling the shape of a 3D object requires an effective state representation that can capture the full 3D geometry of the object. Current methods work around this problem by defining a set of feature points on the object or only deforming the object in 2D image sp… ▽ More In this paper, we propose a novel approach to 3D deformable object manipulation leveraging a deep neural network called DeformerNet. Controlling the shape of a 3D object requires an effective state representation that can capture the full 3D geometry of the object. Current methods work around this problem by defining a set of feature points on the object or only deforming the object in 2D image space, which does not truly address the 3D shape control problem. Instead, we explicitly use 3D point clouds as the state representation and apply Convolutional Neural Network on point clouds to learn the 3D features. These features are then mapped to the robot end-effector's position using a fully-connected neural network. Once trained in an end-to-end fashion, DeformerNet directly maps the current point cloud of a deformable object, as well as a target point cloud shape, to the desired displacement in robot gripper position. In addition, we investigate the problem of predicting the manipulation point location given the initial and goal shape of the object. △ Less

Submitted 16 July, 2021; originally announced July 2021.

Comments: Published at RSS 2021 Workshop on Deformable Object Simulation in Robotics; received Honorable Mention for Best Paper Award

DULA: A Differentiable Ergonomics Model for Postural Optimization in Physical HRI

Authors: Amir Yazdani, Roya Sabbagh Novin, Andrew Merryweather, Tucker Hermans

Abstract: Ergonomics and human comfort are essential concerns in physical human-robot interaction applications. Defining an accurate and easy-to-use ergonomic assessment model stands as an important step in providing feedback for postural correction to improve operator health and comfort. In order to enable efficient computation, previously proposed automated ergonomic assessment and correction tools make a… ▽ More Ergonomics and human comfort are essential concerns in physical human-robot interaction applications. Defining an accurate and easy-to-use ergonomic assessment model stands as an important step in providing feedback for postural correction to improve operator health and comfort. In order to enable efficient computation, previously proposed automated ergonomic assessment and correction tools make approximations or simplifications to gold-standard assessment tools used by ergonomists in practice. In order to retain assessment quality, while improving computational considerations, we introduce DULA, a differentiable and continuous ergonomics model learned to replicate the popular and scientifically validated RULA assessment. We show that DULA provides assessment comparable to RULA while providing computational benefits. We highlight DULA's strength in a demonstration of gradient-based postural optimization for a simulated teleoperation task. △ Less

Submitted 14 July, 2021; originally announced July 2021.

DefGraspSim: Simulation-based gras** of 3D deformable objects

Authors: Isabella Huang, Yashraj Narang, Clemens Eppner, Balakumar Sundaralingam, Miles Macklin, Tucker Hermans, Dieter Fox

Abstract: Robotic gras** of 3D deformable objects (e.g., fruits/vegetables, internal organs, bottles/boxes) is critical for real-world applications such as food processing, robotic surgery, and household automation. However, develo** grasp strategies for such objects is uniquely challenging. In this work, we efficiently simulate grasps on a wide range of 3D deformable objects using a GPU-based implement… ▽ More Robotic gras** of 3D deformable objects (e.g., fruits/vegetables, internal organs, bottles/boxes) is critical for real-world applications such as food processing, robotic surgery, and household automation. However, develo** grasp strategies for such objects is uniquely challenging. In this work, we efficiently simulate grasps on a wide range of 3D deformable objects using a GPU-based implementation of the corotational finite element method (FEM). To facilitate future research, we open-source our simulated dataset (34 objects, 1e5 Pa elasticity range, 6800 grasp evaluations, 1.1M grasp measurements), as well as a code repository that allows researchers to run our full FEM-based grasp evaluation pipeline on arbitrary 3D object models of their choice. We also provide a detailed analysis on 6 object primitives. For each primitive, we methodically describe the effects of different grasp strategies, compute a set of performance metrics (e.g., deformation, stress) that fully capture the object response, and identify simple grasp features (e.g., gripper displacement, contact area) measurable by robots prior to pickup and predictive of these performance metrics. Finally, we demonstrate good correspondence between grasps on simulated objects and their real-world counterparts. △ Less

Submitted 12 July, 2021; originally announced July 2021.

Comments: 11 pages, 19 figures. For associated website and code repository, see https://sites.google.com/nvidia.com/defgraspsim and https://github.com/NVlabs/deformable_object_gras**. Published in DO-Sim: Workshop on Deformable Object Simulation in Robotics at Robotics: Science and Systems (RSS) 2021

A new framework for experimental design using Bayesian Evidential Learning: the case of wellhead protection area

Authors: Robin Thibaut, Eric Laloy, Thomas Hermans

Abstract: In this contribution, we predict the wellhead protection area (WHPA, target), the shape and extent of which is influenced by the distribution of hydraulic conductivity (K), from a small number of tracing experiments (predictor). Our first objective is to make stochastic predictions of the WHPA within the Bayesian Evidential Learning (BEL) framework, which aims to find a direct relationship between… ▽ More In this contribution, we predict the wellhead protection area (WHPA, target), the shape and extent of which is influenced by the distribution of hydraulic conductivity (K), from a small number of tracing experiments (predictor). Our first objective is to make stochastic predictions of the WHPA within the Bayesian Evidential Learning (BEL) framework, which aims to find a direct relationship between predictor and target using machine learning. This relationship is learned from a small set of training models (400) sampled from the prior distribution of K. The associated 400 pairs of simulated predictors and targets are obtained through forward modelling. Newly collected field data can then be directly used to predict the approximate posterior distribution of the corresponding WHPA. The uncertainty range of the posterior WHPA distribution is affected by the number and position of data sources (injection wells). Our second objective is to extend BEL to identify the optimal design of data source locations that minimizes the posterior uncertainty of the WHPA. This can be done explicitly, without averaging or approximating because once trained, the BEL model allows the computation of the posterior uncertainty corresponding to any new input data. We use the Modified Hausdorff Distance and the Structural Similarity index metrics to estimate the posterior uncertainty range of the WHPA. Increasing the number of injection wells effectively reduces the derived posterior WHPA uncertainty. Our approach can also estimate which injection wells are more informative than others, as validated through a k-fold cross-validation procedure. Overall, the application of BEL to experimental design makes it possible to identify the data sources maximizing the information content of any measurement data. △ Less

Submitted 12 May, 2021; originally announced May 2021.

Optimizing Hospital Room Layout to Reduce the Risk of Patient Falls

Authors: Sarvenaz Chaeibakhsh, Roya Sabbagh Novin, Tucker Hermans, Andrew Merryweather, Alan Kuntz

Abstract: Despite years of research into patient falls in hospital rooms, falls and related injuries remain a serious concern to patient safety. In this work, we formulate a gradient-free constrained optimization problem to generate and reconfigure the hospital room interior layout to minimize the risk of falls. We define a cost function built on a hospital room fall model that takes into account the suppor… ▽ More Despite years of research into patient falls in hospital rooms, falls and related injuries remain a serious concern to patient safety. In this work, we formulate a gradient-free constrained optimization problem to generate and reconfigure the hospital room interior layout to minimize the risk of falls. We define a cost function built on a hospital room fall model that takes into account the supportive or hazardous effect of the patient's surrounding objects, as well as simulated patient trajectories inside the room. We define a constraint set that ensures the functionality of the generated room layouts in addition to conforming to architectural guidelines. We solve this problem efficiently using a variant of simulated annealing. We present results for two real-world hospital room types and demonstrate a significant improvement of 18% on average in patient fall risk when compared with a traditional hospital room layout and 41% when compared with randomly generated layouts. △ Less

Submitted 8 January, 2021; originally announced January 2021.

Comments: Accepted in: "10th International Conference on Operations Research and Enterprise Systems". 13 pages, 10 figures

MSC Class: 90C15 (Primary) 68T01 (Secondary) ACM Class: J.3; I.2

Comparing Piezoresistive Substrates for Tactile Sensing in Dexterous Hands

Authors: Rebecca Miles, Martin Matak, Sarah Hood, Mohanraj Devendran Shanthi, Darrin Young, Tucker Hermans

Abstract: While tactile skins have been shown to be useful for detecting collisions between a robotic arm and its environment, they have not been extensively used for improving robotic gras** and in-hand manipulation. We propose a novel sensor design for use in covering existing multi-fingered robot hands. We analyze the performance of four different piezoresistive materials using both fabric and anti-sta… ▽ More While tactile skins have been shown to be useful for detecting collisions between a robotic arm and its environment, they have not been extensively used for improving robotic gras** and in-hand manipulation. We propose a novel sensor design for use in covering existing multi-fingered robot hands. We analyze the performance of four different piezoresistive materials using both fabric and anti-static foam substrates in benchtop experiments. We find that although the piezoresistive foam was designed as packing material and not for use as a sensing substrate, it performs comparably with fabrics specifically designed for this purpose. While these results demonstrate the potential of piezoresistive foams for tactile sensing applications, they do not fully characterize the efficacy of these sensors for use in robot manipulation. As such, we use a low density foam substrate to develop a scalable tactile skin that can be attached to the palm of a robotic hand. We demonstrate several robotic manipulation tasks using this sensor to show its ability to reliably detect and localize contact, as well as analyze contact patterns during gras** and transport tasks. Our project website provides details on all materials, software, and data used in the sensor development and analysis: https://sites.google.com/gcloud.utah.edu/piezoresistive-tactile-sensing/. △ Less

Submitted 14 September, 2022; v1 submitted 11 November, 2020; originally announced November 2020.

Planning under Uncertainty to Goal Distributions

Authors: Adam Conkey, Tucker Hermans

Abstract: Goals for planning problems are typically conceived of as subsets of the state space. However, for many practical planning problems in robotics, we expect the robot to predict goals, e.g. from noisy sensors or by generalizing learned models to novel contexts. In these cases, sets with uncertainty naturally extend to probability distributions. While a few works have used probability distributions a… ▽ More Goals for planning problems are typically conceived of as subsets of the state space. However, for many practical planning problems in robotics, we expect the robot to predict goals, e.g. from noisy sensors or by generalizing learned models to novel contexts. In these cases, sets with uncertainty naturally extend to probability distributions. While a few works have used probability distributions as goals for planning, surprisingly no systematic treatment of planning to goal distributions exists in the literature. This article serves to fill that gap. We argue that goal distributions are a more appropriate goal representation than deterministic sets for many robotics applications. We present a novel approach to planning under uncertainty to goal distributions, which we use to highlight several advantages of the goal distribution formulation. We build on previous results in the literature by formally framing our approach as an instance of planning as inference. We additionally derive reductions of several common planning objectives as special cases of our probabilistic planning framework. Our experiments demonstrate the flexibility of probability distributions as a goal representation on a variety of problems including planar navigation among obstacles, intercepting a moving target, rolling a ball to a target location, and a 7-DOF robot arm reaching to grasp an object. △ Less

Submitted 29 April, 2022; v1 submitted 9 November, 2020; originally announced November 2020.

Comments: Currently under review

Risk-Aware Decision Making in Service Robots to Minimize Risk of Patient Falls in Hospitals

Authors: Roya Sabbagh Novin, Amir Yazdani, Andrew Merryweather, Tucker Hermans

Abstract: Planning under uncertainty is a crucial capability for autonomous systems to operate reliably in uncertain and dynamic environments. The concern of safety becomes even more critical in healthcare settings where robots interact with human patients. In this paper, we propose a novel risk-aware planning framework to minimize the risk of falls by providing a patient with an assistive device. Our appro… ▽ More Planning under uncertainty is a crucial capability for autonomous systems to operate reliably in uncertain and dynamic environments. The concern of safety becomes even more critical in healthcare settings where robots interact with human patients. In this paper, we propose a novel risk-aware planning framework to minimize the risk of falls by providing a patient with an assistive device. Our approach combines learning-based prediction with model-based control to plan for the fall prevention task. This provides advantages compared to end-to-end learning methods in which the robot's performance is limited to specific scenarios, or purely model-based approaches that use relatively simple function approximators and are prone to high modeling errors. We compare various risk metrics and the results from simulated scenarios show that using the proposed cost function, the robot can plan interventions to avoid high fall score events. △ Less

Submitted 25 March, 2021; v1 submitted 15 October, 2020; originally announced October 2020.

Comments: 7 pages + 2 page supplementary

Inversion of electromagnetic induction data using a novel wavelet-based and scale-dependent regularization term

Authors: Wouter Deleersnyder, Benjamin Maveau, Thomas Hermans, David Dudal

Abstract: The inversion of electromagnetic induction data to a conductivity profile is an ill-posed problem. Regularization improves the stability of the inversion and, based on Occam's razor principle, a smoothing constraint is typically used. However, the conductivity profiles are not always expected to be smooth. Here, we develop a new inversion scheme in which we transform the model to the wavelet space… ▽ More The inversion of electromagnetic induction data to a conductivity profile is an ill-posed problem. Regularization improves the stability of the inversion and, based on Occam's razor principle, a smoothing constraint is typically used. However, the conductivity profiles are not always expected to be smooth. Here, we develop a new inversion scheme in which we transform the model to the wavelet space and impose a sparsity constraint. This sparsity constrained inversion scheme will minimize an objective function with a least-squares data misfit and a sparsity measure of the model in the wavelet domain. A model in the wavelet domain has both temporal as spatial resolution, and penalizing small-scale coefficients effectively reduces the complexity of the model. Depending on the expected conductivity profile, an optimal wavelet basis function can be chosen. The scheme is thus adaptive. Finally, we apply this new scheme on a frequency domain electromagnetic sounding (FDEM) dataset, but the scheme could equally apply to any other 1D geophysical method. △ Less

Submitted 7 September, 2020; originally announced September 2020.

Deep generative models in inversion: a review and development of a new approach based on a variational autoencoder

Authors: Jorge Lopez-Alvis, Eric Laloy, Frédéric Nguyen, Thomas Hermans

Abstract: When solving inverse problems in geophysical imaging, deep generative models (DGMs) may be used to enforce the solution to display highly structured spatial patterns which are supported by independent information (e.g. the geological setting) of the subsurface. In such case, inversion may be formulated in a latent space where a low-dimensional parameterization of the patterns is defined and where… ▽ More When solving inverse problems in geophysical imaging, deep generative models (DGMs) may be used to enforce the solution to display highly structured spatial patterns which are supported by independent information (e.g. the geological setting) of the subsurface. In such case, inversion may be formulated in a latent space where a low-dimensional parameterization of the patterns is defined and where Markov chain Monte Carlo or gradient-based methods may be applied. However, the generative map** between the latent and the original (pixel) representations is usually highly nonlinear which may cause some difficulties for inversion, especially for gradient-based methods. In this contribution we review the conceptual framework of inversion with DGMs and study the principal causes of the nonlinearity of the generative map**. As a result, we identify a conflict between two goals: the accuracy of the generated patterns and the feasibility of gradient-based inversion. In addition, we show how some of the training parameters of a variational autoencoder, which is a particular instance of a DGM, may be chosen so that a tradeoff between these two goals is achieved and acceptable inversion results are obtained with a stochastic gradient-descent scheme. A test case using truth models with channel patterns of different complexity and cross-borehole traveltime tomographic data involving both a linear and a nonlinear forward operator is used to assess the performance of the proposed approach. △ Less

Submitted 27 August, 2020; originally announced August 2020.

Development of a Novel Computational Model for Evaluating Fall Risk in Patient Room Design

Authors: Roya Sabbagh Novin, Ellen Taylor, Tucker Hermans, Andrew Merryweather

Abstract: Objectives: The aims of this study are to identify factors in physical environments that contribute to patient falls in hospitals and to propose a computational model to evaluate patient room designs. Background: The existing fall risk assessment tools have an acceptable level of sensitivity and specificity, however, they only consider intrinsic factors and medications, making the prediction ver… ▽ More Objectives: The aims of this study are to identify factors in physical environments that contribute to patient falls in hospitals and to propose a computational model to evaluate patient room designs. Background: The existing fall risk assessment tools have an acceptable level of sensitivity and specificity, however, they only consider intrinsic factors and medications, making the prediction very limited in terms of how the physical environment contributes to fall risk. Methods: We provide a computational model for risk of fall based on physical-environment and patient-motion factors. We use a trajectory optimization approach for patient motion prediction. Results: We run the proposed model on four room designs as examples of various room design categories. Results show the capabilities of the proposed model in identifying risky locations within the room. Conclusions: Our study shows the potential capabilities of the proposed model. Due to lack of enough evidence for the examined factors, it is not possible at this point to gain robust confidence in the final evaluations. More studies using quantitative, relational, or causal designs are recommended to inform the proposed model for patient falls. Application: Develo** a comprehensive fall risk model is a significant step in understanding and solving the problem of patient falls in hospitals. It can provide guidance for healthcare decision makers to optimize effective interventions to reduce risk of falls while promoting safe patient mobility in the hospital room environment. We can also use it in healthcare technologies such as assistive robots to provide informed assistance. △ Less

Submitted 28 August, 2020; v1 submitted 20 August, 2020; originally announced August 2020.

Fluid drag reduction by magnetic confinement

Authors: Arvind Arun Dev, Peter Dunne, Thomas M. Hermans, Bernard Doudin

Abstract: The frictional forces of a viscous liquid flow are a major energy loss issue and severely limit microfluidics practical use. Reducing this drag by more than a few tens of percent remain illusive. Here, we show how cylindrical liquid-in-liquid flow leads to drag reduction of 60-99% for sub mm and mm sized channels, irrespective of whether the viscosity of the transported liquid is larger or smaller… ▽ More The frictional forces of a viscous liquid flow are a major energy loss issue and severely limit microfluidics practical use. Reducing this drag by more than a few tens of percent remain illusive. Here, we show how cylindrical liquid-in-liquid flow leads to drag reduction of 60-99% for sub mm and mm sized channels, irrespective of whether the viscosity of the transported liquid is larger or smaller than that of the encapsulating one. In contrast to lubrication or sheath flow, we do not require the continuous flow of the encapsulating lubricant, here made up of a ferrofluid held in place by magnetic forces. In a laminar flow model with appropriate boundary conditions, we introduce a modified Reynolds number with a scaling that depends on geometrical factors and viscosity ratio of the two liquids. It explains our whole range of data and reveal the key design parameters for optimizing the drag reduction values. Our results therefore open the route to microfluidics designs with pressure gradients possibly reduced by orders of magnitudes. △ Less

Submitted 1 October, 2021; v1 submitted 26 June, 2020; originally announced June 2020.

Comments: MS- 22 pages, 5 figures

Multi-Fingered Active Grasp Learning

Authors: Qingkai Lu, Mark Van der Merwe, Tucker Hermans

Abstract: Learning-based approaches to grasp planning are preferred over analytical methods due to their ability to better generalize to new, partially observed objects. However, data collection remains one of the biggest bottlenecks for grasp learning methods, particularly for multi-fingered hands. The relatively high dimensional configuration space of the hands coupled with the diversity of objects common… ▽ More Learning-based approaches to grasp planning are preferred over analytical methods due to their ability to better generalize to new, partially observed objects. However, data collection remains one of the biggest bottlenecks for grasp learning methods, particularly for multi-fingered hands. The relatively high dimensional configuration space of the hands coupled with the diversity of objects common in daily life requires a significant number of samples to produce robust and confident grasp success classifiers. In this paper, we present the first active deep learning approach to gras** that searches over the grasp configuration space and classifier confidence in a unified manner. We base our approach on recent success in planning multi-fingered grasps as probabilistic inference with a learned neural network likelihood function. We embed this within a multi-armed bandit formulation of sample selection. We show that our active grasp learning approach uses fewer training samples to produce grasp success rates comparable with the passive supervised learning method trained with gras** data generated by an analytical planner. We additionally show that grasps generated by the active learner have greater qualitative and quantitative diversity in shape. △ Less

Submitted 1 August, 2020; v1 submitted 6 June, 2020; originally announced June 2020.

Comments: arXiv admin note: text overlap with arXiv:2001.09242

Helium Ion Microscopy for Reduced Spin Orbit Torque Switching Currents

Authors: Peter Dunne, Ciaran Fowley, Gregor Hlawacek, **u Kurian, Gwenaël Atcheson, Silviu Colis, Niclas Teichert, Bohdan Kundys, M. Venkatesan, Jürgen Lindner, Alina Maria Deac, Thomas M. Hermans, J. M. D. Coey, Bernard Doudin

Abstract: Spin orbit torque driven switching is a favourable way to manipulate nanoscale magnetic objects for both memory and wireless communication devices. The critical current required to switch from one magnetic state to another depends on the geometry and the intrinsic properties of the materials used, which are difficult to control locally. Here we demonstrate how focused helium ion beam irradiation c… ▽ More Spin orbit torque driven switching is a favourable way to manipulate nanoscale magnetic objects for both memory and wireless communication devices. The critical current required to switch from one magnetic state to another depends on the geometry and the intrinsic properties of the materials used, which are difficult to control locally. Here we demonstrate how focused helium ion beam irradiation can modulate the local magnetic anisotropy of a Co thin film at the microscopic scale. Real-time in-situ characterisation using the anomalous Hall effect showed up to an order of magnitude reduction of the magnetic anisotropy under irradiation, and using this, multi-level switching is demonstrated. The result is that spin-switching current densities, down to 800 kA cm$^{-2}$, can be achieved on predetermined areas of the film, without the need for lithography. The ability to vary critical currents spatially has implications not only for storage elements, but also neuromorphic and probabilistic computing. △ Less

Submitted 14 September, 2020; v1 submitted 15 May, 2020; originally announced May 2020.

Comments: Main text: 22 pages, 3 figures, 2 tables, 1 TOC graphic. Included SI: 2 pages, 2 figures

In-Hand Object-Dynamics Inference using Tactile Fingertips

Authors: Balakumar Sundaralingam, Tucker Hermans

Abstract: Having the ability to estimate an object's properties through interaction will enable robots to manipulate novel objects. Object's dynamics, specifically the friction and inertial parameters have only been estimated in a lab environment with precise and often external sensing. Could we infer an object's dynamics in the wild with only the robot's sensors? In this paper, we explore the estimation of… ▽ More Having the ability to estimate an object's properties through interaction will enable robots to manipulate novel objects. Object's dynamics, specifically the friction and inertial parameters have only been estimated in a lab environment with precise and often external sensing. Could we infer an object's dynamics in the wild with only the robot's sensors? In this paper, we explore the estimation of dynamics of a grasped object in motion, with tactile force sensing at multiple fingertips. Our estimation approach does not rely on torque sensing to estimate the dynamics. To estimate friction, we develop a control scheme to actively interact with the object until slip is detected. To robustly perform the inertial estimation, we setup a factor graph that fuses all our sensor measurements on physically consistent manifolds and perform inference. We show that tactile fingertips enable in-hand dynamics estimation of low mass objects. △ Less

Submitted 18 January, 2021; v1 submitted 29 March, 2020; originally announced March 2020.

Comments: Accepted at IEEE Transactions on Robotics (T-RO). Website: https://sites.google.com/view/tactile-obj-dynamics

Is The Leader Robot an Adequate Sensor for Posture Estimation and Ergonomic Assessment of A Human Teleoperator?

Authors: Amir Yazdani, Roya Sabbagh Novin, Andrew Merryweather, Tucker Hermans

Abstract: Ergonomic assessment of human posture plays a vital role in understanding work-related safety and health. Current posture estimation approaches face occlusion challenges in teleoperation and physical human-robot interaction. We investigate if the leader robot is an adequate sensor for posture estimation in teleoperation and we introduce a new probabilistic approach that relies solely on the trajec… ▽ More Ergonomic assessment of human posture plays a vital role in understanding work-related safety and health. Current posture estimation approaches face occlusion challenges in teleoperation and physical human-robot interaction. We investigate if the leader robot is an adequate sensor for posture estimation in teleoperation and we introduce a new probabilistic approach that relies solely on the trajectory of the leader robot for generating observations. We model the human using a redundant, partially-observable dynamical system and we infer the posture using a standard particle filter. We compare our approach with postures from a commercial motion capture system and also two least-squares optimization approaches for human inverse kinematics. The results reveal that the proposed approach successfully estimates human postures and ergonomic risk scores comparable to those estimates from gold-standard motion capture. △ Less

Submitted 19 March, 2021; v1 submitted 24 February, 2020; originally announced February 2020.

Comments: Submitted to IEEE CASE 2021

Multi-Fingered Grasp Planning via Inference in Deep Neural Networks

Authors: Qingkai Lu, Mark Van der Merwe, Balakumar Sundaralingam, Tucker Hermans

Abstract: We propose a novel approach to multi-fingered grasp planning leveraging learned deep neural network models. We train a voxel-based 3D convolutional neural network to predict grasp success probability as a function of both visual information of an object and grasp configuration. We can then formulate grasp planning as inferring the grasp configuration which maximizes the probability of grasp succes… ▽ More We propose a novel approach to multi-fingered grasp planning leveraging learned deep neural network models. We train a voxel-based 3D convolutional neural network to predict grasp success probability as a function of both visual information of an object and grasp configuration. We can then formulate grasp planning as inferring the grasp configuration which maximizes the probability of grasp success. In addition, we learn a prior over grasp configurations as a mixture density network conditioned on our voxel-based object representation. We show that this object conditional prior improves grasp inference when used with the learned grasp success prediction network when compared to a learned, object-agnostic prior, or an uninformed uniform prior. Our work is the first to directly plan high quality multi-fingered grasps in configuration space using a deep neural network without the need of an external planner. We validate our inference method performing multi-finger gras** on a physical robot. Our experimental results show that our planning method outperforms existing grasp planning methods for neural networks. △ Less

Submitted 19 March, 2020; v1 submitted 24 January, 2020; originally announced January 2020.

Neutron imaging of liquid-liquid systems containing paramagnetic salt solutions

Authors: Tim A. Butcher, G. J. M. Formon, P. Dunne, T. M. Hermans, F. Ott, L. Noirez, J. M. D. Coey

Abstract: The method of neutron imaging was adopted to map the concentration evolution of aqueous paramagnetic Gd(NO3)3 solutions. Magnetic manipulation of the paramagnetic liquid within a miscible nonmagnetic liquid is possible by countering density-difference driven convection. The formation of salt fingers caused by double-diffusive convection in a liquid-liquid system of Gd(NO3)3 and Y(NO3)3 solutions c… ▽ More The method of neutron imaging was adopted to map the concentration evolution of aqueous paramagnetic Gd(NO3)3 solutions. Magnetic manipulation of the paramagnetic liquid within a miscible nonmagnetic liquid is possible by countering density-difference driven convection. The formation of salt fingers caused by double-diffusive convection in a liquid-liquid system of Gd(NO3)3 and Y(NO3)3 solutions can be prevented by the magnetic field gradient force. △ Less

Submitted 13 January, 2020; originally announced January 2020.

Comments: 5 Pages of article with 4 figures, 1 page of supplementary material with 2 figures; Time sequenced neutron images available at https://aip.scitation.org/doi/suppl/10.1063/1.5135390

Journal ref: Appl. Phys. Lett. 116, 022405 (2020)

Benchmarking In-Hand Manipulation

Authors: Silvia Cruciani, Balakumar Sundaralingam, Kaiyu Hang, Vikash Kumar, Tucker Hermans, Danica Kragic

Abstract: The purpose of this benchmark is to evaluate the planning and control aspects of robotic in-hand manipulation systems. The goal is to assess the system's ability to change the pose of a hand-held object by either using the fingers, environment or a combination of both. Given an object surface mesh from the YCB data-set, we provide examples of initial and goal states (i.e.\ static object poses and… ▽ More The purpose of this benchmark is to evaluate the planning and control aspects of robotic in-hand manipulation systems. The goal is to assess the system's ability to change the pose of a hand-held object by either using the fingers, environment or a combination of both. Given an object surface mesh from the YCB data-set, we provide examples of initial and goal states (i.e.\ static object poses and fingertip locations) for various in-hand manipulation tasks. We further propose metrics that measure the error in reaching the goal state from a specific initial state, which, when aggregated across all tasks, also serves as a measure of the system's in-hand manipulation capability. We provide supporting software, task examples, and evaluation results associated with the benchmark. All the supporting material is available at https://robot-learning.cs.utah.edu/project/benchmarking_in_hand_manipulation △ Less

Submitted 9 January, 2020; originally announced January 2020.

Comments: Accepted to Robotics Automation and Letters (RA-L)

A Model Predictive Approach for Online Mobile Manipulation of Nonholonomic Objects using Learned Dynamics

Authors: Roya Sabbagh Novin, Amir Yazdani, Andrew Merryweather, Tucker Hermans

Abstract: A particular type of assistive robots designed for physical interaction with objects could play an important role assisting with mobility and fall prevention in healthcare facilities. Autonomous mobile manipulation presents a hurdle prior to safely using robots in real life applications. In this article, we introduce a mobile manipulation framework based on model predictive control using learned d… ▽ More A particular type of assistive robots designed for physical interaction with objects could play an important role assisting with mobility and fall prevention in healthcare facilities. Autonomous mobile manipulation presents a hurdle prior to safely using robots in real life applications. In this article, we introduce a mobile manipulation framework based on model predictive control using learned dynamics models of objects. We focus on the specific problem of manipulating legged objects such as those commonly found in healthcare environments and personal dwellings (e.g. walkers, tables, chairs). We describe a probabilistic method for autonomous learning of an approximate dynamics model for these objects. In this method, we learn dynamic parameters using a small dataset consisting of force and motion data from interactions between the robot and object. Moreover, we account for multiple manipulation strategies by formulating the manipulation planning as a mixed-integer convex optimization. The proposed framework considers the hybrid control system comprised of i) choosing which leg to grasp, and ii) control of continuous applied forces for manipulation. We formalize our algorithm based on model predictive control to compensate for modeling errors and find an optimal path to manipulate the object from one configuration to another. We show results for several objects with various wheel configurations. Simulation and physical experiments show that the obtained dynamics models are sufficiently accurate for safe and collision-free manipulation. When combined with the proposed manipulation planning algorithm, the robot successfully moves the object to a desired pose while avoiding collision. △ Less

Submitted 10 November, 2020; v1 submitted 19 December, 2019; originally announced December 2019.