-
MultiGripperGrasp: A Dataset for Robotic Gras** from Parallel Jaw Grippers to Dexterous Hands
Authors:
Luis Felipe Casas Murrilo,
Ninad Khargonkar,
Balakrishnan Prabhakaran,
Yu Xiang
Abstract:
We introduce a large-scale dataset named MultiGripperGrasp for robotic gras**. Our dataset contains 30.4M grasps from 11 grippers for 345 objects. These grippers range from two-finger grippers to five-finger grippers, including a human hand. All grasps in the dataset are verified in Isaac Sim to classify them as successful and unsuccessful grasps. Additionally, the object fall-off time for each…
▽ More
We introduce a large-scale dataset named MultiGripperGrasp for robotic gras**. Our dataset contains 30.4M grasps from 11 grippers for 345 objects. These grippers range from two-finger grippers to five-finger grippers, including a human hand. All grasps in the dataset are verified in Isaac Sim to classify them as successful and unsuccessful grasps. Additionally, the object fall-off time for each grasp is recorded as a grasp quality measurement. Furthermore, the grippers in our dataset are aligned according to the orientation and position of their palms, allowing us to transfer grasps from one gripper to another. The grasp transfer significantly increases the number of successful grasps for each gripper in the dataset. Our dataset is useful to study generalized grasp planning and grasp transfer across different grippers.
△ Less
Submitted 14 March, 2024;
originally announced March 2024.
-
RISeg: Robot Interactive Object Segmentation via Body Frame-Invariant Features
Authors:
Howard H. Qian,
Yangxiao Lu,
Kejia Ren,
Gaotian Wang,
Ninad Khargonkar,
Yu Xiang,
Kaiyu Hang
Abstract:
In order to successfully perform manipulation tasks in new environments, such as gras**, robots must be proficient in segmenting unseen objects from the background and/or other objects. Previous works perform unseen object instance segmentation (UOIS) by training deep neural networks on large-scale data to learn RGB/RGB-D feature embeddings, where cluttered environments often result in inaccurat…
▽ More
In order to successfully perform manipulation tasks in new environments, such as gras**, robots must be proficient in segmenting unseen objects from the background and/or other objects. Previous works perform unseen object instance segmentation (UOIS) by training deep neural networks on large-scale data to learn RGB/RGB-D feature embeddings, where cluttered environments often result in inaccurate segmentations. We build upon these methods and introduce a novel approach to correct inaccurate segmentation, such as under-segmentation, of static image-based UOIS masks by using robot interaction and a designed body frame-invariant feature. We demonstrate that the relative linear and rotational velocities of frames randomly attached to rigid bodies due to robot interactions can be used to identify objects and accumulate corrected object-level segmentation masks. By introducing motion to regions of segmentation uncertainty, we are able to drastically improve segmentation accuracy in an uncertainty-driven manner with minimal, non-disruptive interactions (ca. 2-3 per scene). We demonstrate the effectiveness of our proposed interactive perception pipeline in accurately segmenting cluttered scenes by achieving an average object segmentation accuracy rate of 80.7%, an increase of 28.2% when compared with other state-of-the-art UOIS methods.
△ Less
Submitted 4 March, 2024;
originally announced March 2024.
-
SCENEREPLICA: Benchmarking Real-World Robot Manipulation by Creating Replicable Scenes
Authors:
Ninad Khargonkar,
Sai Haneesh Allu,
Yangxiao Lu,
Jishnu Jaykumar P,
Balakrishnan Prabhakaran,
Yu Xiang
Abstract:
We present a new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on pick-and-place. Our benchmark uses the YCB objects, a commonly used dataset in the robotics community, to ensure that our results are comparable to other studies. Additionally, the benchmark is designed to be easily reproducible in the real world, making it accessible to researcher…
▽ More
We present a new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on pick-and-place. Our benchmark uses the YCB objects, a commonly used dataset in the robotics community, to ensure that our results are comparable to other studies. Additionally, the benchmark is designed to be easily reproducible in the real world, making it accessible to researchers and practitioners. We also provide our experimental results and analyzes for model-based and model-free 6D robotic gras** on the benchmark, where representative algorithms are evaluated for object perception, gras** planning, and motion planning. We believe that our benchmark will be a valuable tool for advancing the field of robot manipulation. By providing a standardized evaluation framework, researchers can more easily compare different techniques and algorithms, leading to faster progress in develo** robot manipulation methods.
△ Less
Submitted 11 March, 2024; v1 submitted 27 June, 2023;
originally announced June 2023.
-
Skeletal Point Representations with Geometric Deep Learning
Authors:
Ninad Khargonkar,
Beatriz Paniagua,
Jared Vicory
Abstract:
Skeletonization has been a popular shape analysis technique that models both the interior and exterior of an object. Existing template-based calculations of skeletal models from anatomical structures are a time-consuming manual process. Recently, learning-based methods have been used to extract skeletons from 3D shapes. In this work, we propose novel additional geometric terms for calculating skel…
▽ More
Skeletonization has been a popular shape analysis technique that models both the interior and exterior of an object. Existing template-based calculations of skeletal models from anatomical structures are a time-consuming manual process. Recently, learning-based methods have been used to extract skeletons from 3D shapes. In this work, we propose novel additional geometric terms for calculating skeletal structures of objects. The results are similar to traditional fitted s-reps but but are produced much more quickly. Evaluation on real clinical data shows that the learned model predicts accurate skeletal representations and shows the impact of proposed geometric losses along with using s-reps as weak supervision.
△ Less
Submitted 3 March, 2023;
originally announced March 2023.
-
Self-Supervised Unseen Object Instance Segmentation via Long-Term Robot Interaction
Authors:
Yangxiao Lu,
Ninad Khargonkar,
Zesheng Xu,
Charles Averill,
Kamalesh Palanisamy,
Kaiyu Hang,
Yunhui Guo,
Nicholas Ruozzi,
Yu Xiang
Abstract:
We introduce a novel robotic system for improving unseen object instance segmentation in the real world by leveraging long-term robot interaction with objects. Previous approaches either grasp or push an object and then obtain the segmentation mask of the grasped or pushed object after one action. Instead, our system defers the decision on segmenting objects after a sequence of robot pushing actio…
▽ More
We introduce a novel robotic system for improving unseen object instance segmentation in the real world by leveraging long-term robot interaction with objects. Previous approaches either grasp or push an object and then obtain the segmentation mask of the grasped or pushed object after one action. Instead, our system defers the decision on segmenting objects after a sequence of robot pushing actions. By applying multi-object tracking and video object segmentation on the images collected via robot pushing, our system can generate segmentation masks of all the objects in these images in a self-supervised way. These include images where objects are very close to each other, and segmentation errors usually occur on these images for existing object segmentation networks. We demonstrate the usefulness of our system by fine-tuning segmentation networks trained on synthetic data with real-world data collected by our system. We show that, after fine-tuning, the segmentation accuracy of the networks is significantly improved both in the same domain and across different domains. In addition, we verify that the fine-tuned networks improve top-down robotic gras** of unseen objects in the real world.
△ Less
Submitted 7 February, 2023;
originally announced February 2023.
-
NeuralGrasps: Learning Implicit Representations for Grasps of Multiple Robotic Hands
Authors:
Ninad Khargonkar,
Neil Song,
Zesheng Xu,
Balakrishnan Prabhakaran,
Yu Xiang
Abstract:
We introduce a neural implicit representation for grasps of objects from multiple robotic hands. Different grasps across multiple robotic hands are encoded into a shared latent space. Each latent vector is learned to decode to the 3D shape of an object and the 3D shape of a robotic hand in a gras** pose in terms of the signed distance functions of the two 3D shapes. In addition, the distance met…
▽ More
We introduce a neural implicit representation for grasps of objects from multiple robotic hands. Different grasps across multiple robotic hands are encoded into a shared latent space. Each latent vector is learned to decode to the 3D shape of an object and the 3D shape of a robotic hand in a gras** pose in terms of the signed distance functions of the two 3D shapes. In addition, the distance metric in the latent space is learned to preserve the similarity between grasps across different robotic hands, where the similarity of grasps is defined according to contact regions of the robotic hands. This property enables our method to transfer grasps between different grippers including a human hand, and grasp transfer has the potential to share gras** skills between robots and enable robots to learn gras** skills from humans. Furthermore, the encoded signed distance functions of objects and grasps in our implicit representation can be used for 6D object pose estimation with gras** contact optimization from partial point clouds, which enables robotic gras** in the real world.
△ Less
Submitted 6 July, 2022;
originally announced July 2022.
-
Submodular Combinatorial Information Measures with Applications in Machine Learning
Authors:
Rishabh Iyer,
Ninad Khargonkar,
Jeff Bilmes,
Himanshu Asnani
Abstract:
Information-theoretic quantities like entropy and mutual information have found numerous uses in machine learning. It is well known that there is a strong connection between these entropic quantities and submodularity since entropy over a set of random variables is submodular. In this paper, we study combinatorial information measures that generalize independence, (conditional) entropy, (condition…
▽ More
Information-theoretic quantities like entropy and mutual information have found numerous uses in machine learning. It is well known that there is a strong connection between these entropic quantities and submodularity since entropy over a set of random variables is submodular. In this paper, we study combinatorial information measures that generalize independence, (conditional) entropy, (conditional) mutual information, and total correlation defined over sets of (not necessarily random) variables. These measures strictly generalize the corresponding entropic measures since they are all parameterized via submodular functions that themselves strictly generalize entropy. Critically, we show that, unlike entropic mutual information in general, the submodular mutual information is actually submodular in one argument, holding the other fixed, for a large class of submodular functions whose third-order partial derivatives satisfy a non-negativity property. This turns out to include a number of practically useful cases such as the facility location and set-cover functions. We study specific instantiations of the submodular information measures on these, as well as the probabilistic coverage, graph-cut, and saturated coverage functions, and see that they all have mathematically intuitive and practically useful expressions. Regarding applications, we connect the maximization of submodular (conditional) mutual information to problems such as mutual-information-based, query-based, and privacy-preserving summarization -- and we connect optimizing the multi-set submodular mutual information to clustering and robust partitioning.
△ Less
Submitted 2 March, 2021; v1 submitted 27 June, 2020;
originally announced June 2020.
