-
Combining Vision and Tactile Sensation for Video Prediction
Authors:
Willow Mandil,
Amir Ghalamzan-E
Abstract:
In this paper, we explore the impact of adding tactile sensation to video prediction models for physical robot interactions. Predicting the impact of robotic actions on the environment is a fundamental challenge in robotics. Current methods leverage visual and robot action data to generate video predictions over a given time period, which can then be used to adjust robot actions. However, humans r…
▽ More
In this paper, we explore the impact of adding tactile sensation to video prediction models for physical robot interactions. Predicting the impact of robotic actions on the environment is a fundamental challenge in robotics. Current methods leverage visual and robot action data to generate video predictions over a given time period, which can then be used to adjust robot actions. However, humans rely on both visual and tactile feedback to develop and maintain a mental model of their physical surroundings. In this paper, we investigate the impact of integrating tactile feedback into video prediction models for physical robot interactions. We propose three multi-modal integration approaches and compare the performance of these tactile-enhanced video prediction models. Additionally, we introduce two new datasets of robot pushing that use a magnetic-based tactile sensor for unsupervised learning. The first dataset contains visually identical objects with different physical properties, while the second dataset mimics existing robot-pushing datasets of household object clusters. Our results demonstrate that incorporating tactile feedback into video prediction models improves scene prediction accuracy and enhances the agent's perception of physical interactions and understanding of cause-effect relationships during physical robot interactions.
△ Less
Submitted 21 April, 2023;
originally announced April 2023.
-
Towards Autonomous Selective Harvesting: A Review of Robot Perception, Robot Design, Motion Planning and Control
Authors:
Vishnu Rajendran S,
Bappaditya Debnath,
Bappaditya Debnath,
Sariah Mghames,
Willow Mandil,
Soran Parsa,
Simon Parsons,
Amir Ghalamzan-E
Abstract:
This paper provides an overview of the current state-of-the-art in selective harvesting robots (SHRs) and their potential for addressing the challenges of global food production. SHRs have the potential to increase productivity, reduce labour costs, and minimise food waste by selectively harvesting only ripe fruits and vegetables. The paper discusses the main components of SHRs, including percepti…
▽ More
This paper provides an overview of the current state-of-the-art in selective harvesting robots (SHRs) and their potential for addressing the challenges of global food production. SHRs have the potential to increase productivity, reduce labour costs, and minimise food waste by selectively harvesting only ripe fruits and vegetables. The paper discusses the main components of SHRs, including perception, gras**, cutting, motion planning, and control. It also highlights the challenges in develo** SHR technologies, particularly in the areas of robot design, motion planning and control. The paper also discusses the potential benefits of integrating AI and soft robots and data-driven methods to enhance the performance and robustness of SHR systems. Finally, the paper identifies several open research questions in the field and highlights the need for further research and development efforts to advance SHR technologies to meet the challenges of global food production. Overall, this paper provides a starting point for researchers and practitioners interested in develo** SHRs and highlights the need for more research in this field.
△ Less
Submitted 19 April, 2023;
originally announced April 2023.
-
Acoustic Soft Tactile Skin (AST Skin)
Authors:
Vishnu Rajendran S,
Willow Mandil,
Simon Parsons,
Amir Ghalamzan E
Abstract:
This paper presents a novel soft tactile skin (STS) technology operating with sound waves. In this innovative approach, the sound waves generated by a speaker travel in channels embedded in a soft membrane and get modulated due to a deformation of the channel when pressed by an external force and received by a microphone at the end of the channel. The sensor leverages regression and classification…
▽ More
This paper presents a novel soft tactile skin (STS) technology operating with sound waves. In this innovative approach, the sound waves generated by a speaker travel in channels embedded in a soft membrane and get modulated due to a deformation of the channel when pressed by an external force and received by a microphone at the end of the channel. The sensor leverages regression and classification methods for estimating the normal force and its contact location. Our sensor can be affixed to any robot part, e.g., end effectors or arm. We tested several regression and classifier methods to learn the relation between sound wave modulation, the applied force, and its location, respectively and picked the best-performing models for force and location predictions. Our novel tactile sensor yields 93% of the force estimation within 1.5 N tolerances for a range of 0-30+1 N and estimates contact locations with over 96% accuracy. We also demonstrated the performance of STS technology for a real-time grip** force control application.
△ Less
Submitted 29 February, 2024; v1 submitted 30 March, 2023;
originally announced March 2023.
-
Proactive slip control by learned slip model and trajectory adaptation
Authors:
Kiyanoush Nazari,
Willow Mandil,
Amir Ghalamzan E
Abstract:
This paper presents a novel control approach to dealing with object slip during robotic manipulative movements. Slip is a major cause of failure in many robotic gras** and manipulation tasks. Existing works increase grip force to avoid/control slip. However, this may not be feasible when (i) the robot cannot increase the grip** force -- the max grip** force is already applied or (ii) increas…
▽ More
This paper presents a novel control approach to dealing with object slip during robotic manipulative movements. Slip is a major cause of failure in many robotic gras** and manipulation tasks. Existing works increase grip force to avoid/control slip. However, this may not be feasible when (i) the robot cannot increase the grip** force -- the max grip** force is already applied or (ii) increased force damages the grasped object, such as soft fruit. Moreover, the robot fixes the grip** force when it forms a stable grasp on the surface of an object, and changing the grip** force during real-time manipulation may not be an effective control policy. We propose a novel control approach to slip avoidance including a learned action-conditioned slip predictor and a constrained optimiser avoiding a predicted slip given a desired robot action. We show the effectiveness of the proposed trajectory adaptation method with receding horizon controller with a series of real-robot test cases. Our experimental results show our proposed data-driven predictive controller can control slip for objects unseen in training.
△ Less
Submitted 13 September, 2022;
originally announced September 2022.
-
Action Conditioned Tactile Prediction: case study on slip prediction
Authors:
Willow Mandil,
Kiyanoush Nazari,
Amir Ghalamzan E
Abstract:
Tactile predictive models can be useful across several robotic manipulation tasks, e.g. robotic pushing, robotic gras**, slip avoidance, and in-hand manipulation. However, available tactile prediction models are mostly studied for image-based tactile sensors and there is no comparison study indicating the best performing models. In this paper, we presented two novel data-driven action-conditione…
▽ More
Tactile predictive models can be useful across several robotic manipulation tasks, e.g. robotic pushing, robotic gras**, slip avoidance, and in-hand manipulation. However, available tactile prediction models are mostly studied for image-based tactile sensors and there is no comparison study indicating the best performing models. In this paper, we presented two novel data-driven action-conditioned models for predicting tactile signals during real-world physical robot interaction tasks (1) action condition tactile prediction and (2) action conditioned tactile-video prediction models. We use a magnetic-based tactile sensor that is challenging to analyse and test state-of-the-art predictive models and the only existing bespoke tactile prediction model. We compare the performance of these models with those of our proposed models. We perform the comparison study using our novel tactile-enabled dataset containing 51,000 tactile frames of a real-world robotic manipulation task with 11 flat-surfaced household objects. Our experimental results demonstrate the superiority of our proposed tactile prediction models in terms of qualitative, quantitative and slip prediction scores.
△ Less
Submitted 10 May, 2024; v1 submitted 19 May, 2022;
originally announced May 2022.
