-
Learning Barrier-Certified Polynomial Dynamical Systems for Obstacle Avoidance with Robots
Authors:
Martin Schonger,
Hugo T. M. Kussaba,
Lingyun Chen,
Luis Figueredo,
Abdalla Swikir,
Aude Billard,
Sami Haddadin
Abstract:
Established techniques that enable robots to learn from demonstrations are based on learning a stable dynamical system (DS). To increase the robots' resilience to perturbations during tasks that involve static obstacle avoidance, we propose incorporating barrier certificates into an optimization problem to learn a stable and barrier-certified DS. Such optimization problem can be very complex or ex…
▽ More
Established techniques that enable robots to learn from demonstrations are based on learning a stable dynamical system (DS). To increase the robots' resilience to perturbations during tasks that involve static obstacle avoidance, we propose incorporating barrier certificates into an optimization problem to learn a stable and barrier-certified DS. Such optimization problem can be very complex or extremely conservative when the traditional linear parameter-varying formulation is used. Thus, different from previous approaches in the literature, we propose to use polynomial representations for DSs, which yields an optimization problem that can be tackled by sum-of-squares techniques. Finally, our approach can handle obstacle shapes that fall outside the scope of assumptions typically found in the literature concerning obstacle avoidance within the DS learning framework. Supplementary material can be found at the project webpage: https://martinschonger.github.io/abc-ds
△ Less
Submitted 12 March, 2024;
originally announced March 2024.
-
Smooth real-time motion planning based on a cascade dual-quaternion screw-geometry MPC
Authors:
Ainoor Teimoorzadeh,
Frederico Fernandes Afonso Silva,
Luis F. C. Figueredo,
Sami Haddadin
Abstract:
This paper investigates the tracking problem of a smooth coordinate-invariant trajectory using dual quaternion algebra. The proposed architecture consists of a cascade structure in which the outer-loop MPC performs real-time smoothing of the manipulator's end-effector twist while an inner-loop kinematic controller ensures tracking of the instantaneous desired end-effector pose. Experiments on a…
▽ More
This paper investigates the tracking problem of a smooth coordinate-invariant trajectory using dual quaternion algebra. The proposed architecture consists of a cascade structure in which the outer-loop MPC performs real-time smoothing of the manipulator's end-effector twist while an inner-loop kinematic controller ensures tracking of the instantaneous desired end-effector pose. Experiments on a $7$-DoF Franka Emika Panda robotic manipulator validate the proposed method demonstrating its application to constraint the robot twists, accelerations and jerks within prescribed bounds.
△ Less
Submitted 7 February, 2024;
originally announced February 2024.
-
Leveraging Previous Facial Action Units Knowledge for Emotion Recognition on Faces
Authors:
Pietro B. S. Masur,
Willams Costa,
Lucas S. Figueredo,
Veronica Teichrieb
Abstract:
People naturally understand emotions, thus permitting a machine to do the same could open new paths for human-computer interaction. Facial expressions can be very useful for emotion recognition techniques, as these are the biggest transmitters of non-verbal cues capable of being correlated with emotions. Several techniques are based on Convolutional Neural Networks (CNNs) to extract information in…
▽ More
People naturally understand emotions, thus permitting a machine to do the same could open new paths for human-computer interaction. Facial expressions can be very useful for emotion recognition techniques, as these are the biggest transmitters of non-verbal cues capable of being correlated with emotions. Several techniques are based on Convolutional Neural Networks (CNNs) to extract information in a machine learning process. However, simple CNNs are not always sufficient to locate points of interest on the face that can be correlated with emotions. In this work, we intend to expand the capacity of emotion recognition techniques by proposing the usage of Facial Action Units (AUs) recognition techniques to recognize emotions. This recognition will be based on the Facial Action Coding System (FACS) and computed by a machine learning system. In particular, our method expands over EmotiRAM, an approach for multi-cue emotion recognition, in which we improve over their facial encoding module.
△ Less
Submitted 20 November, 2023;
originally announced November 2023.
-
LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulation
Authors:
Shengqiang Zhang,
Philipp Wicke,
Lütfi Kerem Şenel,
Luis Figueredo,
Abdeldjallil Naceri,
Sami Haddadin,
Barbara Plank,
Hinrich Schütze
Abstract:
The convergence of embodied agents and large language models (LLMs) has brought significant advancements to embodied instruction following. Particularly, the strong reasoning capabilities of LLMs make it possible for robots to perform long-horizon tasks without expensive annotated demonstrations. However, public benchmarks for testing the long-horizon reasoning capabilities of language-conditioned…
▽ More
The convergence of embodied agents and large language models (LLMs) has brought significant advancements to embodied instruction following. Particularly, the strong reasoning capabilities of LLMs make it possible for robots to perform long-horizon tasks without expensive annotated demonstrations. However, public benchmarks for testing the long-horizon reasoning capabilities of language-conditioned robots in various scenarios are still missing. To fill this gap, this work focuses on the tabletop manipulation task and releases a simulation benchmark, \textit{LoHoRavens}, which covers various long-horizon reasoning aspects spanning color, size, space, arithmetics and reference. Furthermore, there is a key modality bridging problem for long-horizon manipulation tasks with LLMs: how to incorporate the observation feedback during robot execution for the LLM's closed-loop planning, which is however less studied by prior work. We investigate two methods of bridging the modality gap: caption generation and learnable interface for incorporating explicit and implicit observation feedback to the LLM, respectively. These methods serve as the two baselines for our proposed benchmark. Experiments show that both methods struggle to solve some tasks, indicating long-horizon manipulation tasks are still challenging for current popular models. We expect the proposed public benchmark and baselines can help the community develop better models for long-horizon tabletop manipulation tasks.
△ Less
Submitted 23 October, 2023; v1 submitted 18 October, 2023;
originally announced October 2023.
-
A Concise Overview of Safety Aspects in Human-Robot Interaction
Authors:
Mazin Hamad,
Simone Nertinger,
Robin J. Kirschner,
Luis Figueredo,
Abdeldjallil Naceri,
Sami Haddadin
Abstract:
As of today, robots exhibit impressive agility but also pose potential hazards to humans using/collaborating with them. Consequently, safety is considered the most paramount factor in human-robot interaction (HRI). This paper presents a multi-layered safety architecture, integrating both physical and cognitive aspects for effective HRI. We outline critical requirements for physical safety layers a…
▽ More
As of today, robots exhibit impressive agility but also pose potential hazards to humans using/collaborating with them. Consequently, safety is considered the most paramount factor in human-robot interaction (HRI). This paper presents a multi-layered safety architecture, integrating both physical and cognitive aspects for effective HRI. We outline critical requirements for physical safety layers as service modules that can be arbitrarily queried. Further, we showcase an HRI scheme that addresses human factors and perceived safety as high-level constraints on a validated impact safety paradigm. The aim is to enable safety certification of human-friendly robots across various HRI scenarios.
△ Less
Submitted 18 September, 2023;
originally announced September 2023.
-
Towards Language-Based Modulation of Assistive Robots through Multimodal Models
Authors:
Philipp Wicke,
Lüfti Kerem Şenel,
Shengqiang Zhang,
Luis Figueredo,
Abdeldjallil Naceri,
Sami Haddadin,
Hinrich Schütze
Abstract:
In the field of Geriatronics, enabling effective and transparent communication between humans and robots is crucial for enhancing the acceptance and performance of assistive robots. Our early-stage research project investigates the potential of language-based modulation as a means to improve human-robot interaction. We propose to explore real-time modulation during task execution, leveraging langu…
▽ More
In the field of Geriatronics, enabling effective and transparent communication between humans and robots is crucial for enhancing the acceptance and performance of assistive robots. Our early-stage research project investigates the potential of language-based modulation as a means to improve human-robot interaction. We propose to explore real-time modulation during task execution, leveraging language cues, visual references, and multimodal inputs. By develo** transparent and interpretable methods, we aim to enable robots to adapt and respond to language commands, enhancing their usability and flexibility. Through the exchange of insights and knowledge at the workshop, we seek to gather valuable feedback to advance our research and contribute to the development of interactive robotic systems for Geriatronics and beyond.
△ Less
Submitted 27 June, 2023; v1 submitted 26 June, 2023;
originally announced June 2023.
-
A Solution to Slosh-free Robot Trajectory Optimization
Authors:
Rafael I. Cabral Muchacho,
Riddhiman Laha,
Luis F. C. Figueredo,
Sami Haddadin
Abstract:
This paper is about fast slosh free fluid transportation. Existing approaches are either computationally heavy or only suitable for specific robots and container shapes. We model the end effector as a point mass suspended by a spherical pendulum and study the requirements for slosh free motion and the validity of the point mass model. In this approach, slosh free trajectories are generated by cont…
▽ More
This paper is about fast slosh free fluid transportation. Existing approaches are either computationally heavy or only suitable for specific robots and container shapes. We model the end effector as a point mass suspended by a spherical pendulum and study the requirements for slosh free motion and the validity of the point mass model. In this approach, slosh free trajectories are generated by controlling the pendulum's pivot and simulating the motion of the point mass. We cast the trajectory optimization problem as a quadratic program; this strategy can be used to obtain valid control inputs. Through simulations and experiments on a 7 DoF Franka Emika Panda robot we validate the effectiveness of the proposed approach.
△ Less
Submitted 23 October, 2022;
originally announced October 2022.
-
Human-to-Robot Manipulability Domain Adaptation with Parallel Transport and Manifold-Aware ICP
Authors:
Anna Reithmeir,
Luis Figueredo,
Sami Haddadin
Abstract:
Manipulability ellipsoids efficiently capture the human pose and reveal information about the task at hand. Their use in task-dependent robot teaching - particularly their transfer from a teacher to a learner - can advance emulation of human-like motion. Although in recent literature focus is shifted towards manipulability transfer between two robots, the adaptation to the capabilities of the othe…
▽ More
Manipulability ellipsoids efficiently capture the human pose and reveal information about the task at hand. Their use in task-dependent robot teaching - particularly their transfer from a teacher to a learner - can advance emulation of human-like motion. Although in recent literature focus is shifted towards manipulability transfer between two robots, the adaptation to the capabilities of the other kinematic system is to date not addressed and research in transfer from human to robot is still in its infancy. This work presents a novel manipulability domain adaptation method for the transfer of manipulability information to the domain of another kinematic system. As manipulability matrices/ellipsoids are symmetric positive-definite (SPD) they can be viewed as points on the Riemannian manifold of SPD matrices. We are the first to address the problem of manipulability transfer from the perspective of point cloud registration. We propose a manifold-aware Iterative Closest Point algorithm (ICP) with parallel transport initialization. Furthermore, we introduce a correspondence matching heuristic for manipulability ellipsoids based on inherent geometric features. We confirm our method in simulation experiments with 2-DoF manipulators as well as 7-DoF models representing the human-arm kinematics.
△ Less
Submitted 16 August, 2022;
originally announced August 2022.
-
LATTE: LAnguage Trajectory TransformEr
Authors:
Arthur Bucker,
Luis Figueredo,
Sami Haddadin,
Ashish Kapoor,
Shuang Ma,
Sai Vemprala,
Rogerio Bonatti
Abstract:
Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the real world is far from being an easy task. The challenge of combining a robot's inherent low-level geometric and kinodynamic constraints with a human's high-level semantic instructions traditionally is solved using task-s…
▽ More
Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the real world is far from being an easy task. The challenge of combining a robot's inherent low-level geometric and kinodynamic constraints with a human's high-level semantic instructions traditionally is solved using task-specific solutions with little generalizability between hardware platforms, often with the use of static sets of target actions and commands. This work instead proposes a flexible language-based framework that allows a user to modify generic robotic trajectories. Our method leverages pre-trained language models (BERT and CLIP) to encode the user's intent and target objects directly from a free-form text input and scene images, fuses geometrical features generated by a transformer encoder network, and finally outputs trajectories using a transformer decoder, without the need of priors related to the task or robot information. We significantly extend our own previous work presented in Bucker et al. by expanding the trajectory parametrization space to 3D and velocity as opposed to just XY movements. In addition, we now train the model to use actual images of the objects in the scene for context (as opposed to textual descriptions), and we evaluate the system in a diverse set of scenarios beyond manipulation, such as aerial and legged robots. Our simulated and real-life experiments demonstrate that our transformer model can successfully follow human intent, modifying the shape and speed of trajectories within multiple environments. Codebase available at: https://github.com/arthurfenderbucker/LaTTe-Language-Trajectory-TransformEr.git
△ Less
Submitted 16 September, 2022; v1 submitted 4 August, 2022;
originally announced August 2022.
-
Resha** Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers
Authors:
Arthur Bucker,
Luis Figueredo,
Sami Haddadin,
Ashish Kapoor,
Shuang Ma,
Rogerio Bonatti
Abstract:
Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible language-based…
▽ More
Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible language-based interface for human-robot collaboration, which allows a user to reshape existing trajectories for an autonomous agent. We take advantage of recent advancements in the field of large language models (BERT and CLIP) to encode the user command, and then combine these features with trajectory information using multi-modal attention transformers. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands, and treat the trajectory generation process as a sequence prediction problem, analogously to how language generation architectures operate. We evaluate the system in multiple simulated trajectory scenarios, and show a significant performance increase of our model over baseline approaches. In addition, our real-world experiments with a robot arm show that users significantly prefer our natural language interface over traditional methods such as kinesthetic teaching or cost-function programming. Our study shows how the field of robotics can take advantage of large pre-trained language models towards creating more intuitive interfaces between robots and machines. Project webpage: https://arthurfenderbucker.github.io/NL_trajectory_reshaper/
△ Less
Submitted 24 March, 2022;
originally announced March 2022.
-
Coordinate Invariant User-Guided Constrained Path Planning with Reactive Rapidly Expanding Plane-Oriented Esca** Trees
Authors:
Riddhiman Laha,
Ruiai Sun,
Wenxi Wu,
Dasharadhan Mahalingam,
Nilanjan Chakraborty,
Luis F. C. Figueredo,
Sami Haddadin
Abstract:
As collaborative robots move closer to human environments, motion generation and reactive planning strategies that allow for elaborate task execution with minimal easy-to-implement guidance whilst co** with changes in the environment is of paramount importance. In this paper, we present a novel approach for generating real-time motion plans for point-to-point tasks using a single successful huma…
▽ More
As collaborative robots move closer to human environments, motion generation and reactive planning strategies that allow for elaborate task execution with minimal easy-to-implement guidance whilst co** with changes in the environment is of paramount importance. In this paper, we present a novel approach for generating real-time motion plans for point-to-point tasks using a single successful human demonstration. Our approach is based on screw linear interpolation,which allows us to respect the underlying geometric constraints that characterize the task and are implicitly present in the demonstration. We also integrate an original reactive collision avoidance approach with our planner. We present extensive experimental results to demonstrate that with our approach,by using a single demonstration of moving one block, we can generate motion plans for complex tasks like stacking multiple blocks (in a dynamic environment). Analogous generalization abilities are also shown for tasks like pouring and loading shelves. For the pouring task, we also show that a demonstration given for one-armed pouring can be used for planning pouring with a dual-armed manipulator of different kinematic structure.
△ Less
Submitted 20 March, 2022;
originally announced March 2022.
-
Cooperative Observation of Targets moving over a Planar Graph with Prediction of Positions
Authors:
José E. B. Maia,
Levi P. Figueredo
Abstract:
Consider a team with two types of agents: targets and observers. Observers are aerial UAVs that observe targets moving on land with their movements restricted to the paths that form a planar graph on the surface. Observers have limited range of vision and targets do not avoid observers. The objective is to maximize the integral of the number of targets observed in the observation interval. Taking…
▽ More
Consider a team with two types of agents: targets and observers. Observers are aerial UAVs that observe targets moving on land with their movements restricted to the paths that form a planar graph on the surface. Observers have limited range of vision and targets do not avoid observers. The objective is to maximize the integral of the number of targets observed in the observation interval. Taking advantage of the fact that the future positions of targets in the short term are predictable, we show in this article a modified hill climbing algorithm that surpasses its previous versions in this new setting of the CTO problem.
△ Less
Submitted 12 February, 2020;
originally announced February 2020.
-
Robust H-infinity kinematic control of manipulator robots using dual quaternion algebra
Authors:
Luis Felipe Cruz Figueredo,
Bruno Vilhena Adorno,
João Yoshiyuki Ishihara
Abstract:
This paper proposes a robust dual-quaternion based H-infinity task-space kinematic controller for robot manipulators. To address the manipulator liability to modeling errors, uncertainties, exogenous disturbances, and their influence upon the kinematics of the end-effector pose, we adapt H-infinity techniques\textemdash suitable only for additive noises\textemdash to unit dual quaternions. The noi…
▽ More
This paper proposes a robust dual-quaternion based H-infinity task-space kinematic controller for robot manipulators. To address the manipulator liability to modeling errors, uncertainties, exogenous disturbances, and their influence upon the kinematics of the end-effector pose, we adapt H-infinity techniques\textemdash suitable only for additive noises\textemdash to unit dual quaternions. The noise to error attenuation within the H-infinity framework has the additional advantage of casting aside requirements concerning noise distributions, which are significantly hard to characterize within the group of rigid-body transformations. Using dual quaternion algebra, we provide a connection between performance effects over the end-effector trajectory and different sources of uncertainties and disturbances while satisfying attenuation requirements with minimum instantaneous control effort. The result is an easy-to-implement closed-form H-infinity control design criterion. The performance of the proposed strategy is evaluated within different realistic simulated scenarios and validated through real experiments.
△ Less
Submitted 21 June, 2021; v1 submitted 13 November, 2018;
originally announced November 2018.
-
Planning for Muscular and Peripersonal-Space Comfort during Human-Robot Forceful Collaboration
Authors:
Lipeng Chen,
Luis F C Figueredo,
Mehmet R. Dogar
Abstract:
This paper presents a planning algorithm designed to improve cooperative robot behavior concerning human comfort during forceful human-robot physical interaction. Particularly, we are interested in planning for object gras** and positioning ensuring not only stability against the exerted human force but also empowering the robot with capabilities to address and improve human experience and comfo…
▽ More
This paper presents a planning algorithm designed to improve cooperative robot behavior concerning human comfort during forceful human-robot physical interaction. Particularly, we are interested in planning for object gras** and positioning ensuring not only stability against the exerted human force but also empowering the robot with capabilities to address and improve human experience and comfort. Herein, comfort is addressed as both the muscular activation level required to exert the cooperative task, and the human spatial perception during the interaction, namely, the peripersonal space. By maximizing both comfort criteria, the robotic system can plan for the task (ensuring grasp stability) and for the human (improving human comfort). We believe this to be a key element to achieve intuitive and fluid human-robot interaction in real applications. Real HRI drilling and cutting experiments illustrated the efficiency of the proposed planner in improving overall comfort and HRI experience without compromising grasp stability.
△ Less
Submitted 19 October, 2018; v1 submitted 30 July, 2018;
originally announced July 2018.
-
Manipulation Planning under Changing External Forces
Authors:
Lipeng Chen,
Luis F. C. Figueredo,
Mehmet Dogar
Abstract:
In this work, we present a manipulation planning algorithm for a robot to keep an object stable under changing external forces. We particularly focus on the case where a human may be applying forceful operations, e.g. cutting or drilling, on an object that the robot is holding. The planner produces an efficient plan by intelligently deciding when the robot should change its grasp on the object as…
▽ More
In this work, we present a manipulation planning algorithm for a robot to keep an object stable under changing external forces. We particularly focus on the case where a human may be applying forceful operations, e.g. cutting or drilling, on an object that the robot is holding. The planner produces an efficient plan by intelligently deciding when the robot should change its grasp on the object as the human applies the forces. The planner also tries to choose subsequent grasps such that they will minimize the number of regrasps that will be required in the long-term. Furthermore, as it switches from one grasp to the other, the planner solves the problem of bimanual regrasp planning, where the object is not placed on a support surface, but instead it is held by a single gripper until the second gripper moves to a new position on the object. This requires the planner to also reason about the stability of the object under gravity.We provide an implementation on a bimanual robot and present experiments to show the performance of our planner.
△ Less
Submitted 30 July, 2018; v1 submitted 30 October, 2017;
originally announced October 2017.
-
Disseminando a Aprendizagem Colaborativa atraves do Ambiente Canvas
Authors:
Jose Solenir Lima Figueredo,
Renato Santos Mascarenhas,
Roberto Almeida Bittencourt
Abstract:
Virtual learning environments are actual solutions that facilitate collaborative learning, both in classroom and distance education. However, such environments are not yet fully disseminated in Brazilian universities. This work reports a learning experience designed as a workshop that aims to popularize the use of the Canvas virtual learning environment in an university. The workshop emphasizes th…
▽ More
Virtual learning environments are actual solutions that facilitate collaborative learning, both in classroom and distance education. However, such environments are not yet fully disseminated in Brazilian universities. This work reports a learning experience designed as a workshop that aims to popularize the use of the Canvas virtual learning environment in an university. The workshop emphasizes the support to collaborative learning through resources of the environment. Learned lessons include: the value of adequate planning of workshop activities; the potential of the environment regarding various uses by instructors; and the relevance of collaborative learning for instructors.
△ Less
Submitted 21 September, 2017;
originally announced September 2017.
