-
Follow Anything: Open-set detection, tracking, and following in real-time
Authors:
Alaa Maalouf,
Ninad Jadhav,
Krishna Murthy Jatavallabhula,
Makram Chahine,
Daniel M. Vogt,
Robert J. Wood,
Antonio Torralba,
Daniela Rus
Abstract:
Tracking and following objects of interest is critical to several robotics use cases, ranging from industrial automation to logistics and warehousing, to healthcare and security. In this paper, we present a robotic system to detect, track, and follow any object in real-time. Our approach, dubbed ``follow anything'' (FAn), is an open-vocabulary and multimodal model -- it is not restricted to concep…
▽ More
Tracking and following objects of interest is critical to several robotics use cases, ranging from industrial automation to logistics and warehousing, to healthcare and security. In this paper, we present a robotic system to detect, track, and follow any object in real-time. Our approach, dubbed ``follow anything'' (FAn), is an open-vocabulary and multimodal model -- it is not restricted to concepts seen at training time and can be applied to novel classes at inference time using text, images, or click queries. Leveraging rich visual descriptors from large-scale pre-trained models (foundation models), FAn can detect and segment objects by matching multimodal queries (text, images, clicks) against an input image sequence. These detected and segmented objects are tracked across image frames, all while accounting for occlusion and object re-emergence. We demonstrate FAn on a real-world robotic system (a micro aerial vehicle) and report its ability to seamlessly follow the objects of interest in a real-time control loop. FAn can be deployed on a laptop with a lightweight (6-8 GB) graphics card, achieving a throughput of 6-20 frames per second. To enable rapid adoption, deployment, and extensibility, we open-source all our code on our project webpage at https://github.com/alaamaalouf/FollowAnything . We also encourage the reader to watch our 5-minutes explainer video in this https://www.youtube.com/watch?v=6Mgt3EPytrw .
△ Less
Submitted 9 February, 2024; v1 submitted 10 August, 2023;
originally announced August 2023.
-
Cetacean Translation Initiative: a roadmap to deciphering the communication of sperm whales
Authors:
Jacob Andreas,
Gašper Beguš,
Michael M. Bronstein,
Roee Diamant,
Denley Delaney,
Shane Gero,
Shafi Goldwasser,
David F. Gruber,
Sarah de Haas,
Peter Malkin,
Roger Payne,
Giovanni Petri,
Daniela Rus,
Pratyusha Sharma,
Dan Tchernov,
Pernille Tønnesen,
Antonio Torralba,
Daniel Vogt,
Robert J. Wood
Abstract:
The past decade has witnessed a groundbreaking rise of machine learning for human language analysis, with current methods capable of automatically accurately recovering various aspects of syntax and semantics - including sentence structure and grounded word meaning - from large data collections. Recent research showed the promise of such tools for analyzing acoustic communication in nonhuman speci…
▽ More
The past decade has witnessed a groundbreaking rise of machine learning for human language analysis, with current methods capable of automatically accurately recovering various aspects of syntax and semantics - including sentence structure and grounded word meaning - from large data collections. Recent research showed the promise of such tools for analyzing acoustic communication in nonhuman species. We posit that machine learning will be the cornerstone of future collection, processing, and analysis of multimodal streams of data in animal communication studies, including bioacoustic, behavioral, biological, and environmental data. Cetaceans are unique non-human model species as they possess sophisticated acoustic communications, but utilize a very different encoding system that evolved in an aquatic rather than terrestrial medium. Sperm whales, in particular, with their highly-developed neuroanatomical features, cognitive abilities, social structures, and discrete click-based encoding make for an excellent starting point for advanced machine learning tools that can be applied to other animals in the future. This paper details a roadmap toward this goal based on currently existing technology and multidisciplinary scientific community effort. We outline the key elements required for the collection and processing of massive bioacoustic data of sperm whales, detecting their basic communication units and language-like higher-level structures, and validating these models through interactive playback experiments. The technological capabilities developed by such an undertaking are likely to yield cross-applications and advancements in broader communities investigating non-human communication and animal behavioral research.
△ Less
Submitted 17 April, 2021;
originally announced April 2021.
-
Scaling down an insect-size microrobot, HAMR-VI into HAMR-Jr
Authors:
Kaushik Jayaram,
Jennifer Shum,
Samantha Castellanos,
E. Farrell Helbling,
Robert J. Wood
Abstract:
Here we present HAMR-Jr, a \SI{22.5}{\milli\meter}, \SI{320}{\milli\gram} quadrupedal microrobot. With eight independently actuated degrees of freedom, HAMR-Jr is, to our knowledge, the most mechanically dexterous legged robot at its scale and is capable of high-speed locomotion (\SI{13.91}{bodylengths~\second^{-1}}) at a variety of stride frequencies (\SI{1}{}-\SI{200}{\hertz}) using multiple gai…
▽ More
Here we present HAMR-Jr, a \SI{22.5}{\milli\meter}, \SI{320}{\milli\gram} quadrupedal microrobot. With eight independently actuated degrees of freedom, HAMR-Jr is, to our knowledge, the most mechanically dexterous legged robot at its scale and is capable of high-speed locomotion (\SI{13.91}{bodylengths~\second^{-1}}) at a variety of stride frequencies (\SI{1}{}-\SI{200}{\hertz}) using multiple gaits. We achieved this using a design and fabrication process that is flexible, allowing scaling with minimum changes to our workflow. We further characterized HAMR-Jr's open-loop locomotion and compared it with the larger scale HAMR-VI microrobot to demonstrate the effectiveness of scaling laws in predicting running performance.
△ Less
Submitted 6 March, 2020;
originally announced March 2020.
-
Contact-Implicit Optimization of Locomotion Trajectories for a Quadrupedal Microrobot
Authors:
Neel Doshi,
Kaushik Jayaram,
Benjamin Goldberg,
Zachary Manchester,
Robert J. Wood,
Scott Kuindersma
Abstract:
Planning locomotion trajectories for legged microrobots is challenging because of their complex morphology, high frequency passive dynamics, and discontinuous contact interactions with their environment. Consequently, such research is often driven by time-consuming experimental methods. As an alternative, we present a framework for systematically modeling, planning, and controlling legged microrob…
▽ More
Planning locomotion trajectories for legged microrobots is challenging because of their complex morphology, high frequency passive dynamics, and discontinuous contact interactions with their environment. Consequently, such research is often driven by time-consuming experimental methods. As an alternative, we present a framework for systematically modeling, planning, and controlling legged microrobots. We develop a three-dimensional dynamic model of a 1.5 gram quadrupedal microrobot with complexity (e.g., number of degrees of freedom) similar to larger-scale legged robots. We then adapt a recently developed variational contact-implicit trajectory optimization method to generate feasible whole-body locomotion plans for this microrobot, and we demonstrate that these plans can be tracked with simple joint-space controllers. We plan and execute periodic gaits at multiple stride frequencies and on various surfaces. These gaits achieve high per-cycle velocities, including a maximum of 10.87 mm/cycle, which is 15% faster than previously measured velocities for this microrobot. Furthermore, we plan and execute a vertical jump of 9.96 mm, which is 78% of the microrobot's center-of-mass height. To the best of our knowledge, this is the first end-to-end demonstration of planning and tracking whole-body dynamic locomotion on a millimeter-scale legged microrobot.
△ Less
Submitted 25 January, 2019;
originally announced January 2019.
-
Effective Locomotion at Multiple Stride Frequencies Using Proprioceptive Feedback on a Legged Microrobot
Authors:
Neel Doshi,
Kaushik Jayaram,
Samantha Castellanos,
Scott Kuindersma,
Robert J Wood
Abstract:
Limitations in actuation, sensing, and computation have forced small legged robots to rely on carefully tuned, mechanically mediated leg trajectories for effective locomotion. Recent advances in manufacturing, however, have enabled the development of small legged robots capable of operation at multiple stride frequencies using multi-degree-of-freedom leg trajectories. Proprioceptive sensing and co…
▽ More
Limitations in actuation, sensing, and computation have forced small legged robots to rely on carefully tuned, mechanically mediated leg trajectories for effective locomotion. Recent advances in manufacturing, however, have enabled the development of small legged robots capable of operation at multiple stride frequencies using multi-degree-of-freedom leg trajectories. Proprioceptive sensing and control is key to extending the capabilities of these robots to a broad range of operating conditions. In this work, we use concomitant sensing for piezoelectric actuation with a computationally efficient framework for estimation and control of leg trajectories on a quadrupedal microrobot. We demonstrate accurate position estimation (< 16% root-mean-square error) and control (16% root-mean-square tracking error) during locomotion across a wide range of stride frequencies (10-50 Hz). This capability enables the exploration of two bioinspired parametric leg trajectories designed to reduce leg slip and increase locomotion performance (e.g., speed, cost-of-transport, etc.). Using this approach, we demonstrate high performance locomotion at stride frequencies of (10-30 Hz) where the robot's natural dynamics result in poor open-loop locomotion. Furthermore, we validate the biological hypotheses that inspired the our trajectories and identify regions of highly dynamic locomotion, low cost-of-transport (3.33), and minimal leg slippage (< 10%).
△ Less
Submitted 3 June, 2019; v1 submitted 24 January, 2019;
originally announced January 2019.
-
Framing Human-Robot Task Communication as a POMDP
Authors:
Mark P. Woodward,
Robert J. Wood
Abstract:
As general purpose robots become more capable, pre-programming of all tasks at the factory will become less practical. We would like for non-technical human owners to be able to communicate, through interaction with their robot, the details of a new task; we call this interaction "task communication". During task communication the robot must infer the details of the task from unstructured human si…
▽ More
As general purpose robots become more capable, pre-programming of all tasks at the factory will become less practical. We would like for non-technical human owners to be able to communicate, through interaction with their robot, the details of a new task; we call this interaction "task communication". During task communication the robot must infer the details of the task from unstructured human signals and it must choose actions that facilitate this inference. In this paper we propose the use of a partially observable Markov decision process (POMDP) for representing the task communication problem; with the unobservable task details and unobservable intentions of the human teacher captured in the state, with all signals from the human represented as observations, and with the cost function chosen to penalize uncertainty. We work through an example representation of task communication as a POMDP, and present results from a user experiment on an interactive virtual robot, compared with a human controlled virtual robot, for a task involving a single object movement and binary approval input from the teacher. The results suggest that the proposed POMDP representation produces robots that are robust to teacher error, that can accurately infer task details, and that are perceived to be intelligent.
△ Less
Submitted 1 April, 2012;
originally announced April 2012.
-
Learning from Humans as an I-POMDP
Authors:
Mark P. Woodward,
Robert J. Wood
Abstract:
The interactive partially observable Markov decision process (I-POMDP) is a recently developed framework which extends the POMDP to the multi-agent setting by including agent models in the state space. This paper argues for formulating the problem of an agent learning interactively from a human teacher as an I-POMDP, where the agent \emph{programming} to be learned is captured by random variables…
▽ More
The interactive partially observable Markov decision process (I-POMDP) is a recently developed framework which extends the POMDP to the multi-agent setting by including agent models in the state space. This paper argues for formulating the problem of an agent learning interactively from a human teacher as an I-POMDP, where the agent \emph{programming} to be learned is captured by random variables in the agent's state space, all \emph{signals} from the human teacher are treated as observed random variables, and the human teacher, modeled as a distinct agent, is explicitly represented in the agent's state space. The main benefits of this approach are: i. a principled action selection mechanism, ii. a principled belief update mechanism, iii. support for the most common teacher \emph{signals}, and iv. the anticipated production of complex beneficial interactions. The proposed formulation, its benefits, and several open questions are presented.
△ Less
Submitted 1 April, 2012;
originally announced April 2012.
