-
VADER: Visual Affordance Detection and Error Recovery for Multi Robot Human Collaboration
Authors:
Michael Ahn,
Montserrat Gonzalez Arenas,
Matthew Bennice,
Noah Brown,
Christine Chan,
Byron David,
Anthony Francis,
Gavin Gonzalez,
Rainer Hessmer,
Tomas Jackson,
Nikhil J Joshi,
Daniel Lam,
Tsang-Wei Edward Lee,
Alex Luong,
Sharath Maddineni,
Harsh Patel,
Jodilyn Peralta,
Jornell Quiambao,
Diego Reyes,
Rosario M Jauregui Ruano,
Dorsa Sadigh,
Pannag Sanketi,
Leila Takayama,
Pavel Vodenski,
Fei Xia
Abstract:
Robots today can exploit the rich world knowledge of large language models to chain simple behavioral skills into long-horizon tasks. However, robots often get interrupted during long-horizon tasks due to primitive skill failures and dynamic environments. We propose VADER, a plan, execute, detect framework with seeking help as a new skill that enables robots to recover and complete long-horizon ta…
▽ More
Robots today can exploit the rich world knowledge of large language models to chain simple behavioral skills into long-horizon tasks. However, robots often get interrupted during long-horizon tasks due to primitive skill failures and dynamic environments. We propose VADER, a plan, execute, detect framework with seeking help as a new skill that enables robots to recover and complete long-horizon tasks with the help of humans or other robots. VADER leverages visual question answering (VQA) modules to detect visual affordances and recognize execution errors. It then generates prompts for a language model planner (LMP) which decides when to seek help from another robot or human to recover from errors in long-horizon task execution. We show the effectiveness of VADER with two long-horizon robotic tasks. Our pilot study showed that VADER is capable of performing complex long-horizon tasks by asking for help from another robot to clear a table. Our user study showed that VADER is capable of performing complex long-horizon tasks by asking for help from a human to clear a path. We gathered feedback from people (N=19) about the performance of the VADER performance vs. a robot that did not ask for help. https://google-vader.github.io/
△ Less
Submitted 30 May, 2024; v1 submitted 24 May, 2024;
originally announced May 2024.
-
Perceptually Equivalent Resolution in Handheld Devices for Streaming Bandwidth Saving
Authors:
Mateo Cámara,
César Díaz,
Juan Casal,
Jorge Ruano,
Narciso García
Abstract:
We present the description, results, and analysis of the experiments conducted to find the equivalent resolution associated with handheld devices. That is, the resolution from which users stop perceiving quality improvements if better resolutions are presented to them in such devices. Thus, it is the maximum resolution that it is worth considering for generating and delivering video, as long as se…
▽ More
We present the description, results, and analysis of the experiments conducted to find the equivalent resolution associated with handheld devices. That is, the resolution from which users stop perceiving quality improvements if better resolutions are presented to them in such devices. Thus, it is the maximum resolution that it is worth considering for generating and delivering video, as long as sequences are not too intensively compressed. Therefore, the detection of the equivalent resolutions allows for notable savings in bandwidth consumption. Subjective assessments have been carried out on fifty subjects using a set of video sequences of very different nature and four handheld devices with a broad range of screen dimensions. The results prove that the equivalent resolution in current handheld devices is 720p as higher resolutions are not valued by users.
△ Less
Submitted 7 February, 2024;
originally announced February 2024.
-
Leveraging a realistic synthetic database to learn Shape-from-Shading for estimating the colon depth in colonoscopy images
Authors:
Josué Ruano,
Martín Gómez,
Eduardo Romero,
Antoine Manzanera
Abstract:
Colonoscopy is the choice procedure to diagnose colon and rectum cancer, from early detection of small precancerous lesions (polyps), to confirmation of malign masses. However, the high variability of the organ appearance and the complex shape of both the colon wall and structures of interest make this exploration difficult. Learned visuospatial and perceptual abilities mitigate technical limitati…
▽ More
Colonoscopy is the choice procedure to diagnose colon and rectum cancer, from early detection of small precancerous lesions (polyps), to confirmation of malign masses. However, the high variability of the organ appearance and the complex shape of both the colon wall and structures of interest make this exploration difficult. Learned visuospatial and perceptual abilities mitigate technical limitations in clinical practice by proper estimation of the intestinal depth. This work introduces a novel methodology to estimate colon depth maps in single frames from monocular colonoscopy videos. The generated depth map is inferred from the shading variation of the colon wall with respect to the light source, as learned from a realistic synthetic database. Briefly, a classic convolutional neural network architecture is trained from scratch to estimate the depth map, improving sharp depth estimations in haustral folds and polyps by a custom loss function that minimizes the estimation error in edges and curvatures. The network was trained by a custom synthetic colonoscopy database herein constructed and released, composed of 248,400 frames (47 videos), with depth annotations at the level of pixels. This collection comprehends 5 subsets of videos with progressively higher levels of visual complexity. Evaluation of the depth estimation with the synthetic database reached a threshold accuracy of 95.65%, and a mean-RMSE of 0.451 cm, while a qualitative assessment with a real database showed consistent depth estimations, visually evaluated by the expert gastroenterologist coauthoring this paper. Finally, the method achieved competitive performance with respect to another state-of-the-art method using a public synthetic database and comparable results in a set of images with other five state-of-the-art methods.
△ Less
Submitted 8 November, 2023;
originally announced November 2023.
-
Fourier-Mukai transform for fine compactified Prym varieties
Authors:
Emilio Franco,
Robert Hanson,
João Ruano
Abstract:
Consider a finite covering $β: C \to X$ of a smooth projective curve $X$ by a reduced, projective, planar curve $C$. Associated to two general polarizations on $C$, $q$ and $q'$, one can construct the corresponding compactified Prym varieties $\overline{\mathrm{P}}_β(q)$ and $\overline{\mathrm{P}}_β(q')$. Consider $Γ$ to be the group of line bundles whose torsion coincides with the order of $β$. I…
▽ More
Consider a finite covering $β: C \to X$ of a smooth projective curve $X$ by a reduced, projective, planar curve $C$. Associated to two general polarizations on $C$, $q$ and $q'$, one can construct the corresponding compactified Prym varieties $\overline{\mathrm{P}}_β(q)$ and $\overline{\mathrm{P}}_β(q')$. Consider $Γ$ to be the group of line bundles whose torsion coincides with the order of $β$. In this article we construct a Fourier-Mukai transform between the derived categories of $\overline{\mathrm{P}}_β(q)$ and the $Γ$-equivariant derived category of $\overline{\mathrm{P}}_β(q')$. Hence, we obtain a derived equivalence between the $\mathrm{SL}(n,\mathbb{C})$-Hitchin fibre and its associated $\mathrm{PGL}(n,\mathbb{C})$-Hitchin fibre for a dense class of singular spectral curves. Our work then provides the extension of the Fourier-Mukai transform constructed by Arinkin and Melo-Rapagnetta-Viviani, which corresponds to autoduality of $\mathrm{GL}(n,\mathbb{C})$-Hitchin fibres in this class of singular spectral curves.
△ Less
Submitted 28 February, 2024; v1 submitted 24 June, 2022;
originally announced June 2022.
-
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Authors:
Michael Ahn,
Anthony Brohan,
Noah Brown,
Yevgen Chebotar,
Omar Cortes,
Byron David,
Chelsea Finn,
Chuyuan Fu,
Keerthana Gopalakrishnan,
Karol Hausman,
Alex Herzog,
Daniel Ho,
Jasmine Hsu,
Julian Ibarz,
Brian Ichter,
Alex Irpan,
Eric Jang,
Rosario Jauregui Ruano,
Kyle Jeffrey,
Sally Jesmonth,
Nikhil J Joshi,
Ryan Julian,
Dmitry Kalashnikov,
Yuheng Kuang,
Kuang-Huei Lee
, et al. (20 additional authors not shown)
Abstract:
Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embo…
▽ More
Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model's "hands and eyes," while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project's website and the video can be found at https://say-can.github.io/.
△ Less
Submitted 16 August, 2022; v1 submitted 4 April, 2022;
originally announced April 2022.
-
Spectral Vector Beams for High-Speed Spectroscopic Measurements
Authors:
Lea Kopf,
Juan R. Deop Ruano,
Markus Hiekkamäki,
Timo Stolt,
Mikko J. Huttunen,
Frédéric Bouchard,
Robert Fickler
Abstract:
Structured light harnessing multiple degrees of freedom has become a powerful approach to use complex states of light in fundamental studies and applications. Here, we investigate the light field of an ultrafast laser beam with a wavelength-depended polarization state, a beam we term spectral vector beam. We demonstrate a simple technique to generate and tune such structured beams and demonstrate…
▽ More
Structured light harnessing multiple degrees of freedom has become a powerful approach to use complex states of light in fundamental studies and applications. Here, we investigate the light field of an ultrafast laser beam with a wavelength-depended polarization state, a beam we term spectral vector beam. We demonstrate a simple technique to generate and tune such structured beams and demonstrate their spectroscopic capabilities. By only measuring the polarization state using fast photodetectors, it is possible to track pulse-to-pulse changes in the frequency spectrum caused by, e.g. narrowband transmission or absorption. In our experiments, we reach read-out rates of around 6 MHz, which is limited by our technical ability to modulate the spectrum and can in principle reach GHz read-out rates. In simulations we extend the spectral range to more than 1000 nm by using a supercontinuum light source, thereby paving the way to various applications requiring high-speed spectroscopic measurements.
△ Less
Submitted 17 March, 2021; v1 submitted 4 March, 2021;
originally announced March 2021.
-
A general method to compute numerical dispersion errors and its application to stretched meshes
Authors:
J. Ruano,
A. Baez Vidal,
F. X. Trias,
J. Rigola
Abstract:
This article presents a new spectral analysis approach for dispersion error and a methodology to numerically evaluate it. In practice, this new analysis allows the numerical study of dispersion errors on all types of mesh and for multiple dimensions. Nonetheless, when mesh uniformity and one-dimensionality assumptions are imposed as in the classical method, the results of this new technique coinci…
▽ More
This article presents a new spectral analysis approach for dispersion error and a methodology to numerically evaluate it. In practice, this new analysis allows the numerical study of dispersion errors on all types of mesh and for multiple dimensions. Nonetheless, when mesh uniformity and one-dimensionality assumptions are imposed as in the classical method, the results of this new technique coincide with those of the classic method. We establish the theoretical basis of the approach, derive a numerical methodology to evaluate dispersion errors and assess the method after a set of numerical tests on non-uniform stretched meshes.
△ Less
Submitted 17 September, 2019;
originally announced September 2019.
