Showing 1–11 of 11 results for author: Leal, I

AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents

Authors: Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Isabel Leal, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, Pierre Sermanet, Quan Vuong, Stefan Welker, Fei Xia, Ted Xiao , et al. (3 additional authors not shown)

Abstract: Foundation models that incorporate language, vision, and more recently actions have revolutionized the ability to harness internet scale data to reason about useful tasks. However, one of the key challenges of training embodied foundation models is the lack of data grounded in the physical world. In this paper, we propose AutoRT, a system that leverages existing foundation models to scale up the d… ▽ More Foundation models that incorporate language, vision, and more recently actions have revolutionized the ability to harness internet scale data to reason about useful tasks. However, one of the key challenges of training embodied foundation models is the lack of data grounded in the physical world. In this paper, we propose AutoRT, a system that leverages existing foundation models to scale up the deployment of operational robots in completely unseen scenarios with minimal human supervision. AutoRT leverages vision-language models (VLMs) for scene understanding and grounding, and further uses large language models (LLMs) for proposing diverse and novel instructions to be performed by a fleet of robots. Guiding data collection by tap** into the knowledge of foundation models enables AutoRT to effectively reason about autonomy tradeoffs and safety while significantly scaling up data collection for robot learning. We demonstrate AutoRT proposing instructions to over 20 robots across multiple buildings and collecting 77k real robot episodes via both teleoperation and autonomous robot policies. We experimentally show that such "in-the-wild" data collected by AutoRT is significantly more diverse, and that AutoRT's use of LLMs allows for instruction following data collection robots that can align to human preferences. △ Less

Submitted 1 July, 2024; v1 submitted 23 January, 2024; originally announced January 2024.

Comments: 26 pages, 9 figures, ICRA 2024 VLMNM Workshop

SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention

Authors: Isabel Leal, Krzysztof Choromanski, Deepali Jain, Avinava Dubey, Jake Varley, Michael Ryoo, Yao Lu, Frederick Liu, Vikas Sindhwani, Quan Vuong, Tamas Sarlos, Ken Oslund, Karol Hausman, Kanishka Rao

Abstract: We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment. SARA-RT relies on the new method of fine-tuning proposed by us, called up-training. It converts pre-trained or already fine-tuned Transformer-based robotic policies of quadratic time complexity (includi… ▽ More We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment. SARA-RT relies on the new method of fine-tuning proposed by us, called up-training. It converts pre-trained or already fine-tuned Transformer-based robotic policies of quadratic time complexity (including massive billion-parameter vision-language-action models or VLAs), into their efficient linear-attention counterparts maintaining high quality. We demonstrate the effectiveness of SARA-RT by speeding up: (a) the class of recently introduced RT-2 models, the first VLA robotic policies pre-trained on internet-scale data, as well as (b) Point Cloud Transformer (PCT) robotic policies operating on large point clouds. We complement our results with the rigorous mathematical analysis providing deeper insight into the phenomenon of SARA. △ Less

Submitted 4 December, 2023; originally announced December 2023.

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Authors: Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, A**kya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie , et al. (267 additional authors not shown)

Abstract: Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning method… ▽ More Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website https://robotics-transformer-x.github.io. △ Less

Submitted 1 June, 2024; v1 submitted 13 October, 2023; originally announced October 2023.

Comments: Project website: https://robotics-transformer-x.github.io

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Authors: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal , et al. (29 additional authors not shown)

Abstract: We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web.… ▽ More We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web. To this end, we propose to co-fine-tune state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks, such as visual question answering. In contrast to other approaches, we propose a simple, general recipe to achieve this goal: in order to fit both natural language responses and robotic actions into the same format, we express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens. We refer to such category of models as vision-language-action models (VLA) and instantiate an example of such a model, which we call RT-2. Our extensive evaluation (6k evaluation trials) shows that our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training. This includes significantly improved generalization to novel objects, the ability to interpret commands not present in the robot training data (such as placing an object onto a particular number or icon), and the ability to perform rudimentary reasoning in response to user commands (such as picking up the smallest or largest object, or the one closest to another object). We further show that incorporating chain of thought reasoning allows RT-2 to perform multi-stage semantic reasoning, for example figuring out which object to pick up for use as an improvised hammer (a rock), or which type of drink is best suited for someone who is tired (an energy drink). △ Less

Submitted 28 July, 2023; originally announced July 2023.

Comments: Website: https://robotics-transformer.github.io/

A Multicut Approach to Compute Upper Bounds for Risk-Averse SDDP

Authors: Joaquim Dias Garcia, Iago Leal, Raphael Chabar, Mario Veiga Pereira

Abstract: Stochastic Dual Dynamic Programming (SDDP) is a widely used and fundamental algorithm for solving multistage stochastic optimization problems. Although SDDP has been frequently applied to solve risk-averse models with the Conditional Value-at-Risk (CVaR), it is known that the estimation of upper bounds is a methodological challenge, and many methods are computationally intensive. In practice, this… ▽ More Stochastic Dual Dynamic Programming (SDDP) is a widely used and fundamental algorithm for solving multistage stochastic optimization problems. Although SDDP has been frequently applied to solve risk-averse models with the Conditional Value-at-Risk (CVaR), it is known that the estimation of upper bounds is a methodological challenge, and many methods are computationally intensive. In practice, this leaves most SDDP implementations without a practical and clear stop** criterion. In this paper, we propose using the information already contained in a multicut formulation of SDDP to solve this problem with a simple and computationally efficient methodology. The multicut version of SDDP, in contrast with the typical average cut, preserves the information about which scenarios give rise to the worst costs, thus contributing to the CVaR value. We use this fact to modify the standard sampling method on the forward step so the average of multiple paths approximates the nested CVaR cost. We highlight that minimal changes are required in the SDDP algorithm and there is no additional computational burden for a fixed number of iterations. We present multiple case studies to empirically demonstrate the effectiveness of the method. First, we use a small hydrothermal dispatch test case, in which we can write the deterministic equivalent of the entire scenario tree to show that the method perfectly computes the correct objective values. Then, we present results using a standard approximation of the Brazilian operation problem and a real hydrothermal dispatch case based on data from Colombia. Our numerical experiments showed that this method consistently calculates upper bounds higher than lower bounds for those risk-averse problems and that lower bounds are improved thanks to the better exploration of the scenarios tree. △ Less

Submitted 24 July, 2023; originally announced July 2023.

RT-1: Robotics Transformer for Real-World Control at Scale

Authors: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath , et al. (26 additional authors not shown)

Abstract: By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, wher… ▽ More By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks. The project's website and videos can be found at robotics-transformer1.github.io △ Less

Submitted 11 August, 2023; v1 submitted 13 December, 2022; originally announced December 2022.

Comments: See website at robotics-transformer1.github.io

Refined Swan conductors mod p of one-dimensional Galois representations

Authors: Kazuya Kato, Isabel Leal, Takeshi Saito

Abstract: For a character of the absolute Galois group of a complete discrete valuation field, we define a lifting of the refined Swan conductor, using higher dimensional class field theory. For a character of the absolute Galois group of a complete discrete valuation field, we define a lifting of the refined Swan conductor, using higher dimensional class field theory. △ Less

Submitted 5 September, 2018; originally announced September 2018.

Comments: 39 pages

MSC Class: 14F30; 11S31

Journal ref: Nagoya Mathematical Journal 236 (2019), 134--182

Generalized Hasse-Herbrand functions in positive characteristic

Authors: Isabel Leal

Abstract: Let $L/K$ be an extension of complete discrete valuation fields of positive characteristic, and assume that the residue field of $K$ is perfect. The residue field of $L$ is not assumed to be perfect. In this paper, we show that the generalized Hasse-Herbrand function $ψ_{L/K}^{\mathrm{ab}}$ has properties similar to those of its classical counterpart. In particular, we prove that… ▽ More Let $L/K$ be an extension of complete discrete valuation fields of positive characteristic, and assume that the residue field of $K$ is perfect. The residue field of $L$ is not assumed to be perfect. In this paper, we show that the generalized Hasse-Herbrand function $ψ_{L/K}^{\mathrm{ab}}$ has properties similar to those of its classical counterpart. In particular, we prove that $ψ_{L/K}^{\mathrm{ab}}$ is continuous, piecewise linear, increasing, convex, and satisfies certain integrality properties. △ Less

Submitted 27 April, 2018; originally announced April 2018.

Comments: arXiv admin note: text overlap with arXiv:1703.00652

Zero-cycles on a product of elliptic curves over a $p$-adic field

Authors: Evangelia Gazaki, Isabel Leal

Abstract: We consider a product $X=E_1\times\cdots\times E_d$ of elliptic curves over a finite extension $K$ of $\mathbb{Q}_p$ with a combination of good or split multiplicative reduction. We assume that at most one of the elliptic curves has supersingular reduction. Under these assumptions, we prove that the Albanese kernel of $X$ is the direct sum of a finite group and a divisible group, extending work of… ▽ More We consider a product $X=E_1\times\cdots\times E_d$ of elliptic curves over a finite extension $K$ of $\mathbb{Q}_p$ with a combination of good or split multiplicative reduction. We assume that at most one of the elliptic curves has supersingular reduction. Under these assumptions, we prove that the Albanese kernel of $X$ is the direct sum of a finite group and a divisible group, extending work of Raskind and Spiess to cases that include supersingular phenomena. Our method involves studying the kernel of the cycle map $CH_0(X)/p^n\rightarrow H^{2d}_{\text{ét}}(X, μ_{p^n}^{\otimes d})$. We give specific criteria that guarantee this map is injective for every $n\geq 1$. When all curves have good ordinary reduction, we show that it suffices to extend to a specific finite extension $L$ of $K$ for these criteria to be satisfied. This extends previous work of Yamazaki and Hiranouchi. △ Less

Submitted 26 March, 2021; v1 submitted 11 February, 2018; originally announced February 2018.

Comments: 28 pages

On ramification in transcendental extensions of local fields

Authors: Isabel Leal

Abstract: Let $L/K$ be an extension of complete discrete valuation fields, and assume that the residue field of $K$ is perfect and of positive characteristic. The residue field of $L$ is not assumed to be perfect. In this paper, we prove a formula for the Swan conductor of the image of a character $χ\in H^1(K, \mathbb{Q}/\mathbb{Z})$ in $H^1(L, \mathbb{Q}/\mathbb{Z})$ for $χ$ sufficiently ramified. Furthe… ▽ More Let $L/K$ be an extension of complete discrete valuation fields, and assume that the residue field of $K$ is perfect and of positive characteristic. The residue field of $L$ is not assumed to be perfect. In this paper, we prove a formula for the Swan conductor of the image of a character $χ\in H^1(K, \mathbb{Q}/\mathbb{Z})$ in $H^1(L, \mathbb{Q}/\mathbb{Z})$ for $χ$ sufficiently ramified. Further, we define generalizations $ψ_{L/K}^{\mathrm{ab}}$ and $ψ_{L/K}^{\mathrm{AS}}$ of the classical Hasse-Herbrand $ψ$-function and prove a formula for $ψ_{L/K}^{\mathrm{ab}}(t)$ for sufficiently large $t\in \mathbb{R}$. △ Less

Submitted 28 October, 2017; v1 submitted 2 March, 2017; originally announced March 2017.

On the ramification of étale cohomology groups

Authors: Isabel Leal

Abstract: Let $K$ be a complete discrete valuation field whose residue field is perfect and of positive characteristic, let $X$ be a connected, proper scheme over $\mathcal{O}_K$, and let $U$ be the complement in $X$ of a divisor with simple normal crossings. Assume that the pair $(X,U)$ is strictly semi-stable over $\mathcal{O}_K$ of relative dimension one and $K$ is of equal characteristic. We prove tha… ▽ More Let $K$ be a complete discrete valuation field whose residue field is perfect and of positive characteristic, let $X$ be a connected, proper scheme over $\mathcal{O}_K$, and let $U$ be the complement in $X$ of a divisor with simple normal crossings. Assume that the pair $(X,U)$ is strictly semi-stable over $\mathcal{O}_K$ of relative dimension one and $K$ is of equal characteristic. We prove that, for any smooth $\ell$-adic sheaf $\mathscr{G}$ on $U$ of rank one, at most tamely ramified on the generic fiber, if the ramification of $\mathscr{G}$ is bounded by $t+$ for the logarithmic upper ramification groups of Abbes-Saito at points of codimension one of $X$, then the ramification of the étale cohomology groups with compact support of $\mathscr{G}$ is bounded by $t+$ in the same sense. △ Less

Submitted 22 June, 2016; v1 submitted 4 December, 2015; originally announced December 2015.