-
Idiolect: A Reconfigurable Voice Coding Assistant
Authors:
Breandan Considine,
Nicholas Albion,
Xujie Si
Abstract:
This paper presents Idiolect, an open source (https://github.com/OpenASR/idiolect) IDE plugin for voice coding and a novel approach to building bots that allows for users to define custom commands on-the-fly. Unlike traditional chatbots, Idiolect does not pretend to be an omniscient virtual assistant but rather a reconfigurable voice programming system that empowers users to create their own comma…
▽ More
This paper presents Idiolect, an open source (https://github.com/OpenASR/idiolect) IDE plugin for voice coding and a novel approach to building bots that allows for users to define custom commands on-the-fly. Unlike traditional chatbots, Idiolect does not pretend to be an omniscient virtual assistant but rather a reconfigurable voice programming system that empowers users to create their own commands and actions dynamically, without rebuilding or restarting the application. We offer an experience report describing the tool itself, illustrate some example use cases, and reflect on several lessons learned during the tool's development.
△ Less
Submitted 4 May, 2023;
originally announced May 2023.
-
gradSim: Differentiable simulation for system identification and visuomotor control
Authors:
Krishna Murthy Jatavallabhula,
Miles Macklin,
Florian Golemo,
Vikram Voleti,
Linda Petrini,
Martin Weiss,
Breandan Considine,
Jerome Parent-Levesque,
Kevin Xie,
Kenny Erleben,
Liam Paull,
Florian Shkurti,
Derek Nowrouzezahrai,
Sanja Fidler
Abstract:
We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable…
▽ More
We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. We present gradSim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Moreover, our unified computation graph -- spanning from the dynamics and through the rendering process -- enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.
△ Less
Submitted 6 April, 2021;
originally announced April 2021.
-
The AI Driving Olympics at NeurIPS 2018
Authors:
Julian Zilly,
Jacopo Tani,
Breandan Considine,
Bhairav Mehta,
Andrea F. Daniele,
Manfred Diaz,
Gianmarco Bernasconi,
Claudio Ruch,
Jan Hakenberg,
Florian Golemo,
A. Kirsten Bowser,
Matthew R. Walter,
Ruslan Hristov,
Sunil Mallya,
Emilio Frazzoli,
Andrea Censi,
Liam Paull
Abstract:
Despite recent breakthroughs, the ability of deep learning and reinforcement learning to outperform traditional approaches to control physically embodied robotic agents remains largely unproven. To help bridge this gap, we created the 'AI Driving Olympics' (AI-DO), a competition with the objective of evaluating the state of the art in machine learning and artificial intelligence for mobile robotic…
▽ More
Despite recent breakthroughs, the ability of deep learning and reinforcement learning to outperform traditional approaches to control physically embodied robotic agents remains largely unproven. To help bridge this gap, we created the 'AI Driving Olympics' (AI-DO), a competition with the objective of evaluating the state of the art in machine learning and artificial intelligence for mobile robotics. Based on the simple and well specified autonomous driving and navigation environment called 'Duckietown', AI-DO includes a series of tasks of increasing complexity -- from simple lane-following to fleet management. For each task, we provide tools for competitors to use in the form of simulators, logs, code templates, baseline implementations and low-cost access to robotic hardware. We evaluate submissions in simulation online, on standardized hardware environments, and finally at the competition event. The first AI-DO, AI-DO 1, occurred at the Neural Information Processing Systems (NeurIPS) conference in December 2018. The results of AI-DO 1 highlight the need for better benchmarks, which are lacking in robotics, as well as improved mechanisms to bridge the gap between simulation and reality.
△ Less
Submitted 6 March, 2019;
originally announced March 2019.
-
Multi-objective training of Generative Adversarial Networks with multiple discriminators
Authors:
Isabela Albuquerque,
João Monteiro,
Thang Doan,
Breandan Considine,
Tiago Falk,
Ioannis Mitliagkas
Abstract:
Recent literature has demonstrated promising results for training Generative Adversarial Networks by employing a set of discriminators, in contrast to the traditional game involving one generator against a single adversary. Such methods perform single-objective optimization on some simple consolidation of the losses, e.g. an arithmetic average. In this work, we revisit the multiple-discriminator s…
▽ More
Recent literature has demonstrated promising results for training Generative Adversarial Networks by employing a set of discriminators, in contrast to the traditional game involving one generator against a single adversary. Such methods perform single-objective optimization on some simple consolidation of the losses, e.g. an arithmetic average. In this work, we revisit the multiple-discriminator setting by framing the simultaneous minimization of losses provided by different models as a multi-objective optimization problem. Specifically, we evaluate the performance of multiple gradient descent and the hypervolume maximization algorithm on a number of different datasets. Moreover, we argue that the previously proposed methods and hypervolume maximization can all be seen as variations of multiple gradient descent in which the update direction can be computed efficiently. Our results indicate that hypervolume maximization presents a better compromise between sample quality and computational cost than previous methods.
△ Less
Submitted 24 January, 2019;
originally announced January 2019.
-
Deep Pepper: Expert Iteration based Chess agent in the Reinforcement Learning Setting
Authors:
Sai Krishna G. V.,
Kyle Goyette,
Ahmad Chamseddine,
Breandan Considine
Abstract:
An almost-perfect chess playing agent has been a long standing challenge in the field of Artificial Intelligence. Some of the recent advances demonstrate we are approaching that goal. In this project, we provide methods for faster training of self-play style algorithms, mathematical details of the algorithm used, various potential future directions, and discuss most of the relevant work in the are…
▽ More
An almost-perfect chess playing agent has been a long standing challenge in the field of Artificial Intelligence. Some of the recent advances demonstrate we are approaching that goal. In this project, we provide methods for faster training of self-play style algorithms, mathematical details of the algorithm used, various potential future directions, and discuss most of the relevant work in the area of computer chess. Deep Pepper uses embedded knowledge to accelerate the training of the chess engine over a "tabula rasa" system such as Alpha Zero. We also release our code to promote further research.
△ Less
Submitted 17 October, 2018; v1 submitted 2 June, 2018;
originally announced June 2018.
