-
SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research
Authors:
Dan Bohus,
Sean Andrist,
Nick Saw,
Ann Paradiso,
Ishani Chakraborty,
Mahdi Rad
Abstract:
We introduce an open-source system called SIGMA (short for "Situated Interactive Guidance, Monitoring, and Assistance") as a platform for conducting research on task-assistive agents in mixed-reality scenarios. The system leverages the sensing and rendering affordances of a head-mounted mixed-reality device in conjunction with large language and vision models to guide users step by step through pr…
▽ More
We introduce an open-source system called SIGMA (short for "Situated Interactive Guidance, Monitoring, and Assistance") as a platform for conducting research on task-assistive agents in mixed-reality scenarios. The system leverages the sensing and rendering affordances of a head-mounted mixed-reality device in conjunction with large language and vision models to guide users step by step through procedural tasks. We present the system's core capabilities, discuss its overall design and implementation, and outline directions for future research enabled by the system. SIGMA is easily extensible and provides a useful basis for future research at the intersection of mixed reality and AI. By open-sourcing an end-to-end implementation, we aim to lower the barrier to entry, accelerate research in this space, and chart a path towards community-driven end-to-end evaluation of large language, vision, and multimodal models in the context of real-world interactive applications.
△ Less
Submitted 16 May, 2024;
originally announced May 2024.
-
HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
Authors:
Xin Wang,
Taein Kwon,
Mahdi Rad,
Bowen Pan,
Ishani Chakraborty,
Sean Andrist,
Dan Bohus,
Ashley Feniello,
Bugra Tekin,
Felipe Vieira Frujeri,
Neel Joshi,
Marc Pollefeys
Abstract:
Building an interactive AI assistant that can perceive, reason, and collaborate with humans in the real world has been a long-standing pursuit in the AI community. This work is part of a broader research effort to develop intelligent agents that can interactively guide humans through performing tasks in the physical world. As a first step in this direction, we introduce HoloAssist, a large-scale e…
▽ More
Building an interactive AI assistant that can perceive, reason, and collaborate with humans in the real world has been a long-standing pursuit in the AI community. This work is part of a broader research effort to develop intelligent agents that can interactively guide humans through performing tasks in the physical world. As a first step in this direction, we introduce HoloAssist, a large-scale egocentric human interaction dataset, where two people collaboratively complete physical manipulation tasks. The task performer executes the task while wearing a mixed-reality headset that captures seven synchronized data streams. The task instructor watches the performer's egocentric video in real time and guides them verbally. By augmenting the data with action and conversational annotations and observing the rich behaviors of various participants, we present key insights into how human assistants correct mistakes, intervene in the task completion procedure, and ground their instructions to the environment. HoloAssist spans 166 hours of data captured by 350 unique instructor-performer pairs. Furthermore, we construct and present benchmarks on mistake detection, intervention type prediction, and hand forecasting, along with detailed analysis. We expect HoloAssist will provide an important resource for building AI assistants that can fluidly collaborate with humans in the real world. Data can be downloaded at https://holoassist.github.io/.
△ Less
Submitted 29 September, 2023;
originally announced September 2023.
-
Hardness of the Maximum Independent Set Problem on Unit-Disk Graphs and Prospects for Quantum Speedups
Authors:
Ruben S. Andrist,
Martin J. A. Schuetz,
Pierre Minssen,
Romina Yalovetzky,
Shouvanik Chakrabarti,
Dylan Herman,
Niraj Kumar,
Grant Salton,
Ruslan Shaydulin,
Yue Sun,
Marco Pistoia,
Helmut G. Katzgraber
Abstract:
Rydberg atom arrays are among the leading contenders for the demonstration of quantum speedups. Motivated by recent experiments with up to 289 qubits [Ebadi et al., Science 376, 1209 (2022)] we study the maximum independent set problem on unit-disk graphs with a broader range of classical solvers beyond the scope of the original paper. We carry out extensive numerical studies and assess problem ha…
▽ More
Rydberg atom arrays are among the leading contenders for the demonstration of quantum speedups. Motivated by recent experiments with up to 289 qubits [Ebadi et al., Science 376, 1209 (2022)] we study the maximum independent set problem on unit-disk graphs with a broader range of classical solvers beyond the scope of the original paper. We carry out extensive numerical studies and assess problem hardness, using both exact and heuristic algorithms. We find that quasi-planar instances with Union-Jack-like connectivity can be solved to optimality for up to thousands of nodes within minutes, with both custom and generic commercial solvers on commodity hardware, without any instance-specific fine-tuning. We also perform a scaling analysis, showing that by relaxing the constraints on the classical simulated annealing algorithms considered in Ebadi et al., our implementation is competitive with the quantum algorithms. Conversely, instances with larger connectivity or less structure are shown to display a time-to-solution potentially orders of magnitudes larger. Based on these results we propose protocols to systematically tune problem hardness, motivating experiments with Rydberg atom arrays on instances orders of magnitude harder (for established classical solvers) than previously studied.
△ Less
Submitted 9 January, 2024; v1 submitted 18 July, 2023;
originally announced July 2023.
-
Platform for Situated Intelligence
Authors:
Dan Bohus,
Sean Andrist,
Ashley Feniello,
Nick Saw,
Mihai Jalobeanu,
Patrick Sweeney,
Anne Loomis Thompson,
Eric Horvitz
Abstract:
We introduce Platform for Situated Intelligence, an open-source framework created to support the rapid development and study of multimodal, integrative-AI systems. The framework provides infrastructure for sensing, fusing, and making inferences from temporal streams of data across different modalities, a set of tools that enable visualization and debugging, and an ecosystem of components that enca…
▽ More
We introduce Platform for Situated Intelligence, an open-source framework created to support the rapid development and study of multimodal, integrative-AI systems. The framework provides infrastructure for sensing, fusing, and making inferences from temporal streams of data across different modalities, a set of tools that enable visualization and debugging, and an ecosystem of components that encapsulate a variety of perception and processing technologies. These assets jointly provide the means for rapidly constructing and refining multimodal, integrative-AI systems, while retaining the efficiency and performance characteristics required for deployment in open-world settings.
△ Less
Submitted 29 March, 2021;
originally announced March 2021.
-
Accelerating the Development of Multimodal, Integrative-AI Systems with Platform for Situated Intelligence
Authors:
Sean Andrist,
Dan Bohus
Abstract:
We describe Platform for Situated Intelligence, an open-source framework for multimodal, integrative-AI systems. The framework provides infrastructure, tools, and components that enable and accelerate the development of applications that process multimodal streams of data and in which timing is critical. The framework is particularly well-suited for develo** physically situated interactive syste…
▽ More
We describe Platform for Situated Intelligence, an open-source framework for multimodal, integrative-AI systems. The framework provides infrastructure, tools, and components that enable and accelerate the development of applications that process multimodal streams of data and in which timing is critical. The framework is particularly well-suited for develo** physically situated interactive systems that perceive and reason about their surroundings in order to better interact with people, such as social robots, virtual assistants, smart meeting rooms, etc. In this paper, we provide a brief, high-level overview of the framework and its main affordances, and discuss its implications for HRI.
△ Less
Submitted 12 October, 2020;
originally announced October 2020.
-
REFORM: Recognizing F-formations for Social Robots
Authors:
Hooman Hedayati,
Annika Muehlbradt,
Daniel J. Szafir,
Sean Andrist
Abstract:
Recognizing and understanding conversational groups, or F-formations, is a critical task for situated agents designed to interact with humans. F-formations contain complex structures and dynamics, yet are used intuitively by people in everyday face-to-face conversations. Prior research exploring ways of identifying F-formations has largely relied on heuristic algorithms that may not capture the ri…
▽ More
Recognizing and understanding conversational groups, or F-formations, is a critical task for situated agents designed to interact with humans. F-formations contain complex structures and dynamics, yet are used intuitively by people in everyday face-to-face conversations. Prior research exploring ways of identifying F-formations has largely relied on heuristic algorithms that may not capture the rich dynamic behaviors employed by humans. We introduce REFORM (REcognize F-FORmations with Machine learning), a data-driven approach for detecting F-formations given human and agent positions and orientations. REFORM decomposes the scene into all possible pairs and then reconstructs F-formations with a voting-based scheme. We evaluated our approach across three datasets: the SALSA dataset, a newly collected human-only dataset, and a new set of acted human-robot scenarios, and found that REFORM yielded improved accuracy over a state-of-the-art F-formation detection algorithm. We also introduce symmetry and tightness as quantitative measures to characterize F-formations. Supplementary video: https://youtu.be/Fp7ETdkKvdA , Dataset available at: github.com/cu-ironlab/Babble
△ Less
Submitted 17 August, 2020;
originally announced August 2020.
-
Metareasoning in Modular Software Systems: On-the-Fly Configuration using Reinforcement Learning with Rich Contextual Representations
Authors:
Aditya Modi,
Debadeepta Dey,
Alekh Agarwal,
Adith Swaminathan,
Besmira Nushi,
Sean Andrist,
Eric Horvitz
Abstract:
Assemblies of modular subsystems are being pressed into service to perform sensing, reasoning, and decision making in high-stakes, time-critical tasks in such areas as transportation, healthcare, and industrial automation. We address the opportunity to maximize the utility of an overall computing system by employing reinforcement learning to guide the configuration of the set of interacting module…
▽ More
Assemblies of modular subsystems are being pressed into service to perform sensing, reasoning, and decision making in high-stakes, time-critical tasks in such areas as transportation, healthcare, and industrial automation. We address the opportunity to maximize the utility of an overall computing system by employing reinforcement learning to guide the configuration of the set of interacting modules that comprise the system. The challenge of doing system-wide optimization is a combinatorial problem. Local attempts to boost the performance of a specific module by modifying its configuration often leads to losses in overall utility of the system's performance as the distribution of inputs to downstream modules changes drastically. We present metareasoning techniques which consider a rich representation of the input, monitor the state of the entire pipeline, and adjust the configuration of modules on-the-fly so as to maximize the utility of a system's operation. We show significant improvement in both real-world and synthetic pipelines across a variety of reinforcement learning techniques.
△ Less
Submitted 12 May, 2019;
originally announced May 2019.
-
Error tolerance of topological codes with independent bit-flip and measurement errors
Authors:
Ruben S. Andrist,
Helmut G. Katzgraber,
H. Bombin,
M. A. Martin-Delgado
Abstract:
Topological quantum error correction codes are currently among the most promising candidates for efficiently dealing with the decoherence effects inherently present in quantum devices. Numerically, their theoretical error threshold can be calculated by map** the underlying quantum problem to a related classical statistical-mechanical spin system with quenched disorder. Here, we present results f…
▽ More
Topological quantum error correction codes are currently among the most promising candidates for efficiently dealing with the decoherence effects inherently present in quantum devices. Numerically, their theoretical error threshold can be calculated by map** the underlying quantum problem to a related classical statistical-mechanical spin system with quenched disorder. Here, we present results for the general fault-tolerant regime, where we consider both qubit and measurement errors. However, unlike in previous studies, here we vary the strength of the different error sources independently. Our results highlight peculiar differences between toric and color codes. This study complements previous results published in New J. Phys. 13, 083006 (2011).
△ Less
Submitted 12 July, 2016; v1 submitted 29 March, 2016;
originally announced March 2016.
-
Erratum: Glassy Chimeras Could Be Blind to Quantum Speedup. . . [Phys. Rev. X 4, 021008 (2014)]
Authors:
Martin Weigel,
Helmut G. Katzgraber,
Jonathan Machta,
Firas Hamze,
Ruben S. Andrist
Abstract:
Erratum to Phys. Rev. X 4, 021008 (2014): The critical exponent associated with the ferromagnetic susceptibility was computed incorrectly. Furthermore, Ising ferromagnets on the Chimera topology have the same universality class as two-dimensional Ising ferromagnets.
Erratum to Phys. Rev. X 4, 021008 (2014): The critical exponent associated with the ferromagnetic susceptibility was computed incorrectly. Furthermore, Ising ferromagnets on the Chimera topology have the same universality class as two-dimensional Ising ferromagnets.
△ Less
Submitted 31 January, 2015; v1 submitted 19 December, 2014;
originally announced December 2014.
-
Error Thresholds for Abelian Quantum Double Models: Increasing the bit-flip Stability of Topological Quantum Memory
Authors:
Ruben S. Andrist,
James R. Wootton,
Helmut G. Katzgraber
Abstract:
Current approaches for building quantum computing devices focus on two-level quantum systems which nicely mimic the concept of a classical bit, albeit enhanced with additional quantum properties. However, rather than artificially limiting the number of states to two, the use of d-level quantum systems (qudits) could provide advantages for quantum information processing. Among other merits, it has…
▽ More
Current approaches for building quantum computing devices focus on two-level quantum systems which nicely mimic the concept of a classical bit, albeit enhanced with additional quantum properties. However, rather than artificially limiting the number of states to two, the use of d-level quantum systems (qudits) could provide advantages for quantum information processing. Among other merits, it has recently been shown that multi-level quantum systems can offer increased stability to external disturbances - a key problem in current technologies. In this study we demonstrate that topological quantum memories built from qudits, also known as abelian quantum double models, exhibit a substantially increased resilience to noise. That is, even when taking into account the multitude of errors possible for multi-level quantum systems, topological quantum error correction codes employing qudits can sustain a larger error rate than their two-level counterparts. In particular, we find strong numerical evidence that the thresholds of these error-correction codes are given by the hashing bound. Considering the significantly increased error thresholds attained, this might well outweigh the added complexity of engineering and controlling higher dimensional quantum systems.
△ Less
Submitted 23 June, 2014;
originally announced June 2014.
-
Glassy Chimeras could be blind to quantum speedup: Designing better benchmarks for quantum annealing machines
Authors:
Helmut G. Katzgraber,
Firas Hamze,
Ruben S. Andrist
Abstract:
Recently, a programmable quantum annealing machine has been built that minimizes the cost function of hard optimization problems by adiabatically quenching quantum fluctuations. Tests performed by different research teams have shown that, indeed, the machine seems to exploit quantum effects. However experiments on a class of random-bond instances have not yet demonstrated an advantage over classic…
▽ More
Recently, a programmable quantum annealing machine has been built that minimizes the cost function of hard optimization problems by adiabatically quenching quantum fluctuations. Tests performed by different research teams have shown that, indeed, the machine seems to exploit quantum effects. However experiments on a class of random-bond instances have not yet demonstrated an advantage over classical optimization algorithms on traditional computer hardware. Here we present evidence as to why this might be the case. These engineered quantum annealing machines effectively operate coupled to a decohering thermal bath. Therefore, we study the finite-temperature critical behavior of the standard benchmark problem used to assess the computational capabilities of these complex machines. We simulate both random-bond Ising models and spin glasses with bimodal and Gaussian disorder on the D-Wave Chimera topology. Our results show that while the worst-case complexity of finding a ground state of an Ising spin glass on the Chimera graph is not polynomial, the finite-temperature phase space is likely rather simple: Spin glasses on Chimera have only a zero-temperature transition. This means that benchmarking optimization methods using spin glasses on the Chimera graph might not be the best benchmark problems to test quantum speedup. We propose alternative benchmarks by embedding potentially harder problems on the Chimera topology. Finally, we also study the (reentrant) disorder-temperature phase diagram of the random-bond Ising model on the Chimera graph and show that a finite-temperature ferromagnetic phase is stable up to 19.85(15)% antiferromagnetic bonds. Beyond this threshold the system only displays a zero-temperature spin-glass phase. Our results therefore show that a careful design of the hardware architecture and benchmark problems is key when building quantum annealing machines.
△ Less
Submitted 10 April, 2014; v1 submitted 7 January, 2014;
originally announced January 2014.
-
Stability of topologically-protected quantum computing proposals as seen through spin glasses
Authors:
Helmut G. Katzgraber,
Ruben S. Andrist
Abstract:
Sensitivity to noise makes most of the current quantum computing schemes prone to error and nonscalable, allowing only for small proof-of-principle devices. Topologically-protected quantum computing aims at solving this problem by encoding quantum bits and gates in topological properties of the hardware medium that are immune to noise that does not impact the entire system at once. There are diffe…
▽ More
Sensitivity to noise makes most of the current quantum computing schemes prone to error and nonscalable, allowing only for small proof-of-principle devices. Topologically-protected quantum computing aims at solving this problem by encoding quantum bits and gates in topological properties of the hardware medium that are immune to noise that does not impact the entire system at once. There are different approaches to achieve topological stability or active error correction, ranging from quasiparticle braidings to spin models and topological color codes. The stability of these proposals against noise can be quantified by their error threshold. This figure of merit can be computed by map** the problem onto complex statistical-mechanical spin-glass models with local disorder on nontrival lattices that can have many-body interactions and are sometimes described by lattice gauge theories. The error threshold for a given source of error then represents the point in the temperature-disorder phase diagram where a stable symmetry-broken phase vanishes. An overview of the techniques used to estimate the error thresholds is given, as well as a summary of recent results on the stability of different topologically-protected quantum computing schemes to different error sources.
△ Less
Submitted 16 December, 2013; v1 submitted 3 June, 2013;
originally announced June 2013.
-
Self-Organized Criticality in Glassy Spin Systems Requires a Diverging Number of Neighbors
Authors:
Juan Carlos Andresen,
Zheng Zhu,
Ruben S. Andrist,
Helmut G. Katzgraber,
V. Dobrosavljevic,
Gergely T. Zimanyi
Abstract:
We investigate the conditions required for general spin systems with frustration and disorder to display self-organized criticality, a property which so far has been established only for the fully-connected infinite-range Sherrington-Kirkpatrick Ising spin-glass model [Phys. Rev. Lett. 83, 1034 (1999)]. Here we study both avalanche and magnetization jump distributions triggered by an external magn…
▽ More
We investigate the conditions required for general spin systems with frustration and disorder to display self-organized criticality, a property which so far has been established only for the fully-connected infinite-range Sherrington-Kirkpatrick Ising spin-glass model [Phys. Rev. Lett. 83, 1034 (1999)]. Here we study both avalanche and magnetization jump distributions triggered by an external magnetic field, as well as internal field distributions in the short-range Edwards-Anderson Ising spin glass for various space dimensions between 2 and 8, as well as the fixed-connectivity mean-field Viana-Bray model. Our numerical results, obtained on systems of unprecedented size, demonstrate that self-organized criticality is recovered only in the strict limit of a diverging number of neighbors, and is not a generic property of spin-glass models in finite space dimensions.
△ Less
Submitted 27 August, 2013; v1 submitted 14 October, 2012;
originally announced October 2012.
-
Optimal error correction in topological subsystem codes
Authors:
Ruben S. Andrist,
H. Bombin,
Helmut G. Katzgraber,
M. A. Martin-Delgado
Abstract:
A promising approach to overcome decoherence in quantum computing schemes is to perform active quantum error correction using topology. Topological subsystem codes incorporate both the benefits of topological and subsystem codes, allowing for error syndrome recovery with only 2-local measurements in a two-dimensional array of qubits. We study the error threshold for topological subsystem color cod…
▽ More
A promising approach to overcome decoherence in quantum computing schemes is to perform active quantum error correction using topology. Topological subsystem codes incorporate both the benefits of topological and subsystem codes, allowing for error syndrome recovery with only 2-local measurements in a two-dimensional array of qubits. We study the error threshold for topological subsystem color codes under very general external noise conditions. By transforming the problem into a classical disordered spin model, we estimate using Monte Carlo simulations that topological subsystem codes have an optimal error tolerance of 5.5(2)%. This means there is ample space for improvement in existing error-correcting algorithms that typically find a threshold of approximately 2%.
△ Less
Submitted 14 May, 2012; v1 submitted 9 April, 2012;
originally announced April 2012.
-
Strong Resilience of Topological Codes to Depolarization
Authors:
H. Bombin,
Ruben S. Andrist,
Masayuki Ohzeki,
Helmut G. Katzgraber,
M. A. Martin-Delgado
Abstract:
The inevitable presence of decoherence effects in systems suitable for quantum computation necessitates effective error-correction schemes to protect information from noise. We compute the stability of the toric code to depolarization by map** the quantum problem onto a classical disordered eight-vertex Ising model. By studying the stability of the related ferromagnetic phase both via large-scal…
▽ More
The inevitable presence of decoherence effects in systems suitable for quantum computation necessitates effective error-correction schemes to protect information from noise. We compute the stability of the toric code to depolarization by map** the quantum problem onto a classical disordered eight-vertex Ising model. By studying the stability of the related ferromagnetic phase both via large-scale Monte Carlo simulations and via the duality method, we are able to demonstrate an increased error threshold of 18.9(3)% when noise correlations are taken into account. Remarkably, this agrees within error bars with the result for a different class of codes-topological color codes-where the map** yields interesting new types of interacting eight-vertex models.
△ Less
Submitted 30 April, 2012; v1 submitted 8 February, 2012;
originally announced February 2012.
-
Evidence of a glass transition in a 10-state non-mean-field Potts glass
Authors:
Ruben S. Andrist,
Derek Larson,
Helmut G. Katzgraber
Abstract:
Potts glasses are prototype models that have been used to understand the structural glass transition. However, in finite space dimensions a glass transition remains to be detected in the 10-state Potts glass. Using a one-dimensional model with long-range power-law interactions we present evidence that a glass transition below the upper critical dimension can exist for short-range systems at low en…
▽ More
Potts glasses are prototype models that have been used to understand the structural glass transition. However, in finite space dimensions a glass transition remains to be detected in the 10-state Potts glass. Using a one-dimensional model with long-range power-law interactions we present evidence that a glass transition below the upper critical dimension can exist for short-range systems at low enough temperatures. Gaining insights into the structural glass transition for short-range systems using spin models is thus potentially possible, yet difficult.
△ Less
Submitted 29 March, 2011; v1 submitted 9 September, 2010;
originally announced September 2010.
-
Tricolored Lattice Gauge Theory with Randomness: Fault-Tolerance in Topological Color Codes
Authors:
Ruben S. Andrist,
Helmut G. Katzgraber,
H. Bombin,
M. A. Martin-Delgado
Abstract:
We compute the error threshold of color codes, a class of topological quantum codes that allow a direct implementation of quantum Clifford gates, when both qubit and measurement errors are present. By map** the problem onto a statistical-mechanical three-dimensional disordered Ising lattice gauge theory, we estimate via large-scale Monte Carlo simulations that color codes are stable against 4.5(…
▽ More
We compute the error threshold of color codes, a class of topological quantum codes that allow a direct implementation of quantum Clifford gates, when both qubit and measurement errors are present. By map** the problem onto a statistical-mechanical three-dimensional disordered Ising lattice gauge theory, we estimate via large-scale Monte Carlo simulations that color codes are stable against 4.5(2)% errors. Furthermore, by evaluating the skewness of the Wilson loop distributions, we introduce a very sensitive probe to locate first-order phase transitions in lattice gauge theories.
△ Less
Submitted 8 August, 2011; v1 submitted 5 May, 2010;
originally announced May 2010.
-
Topological color codes on Union Jack lattices: A stable implementation of the whole Clifford group
Authors:
Helmut G. Katzgraber,
H. Bombin,
Ruben S. Andrist,
M. A. Martin-Delgado
Abstract:
We study the error threshold of topological color codes on Union Jack lattices that allow for the full implementation of the whole Clifford group of quantum gates. After map** the error-correction process onto a statistical mechanical random 3-body Ising model on a Union Jack lattice, we compute its phase diagram in the temperature-disorder plane using Monte Carlo simulations. Surprisingly, to…
▽ More
We study the error threshold of topological color codes on Union Jack lattices that allow for the full implementation of the whole Clifford group of quantum gates. After map** the error-correction process onto a statistical mechanical random 3-body Ising model on a Union Jack lattice, we compute its phase diagram in the temperature-disorder plane using Monte Carlo simulations. Surprisingly, topological color codes on Union Jack lattices have similar error stability than color codes on triangular lattices, as well as the Kitaev toric code. The enhanced computational capabilities of the topological color codes on Union Jack lattices with respect to triangular lattices and the toric code demonstrate the inherent robustness of this implementation.
△ Less
Submitted 25 January, 2010; v1 submitted 3 October, 2009;
originally announced October 2009.
