Showing 1–50 of 144 results for author: Schuster, T

Enhancements for Real-Time Monte-Carlo Tree Search in General Video Game Playing

Authors: Dennis J. N. J. Soemers, Chiara F. Sironi, Torsten Schuster, Mark H. M. Winands

Abstract: General Video Game Playing (GVGP) is a field of Artificial Intelligence where agents play a variety of real-time video games that are unknown in advance. This limits the use of domain-specific heuristics. Monte-Carlo Tree Search (MCTS) is a search technique for game playing that does not rely on domain-specific knowledge. This paper discusses eight enhancements for MCTS in GVGP; Progressive Histor… ▽ More General Video Game Playing (GVGP) is a field of Artificial Intelligence where agents play a variety of real-time video games that are unknown in advance. This limits the use of domain-specific heuristics. Monte-Carlo Tree Search (MCTS) is a search technique for game playing that does not rely on domain-specific knowledge. This paper discusses eight enhancements for MCTS in GVGP; Progressive History, N-Gram Selection Technique, Tree Reuse, Breadth-First Tree Initialization, Loss Avoidance, Novelty-Based Pruning, Knowledge-Based Evaluations, and Deterministic Game Detection. Some of these are known from existing literature, and are either extended or introduced in the context of GVGP, and some are novel enhancements for MCTS. Most enhancements are shown to provide statistically significant increases in win percentages when applied individually. When combined, they increase the average win percentage over sixty different games from 31.0% to 48.4% in comparison to a vanilla MCTS implementation, approaching a level that is competitive with the best agents of the GVG-AI competition in 2015. △ Less

Submitted 3 July, 2024; originally announced July 2024.

Comments: Green Open Access version of conference paper published in 2016

Journal ref: 2016 IEEE Conference on Computational Intelligence and Games (CIG 2016), pp. 436-443

TACT: Advancing Complex Aggregative Reasoning with Information Extraction Tools

Authors: Avi Caciularu, Alon Jacovi, Eyal Ben-David, Sasha Goldshtein, Tal Schuster, Jonathan Herzig, Gal Elidan, Amir Globerson

Abstract: Large Language Models (LLMs) often do not perform well on queries that require the aggregation of information across texts. To better evaluate this setting and facilitate modeling efforts, we introduce TACT - Text And Calculations through Tables, a dataset crafted to evaluate LLMs' reasoning and computational abilities using complex instructions. TACT contains challenging instructions that demand… ▽ More Large Language Models (LLMs) often do not perform well on queries that require the aggregation of information across texts. To better evaluate this setting and facilitate modeling efforts, we introduce TACT - Text And Calculations through Tables, a dataset crafted to evaluate LLMs' reasoning and computational abilities using complex instructions. TACT contains challenging instructions that demand stitching information scattered across one or more texts, and performing complex integration on this information to generate the answer. We construct this dataset by leveraging an existing dataset of texts and their associated tables. For each such tables, we formulate new queries, and gather their respective answers. We demonstrate that all contemporary LLMs perform poorly on this dataset, achieving an accuracy below 38\%. To pinpoint the difficulties and thoroughly dissect the problem, we analyze model performance across three components: table-generation, Pandas command-generation, and execution. Unexpectedly, we discover that each component presents substantial challenges for current LLMs. These insights lead us to propose a focused modeling framework, which we refer to as IE as a tool. Specifically, we propose to add "tools" for each of the above steps, and implement each such tool with few-shot prompting. This approach shows an improvement over existing prompting techniques, offering a promising direction for enhancing model capabilities in these tasks. △ Less

Submitted 5 June, 2024; originally announced June 2024.

Comments: Website (https://tact-benchmark.github.io), Huggingface (https://huggingface.co/datasets/google/TACT)

Block Transformer: Global-to-Local Language Modeling for Fast Inference

Authors: Namgyu Ho, Sangmin Bae, Taehyeon Kim, Hyunjik Jo, Yireun Kim, Tal Schuster, Adam Fisch, James Thorne, Se-Young Yun

Abstract: This paper presents the Block Transformer architecture which adopts hierarchical global-to-local modeling to autoregressive transformers to mitigate the inference bottlenecks of self-attention. To apply self-attention, the key-value (KV) cache of all previous sequences must be retrieved from memory at every decoding step. Thereby, this KV cache IO becomes a significant bottleneck in batch inferenc… ▽ More This paper presents the Block Transformer architecture which adopts hierarchical global-to-local modeling to autoregressive transformers to mitigate the inference bottlenecks of self-attention. To apply self-attention, the key-value (KV) cache of all previous sequences must be retrieved from memory at every decoding step. Thereby, this KV cache IO becomes a significant bottleneck in batch inference. We notice that these costs stem from applying self-attention on the global context, therefore we isolate the expensive bottlenecks of global modeling to lower layers and apply fast local modeling in upper layers. To mitigate the remaining costs in the lower layers, we aggregate input tokens into fixed size blocks and then apply self-attention at this coarse level. Context information is aggregated into a single embedding to enable upper layers to decode the next block of tokens, without global attention. Free of global attention bottlenecks, the upper layers can fully utilize the compute hardware to maximize inference throughput. By leveraging global and local modules, the Block Transformer architecture demonstrates 10-20x gains in inference throughput compared to vanilla transformers with equivalent perplexity. Our work introduces a new approach to optimize language model inference through novel application of global-to-local modeling. Code is available at https://github.com/itsnamgyu/block-transformer. △ Less

Submitted 4 June, 2024; originally announced June 2024.

Comments: 30 pages, 21 figures, 5 tables

Attribute First, then Generate: Locally-attributable Grounded Text Generation

Authors: Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, Ido Dagan

Abstract: Recent efforts to address hallucinations in Large Language Models (LLMs) have focused on attributed text generation, which supplements generated texts with citations of supporting sources for post-generation fact-checking and corrections. Yet, these citations often point to entire documents or paragraphs, burdening users with extensive verification work. In this paper, we introduce a locally-attri… ▽ More Recent efforts to address hallucinations in Large Language Models (LLMs) have focused on attributed text generation, which supplements generated texts with citations of supporting sources for post-generation fact-checking and corrections. Yet, these citations often point to entire documents or paragraphs, burdening users with extensive verification work. In this paper, we introduce a locally-attributable text generation approach, prioritizing concise attributions. Our method, named "Attribute First, then Generate", breaks down the conventional end-to-end generation process into three intuitive steps: content selection, sentence planning, and sequential sentence generation. By initially identifying relevant source segments ("select first") and then conditioning the generation process on them ("then generate"), we ensure these segments also act as the output's fine-grained attributions ("select" becomes "attribute"). Tested on Multi-document Summarization and Long-form Question-answering, our method not only yields more concise citations than the baselines but also maintains - and in some cases enhances - both generation quality and attribution accuracy. Furthermore, it significantly reduces the time required for fact verification by human assessors. △ Less

Submitted 4 July, 2024; v1 submitted 25 March, 2024; originally announced March 2024.

Comments: ACL 2024

A holographic view of topological stabilizer codes

Authors: Thomas Schuster, Nathanan Tantivasadakarn, Ashvin Vishwanath, Norman Y. Yao

Abstract: The bulk-boundary correspondence is a hallmark feature of topological phases of matter. Nonetheless, our understanding of the correspondence remains incomplete for phases with intrinsic topological order, and is nearly entirely lacking for more exotic phases, such as fractons. Intriguingly, for the former, recent work suggests that bulk topological order manifests in a non-local structure in the b… ▽ More The bulk-boundary correspondence is a hallmark feature of topological phases of matter. Nonetheless, our understanding of the correspondence remains incomplete for phases with intrinsic topological order, and is nearly entirely lacking for more exotic phases, such as fractons. Intriguingly, for the former, recent work suggests that bulk topological order manifests in a non-local structure in the boundary Hilbert space; however, a concrete understanding of how and where this perspective applies remains limited. Here, we provide an explicit and general framework for understanding the bulk-boundary correspondence in Pauli topological stabilizer codes. We show -- for any boundary termination of any two-dimensional topological stabilizer code -- that the boundary Hilbert space cannot be realized via local degrees of freedom, in a manner precisely determined by the anyon data of the bulk topological order. We provide a simple method to compute this "obstruction" using a well-known map** to polynomials over finite fields. Leveraging this map**, we generalize our framework to fracton models in three-dimensions, including both the X-Cube model and Haah's code. An important consequence of our results is that the boundaries of topological phases can exhibit emergent symmetries that are impossible to otherwise achieve without an unrealistic degree of fine tuning. For instance, we show how linear and fractal subsystem symmetries naturally arise at the boundaries of fracton phases. △ Less

Submitted 7 December, 2023; originally announced December 2023.

Comments: 29+18 pages. 19 figures

A knowledge-driven framework for synthesizing designs from modular components

Authors: Constantin Chaumet, Jakob Rehof, Thomas Schuster

Abstract: Creating a design from modular components necessitates three steps: Acquiring knowledge about available components, conceiving an abstract design concept, and implementing that concept in a concrete design. The third step entails many repetitive and menial tasks, such as inserting parts and creating joints between them. Especially when comparing and implementing design alternatives, this issue is… ▽ More Creating a design from modular components necessitates three steps: Acquiring knowledge about available components, conceiving an abstract design concept, and implementing that concept in a concrete design. The third step entails many repetitive and menial tasks, such as inserting parts and creating joints between them. Especially when comparing and implementing design alternatives, this issue is compounded. We propose a use-case agnostic knowledge-driven framework to automate the implementation step. In particular, the framework catalogues the acquired knowledge and the design concept, as well as utilizes Combinatory Logic Synthesis to synthesize concrete design alternatives. This minimizes the effort required to create designs, allowing the design space to be thoroughly explored. We implemented the framework as a plugin for the CAD software Autodesk Fusion 360. We conducted a case study in which robotic arms were synthesized from a set of 28 modular components. Based on the case study, the applicability of the framework is analyzed and discussed. △ Less

Submitted 30 November, 2023; originally announced November 2023.

ACM Class: J.6; F.4.1; D.2.2

SEMQA: Semi-Extractive Multi-Source Question Answering

Authors: Tal Schuster, Adam D. Lelkes, Haitian Sun, Jai Gupta, Jonathan Berant, William W. Cohen, Donald Metzler

Abstract: Recently proposed long-form question answering (QA) systems, supported by large language models (LLMs), have shown promising capabilities. Yet, attributing and verifying their generated abstractive answers can be difficult, and automatically evaluating their accuracy remains an ongoing challenge. In this work, we introduce a new QA task for answering multi-answer questions by summarizing multipl… ▽ More Recently proposed long-form question answering (QA) systems, supported by large language models (LLMs), have shown promising capabilities. Yet, attributing and verifying their generated abstractive answers can be difficult, and automatically evaluating their accuracy remains an ongoing challenge. In this work, we introduce a new QA task for answering multi-answer questions by summarizing multiple diverse sources in a semi-extractive fashion. Specifically, Semi-extractive Multi-source QA (SEMQA) requires models to output a comprehensive answer, while mixing factual quoted spans -- copied verbatim from given input sources -- and non-factual free-text connectors that glue these spans together into a single cohesive passage. This setting bridges the gap between the outputs of well-grounded but constrained extractive QA systems and more fluent but harder to attribute fully abstractive answers. Particularly, it enables a new mode for language models that leverages their advanced language generation capabilities, while also producing fine in-line attributions by-design that are easy to verify, interpret, and evaluate. To study this task, we create the first dataset of this kind, QuoteSum, with human-written semi-extractive answers to natural and generated questions, and define text-based evaluation metrics. Experimenting with several LLMs in various settings, we find this task to be surprisingly challenging, demonstrating the importance of QuoteSum for develo** and studying such consolidation capabilities. △ Less

Submitted 30 June, 2024; v1 submitted 8 November, 2023; originally announced November 2023.

Comments: NAACL 2024

SDOH-NLI: a Dataset for Inferring Social Determinants of Health from Clinical Notes

Authors: Adam D. Lelkes, Eric Loreaux, Tal Schuster, Ming-Jun Chen, Alvin Rajkomar

Abstract: Social and behavioral determinants of health (SDOH) play a significant role in sha** health outcomes, and extracting these determinants from clinical notes is a first step to help healthcare providers systematically identify opportunities to provide appropriate care and address disparities. Progress on using NLP methods for this task has been hindered by the lack of high-quality publicly availab… ▽ More Social and behavioral determinants of health (SDOH) play a significant role in sha** health outcomes, and extracting these determinants from clinical notes is a first step to help healthcare providers systematically identify opportunities to provide appropriate care and address disparities. Progress on using NLP methods for this task has been hindered by the lack of high-quality publicly available labeled data, largely due to the privacy and regulatory constraints on the use of real patients' information. This paper introduces a new dataset, SDOH-NLI, that is based on publicly available notes and which we release publicly. We formulate SDOH extraction as a natural language inference (NLI) task, and provide binary textual entailment labels obtained from human raters for a cross product of a set of social history snippets as premises and SDOH factors as hypotheses. Our dataset differs from standard NLI benchmarks in that our premises and hypotheses are obtained independently. We evaluate both "off-the-shelf" entailment models as well as models fine-tuned on our data, and highlight the ways in which our dataset appears more challenging than commonly used NLI datasets. △ Less

Submitted 27 October, 2023; originally announced October 2023.

Comments: Findings of EMNLP 2023

Ill-posedness of time-dependent inverse problems in Lebesgue-Bochner spaces

Authors: Martin Burger, Thomas Schuster, Anne Wald

Abstract: We consider time-dependent inverse problems in a mathematical setting using Lebesgue-Bochner spaces. Such problems arise when one aims to recover parameters from given observations where the parameters or the data depend on time. There are various important applications being subject of current research that belong to this class of problems. Typically inverse problems are ill-posed in the sense th… ▽ More We consider time-dependent inverse problems in a mathematical setting using Lebesgue-Bochner spaces. Such problems arise when one aims to recover parameters from given observations where the parameters or the data depend on time. There are various important applications being subject of current research that belong to this class of problems. Typically inverse problems are ill-posed in the sense that already small noise in the data causes tremendous errors in the solution. In this article we present two different concepts of ill-posedness: temporally (pointwise) ill-posedness and uniform ill-posedness with respect to the Lebesgue-Bochner setting. We investigate the two concepts by means of a typical setting consisting of a time-depending observation operator composed by a compact operator. Furthermore we develop regularization methods that are adapted to the respective class of ill-posedness. △ Less

Submitted 6 October, 2023; originally announced October 2023.

Comments: 21 pages, no figures

MSC Class: 65J22

Overview of Physics-Informed Machine Learning Inversion of Geophysical Data

Authors: Gerard T. Schuster, Shihang Feng

Abstract: We review four types of algorithms for physics-informed machine learning (PIML) inversion of geophysical data. The unifying equation is given by the joint objective function $ε$: \begin{eqnarray} ε^{||-PIML}&=&λ_1 \overbrace{||{\bf W}^{ML}({\bf H}_{\bf w} {\bf d}^{obs}-{\bf m})||^2}^{NN} + λ_2 \overbrace{{||{\bf W}^{FWI}({\bf L} {\bf m}-{\bf d}^{obs})||^2}}^{FWI} ~+ \nonumber\\ \nonumber\\ && +… ▽ More We review four types of algorithms for physics-informed machine learning (PIML) inversion of geophysical data. The unifying equation is given by the joint objective function $ε$: \begin{eqnarray} ε^{||-PIML}&=&λ_1 \overbrace{||{\bf W}^{ML}({\bf H}_{\bf w} {\bf d}^{obs}-{\bf m})||^2}^{NN} + λ_2 \overbrace{{||{\bf W}^{FWI}({\bf L} {\bf m}-{\bf d}^{obs})||^2}}^{FWI} ~+ \nonumber\\ \nonumber\\ && + ~~Regularizer, \label{PIML.eq120} \end{eqnarray}where the optimal model ${\bf m}^*$ and weights $\bf w^*$ minimize $ε$. Here, The matrix weights are given by the boldface symbol $\bf W$, and full waveform inversion (FWI) is typically computed using a finite-difference solution of the wave equation, where $\bf L$ represents the forward modeling operation of the wave equation as a function of the model $\bf m$. Also, a fully-connected neural network (NN) is used to compute the model ${\bf H_w}{\bf d}^{obs} \approx \bf m$ from the observed input data ${\bf d}^{obs}$. The selection of weights $λ_i$ and the NN operations determine one of four different PIML algorithms. PIML offers potential advantages over standard FWI through its enhanced ability to avoid local minima and the option to locally train the inversion operator, minimizing the requirement for extensive training data for global applicability. However, the effectiveness of PIML relies on the similarity between the test and trained data. Nevertheless, a possible strategy to overcome this limitation involves initial pretraining of a PIML architecture with data from a broader region, followed by fine-tuning for specific data-a method reminiscent of the way large language models are pretrained and adapted for various tasks. △ Less

Submitted 12 October, 2023; originally announced October 2023.

Comments: 37 pages, 16 figures

Causal Discovery with Language Models as Imperfect Experts

Authors: Stephanie Long, Alexandre Piché, Valentina Zantedeschi, Tibor Schuster, Alexandre Drouin

Abstract: Understanding the causal relationships that underlie a system is a fundamental prerequisite to accurate decision-making. In this work, we explore how expert knowledge can be used to improve the data-driven identification of causal graphs, beyond Markov equivalence classes. In doing so, we consider a setting where we can query an expert about the orientation of causal relationships between variable… ▽ More Understanding the causal relationships that underlie a system is a fundamental prerequisite to accurate decision-making. In this work, we explore how expert knowledge can be used to improve the data-driven identification of causal graphs, beyond Markov equivalence classes. In doing so, we consider a setting where we can query an expert about the orientation of causal relationships between variables, but where the expert may provide erroneous information. We propose strategies for amending such expert knowledge based on consistency properties, e.g., acyclicity and conditional independencies in the equivalence class. We then report a case study, on real data, where a large language model is used as an imperfect expert. △ Less

Submitted 5 July, 2023; originally announced July 2023.

Comments: Peer reviewed and accepted for presentation at the Structured Probabilistic Inference & Generative Modeling (SPIGM) workshop at ICML 2023, Hawaii, USA

Conformal Language Modeling

Authors: Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, Regina Barzilay

Abstract: We propose a novel approach to conformal prediction for generative language models (LMs). Standard conformal prediction produces prediction sets -- in place of single predictions -- that have rigorous, statistical performance guarantees. LM responses are typically sampled from the model's predicted distribution over the large, combinatorial output space of natural language. Translating this proces… ▽ More We propose a novel approach to conformal prediction for generative language models (LMs). Standard conformal prediction produces prediction sets -- in place of single predictions -- that have rigorous, statistical performance guarantees. LM responses are typically sampled from the model's predicted distribution over the large, combinatorial output space of natural language. Translating this process to conformal prediction, we calibrate a stop** rule for sampling different outputs from the LM that get added to a growing set of candidates until we are confident that the output set is sufficient. Since some samples may be low-quality, we also simultaneously calibrate and apply a rejection rule for removing candidates from the output set to reduce noise. Similar to conformal prediction, we prove that the sampled set returned by our procedure contains at least one acceptable answer with high probability, while still being empirically precise (i.e., small) on average. Furthermore, within this set of candidate responses, we show that we can also accurately identify subsets of individual components -- such as phrases or sentences -- that are each independently correct (e.g., that are not "hallucinations"), again with statistical guarantees. We demonstrate the promise of our approach on multiple tasks in open-domain question answering, text summarization, and radiology report generation using different LM variants. △ Less

Submitted 1 June, 2024; v1 submitted 16 June, 2023; originally announced June 2023.

Comments: ICLR 2024

LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction

Authors: Jeremiah Milbauer, Annie Louis, Mohammad Javad Hosseini, Alex Fabrikant, Donald Metzler, Tal Schuster

Abstract: Transformer encoders contextualize token representations by attending to all other tokens at each layer, leading to quadratic increase in compute effort with the input length. In practice, however, the input text of many NLP tasks can be seen as a sequence of related segments (e.g., the sequence of sentences within a passage, or the hypothesis and premise in NLI). While attending across these segm… ▽ More Transformer encoders contextualize token representations by attending to all other tokens at each layer, leading to quadratic increase in compute effort with the input length. In practice, however, the input text of many NLP tasks can be seen as a sequence of related segments (e.g., the sequence of sentences within a passage, or the hypothesis and premise in NLI). While attending across these segments is highly beneficial for many tasks, we hypothesize that this interaction can be delayed until later encoding stages. To this end, we introduce Layer-Adjustable Interactions in Transformers (LAIT). Within LAIT, segmented inputs are first encoded independently, and then jointly. This partial two-tower architecture bridges the gap between a Dual Encoder's ability to pre-compute representations for segments and a fully self-attentive Transformer's capacity to model cross-segment attention. The LAIT framework effectively leverages existing pretrained Transformers and converts them into the hybrid of the two aforementioned architectures, allowing for easy and intuitive control over the performance-efficiency tradeoff. Experimenting on a wide range of NLP tasks, we find LAIT able to reduce 30-50% of the attention FLOPs on many tasks, while preserving high accuracy; in some practical settings, LAIT could reduce actual latency by orders of magnitude. △ Less

Submitted 31 May, 2023; originally announced May 2023.

Comments: ACL 2023

The advantage of quantum control in many-body Hamiltonian learning

Authors: Alicja Dutkiewicz, Thomas E. O'Brien, Thomas Schuster

Abstract: We study the problem of learning the Hamiltonian of a many-body quantum system from experimental data. We show that the rate of learning depends on the amount of control available during the experiment. We consider three control models: one where time evolution can be augmented with instantaneous quantum operations, one where the Hamiltonian itself can be augmented by adding constant terms, and on… ▽ More We study the problem of learning the Hamiltonian of a many-body quantum system from experimental data. We show that the rate of learning depends on the amount of control available during the experiment. We consider three control models: one where time evolution can be augmented with instantaneous quantum operations, one where the Hamiltonian itself can be augmented by adding constant terms, and one where the experimentalist has no control over the system's time evolution. With continuous quantum control, we provide an adaptive algorithm for learning a many-body Hamiltonian at the Heisenberg limit: $T = \mathcal{O}(ε^{-1})$, where $T$ is the total amount of time evolution across all experiments and $ε$ is the target precision. This requires only preparation of product states, time-evolution, and measurement in a product basis. In the absence of quantum control, we prove that learning is standard quantum limited, $T = Ω(ε^{-2})$, for large classes of many-body Hamiltonians, including any Hamiltonian that thermalizes via the eigenstate thermalization hypothesis. These results establish a quadratic advantage in experimental runtime for learning with quantum control. △ Less

Submitted 16 November, 2023; v1 submitted 14 April, 2023; originally announced April 2023.

Comments: 10 pages + 27 page appendix, 3 figures

Can large language models build causal graphs?

Authors: Stephanie Long, Tibor Schuster, Alexandre Piché

Abstract: Building causal graphs can be a laborious process. To ensure all relevant causal pathways have been captured, researchers often have to discuss with clinicians and experts while also reviewing extensive relevant medical literature. By encoding common and medical knowledge, large language models (LLMs) represent an opportunity to ease this process by automatically scoring edges (i.e., connections b… ▽ More Building causal graphs can be a laborious process. To ensure all relevant causal pathways have been captured, researchers often have to discuss with clinicians and experts while also reviewing extensive relevant medical literature. By encoding common and medical knowledge, large language models (LLMs) represent an opportunity to ease this process by automatically scoring edges (i.e., connections between two variables) in potential graphs. LLMs however have been shown to be brittle to the choice of probing words, context, and prompts that the user employs. In this work, we evaluate if LLMs can be a useful tool in complementing causal graph development. △ Less

Submitted 23 February, 2024; v1 submitted 7 March, 2023; originally announced March 2023.

Comments: Peer reviewed and accepted for presentation at the Causal Machine Learning for Real-World Impact Workshop (CML4Impact) at NeuRIPs2022 Fixed author list

Comment on "Traversable wormhole dynamics on a quantum processor"

Authors: Bryce Kobrin, Thomas Schuster, Norman Y. Yao

Abstract: A recent article [Nature 612, 51-55 (2022)] claims to observe traversable wormhole dynamics in an experiment. This claim is based upon performing a teleportation protocol using a Hamiltonian that consists of seven Majorana fermions with five fully-commuting terms. The Hamiltonian is generated via a machine-learning procedure designed to replicate the teleportation behavior of the Sachdev-Ye-Kitaev… ▽ More A recent article [Nature 612, 51-55 (2022)] claims to observe traversable wormhole dynamics in an experiment. This claim is based upon performing a teleportation protocol using a Hamiltonian that consists of seven Majorana fermions with five fully-commuting terms. The Hamiltonian is generated via a machine-learning procedure designed to replicate the teleportation behavior of the Sachdev-Ye-Kitaev (SYK) model. The authors claim that the learned Hamiltonian reproduces gravitational dynamics of the SYK model and demonstrates gravitational teleportation through an emergent wormhole. We find: (i) in contrast to these claims, the learned Hamiltonian does not exhibit thermalization; (ii) the teleportation signal only resembles the SYK model for operators that were used in the machine-learning training; (iii) the observed perfect size winding is in fact a generic feature of small-size, fully-commuting models, and does not appear to persist in larger-size fully-commuting models or in non-commuting models at equivalent system sizes △ Less

Submitted 15 February, 2023; originally announced February 2023.

Comments: 5+4 pages, 3+4 figures

Development of a photothermal measurement model to determine layer thickness of multi-layered coating systems with unknown thermal properties

Authors: Dimitri Rothermel, Thomas Schuster

Abstract: In this article, a general model for 1D thermal wave interference is derived for multi-layered coating systems on a thermally thick substrate using the same principles as for the well established one-layered and two-layered coating cases. Using the lock-in thermography principle, an illumination source modulates the surface of those systems periodically by a planar, sinusoidal wave form with a fix… ▽ More In this article, a general model for 1D thermal wave interference is derived for multi-layered coating systems on a thermally thick substrate using the same principles as for the well established one-layered and two-layered coating cases. Using the lock-in thermography principle, an illumination source modulates the surface of those systems periodically by a planar, sinusoidal wave form with a fixed frequency. The coating systems absorb the optical energy on its surface and convert it into thermal energy, resulting in the propagation of a spatially and temporally periodic thermal wave with the same frequency. These thermal waves, originating at the surface, are reflected and transmitted at each interface leading to infinitely many wave trains that need to be tracked in order to formulate the final surface temperature as a superposition of all these waves. The heat transfer inside the object depends on the layer thickness of each coating, but also on the thermal properties of each layer material. The goal is to have a mathematical and physical model which describes the phase angle data measured by an infrared camera. Having these data, the main objective of this paper is to determine the thickness of each coating layer. In practice, the thermal properties of the layers usually are unknown, which makes the process even more difficult. For that reason, this article presents a concept to determine the thermal properties in advance. △ Less

Submitted 24 January, 2023; originally announced January 2023.

Comments: 16 pages, 8 figures

MSC Class: 35R30

PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition

Authors: Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Dan Roth, Tal Schuster

Abstract: The widely studied task of Natural Language Inference (NLI) requires a system to recognize whether one piece of text is textually entailed by another, i.e. whether the entirety of its meaning can be inferred from the other. In current NLI datasets and models, textual entailment relations are typically defined on the sentence- or paragraph-level. However, even a simple sentence often contains multi… ▽ More The widely studied task of Natural Language Inference (NLI) requires a system to recognize whether one piece of text is textually entailed by another, i.e. whether the entirety of its meaning can be inferred from the other. In current NLI datasets and models, textual entailment relations are typically defined on the sentence- or paragraph-level. However, even a simple sentence often contains multiple propositions, i.e. distinct units of meaning conveyed by the sentence. As these propositions can carry different truth values in the context of a given premise, we argue for the need to recognize the textual entailment relation of each proposition in a sentence individually. We propose PropSegmEnt, a corpus of over 45K propositions annotated by expert human raters. Our dataset structure resembles the tasks of (1) segmenting sentences within a document to the set of propositions, and (2) classifying the entailment relation of each proposition with respect to a different yet topically-aligned document, i.e. documents describing the same event or entity. We establish strong baselines for the segmentation and entailment tasks. Through case studies on summary hallucination detection and document-level NLI, we demonstrate that our conceptual framework is potentially useful for understanding and explaining the compositionality of NLI labels. △ Less

Submitted 24 May, 2023; v1 submitted 20 December, 2022; originally announced December 2022.

Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models

Authors: Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Lierni Sestorain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, Kellie Webster

Abstract: Large language models (LLMs) have shown impressive results while requiring little or no direct supervision. Further, there is mounting evidence that LLMs may have potential in information-seeking scenarios. We believe the ability of an LLM to attribute the text that it generates is likely to be crucial in this setting. We formulate and study Attributed QA as a key first step in the development of… ▽ More Large language models (LLMs) have shown impressive results while requiring little or no direct supervision. Further, there is mounting evidence that LLMs may have potential in information-seeking scenarios. We believe the ability of an LLM to attribute the text that it generates is likely to be crucial in this setting. We formulate and study Attributed QA as a key first step in the development of attributed LLMs. We propose a reproducible evaluation framework for the task and benchmark a broad set of architectures. We take human annotations as a gold standard and show that a correlated automatic metric is suitable for development. Our experimental work gives concrete answers to two key questions (How to measure attribution?, and How well do current state-of-the-art methods perform on attribution?), and give some hints as to how to address a third (How to build LLMs with attribution?). △ Less

Submitted 10 February, 2023; v1 submitted 15 December, 2022; originally announced December 2022.

Is margin all you need? An extensive empirical study of active learning on tabular data

Authors: Dara Bahri, Heinrich Jiang, Tal Schuster, Afshin Rostamizadeh

Abstract: Given a labeled training set and a collection of unlabeled data, the goal of active learning (AL) is to identify the best unlabeled points to label. In this comprehensive study, we analyze the performance of a variety of AL algorithms on deep neural networks trained on 69 real-world tabular classification datasets from the OpenML-CC18 benchmark. We consider different data regimes and the effect of… ▽ More Given a labeled training set and a collection of unlabeled data, the goal of active learning (AL) is to identify the best unlabeled points to label. In this comprehensive study, we analyze the performance of a variety of AL algorithms on deep neural networks trained on 69 real-world tabular classification datasets from the OpenML-CC18 benchmark. We consider different data regimes and the effect of self-supervised model pre-training. Surprisingly, we find that the classical margin sampling technique matches or outperforms all others, including current state-of-art, in a wide range of experimental settings. To researchers, we hope to encourage rigorous benchmarking against margin, and to practitioners facing tabular data labeling constraints that hyper-parameter-free margin may often be all they need. △ Less

Submitted 7 October, 2022; originally announced October 2022.

Operator Growth in Open Quantum Systems

Authors: Thomas Schuster, Norman Y. Yao

Abstract: The spreading of quantum information in closed systems, often termed scrambling, is a hallmark of many-body quantum dynamics. In open systems, scrambling competes with noise, errors and decoherence. Here, we provide a universal framework that describes the scrambling of quantum information in open systems: we predict that the effect of open-system dynamics is fundamentally controlled by operator s… ▽ More The spreading of quantum information in closed systems, often termed scrambling, is a hallmark of many-body quantum dynamics. In open systems, scrambling competes with noise, errors and decoherence. Here, we provide a universal framework that describes the scrambling of quantum information in open systems: we predict that the effect of open-system dynamics is fundamentally controlled by operator size distributions and independent of the microscopic error mechanism. This framework allows us to demonstrate that open quantum systems exhibit universal classes of information dynamics that fundamentally differ from their unitary counterparts. Implications for Loschmidt echo experiments and the classical simulability of open quantum dynamics will be discussed. △ Less

Submitted 25 August, 2022; originally announced August 2022.

Comments: 5+18 pages, 4+2 figures

A note on $Γ$-convergence of Tikhonov functionals for nonlinear inverse problems

Authors: Alexey Belenkin, Michael Hartz, Thomas Schuster

Abstract: We consider variational regularization of nonlinear inverse problems in Banach spaces using Tikhonov functionals. This article addresses the problem of $Γ$-convergence of a family of Tikhonov functionals and assertions of the convergence of their respective infima. Such questions arise, if model uncertainties, inaccurate forward operators, finite dimensional approximations of the forward solutions… ▽ More We consider variational regularization of nonlinear inverse problems in Banach spaces using Tikhonov functionals. This article addresses the problem of $Γ$-convergence of a family of Tikhonov functionals and assertions of the convergence of their respective infima. Such questions arise, if model uncertainties, inaccurate forward operators, finite dimensional approximations of the forward solutions and / or data, etc. make the evaluation of the original functional impossible and, thus, its minimizer not computable. But for applications it is of utmost importance that the minimizer of the replacement functional approximates the original minimizer. Under certain additional conditions this is satisfied if the approximated functionals converge to the original functional in the sense of $Γ$-convergence. We deduce simple criteria in different topologies which guarantee $Γ$-convergence as well as convergence of minimizing sequences. △ Less

Submitted 11 August, 2022; originally announced August 2022.

Comments: 14 pages, 0 figures

MSC Class: 58E50; 65J22

Conformal Risk Control

Authors: Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, Tal Schuster

Abstract: We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarantee. Like conformal prediction, the conformal risk control procedure is tight up to an $\mathcal{O}(1/n)$ factor. We also introduce extensions of the idea to distribution shift, quantile risk control, multiple and adversar… ▽ More We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarantee. Like conformal prediction, the conformal risk control procedure is tight up to an $\mathcal{O}(1/n)$ factor. We also introduce extensions of the idea to distribution shift, quantile risk control, multiple and adversarial risk control, and expectations of U-statistics. Worked examples from computer vision and natural language processing demonstrate the usage of our algorithm to bound the false negative rate, graph distance, and token-level F1-score. △ Less

Submitted 29 April, 2023; v1 submitted 4 August, 2022; originally announced August 2022.

Comments: Code available at https://github.com/aangelopoulos/conformal-risk

Information-theoretic Hardness of Out-of-time-order Correlators

Authors: Jordan Cotler, Thomas Schuster, Masoud Mohseni

Abstract: We establish that there are properties of quantum many-body dynamics which are efficiently learnable if we are given access to out-of-time-order correlators (OTOCs), but which require exponentially many operations in the system size if we can only measure time-ordered correlators. This implies that any experimental protocol which reconstructs OTOCs solely from time-ordered correlators must be, in… ▽ More We establish that there are properties of quantum many-body dynamics which are efficiently learnable if we are given access to out-of-time-order correlators (OTOCs), but which require exponentially many operations in the system size if we can only measure time-ordered correlators. This implies that any experimental protocol which reconstructs OTOCs solely from time-ordered correlators must be, in certain cases, exponentially inefficient. Our proofs leverage and generalize recent techniques in quantum learning theory. Along the way, we elucidate a general definition of time-ordered versus out-of-time-order experimental measurement protocols, which can be considered as classes of adaptive quantum learning algorithms. Moreover, our results provide a theoretical foundation for novel applications of OTOCs in quantum simulations. △ Less

Submitted 3 August, 2022; originally announced August 2022.

Comments: 5+13 pages, 2 figures and many diagrams

Learning quantum systems via out-of-time-order correlators

Authors: Thomas Schuster, Murphy Niu, Jordan Cotler, Thomas O'Brien, Jarrod R. McClean, Masoud Mohseni

Abstract: Learning the properties of dynamical quantum systems underlies applications ranging from nuclear magnetic resonance spectroscopy to quantum device characterization. A central challenge in this pursuit is the learning of strongly-interacting systems, where conventional observables decay quickly in time and space, limiting the information that can be learned from their measurement. In this work, we… ▽ More Learning the properties of dynamical quantum systems underlies applications ranging from nuclear magnetic resonance spectroscopy to quantum device characterization. A central challenge in this pursuit is the learning of strongly-interacting systems, where conventional observables decay quickly in time and space, limiting the information that can be learned from their measurement. In this work, we introduce a new class of observables into the context of quantum learning -- the out-of-time-order correlator -- which we show can substantially improve the learnability of strongly-interacting systems by virtue of displaying informative physics at large times and distances. We identify two general scenarios in which out-of-time-order correlators provide a significant advantage for learning tasks in locally-interacting systems: (i) when experimental access to the system is spatially-restricted, for example via a single "probe" degree of freedom, and (ii) when one desires to characterize weak interactions whose strength is much less than the typical interaction strength. We numerically characterize these advantages across a variety of learning problems, and find that they are robust to both read-out error and decoherence. Finally, we introduce a binary classification task that can be accomplished in constant time with out-of-time-order measurements. In a companion paper, we prove that this task is exponentially hard with any adaptive learning protocol that only involves time-ordered operations. △ Less

Submitted 3 August, 2022; originally announced August 2022.

Comments: 18 pages, 8 figures

Confident Adaptive Language Modeling

Authors: Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, Donald Metzler

Abstract: Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks. These gains come with a drastic increase in the models' size, potentially leading to slow and costly use at inference time. In practice, however, the series of generations made by LLMs is composed of varying levels of difficulty. While certain predictions truly bene… ▽ More Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks. These gains come with a drastic increase in the models' size, potentially leading to slow and costly use at inference time. In practice, however, the series of generations made by LLMs is composed of varying levels of difficulty. While certain predictions truly benefit from the models' full capacity, other continuations are more trivial and can be solved with reduced compute. In this work, we introduce Confident Adaptive Language Modeling (CALM), a framework for dynamically allocating different amounts of compute per input and generation timestep. Early exit decoding involves several challenges that we address here, such as: (1) what confidence measure to use; (2) connecting sequence-level constraints to local per-token exit decisions; and (3) attending back to missing hidden representations due to early exits in previous tokens. Through theoretical analysis and empirical experiments on three diverse text generation tasks, we demonstrate the efficacy of our framework in reducing compute -- potential speedup of up to $\times 3$ -- while provably maintaining high performance. △ Less

Submitted 25 October, 2022; v1 submitted 14 July, 2022; originally announced July 2022.

Comments: NeurIPS 2022 (selected as Oral)

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Authors: Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza , et al. (426 additional authors not shown)

Abstract: Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-futur… ▽ More Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting. △ Less

Submitted 12 June, 2023; v1 submitted 9 June, 2022; originally announced June 2022.

Comments: 27 pages, 17 figures + references and appendices, repo: https://github.com/google/BIG-bench

Journal ref: Transactions on Machine Learning Research, May/2022, https://openreview.net/forum?id=uyTL5Bvosj

UL2: Unifying Language Learning Paradigms

Authors: Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, Donald Metzler

Abstract: Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectiv… ▽ More Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized & unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 & GPT-like models across multiple diverse setups. By scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised finetuning based NLP tasks. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization. On 0-shot MMLU, UL2 20B outperforms T0 and T5 models. UL2 20B also works well with chain-of-thought prompting and reasoning, making it an appealing choice for research into reasoning at a small to medium scale of 20B parameters. Finally, we apply FLAN instruction tuning to the UL2 20B model, achieving MMLU and Big-Bench scores competitive to FLAN-PaLM 62B. We release Flax-based T5X checkpoints for the UL2 20B & Flan-UL2 20B. △ Less

Submitted 28 February, 2023; v1 submitted 10 May, 2022; originally announced May 2022.

Comments: Updated Q1 2023 with Flan-UL2 20B release! :)

Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters

Authors: Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Donald Metzler

Abstract: Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs. While early work identified certain biases in NLI models, recent advancements in modeling and datasets demonstrated promising performance. In this work, we further explore the direct zero-shot applicability of NLI models to real applications… ▽ More Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs. While early work identified certain biases in NLI models, recent advancements in modeling and datasets demonstrated promising performance. In this work, we further explore the direct zero-shot applicability of NLI models to real applications, beyond the sentence-pair setting they were trained on. First, we analyze the robustness of these models to longer and out-of-domain inputs. Then, we develop new aggregation methods to allow operating over full documents, reaching state-of-the-art performance on the ContractNLI dataset. Interestingly, we find NLI scores to provide strong retrieval signals, leading to more relevant evidence extractions compared to common similarity-based methods. Finally, we go further and investigate whole document clusters to identify both discrepancies and consensus among sources. In a test case, we find real inconsistencies between Wikipedia pages in different languages about the same topic. △ Less

Submitted 1 November, 2022; v1 submitted 15 April, 2022; originally announced April 2022.

Comments: Findings of EMNLP 2022

Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation

Authors: Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, Tal Schuster

Abstract: The predictions of question answering (QA)systems are typically evaluated against manually annotated finite sets of one or more answers. This leads to a coverage limitation that results in underestimating the true performance of systems, and is typically addressed by extending over exact match (EM) with pre-defined rules or with the token-level F1 measure. In this paper, we present the first syste… ▽ More The predictions of question answering (QA)systems are typically evaluated against manually annotated finite sets of one or more answers. This leads to a coverage limitation that results in underestimating the true performance of systems, and is typically addressed by extending over exact match (EM) with pre-defined rules or with the token-level F1 measure. In this paper, we present the first systematic conceptual and data-driven analysis to examine the shortcomings of token-level equivalence measures. To this end, we define the asymmetric notion of answer equivalence (AE), accepting answers that are equivalent to or improve over the reference, and publish over 23k human judgments for candidates produced by multiple QA systems on SQuAD. Through a careful analysis of this data, we reveal and quantify several concrete limitations of the F1 measure, such as a false impression of graduality, or missing dependence on the question. Since collecting AE annotations for each evaluated model is expensive, we learn a BERT matching (BEM) measure to approximate this task. Being a simpler task than QA, we find BEM to provide significantly better AE approximations than F1, and to more accurately reflect the performance of systems. Finally, we demonstrate the practical utility of AE and BEM on the concrete application of minimal accurate prediction sets, reducing the number of required answers by up to x2.6. △ Less

Submitted 26 October, 2022; v1 submitted 15 February, 2022; originally announced February 2022.

Conformal Prediction Sets with Limited False Positives

Authors: Adam Fisch, Tal Schuster, Tommi Jaakkola, Regina Barzilay

Abstract: We develop a new approach to multi-label conformal prediction in which we aim to output a precise set of promising prediction candidates with a bounded number of incorrect answers. Standard conformal prediction provides the ability to adapt to model uncertainty by constructing a calibrated candidate set in place of a single prediction, with guarantees that the set contains the correct answer with… ▽ More We develop a new approach to multi-label conformal prediction in which we aim to output a precise set of promising prediction candidates with a bounded number of incorrect answers. Standard conformal prediction provides the ability to adapt to model uncertainty by constructing a calibrated candidate set in place of a single prediction, with guarantees that the set contains the correct answer with high probability. In order to obey this coverage property, however, conformal sets can become inundated with noisy candidates -- which can render them unhelpful in practice. This is particularly relevant to practical applications where there is a limited budget, and the cost (monetary or otherwise) associated with false positives is non-negligible. We propose to trade coverage for a notion of precision by enforcing that the presence of incorrect candidates in the predicted conformal sets (i.e., the total number of false positives) is bounded according to a user-specified tolerance. Subject to this constraint, our algorithm then optimizes for a generalized notion of set coverage (i.e., the true positive rate) that allows for any number of true answers for a given query (including zero). We demonstrate the effectiveness of this approach across a number of classification tasks in natural language processing, computer vision, and computational chemistry. △ Less

Submitted 15 February, 2022; originally announced February 2022.

Transformer Memory as a Differentiable Search Index

Authors: Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, Donald Metzler

Abstract: In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries… ▽ More In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup. △ Less

Submitted 21 October, 2022; v1 submitted 14 February, 2022; originally announced February 2022.

Comments: NeurIPS 2022

Unaligned but Safe -- Formally Compensating Performance Limitations for Imprecise 2D Object Detection

Authors: Tobias Schuster, Emmanouil Seferis, Simon Burton, Chih-Hong Cheng

Abstract: In this paper, we consider the imperfection within machine learning-based 2D object detection and its impact on safety. We address a special sub-type of performance limitations: the prediction bounding box cannot be perfectly aligned with the ground truth, but the computed Intersection-over-Union metric is always larger than a given threshold. Under such type of performance limitation, we formally… ▽ More In this paper, we consider the imperfection within machine learning-based 2D object detection and its impact on safety. We address a special sub-type of performance limitations: the prediction bounding box cannot be perfectly aligned with the ground truth, but the computed Intersection-over-Union metric is always larger than a given threshold. Under such type of performance limitation, we formally prove the minimum required bounding box enlargement factor to cover the ground truth. We then demonstrate that the factor can be mathematically adjusted to a smaller value, provided that the motion planner takes a fixed-length buffer in making its decisions. Finally, observing the difference between an empirically measured enlargement factor and our formally derived worst-case enlargement factor offers an interesting connection between the quantitative evidence (demonstrated by statistics) and the qualitative evidence (demonstrated by worst-case analysis). △ Less

Submitted 10 February, 2022; originally announced February 2022.

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

Authors: Vamsi Aribandi, Yi Tay, Tal Schuster, **feng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, Donald Metzler

Abstract: Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces ExMix (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using ExMix, we study the ef… ▽ More Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces ExMix (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using ExMix, we study the effect of multi-task pre-training at the largest scale to date, and analyze co-training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose ExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised ExMix. Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of ExMix. ExT5 also significantly improves sample efficiency while pre-training. △ Less

Submitted 29 January, 2022; v1 submitted 21 November, 2021; originally announced November 2021.

Comments: ICLR 2022; see https://youtu.be/FbRcbM4T-50 for a video overview of the paper

Well-defined forward operators in dynamic diffractive tensor tomography using viscosity solutions of transport equations

Authors: Lukas Vierus, Thomas Schuster

Abstract: We consider a general setting for dynamic tensor field tomography in an inhomogeneous refracting and absorbing medium as inverse source problem for the associated transport equation. Following Fermat's principle the Riemannian metric in the considered domain is generated by the refractive index of the medium. There is wealth of results for the inverse problem of recovering a tensor field from its… ▽ More We consider a general setting for dynamic tensor field tomography in an inhomogeneous refracting and absorbing medium as inverse source problem for the associated transport equation. Following Fermat's principle the Riemannian metric in the considered domain is generated by the refractive index of the medium. There is wealth of results for the inverse problem of recovering a tensor field from its longitudinal ray transform in a static euclidean setting, whereas there are only few inversion formulas and algorithms existing for general Riemannian metrics and time-dependent tensor fields. It is a well-known fact that tensor field tomography is equivalent to an inverse source problem for a transport equation where the ray transform serves as given boundary data. We prove that this result extends to the dynamic case. Interpreting dynamic tensor tomography as inverse source problem represents a holistic approach in this field. To guarantee that the forward map**s are well-defined, it is necessary to prove existence and uniqueness for the underlying transport equations. Unfortunately, the bilinear forms of the associated weak formulations do not satisfy the coercivity condition. To this end we transfer to viscosity solutions and prove their unique existence in appropriate Sobolev (static case) and Sobolev-Bochner (dynamic case) spaces under a certain assumption that allows only small variations of the refractive index. Numerical evidence is given that the viscosity solution solves the original transport equation if the viscosity term turns to zero. △ Less

Submitted 10 November, 2021; originally announced November 2021.

Comments: 23 pages, 4 figures

MSC Class: 35F10; 35F16; 45Q05

Logically Sound Arguments for the Effectiveness of ML Safety Measures

Authors: Chih-Hong Cheng, Tobias Schuster, Simon Burton

Abstract: We investigate the issues of achieving sufficient rigor in the arguments for the safety of machine learning functions. By considering the known weaknesses of DNN-based 2D bounding box detection algorithms, we sharpen the metric of imprecise pedestrian localization by associating it with the safety goal. The sharpening leads to introducing a conservative post-processor after the standard non-max-su… ▽ More We investigate the issues of achieving sufficient rigor in the arguments for the safety of machine learning functions. By considering the known weaknesses of DNN-based 2D bounding box detection algorithms, we sharpen the metric of imprecise pedestrian localization by associating it with the safety goal. The sharpening leads to introducing a conservative post-processor after the standard non-max-suppression as a counter-measure. We then propose a semi-formal assurance case for arguing the effectiveness of the post-processor, which is further translated into formal proof obligations for demonstrating the soundness of the arguments. Applying theorem proving not only discovers the need to introduce missing claims and mathematical concepts but also reveals the limitation of Dempster-Shafer's rules used in semi-formal argumentation. △ Less

Submitted 10 January, 2022; v1 submitted 4 November, 2021; originally announced November 2021.

Comments: v2: fix typos and change some phrases to make the context clear

Dynamic Process Isolation

Authors: Martin Schwarzl, Pietro Borrello, Andreas Kogler, Kenton Varda, Thomas Schuster, Daniel Gruss, Michael Schwarz

Abstract: In the quest for efficiency and performance, edge-computing providers eliminate isolation boundaries between tenants, such as strict process isolation, and instead let them compute in a more lightweight multi-threaded single-process design. Edge-computing providers support a high number of tenants per machine to reduce the physical distance to customers without requiring a large number of machines… ▽ More In the quest for efficiency and performance, edge-computing providers eliminate isolation boundaries between tenants, such as strict process isolation, and instead let them compute in a more lightweight multi-threaded single-process design. Edge-computing providers support a high number of tenants per machine to reduce the physical distance to customers without requiring a large number of machines. Isolation is provided by sandboxing mechanisms, e.g., tenants can only run sandboxed V8 JavaScript code. While this is as secure as a sandbox for software vulnerabilities, microarchitectural attacks can bypass these sandboxes. In this paper, we show that it is possible to mount a Spectre attack on such a restricted environment, leaking secrets from co-located tenants. Cloudflare Workers is one of the top three edge-computing solutions and handles millions of HTTP requests per second worldwide across tens of thousands of web sites every day. We demonstrate a remote Spectre attack using amplification techniques in combination with a remote timing server, which is capable of leaking 120 bit/h. This motivates our main contribution, Dynamic Process Isolation, a process isolation mechanism that only isolates suspicious worker scripts following a detection mechanism. In the worst case of only false positives, Dynamic Process Isolation simply degrades to process isolation. Our proof-of-concept implementation augments a real-world cloud infrastructure framework, Cloudflare Workers, which is used in production at large scale. With a false-positive rate of only 0.61%, we demonstrate that our solution vastly outperforms strict process isolation in terms of performance. In our security evaluation, we show that Dynamic Process Isolation statistically provides the same security guarantees as strict process isolation, fully mitigating Spectre attacks between multiple tenants. △ Less

Submitted 10 October, 2021; originally announced October 2021.

Programming Puzzles

Authors: Tal Schuster, Ashwin Kalyan, Oleksandr Polozov, Adam Tauman Kalai

Abstract: We introduce a new type of programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis, and release an open-source dataset of Python Programming Puzzles (P3). Each puzzle is defined by a short Python program $f$, and the goal is to find an input which makes $f$ return True. The puzzles are objective in that each one is specified entirely by t… ▽ More We introduce a new type of programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis, and release an open-source dataset of Python Programming Puzzles (P3). Each puzzle is defined by a short Python program $f$, and the goal is to find an input which makes $f$ return True. The puzzles are objective in that each one is specified entirely by the source code of its verifier $f$, so evaluating $f$ is all that is needed to test a candidate solution. They do not require an answer key or input/output examples, nor do they depend on natural language understanding. The dataset is comprehensive in that it spans problems of a range of difficulties and domains, ranging from trivial string manipulation problems, to classic programming puzzles (e.g., Tower of Hanoi), to interview/competitive-programming problems (e.g., dynamic programming), to longstanding open problems in algorithms and mathematics (e.g., factoring). We develop baseline enumerative program synthesis, GPT-3 and Codex solvers that are capable of solving puzzles -- even without access to any reference solutions -- by learning from their own past solutions. Codex performs best, solving up to 18% of 397 test problems with a single try and 80% of the problems with 1,000 tries per problem. In a small user study, we find a positive correlation between puzzle-solving performance and coding experience, and between the puzzle difficulty for humans and AI solvers. Therefore, further improvements on P3 could have a significant impact on many program synthesis areas. △ Less

Submitted 6 November, 2021; v1 submitted 10 June, 2021; originally announced June 2021.

Comments: NeurIPS 2021 (Datasets and Benchmarks Track). Puzzles repository: https://github.com/microsoft/PythonProgrammingPuzzles

Floquet Engineering Ultracold Polar Molecules to Simulate Topological Insulators

Authors: Thomas Schuster, Felix Flicker, Ming Li, Svetlana Kotochigova, Joel E. Moore, Jun Ye, Norman Y. Yao

Abstract: We present a quantitative, near-term experimental blueprint for the quantum simulation of topological insulators using lattice-trapped ultracold polar molecules. In particular, we focus on the so-called Hopf insulator, which represents a three-dimensional topological state of matter existing outside the conventional tenfold way and crystalline-symmetry-based classifications of topological insulato… ▽ More We present a quantitative, near-term experimental blueprint for the quantum simulation of topological insulators using lattice-trapped ultracold polar molecules. In particular, we focus on the so-called Hopf insulator, which represents a three-dimensional topological state of matter existing outside the conventional tenfold way and crystalline-symmetry-based classifications of topological insulators. Its topology is protected by a \emph{linking number} invariant, which necessitates long-range spin-orbit coupled hop**s for its realization. While these ingredients have so far precluded its realization in solid state systems and other quantum simulation architectures, in a companion manuscript [1901.08597] we predict that Hopf insulators can in fact arise naturally in dipolar interacting systems. Here, we investigate a specific such architecture in lattices of polar molecules, where the effective `spin' is formed from sublattice degrees of freedom. We introduce two techniques that allow one to optimize dipolar Hopf insulators with large band gaps, and which should also be readily applicable to the simulation of other exotic bandstructures. First, we describe the use of Floquet engineering to control the range and functional form of dipolar hop**s and second, we demonstrate that molecular AC polarizabilities (under circularly polarized light) can be used to precisely tune the resonance condition between different rotational states. To verify that this latter technique is amenable to current generation experiments, we calculate from first principles the AC polarizability for $σ^+$ light for ${}^{40}$K$^{87}$Rb. Finally, we show that experiments are capable of detecting the unconventional topology of the Hopf insulator by varying the termination of the lattice at its edges, which gives rise to three distinct classes of edge mode spectra. △ Less

Submitted 21 May, 2021; originally announced May 2021.

Comments: 15 pages, 8 figures. See companion manuscript arxiv:1901.08597 for an overview on realizing the Hopf insulator via dipolar interactions

Consistent Accelerated Inference via Confident Adaptive Transformers

Authors: Tal Schuster, Adam Fisch, Tommi Jaakkola, Regina Barzilay

Abstract: We develop a novel approach for confidently accelerating inference in the large and expensive multilayer Transformers that are now ubiquitous in natural language processing (NLP). Amortized or approximate computational methods increase efficiency, but can come with unpredictable performance costs. In this work, we present CATs -- Confident Adaptive Transformers -- in which we simultaneously increa… ▽ More We develop a novel approach for confidently accelerating inference in the large and expensive multilayer Transformers that are now ubiquitous in natural language processing (NLP). Amortized or approximate computational methods increase efficiency, but can come with unpredictable performance costs. In this work, we present CATs -- Confident Adaptive Transformers -- in which we simultaneously increase computational efficiency, while guaranteeing a specifiable degree of consistency with the original model with high confidence. Our method trains additional prediction heads on top of intermediate layers, and dynamically decides when to stop allocating computational effort to each input using a meta consistency classifier. To calibrate our early prediction stop** rule, we formulate a unique extension of conformal prediction. We demonstrate the effectiveness of this approach on four classification and regression tasks. △ Less

Submitted 9 September, 2021; v1 submitted 18 April, 2021; originally announced April 2021.

Comments: EMNLP 2021

Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence

Authors: Tal Schuster, Adam Fisch, Regina Barzilay

Abstract: Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VitaminC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight… ▽ More Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VitaminC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight factual changes. We collect over 100,000 Wikipedia revisions that modify an underlying fact, and leverage these revisions, together with additional synthetically constructed ones, to create a total of over 400,000 claim-evidence pairs. Unlike previous resources, the examples in VitaminC are contrastive, i.e., they contain evidence pairs that are nearly identical in language and content, with the exception that one supports a given claim while the other does not. We show that training using this design increases robustness -- improving accuracy by 10% on adversarial fact verification and 6% on adversarial natural language inference (NLI). Moreover, the structure of VitaminC leads us to define additional tasks for fact-checking resources: tagging relevant words in the evidence for verifying the claim, identifying factual revisions, and providing automatic edits via factually consistent text generation. △ Less

Submitted 15 March, 2021; originally announced March 2021.

Comments: NAACL 2021

A method for determining the parameters in a rheological model for viscoelastic materials by minimizing Tikhonov functionals

Authors: Rebecca Rothermel, Wladimir Panfilenko, Prateek Sharma, Anne Wald, Thomas Schuster, Anne Jung, Stefan Diebels

Abstract: Mathematical models describing the behavior of viscoelastic materials are often based on evolution equations that measure the change in stress depending on its material parameters such as stiffness, viscosity or relaxation time. In this article, we introduce a Maxwell-based rheological model, define the associated forward operator and the inverse problem in order to determine the number of Maxwell… ▽ More Mathematical models describing the behavior of viscoelastic materials are often based on evolution equations that measure the change in stress depending on its material parameters such as stiffness, viscosity or relaxation time. In this article, we introduce a Maxwell-based rheological model, define the associated forward operator and the inverse problem in order to determine the number of Maxwell elements and the material parameters of the underlying viscoelastic material. We perform a relaxation experiment by applying a strain to the material and measure the generated stress. Since the measured data varies with the number of Maxwell elements, the forward operator of the underlying inverse problem depends on parts of the solution. By introducing assumptions on the relaxation times, we propose a clustering algorithm to resolve this problem. We provide the calculations that are necessary for the minimization process and conclude with numerical results by investigating unperturbed as well as noisy data. We present different reconstruction approaches based on minimizing a least squares functional. Furthermore, we look at individual stress components to analyze different displacement rates. Finally, we study reconstructions with shortened data sets to obtain assertions on how long experiments have to be performed to identify conclusive material parameters. △ Less

Submitted 26 February, 2021; originally announced February 2021.

Comments: 23 pages, 11 figures, 6 tables

MSC Class: 34A55; 74D05; 74P10

Few-shot Conformal Prediction with Auxiliary Tasks

Authors: Adam Fisch, Tal Schuster, Tommi Jaakkola, Regina Barzilay

Abstract: We develop a novel approach to conformal prediction when the target task has limited data available for training. Conformal prediction identifies a small set of promising output candidates in place of a single prediction, with guarantees that the set contains the correct answer with high probability. When training data is limited, however, the predicted set can easily become unusably large. In thi… ▽ More We develop a novel approach to conformal prediction when the target task has limited data available for training. Conformal prediction identifies a small set of promising output candidates in place of a single prediction, with guarantees that the set contains the correct answer with high probability. When training data is limited, however, the predicted set can easily become unusably large. In this work, we obtain substantially tighter prediction sets while maintaining desirable marginal guarantees by casting conformal prediction as a meta-learning paradigm over exchangeable collections of auxiliary tasks. Our conformalization algorithm is simple, fast, and agnostic to the choice of underlying model, learning algorithm, or dataset. We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery. △ Less

Submitted 20 July, 2021; v1 submitted 17 February, 2021; originally announced February 2021.

Comments: ICML camera ready

Many-body quantum teleportation via operator spreading in the traversable wormhole protocol

Authors: Thomas Schuster, Bryce Kobrin, ** Gao, Iris Cong, Emil T. Khabiboulline, Norbert M. Linke, Mikhail D. Lukin, Christopher Monroe, Beni Yoshida, Norman Y. Yao

Abstract: By leveraging shared entanglement between a pair of qubits, one can teleport a quantum state from one particle to another. Recent advances have uncovered an intrinsically many-body generalization of quantum teleportation, with an elegant and surprising connection to gravity. In particular, the teleportation of quantum information relies on many-body dynamics, which originate from strongly-interact… ▽ More By leveraging shared entanglement between a pair of qubits, one can teleport a quantum state from one particle to another. Recent advances have uncovered an intrinsically many-body generalization of quantum teleportation, with an elegant and surprising connection to gravity. In particular, the teleportation of quantum information relies on many-body dynamics, which originate from strongly-interacting systems that are holographically dual to gravity; from the gravitational perspective, such quantum teleportation can be understood as the transmission of information through a traversable wormhole. Here, we propose and analyze a new mechanism for many-body quantum teleportation -- dubbed peaked-size teleportation. Intriguingly, peaked-size teleportation utilizes precisely the same type of quantum circuit as traversable wormhole teleportation, yet has a completely distinct microscopic origin: it relies upon the spreading of local operators under generic thermalizing dynamics and not gravitational physics. We demonstrate the ubiquity of peaked-size teleportation, both analytically and numerically, across a diverse landscape of physical systems, including random unitary circuits, the Sachdev-Ye-Kitaev model (at high temperatures), one-dimensional spin chains and a bulk theory of gravity with stringy corrections. Our results pave the way towards using many-body quantum teleportation as a powerful experimental tool for: (i) characterizing the size distributions of operators in strongly-correlated systems and (ii) distinguishing between generic and intrinsically gravitational scrambling dynamics. To this end, we provide a detailed experimental blueprint for realizing many-body quantum teleportation in both trapped ions and Rydberg atom arrays; effects of decoherence and experimental imperfections are analyzed. △ Less

Submitted 5 August, 2022; v1 submitted 29 January, 2021; originally announced February 2021.

Comments: 41 + 24 pages, 12 figures

Journal ref: Physical Review X 12, 031013 (2022)

Solving an inverse heat convection problem with an implicit forward operator by using a Projected Quasi-Newton method

Authors: Dimitri Rothermel, Thomas Schuster

Abstract: We consider the quasilinear 1D inverse heat convection problem (IHCP) of determining the enthalpy-dependent heat fluxes from noisy internal enthalpy measurements. This problem arises in the Accelerated Cooling (ACC) process of producing thermomechanically controlled processed (TMCP) heavy plates made of steel. In order to adjust the complex microstructure of the underlying material, the Leidenfros… ▽ More We consider the quasilinear 1D inverse heat convection problem (IHCP) of determining the enthalpy-dependent heat fluxes from noisy internal enthalpy measurements. This problem arises in the Accelerated Cooling (ACC) process of producing thermomechanically controlled processed (TMCP) heavy plates made of steel. In order to adjust the complex microstructure of the underlying material, the Leidenfrost behavior of the hot surfaces with respect to the application of the cooling fluid has to be studied. Since the heat fluxes depend on the enthalpy and hence on the solution of the underlying initial boundary value problem (IBVP), the parameter-to-solution operator, and thus the forward operator of the inverse problem, can only be defined implicitly. To guarantee well-defined operators, we study two approaches for showing existence and uniqueness of solutions of the IBVP. One approach deals with the theory of pseudomonotone operators and so-called strong solutions in Sobolev-Bochner spaces. The other theory uses classical solutions in Hölder spaces. Whereas the first approach yields a solution under milder assumptions, it fails to show the uniqueness result in contrast to the second approach. Furthermore, we propose a convenient parametrization approach for the nonlinear heat fluxes in order to decouple the parameter-to-solution relation and use an iterative solver based on a Projected Quasi-Newton (PQN) method together with box-constraints to solve the inverse problem. For numerical experiments, we derive the necessary gradient information of the objective functional and use the discrepancy principle as a stop** rule. Numerical tests show that the PQN method outperforms the Landweber method with respect to computing time and approximation accuracy. △ Less

Submitted 5 October, 2020; originally announced October 2020.

Comments: 28 pages, 8 figures

MSC Class: 35K61; 65M32

Speculative Dereferencing of Registers:Reviving Foreshadow

Authors: Martin Schwarzl, Thomas Schuster, Michael Schwarz, Daniel Gruss

Abstract: Since 2016, multiple microarchitectural attacks have exploited an effect that is attributed to prefetching. These works observe that certain user-space operations can fetch kernel addresses into the cache. Fetching user-inaccessible data into the cache enables KASLR breaks and assists various Meltdown-type attacks, especially Foreshadow. In this paper, we provide a systematic analysis of the roo… ▽ More Since 2016, multiple microarchitectural attacks have exploited an effect that is attributed to prefetching. These works observe that certain user-space operations can fetch kernel addresses into the cache. Fetching user-inaccessible data into the cache enables KASLR breaks and assists various Meltdown-type attacks, especially Foreshadow. In this paper, we provide a systematic analysis of the root cause of this prefetching effect. While we confirm the empirical results of previous papers, we show that the attribution to a prefetching mechanism is fundamentally incorrect in all previous papers describing or exploiting this effect. In particular, neither the prefetch instruction nor other user-space instructions actually prefetch kernel addresses into the cache, leading to incorrect conclusions and ineffectiveness of proposed defenses. The effect exploited in all of these papers is, in fact, caused by speculative dereferencing of user-space registers in the kernel. Hence, mitigation techniques such as KAISER do not eliminate this leakage as previously believed. Beyond our thorough analysis of these previous works, we also demonstrate new attacks enabled by understanding the root cause, namely an address-translation attack in more restricted contexts, direct leakage of register values in certain scenarios, and the first end-to-end Foreshadow (L1TF) exploit targeting non-L1 data. The latter is effective even with the recommended Foreshadow mitigations enabled and thus revives the Foreshadow attack. We demonstrate that these dereferencing effects exist even on the most recent Intel CPUs with the latest hardware mitigations, and on CPUs previously believed to be unaffected, i.e., ARM, IBM, and AMD CPUs. △ Less

Submitted 5 August, 2020; originally announced August 2020.

Comments: 16 pages, 6 figures

Efficient Conformal Prediction via Cascaded Inference with Expanded Admission

Authors: Adam Fisch, Tal Schuster, Tommi Jaakkola, Regina Barzilay

Abstract: In this paper, we present a novel approach for conformal prediction (CP), in which we aim to identify a set of promising prediction candidates -- in place of a single prediction. This set is guaranteed to contain a correct answer with high probability, and is well-suited for many open-ended classification tasks. In the standard CP paradigm, the predicted set can often be unusably large and also co… ▽ More In this paper, we present a novel approach for conformal prediction (CP), in which we aim to identify a set of promising prediction candidates -- in place of a single prediction. This set is guaranteed to contain a correct answer with high probability, and is well-suited for many open-ended classification tasks. In the standard CP paradigm, the predicted set can often be unusably large and also costly to obtain. This is particularly pervasive in settings where the correct answer is not unique, and the number of total possible answers is high. We first expand the CP correctness criterion to allow for additional, inferred "admissible" answers, which can substantially reduce the size of the predicted set while still providing valid performance guarantees. Second, we amortize costs by conformalizing prediction cascades, in which we aggressively prune implausible labels early on by using progressively stronger classifiers -- again, while still providing valid performance guarantees. We demonstrate the empirical effectiveness of our approach for multiple applications in natural language processing and computational chemistry for drug discovery. △ Less

Submitted 2 February, 2021; v1 submitted 6 July, 2020; originally announced July 2020.

Comments: ICLR 2021. Revision of "Relaxed Conformal Prediction Cascades for Efficient Inference Over Many Labels"

Quantum Information Scrambling in a Superconducting Qutrit Processor

Authors: M. S. Blok, V. V. Ramasesh, T. Schuster, K. O'Brien, J. M. Kreikebaum, D. Dahlen, A. Morvan, B. Yoshida, N. Y. Yao, I. Siddiqi

Abstract: The theory of quantum information provides a common language which links disciplines ranging from cosmology to condensed-matter physics. For example, the delocalization of quantum information in strongly-interacting many-body systems, known as quantum information scrambling, has recently begun to unite our understanding of black hole dynamics, transport in exotic non-Fermi liquids, and many-body a… ▽ More The theory of quantum information provides a common language which links disciplines ranging from cosmology to condensed-matter physics. For example, the delocalization of quantum information in strongly-interacting many-body systems, known as quantum information scrambling, has recently begun to unite our understanding of black hole dynamics, transport in exotic non-Fermi liquids, and many-body analogs of quantum chaos. To date, verified experimental implementations of scrambling have dealt only with systems comprised of two-level qubits. Higher-dimensional quantum systems, however, may exhibit different scrambling modalities and are predicted to saturate conjectured speed limits on the rate of quantum information scrambling. We take the first steps toward accessing such phenomena, by realizing a quantum processor based on superconducting qutrits (three-level quantum systems). We implement two-qutrit scrambling operations and embed them in a five-qutrit teleportation algorithm to directly measure the associated out of-time-ordered correlation functions. Measured teleportation fidelities, Favg = 0.568 +- 0001, confirm the occurrence of scrambling even in the presence of experimental imperfections. Our teleportation algorithm, which connects to recent proposals for studying traversable wormholes in the laboratory, demonstrates how quantum information processing technology based on higher dimensional systems can exploit a larger and more connected state space to achieve the resource efficient encoding of complex quantum circuits. △ Less

Submitted 10 February, 2021; v1 submitted 6 March, 2020; originally announced March 2020.

Journal ref: Phys. Rev. X 11, 021010 (2021)

Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning

Authors: Roei Schuster, Tal Schuster, Yoav Meri, Vitaly Shmatikov

Abstract: Word embeddings, i.e., low-dimensional vector representations such as GloVe and SGNS, encode word "meaning" in the sense that distances between words' vectors correspond to their semantic proximity. This enables transfer learning of semantics for a variety of natural language processing tasks. Word embeddings are typically trained on large public corpora such as Wikipedia or Twitter. We demonstr… ▽ More Word embeddings, i.e., low-dimensional vector representations such as GloVe and SGNS, encode word "meaning" in the sense that distances between words' vectors correspond to their semantic proximity. This enables transfer learning of semantics for a variety of natural language processing tasks. Word embeddings are typically trained on large public corpora such as Wikipedia or Twitter. We demonstrate that an attacker who can modify the corpus on which the embedding is trained can control the "meaning" of new and existing words by changing their locations in the embedding space. We develop an explicit expression over corpus features that serves as a proxy for distance between words and establish a causative relationship between its values and embedding distances. We then show how to use this relationship for two adversarial objectives: (1) make a word a top-ranked neighbor of another word, and (2) move a word from one semantic cluster to another. An attack on the embedding can affect diverse downstream tasks, demonstrating for the first time the power of data poisoning in transfer learning scenarios. We use this attack to manipulate query expansion in information retrieval systems such as resume search, make certain names more or less visible to named entity recognition models, and cause new words to be translated to a particular target word regardless of the language. Finally, we show how the attacker can generate linguistically likely corpus modifications, thus fooling defenses that attempt to filter implausible sentences from the corpus using a language model. △ Less

Submitted 14 January, 2020; originally announced January 2020.

Comments: Accepted at IEEE S&P 2020

Parameter estimation of temperature dependent material parameters in the cooling process of TMCP steel plates

Authors: Dimitri Rothermel, Thomas Schuster, Roland Schorr, Martin Peglow

Abstract: Accelerated cooling is a key technology in producing thermomechanically controlled processed (TMCP) steel plates. In a TMCP process hot plates are subjected to a strong cooling what results in a complex microstructure leading to increased strength and fracture toughness. The microstructure is strongly affected by the temperature evolution during the cooling process as well as residual stresses and… ▽ More Accelerated cooling is a key technology in producing thermomechanically controlled processed (TMCP) steel plates. In a TMCP process hot plates are subjected to a strong cooling what results in a complex microstructure leading to increased strength and fracture toughness. The microstructure is strongly affected by the temperature evolution during the cooling process as well as residual stresses and flatness deformations. Therefore, the full control (quantification) of the temperature evolution is very important regarding plate design and processing. It can only be achieved by a thermophysical characterization of the material and the cooling system. In this paper, we focus on the thermophysical characterization of the material parameters. Mathematically, we consider a specific inverse heat conduction problem. The temperature evolution of a heated steel plate passing through the cooling device is modeled by a 1D nonlinear partial differential equation (PDE) with unknown temperature dependent material parameters, which describe the characteristics of the underlying material. We present a numerical approach to identify these material parameters up to some canonical ambiguity without any a priori information. △ Less

Submitted 18 December, 2019; originally announced December 2019.

Comments: 14 pages, 10 figures

MSC Class: 35K55 (Primary) 49M30; 65M06; 65M32 (Secondary)