-
An Economic Solution to Copyright Challenges of Generative AI
Authors:
Jiachen T. Wang,
Zhun Deng,
Hiroaki Chiba-Okabe,
Boaz Barak,
Weijie J. Su
Abstract:
Generative artificial intelligence (AI) systems are trained on large data corpora to generate new pieces of text, images, videos, and other media. There is growing concern that such systems may infringe on the copyright interests of training data contributors. To address the copyright challenges of generative AI, we propose a framework that compensates copyright owners proportionally to their cont…
▽ More
Generative artificial intelligence (AI) systems are trained on large data corpora to generate new pieces of text, images, videos, and other media. There is growing concern that such systems may infringe on the copyright interests of training data contributors. To address the copyright challenges of generative AI, we propose a framework that compensates copyright owners proportionally to their contributions to the creation of AI-generated content. The metric for contributions is quantitatively determined by leveraging the probabilistic nature of modern generative AI models and using techniques from cooperative game theory in economics. This framework enables a platform where AI developers benefit from access to high-quality training data, thus improving model performance. Meanwhile, copyright owners receive fair compensation, driving the continued provision of relevant data for generative model training. Experiments demonstrate that our framework successfully identifies the most relevant data sources used in artwork generation, ensuring a fair and interpretable distribution of revenues among copyright owners.
△ Less
Submitted 24 April, 2024; v1 submitted 22 April, 2024;
originally announced April 2024.
-
Distinguishing the Knowable from the Unknowable with Language Models
Authors:
Gustaf Ahdritz,
Tian Qin,
Nikhil Vyas,
Boaz Barak,
Benjamin L. Edelman
Abstract:
We study the feasibility of identifying epistemic uncertainty (reflecting a lack of knowledge), as opposed to aleatoric uncertainty (reflecting entropy in the underlying distribution), in the outputs of large language models (LLMs) over free-form text. In the absence of ground-truth probabilities, we explore a setting where, in order to (approximately) disentangle a given LLM's uncertainty, a sign…
▽ More
We study the feasibility of identifying epistemic uncertainty (reflecting a lack of knowledge), as opposed to aleatoric uncertainty (reflecting entropy in the underlying distribution), in the outputs of large language models (LLMs) over free-form text. In the absence of ground-truth probabilities, we explore a setting where, in order to (approximately) disentangle a given LLM's uncertainty, a significantly larger model stands in as a proxy for the ground truth. We show that small linear probes trained on the embeddings of frozen, pretrained models accurately predict when larger models will be more confident at the token level and that probes trained on one text domain generalize to others. Going further, we propose a fully unsupervised method that achieves non-trivial accuracy on the same task. Taken together, we interpret these results as evidence that LLMs naturally contain internal representations of different types of uncertainty that could potentially be leveraged to devise more informative indicators of model confidence in diverse practical settings.
△ Less
Submitted 27 February, 2024; v1 submitted 5 February, 2024;
originally announced February 2024.
-
Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models
Authors:
Hanlin Zhang,
Benjamin L. Edelman,
Danilo Francati,
Daniele Venturi,
Giuseppe Ateniese,
Boaz Barak
Abstract:
Watermarking generative models consists of planting a statistical signal (watermark) in a model's output so that it can be later verified that the output was generated by the given model. A strong watermarking scheme satisfies the property that a computationally bounded attacker cannot erase the watermark without causing significant quality degradation. In this paper, we study the (im)possibility…
▽ More
Watermarking generative models consists of planting a statistical signal (watermark) in a model's output so that it can be later verified that the output was generated by the given model. A strong watermarking scheme satisfies the property that a computationally bounded attacker cannot erase the watermark without causing significant quality degradation. In this paper, we study the (im)possibility of strong watermarking schemes. We prove that, under well-specified and natural assumptions, strong watermarking is impossible to achieve. This holds even in the private detection algorithm setting, where the watermark insertion and detection algorithms share a secret key, unknown to the attacker. To prove this result, we introduce a generic efficient watermark attack; the attacker is not required to know the private key of the scheme or even which scheme is used. Our attack is based on two assumptions: (1) The attacker has access to a "quality oracle" that can evaluate whether a candidate output is a high-quality response to a prompt, and (2) The attacker has access to a "perturbation oracle" which can modify an output with a nontrivial probability of maintaining quality, and which induces an efficiently mixing random walk on high-quality outputs. We argue that both assumptions can be satisfied in practice by an attacker with weaker computational capabilities than the watermarked model itself, to which the attacker has only black-box access. Furthermore, our assumptions will likely only be easier to satisfy over time as models grow in capabilities and modalities. We demonstrate the feasibility of our attack by instantiating it to attack three existing watermarking schemes for large language models: Kirchenbauer et al. (2023), Kuditipudi et al. (2023), and Zhao et al. (2023). The same attack successfully removes the watermarks planted by all three schemes, with only minor quality degradation.
△ Less
Submitted 14 November, 2023; v1 submitted 7 November, 2023;
originally announced November 2023.
-
On the works of Avi Wigderson
Authors:
Boaz Barak,
Yael Kalai,
Ran Raz,
Salil Vadhan,
Nisheeth K. Vishnoi
Abstract:
This is an overview of some of the works of Avi Wigderson, 2021 Abel prize laureate. Wigderson's contributions span many fields of computer science and mathematics. In this survey we focus on four subfields: cryptography, pseudorandomness, computational complexity lower bounds, and the theory of optimization over symmetric manifolds. Even within those fields, we are not able to mention all of Wigd…
▽ More
This is an overview of some of the works of Avi Wigderson, 2021 Abel prize laureate. Wigderson's contributions span many fields of computer science and mathematics. In this survey we focus on four subfields: cryptography, pseudorandomness, computational complexity lower bounds, and the theory of optimization over symmetric manifolds. Even within those fields, we are not able to mention all of Wigderson's results, let alone cover them in full detail. However, we attempt to give a broad view of each field, as well as describe how Wigderson's papers have answered central questions, made key definitions, forged unexpected connections, or otherwise made lasting changes to our ways of thinking in that field.
△ Less
Submitted 18 July, 2023;
originally announced July 2023.
-
Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning
Authors:
Nikhil Vyas,
Depen Morwani,
Rosie Zhao,
Gal Kaplun,
Sham Kakade,
Boaz Barak
Abstract:
The success of SGD in deep learning has been ascribed by prior works to the implicit bias induced by finite batch sizes ("SGD noise"). While prior works focused on offline learning (i.e., multiple-epoch training), we study the impact of SGD noise on online (i.e., single epoch) learning. Through an extensive empirical analysis of image and language data, we demonstrate that small batch sizes do not…
▽ More
The success of SGD in deep learning has been ascribed by prior works to the implicit bias induced by finite batch sizes ("SGD noise"). While prior works focused on offline learning (i.e., multiple-epoch training), we study the impact of SGD noise on online (i.e., single epoch) learning. Through an extensive empirical analysis of image and language data, we demonstrate that small batch sizes do not confer any implicit bias advantages in online learning. In contrast to offline learning, the benefits of SGD noise in online learning are strictly computational, facilitating more cost-effective gradient steps. This suggests that SGD in the online regime can be construed as taking noisy steps along the "golden path" of the noiseless gradient descent algorithm. We study this hypothesis and provide supporting evidence in loss and function space. Our findings challenge the prevailing understanding of SGD and offer novel insights into its role in online learning.
△ Less
Submitted 7 June, 2024; v1 submitted 14 June, 2023;
originally announced June 2023.
-
Scaling Data-Constrained Language Models
Authors:
Niklas Muennighoff,
Alexander M. Rush,
Boaz Barak,
Teven Le Scao,
Aleksandra Piktus,
Nouamane Tazi,
Sampo Pyysalo,
Thomas Wolf,
Colin Raffel
Abstract:
The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the…
▽ More
The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.
△ Less
Submitted 25 October, 2023; v1 submitted 25 May, 2023;
originally announced May 2023.
-
On Provable Copyright Protection for Generative Models
Authors:
Nikhil Vyas,
Sham Kakade,
Boaz Barak
Abstract:
There is a growing concern that learned conditional generative models may output samples that are substantially similar to some copyrighted data $C$ that was in their training set. We give a formal definition of $\textit{near access-freeness (NAF)}$ and prove bounds on the probability that a model satisfying this definition outputs a sample similar to $C$, even if $C$ is included in its training s…
▽ More
There is a growing concern that learned conditional generative models may output samples that are substantially similar to some copyrighted data $C$ that was in their training set. We give a formal definition of $\textit{near access-freeness (NAF)}$ and prove bounds on the probability that a model satisfying this definition outputs a sample similar to $C$, even if $C$ is included in its training set. Roughly speaking, a generative model $p$ is $\textit{$k$-NAF}$ if for every potentially copyrighted data $C$, the output of $p$ diverges by at most $k$-bits from the output of a model $q$ that $\textit{did not access $C$ at all}$. We also give generative model learning algorithms, which efficiently modify the original generative model learning algorithm in a black box manner, that output generative models with strong bounds on the probability of sampling protected content. Furthermore, we provide promising experiments for both language (transformers) and image (diffusion) generative models, showing minimal degradation in output quality while ensuring strong protections against sampling protected content.
△ Less
Submitted 21 July, 2023; v1 submitted 21 February, 2023;
originally announced February 2023.
-
Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
Authors:
Boaz Barak,
Benjamin L. Edelman,
Surbhi Goel,
Sham Kakade,
Eran Malach,
Cyril Zhang
Abstract:
There is mounting evidence of emergent phenomena in the capabilities of deep learning methods as we scale up datasets, model sizes, and training times. While there are some accounts of how these resources modulate statistical capacity, far less is known about their effect on the computational problem of model training. This work conducts such an exploration through the lens of learning a $k$-spars…
▽ More
There is mounting evidence of emergent phenomena in the capabilities of deep learning methods as we scale up datasets, model sizes, and training times. While there are some accounts of how these resources modulate statistical capacity, far less is known about their effect on the computational problem of model training. This work conducts such an exploration through the lens of learning a $k$-sparse parity of $n$ bits, a canonical discrete search problem which is statistically easy but computationally hard. Empirically, we find that a variety of neural networks successfully learn sparse parities, with discontinuous phase transitions in the training curves. On small instances, learning abruptly occurs at approximately $n^{O(k)}$ iterations; this nearly matches SQ lower bounds, despite the apparent lack of a sparse prior. Our theoretical analysis shows that these observations are not explained by a Langevin-like mechanism, whereby SGD "stumbles in the dark" until it finds the hidden set of features (a natural algorithm which also runs in $n^{O(k)}$ time). Instead, we show that SGD gradually amplifies the sparse solution via a Fourier gap in the population gradient, making continual progress that is invisible to loss and error metrics.
△ Less
Submitted 15 January, 2023; v1 submitted 18 July, 2022;
originally announced July 2022.
-
Deconstructing Distributions: A Pointwise Framework of Learning
Authors:
Gal Kaplun,
Nikhil Ghosh,
Saurabh Garg,
Boaz Barak,
Preetum Nakkiran
Abstract:
In machine learning, we traditionally evaluate the performance of a single model, averaged over a collection of test inputs. In this work, we propose a new approach: we measure the performance of a collection of models when evaluated on a $\textit{single input point}$. Specifically, we study a point's $\textit{profile}$: the relationship between models' average performance on the test distribution…
▽ More
In machine learning, we traditionally evaluate the performance of a single model, averaged over a collection of test inputs. In this work, we propose a new approach: we measure the performance of a collection of models when evaluated on a $\textit{single input point}$. Specifically, we study a point's $\textit{profile}$: the relationship between models' average performance on the test distribution and their pointwise performance on this individual point. We find that profiles can yield new insights into the structure of both models and data -- in and out-of-distribution. For example, we empirically show that real data distributions consist of points with qualitatively different profiles. On one hand, there are "compatible" points with strong correlation between the pointwise and average performance. On the other hand, there are points with weak and even $\textit{negative}$ correlation: cases where improving overall model accuracy actually $\textit{hurts}$ performance on these inputs. We prove that these experimental observations are inconsistent with the predictions of several simplified models of learning proposed in prior work. As an application, we use profiles to construct a dataset we call CIFAR-10-NEG: a subset of CINIC-10 such that for standard models, accuracy on CIFAR-10-NEG is $\textit{negatively correlated}$ with accuracy on CIFAR-10 test. This illustrates, for the first time, an OOD dataset that completely inverts "accuracy-on-the-line" (Miller, Taori, Raghunathan, Sagawa, Koh, Shankar, Liang, Carmon, and Schmidt 2021)
△ Less
Submitted 7 June, 2022; v1 submitted 20 February, 2022;
originally announced February 2022.
-
Quantum Optimization of Maximum Independent Set using Rydberg Atom Arrays
Authors:
Sepehr Ebadi,
Alexander Keesling,
Madelyn Cain,
Tout T. Wang,
Harry Levine,
Dolev Bluvstein,
Giulia Semeghini,
Ahmed Omran,
**guo Liu,
Rhine Samajdar,
Xiu-Zhe Luo,
Beatrice Nash,
Xun Gao,
Boaz Barak,
Edward Farhi,
Subir Sachdev,
Nathan Gemelke,
Leo Zhou,
Soonwon Choi,
Hannes Pichler,
Shengtao Wang,
Markus Greiner,
Vladan Vuletic,
Mikhail D. Lukin
Abstract:
Realizing quantum speedup for practically relevant, computationally hard problems is a central challenge in quantum information science. Using Rydberg atom arrays with up to 289 qubits in two spatial dimensions, we experimentally investigate quantum algorithms for solving the Maximum Independent Set problem. We use a hardware-efficient encoding associated with Rydberg blockade, realize closed-loop…
▽ More
Realizing quantum speedup for practically relevant, computationally hard problems is a central challenge in quantum information science. Using Rydberg atom arrays with up to 289 qubits in two spatial dimensions, we experimentally investigate quantum algorithms for solving the Maximum Independent Set problem. We use a hardware-efficient encoding associated with Rydberg blockade, realize closed-loop optimization to test several variational algorithms, and subsequently apply them to systematically explore a class of graphs with programmable connectivity. We find the problem hardness is controlled by the solution degeneracy and number of local minima, and experimentally benchmark the quantum algorithm's performance against classical simulated annealing. On the hardest graphs, we observe a superlinear quantum speedup in finding exact solutions in the deep circuit regime and analyze its origins.
△ Less
Submitted 18 February, 2022;
originally announced February 2022.
-
Limitations of Linear Cross-Entropy as a Measure for Quantum Advantage
Authors:
Xun Gao,
Marcin Kalinowski,
Chi-Ning Chou,
Mikhail D. Lukin,
Boaz Barak,
Soonwon Choi
Abstract:
Demonstrating quantum advantage requires experimental implementation of a computational task that is hard to achieve using state-of-the-art classical systems. One approach is to perform sampling from a probability distribution associated with a class of highly entangled many-body wavefunctions. It has been suggested that this approach can be certified with the Linear Cross-Entropy Benchmark (XEB).…
▽ More
Demonstrating quantum advantage requires experimental implementation of a computational task that is hard to achieve using state-of-the-art classical systems. One approach is to perform sampling from a probability distribution associated with a class of highly entangled many-body wavefunctions. It has been suggested that this approach can be certified with the Linear Cross-Entropy Benchmark (XEB). We critically examine this notion. First, in a "benign" setting where an honest implementation of noisy quantum circuits is assumed, we characterize the conditions under which the XEB approximates the fidelity. Second, in an "adversarial" setting where all possible classical algorithms are considered for comparison, we show that achieving relatively high XEB values does not imply faithful simulation of quantum dynamics. We present an efficient classical algorithm that, with 1 GPU within 2s, yields high XEB values, namely 2-12% of those obtained in experiments. By identifying and exploiting several vulnerabilities of the XEB, we achieve high XEB values without full simulation of quantum circuits. Remarkably, our algorithm features better scaling with the system size than noisy quantum devices for commonly studied random circuit ensembles. To quantitatively explain the success of our algorithm and the limitations of the XEB, we use a theoretical framework in which the average XEB and fidelity are mapped to statistical models. We illustrate the relation between the XEB and the fidelity for quantum circuits in various architectures, with different gate choices, and in the presence of noise. Our results show that XEB's utility as a proxy for fidelity hinges on several conditions, which must be checked in the benign setting but cannot be assumed in the adversarial setting. Thus, the XEB alone has limited utility as a benchmark for quantum advantage. We discuss ways to overcome these limitations.
△ Less
Submitted 2 December, 2021;
originally announced December 2021.
-
Revisiting Model Stitching to Compare Neural Representations
Authors:
Yamini Bansal,
Preetum Nakkiran,
Boaz Barak
Abstract:
We revisit and extend model stitching (Lenc & Vedaldi 2015) as a methodology to study the internal representations of neural networks. Given two trained and frozen models $A$ and $B$, we consider a "stitched model'' formed by connecting the bottom-layers of $A$ to the top-layers of $B$, with a simple trainable layer between them. We argue that model stitching is a powerful and perhaps under-apprec…
▽ More
We revisit and extend model stitching (Lenc & Vedaldi 2015) as a methodology to study the internal representations of neural networks. Given two trained and frozen models $A$ and $B$, we consider a "stitched model'' formed by connecting the bottom-layers of $A$ to the top-layers of $B$, with a simple trainable layer between them. We argue that model stitching is a powerful and perhaps under-appreciated tool, which reveals aspects of representations that measures such as centered kernel alignment (CKA) cannot. Through extensive experiments, we use model stitching to obtain quantitative verifications for intuitive statements such as "good networks learn similar representations'', by demonstrating that good networks of the same architecture, but trained in very different ways (e.g.: supervised vs. self-supervised learning), can be stitched to each other without drop in performance. We also give evidence for the intuition that "more is better'' by showing that representations learnt with (1) more data, (2) bigger width, or (3) more training time can be "plugged in'' to weaker models to improve performance. Finally, our experiments reveal a new structural property of SGD which we call "stitching connectivity'', akin to mode-connectivity: typical minima reached by SGD can all be stitched to each other with minimal change in accuracy.
△ Less
Submitted 14 June, 2021;
originally announced June 2021.
-
Classical algorithms and quantum limitations for maximum cut on high-girth graphs
Authors:
Boaz Barak,
Kunal Marwaha
Abstract:
We study the performance of local quantum algorithms such as the Quantum Approximate Optimization Algorithm (QAOA) for the maximum cut problem, and their relationship to that of classical algorithms.
(1) We prove that every (quantum or classical) one-local algorithm achieves on $D$-regular graphs of girth $> 5$ a maximum cut of at most $1/2 + C/\sqrt{D}$ for $C=1/\sqrt{2} \approx 0.7071$. This i…
▽ More
We study the performance of local quantum algorithms such as the Quantum Approximate Optimization Algorithm (QAOA) for the maximum cut problem, and their relationship to that of classical algorithms.
(1) We prove that every (quantum or classical) one-local algorithm achieves on $D$-regular graphs of girth $> 5$ a maximum cut of at most $1/2 + C/\sqrt{D}$ for $C=1/\sqrt{2} \approx 0.7071$. This is the first such result showing that one-local algorithms achieve a value bounded away from the true optimum for random graphs, which is $1/2 + P_*/\sqrt{D} + o(1/\sqrt{D})$ for $P_* \approx 0.7632$. (2) We show that there is a classical $k$-local algorithm that achieves a value of $1/2 + C/\sqrt{D} - O(1/\sqrt{k})$ for $D$-regular graphs of girth $> 2k+1$, where $C = 2/π\approx 0.6366$. This is an algorithmic version of the existential bound of Lyons and is related to the algorithm of Aizenman, Lebowitz, and Ruelle (ALR) for the Sherrington-Kirkpatrick model. This bound is better than that achieved by the one-local and two-local versions of QAOA on high-girth graphs. (3) Through computational experiments, we give evidence that the ALR algorithm achieves better performance than constant-locality QAOA for random $D$-regular graphs, as well as other natural instances, including graphs that do have short cycles.
Our experimental work suggests that it could be possible to extend beyond our theoretical constraints. This points at the tantalizing possibility that $O(1)$-local quantum maximum-cut algorithms might be *pointwise dominated* by polynomial-time classical algorithms, in the sense that there is a classical algorithm outputting cuts of equal or better quality *on every possible instance*. This is in contrast to the evidence that polynomial-time algorithms cannot simulate the probability distributions induced by local quantum algorithms.
△ Less
Submitted 10 June, 2021;
originally announced June 2021.
-
Named Tensor Notation
Authors:
David Chiang,
Alexander M. Rush,
Boaz Barak
Abstract:
We propose a notation for tensors with named axes, which relieves the author, reader, and future implementers of machine learning models from the burden of kee** track of the order of axes and the purpose of each. The notation makes it easy to lift operations on low-order tensors to higher order ones, for example, from images to minibatches of images, or from an attention mechanism to multiple a…
▽ More
We propose a notation for tensors with named axes, which relieves the author, reader, and future implementers of machine learning models from the burden of kee** track of the order of axes and the purpose of each. The notation makes it easy to lift operations on low-order tensors to higher order ones, for example, from images to minibatches of images, or from an attention mechanism to multiple attention heads.
After a brief overview and formal definition of the notation, we illustrate it through several examples from modern machine learning, from building blocks like attention and convolution to full models like Transformers and LeNet. We then discuss differential calculus in our notation and compare with some alternative notations. Our proposals build on ideas from many previous papers and software libraries. We hope that our notation will encourage more authors to use named tensors, resulting in clearer papers and more precise implementations.
△ Less
Submitted 17 January, 2023; v1 submitted 25 February, 2021;
originally announced February 2021.
-
For self-supervised learning, Rationality implies generalization, provably
Authors:
Yamini Bansal,
Gal Kaplun,
Boaz Barak
Abstract:
We prove a new upper bound on the generalization gap of classifiers that are obtained by first using self-supervision to learn a representation $r$ of the training data, and then fitting a simple (e.g., linear) classifier $g$ to the labels. Specifically, we show that (under the assumptions described below) the generalization gap of such classifiers tends to zero if $\mathsf{C}(g) \ll n$, where…
▽ More
We prove a new upper bound on the generalization gap of classifiers that are obtained by first using self-supervision to learn a representation $r$ of the training data, and then fitting a simple (e.g., linear) classifier $g$ to the labels. Specifically, we show that (under the assumptions described below) the generalization gap of such classifiers tends to zero if $\mathsf{C}(g) \ll n$, where $\mathsf{C}(g)$ is an appropriately-defined measure of the simple classifier $g$'s complexity, and $n$ is the number of training samples. We stress that our bound is independent of the complexity of the representation $r$. We do not make any structural or conditional-independence assumptions on the representation-learning task, which can use the same training dataset that is later used for classification. Rather, we assume that the training procedure satisfies certain natural noise-robustness (adding small amount of label noise causes small degradation in performance) and rationality (getting the wrong label is not better than getting no label at all) conditions that widely hold across many standard architectures. We show that our bound is non-vacuous for many popular representation-learning based classifiers on CIFAR-10 and ImageNet, including SimCLR, AMDIM and MoCo.
△ Less
Submitted 16 October, 2020;
originally announced October 2020.
-
Optimizing testing policies for detecting COVID-19 outbreaks
Authors:
Janni Yuval,
Mor Nitzan,
Neta Ravid Tannenbaum,
Boaz Barak
Abstract:
The COVID-19 pandemic poses challenges for continuing economic activity while reducing health risks. While these challenges can be mitigated through testing, testing budget is often limited. Here we study how institutions, such as nursing homes, should utilize a fixed test budget for early detection of an outbreak. Using an extended network-SEIR model, we show that given a certain budget of tests,…
▽ More
The COVID-19 pandemic poses challenges for continuing economic activity while reducing health risks. While these challenges can be mitigated through testing, testing budget is often limited. Here we study how institutions, such as nursing homes, should utilize a fixed test budget for early detection of an outbreak. Using an extended network-SEIR model, we show that given a certain budget of tests, it is generally better to test smaller subgroups of the population frequently than to test larger groups but less frequently. The numerical results are consistent with an analytical expression we derive for the size of the outbreak at detection in an exponential spread model. Our work provides a simple guideline for institutions: distribute your total tests over several batches instead of using them all at once. We expect that in the appropriate scenarios, this easy-to-implement policy recommendation will lead to earlier detection and better mitigation of local COVID-19 outbreaks.
△ Less
Submitted 1 January, 2021; v1 submitted 9 July, 2020;
originally announced July 2020.
-
Playing Unique Games on Certified Small-Set Expanders
Authors:
Mitali Bafna,
Boaz Barak,
Pravesh Kothari,
Tselil Schramm,
David Steurer
Abstract:
We give an algorithm for solving unique games (UG) instances whenever low-degree sum-of-squares proofs certify good bounds on the small-set-expansion of the underlying constraint graph via a hypercontractive inequality. Our algorithm is in fact more versatile, and succeeds even when the constraint graph is not a small-set expander as long as the structure of non-expanding small sets is (informally…
▽ More
We give an algorithm for solving unique games (UG) instances whenever low-degree sum-of-squares proofs certify good bounds on the small-set-expansion of the underlying constraint graph via a hypercontractive inequality. Our algorithm is in fact more versatile, and succeeds even when the constraint graph is not a small-set expander as long as the structure of non-expanding small sets is (informally speaking) "characterized" by a low-degree sum-of-squares proof. Our results are obtained by rounding \emph{low-entropy} solutions -- measured via a new global potential function -- to sum-of-squares (SoS) semidefinite programs. This technique adds to the (currently short) list of general tools for analyzing SoS relaxations for \emph{worst-case} optimization problems.
As corollaries, we obtain the first polynomial-time algorithms for solving any UG instance where the constraint graph is either the \emph{noisy hypercube}, the \emph{short code} or the \emph{Johnson} graph. The prior best algorithm for such instances was the eigenvalue enumeration algorithm of Arora, Barak, and Steurer (2010) which requires quasi-polynomial time for the noisy hypercube and nearly-exponential time for the short code and Johnson graphs. All of our results achieve an approximation of $1-ε$ vs $δ$ for UG instances, where $ε>0$ and $δ> 0$ depend on the expansion parameters of the graph but are independent of the alphabet size.
△ Less
Submitted 26 June, 2021; v1 submitted 17 June, 2020;
originally announced June 2020.
-
Spoofing Linear Cross-Entropy Benchmarking in Shallow Quantum Circuits
Authors:
Boaz Barak,
Chi-Ning Chou,
Xun Gao
Abstract:
The linear cross-entropy benchmark (Linear XEB) has been used as a test for procedures simulating quantum circuits. Given a quantum circuit $C$ with $n$ inputs and outputs and purported simulator whose output is distributed according to a distribution $p$ over $\{0,1\}^n$, the linear XEB fidelity of the simulator is $\mathcal{F}_{C}(p) = 2^n \mathbb{E}_{x \sim p} q_C(x) -1$ where $q_C(x)$ is the p…
▽ More
The linear cross-entropy benchmark (Linear XEB) has been used as a test for procedures simulating quantum circuits. Given a quantum circuit $C$ with $n$ inputs and outputs and purported simulator whose output is distributed according to a distribution $p$ over $\{0,1\}^n$, the linear XEB fidelity of the simulator is $\mathcal{F}_{C}(p) = 2^n \mathbb{E}_{x \sim p} q_C(x) -1$ where $q_C(x)$ is the probability that $x$ is output from the distribution $C|0^n\rangle$. A trivial simulator (e.g., the uniform distribution) satisfies $\mathcal{F}_C(p)=0$, while Google's noisy quantum simulation of a 53 qubit circuit $C$ achieved a fidelity value of $(2.24\pm0.21)\times10^{-3}$ (Arute et. al., Nature'19).
In this work we give a classical randomized algorithm that for a given circuit $C$ of depth $d$ with Haar random 2-qubit gates achieves in expectation a fidelity value of $Ω(\tfrac{n}{L} \cdot 15^{-d})$ in running time $\textsf{poly}(n,2^L)$. Here $L$ is the size of the \emph{light cone} of $C$: the maximum number of input bits that each output bit depends on. In particular, we obtain a polynomial-time algorithm that achieves large fidelity of $ω(1)$ for depth $O(\sqrt{\log n})$ two-dimensional circuits. To our knowledge, this is the first such result for two dimensional circuits of super-constant depth. Our results can be considered as an evidence that fooling the linear XEB test might be easier than achieving a full simulation of the quantum circuit.
△ Less
Submitted 5 May, 2020;
originally announced May 2020.
-
On Higher-Order Cryptography (Long Version)
Authors:
Boaz Barak,
Raphaëlle Crubillé,
Ugo Dal Lago
Abstract:
Type-two constructions abound in cryptography: adversaries for encryption and authentication schemes, if active, are modeled as algorithms having access to oracles, i.e. as second-order algorithms. But how about making cryptographic schemes themselves higher-order? This paper gives an answer to this question, by first describing why higher-order cryptography is interesting as an object of study, t…
▽ More
Type-two constructions abound in cryptography: adversaries for encryption and authentication schemes, if active, are modeled as algorithms having access to oracles, i.e. as second-order algorithms. But how about making cryptographic schemes themselves higher-order? This paper gives an answer to this question, by first describing why higher-order cryptography is interesting as an object of study, then showing how the concept of probabilistic polynomial time algorithm can be generalized so as to encompass algorithms of order strictly higher than two, and finally proving some positive and negative results about the existence of higher-order cryptographic primitives, namely authentication schemes and pseudorandom functions.
△ Less
Submitted 17 February, 2020;
originally announced February 2020.
-
Deep Double Descent: Where Bigger Models and More Data Hurt
Authors:
Preetum Nakkiran,
Gal Kaplun,
Yamini Bansal,
Tristan Yang,
Boaz Barak,
Ilya Sutskever
Abstract:
We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effect…
▽ More
We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.
△ Less
Submitted 4 December, 2019;
originally announced December 2019.
-
SGD on Neural Networks Learns Functions of Increasing Complexity
Authors:
Preetum Nakkiran,
Gal Kaplun,
Dimitris Kalimeris,
Tristan Yang,
Benjamin L. Edelman,
Fred Zhang,
Boaz Barak
Abstract:
We perform an experimental study of the dynamics of Stochastic Gradient Descent (SGD) in learning deep neural networks for several real and synthetic classification tasks. We show that in the initial epochs, almost all of the performance improvement of the classifier obtained by SGD can be explained by a linear classifier. More generally, we give evidence for the hypothesis that, as iterations pro…
▽ More
We perform an experimental study of the dynamics of Stochastic Gradient Descent (SGD) in learning deep neural networks for several real and synthetic classification tasks. We show that in the initial epochs, almost all of the performance improvement of the classifier obtained by SGD can be explained by a linear classifier. More generally, we give evidence for the hypothesis that, as iterations progress, SGD learns functions of increasing complexity. This hypothesis can be helpful in explaining why SGD-learned classifiers tend to generalize well even in the over-parameterized regime. We also show that the linear classifier learned in the initial stages is "retained" throughout the execution even if training is continued to the point of zero training error, and complement this with a theoretical result in a simplified model. Key to our work is a new measure of how well one classifier explains the performance of another, based on conditional mutual information.
△ Less
Submitted 28 May, 2019;
originally announced May 2019.
-
(Nearly) Efficient Algorithms for the Graph Matching Problem on Correlated Random Graphs
Authors:
Boaz Barak,
Chi-Ning Chou,
Zhixian Lei,
Tselil Schramm,
Yueqi Sheng
Abstract:
We give a quasipolynomial time algorithm for the graph matching problem (also known as noisy or robust graph isomorphism) on correlated random graphs. Specifically, for every $γ>0$, we give a $n^{O(\log n)}$ time algorithm that given a pair of $γ$-correlated $G(n,p)$ graphs $G_0,G_1$ with average degree between $n^{\varepsilon}$ and $n^{1/153}$ for $\varepsilon = o(1)$, recovers the "ground truth"…
▽ More
We give a quasipolynomial time algorithm for the graph matching problem (also known as noisy or robust graph isomorphism) on correlated random graphs. Specifically, for every $γ>0$, we give a $n^{O(\log n)}$ time algorithm that given a pair of $γ$-correlated $G(n,p)$ graphs $G_0,G_1$ with average degree between $n^{\varepsilon}$ and $n^{1/153}$ for $\varepsilon = o(1)$, recovers the "ground truth" permutation $π\in S_n$ that matches the vertices of $G_0$ to the vertices of $G_n$ in the way that minimizes the number of mismatched edges. We also give a recovery algorithm for a denser regime, and a polynomial-time algorithm for distinguishing between correlated and uncorrelated graphs.
Prior work showed that recovery is information-theoretically possible in this model as long the average degree was at least $\log n$, but sub-exponential time algorithms were only known in the dense case (i.e., for $p > n^{-o(1)}$). Moreover, "Percolation Graph Matching", which is the most common heuristic for this problem, has been shown to require knowledge of $n^{Ω(1)}$ "seeds" (i.e., input/output pairs of the permutation $π$) to succeed in this regime. In contrast our algorithms require no seed and succeed for $p$ which is as low as $n^{o(1)-1}$.
△ Less
Submitted 30 January, 2019; v1 submitted 7 May, 2018;
originally announced May 2018.
-
Small-Set Expansion in Shortcode Graph and the 2-to-2 Conjecture
Authors:
Boaz Barak,
Pravesh K. Kothari,
David Steurer
Abstract:
Dinur, Khot, Kindler, Minzer and Safra (2016) recently showed that the (imperfect completeness variant of) Khot's 2 to 2 games conjecture follows from a combinatorial hypothesis about the soundness of a certain "Grassmanian agreement tester". In this work, we show that the hypothesis of Dinur et. al. follows from a conjecture we call the "Inverse Shortcode Hypothesis" characterizing the non-expand…
▽ More
Dinur, Khot, Kindler, Minzer and Safra (2016) recently showed that the (imperfect completeness variant of) Khot's 2 to 2 games conjecture follows from a combinatorial hypothesis about the soundness of a certain "Grassmanian agreement tester". In this work, we show that the hypothesis of Dinur et. al. follows from a conjecture we call the "Inverse Shortcode Hypothesis" characterizing the non-expanding sets of the degree-two shortcode graph. We also show the latter conjecture is equivalent to a characterization of the non-expanding sets in the Grassman graph, as hypothesized by a follow-up paper of Dinur et. al. (2017).
Following our work, Khot, Minzer and Safra (2018) proved the "Inverse Shortcode Hypothesis". Combining their proof with our result and the reduction of Dinur et. al. (2016), completes the proof of the 2 to 2 conjecture with imperfect completeness. Moreover, we believe that the shortcode graph provides a useful view of both the hypothesis and the reduction, and might be useful in extending it further.
△ Less
Submitted 23 April, 2018;
originally announced April 2018.
-
Quantum entanglement, sum of squares, and the log rank conjecture
Authors:
Boaz Barak,
Pravesh Kothari,
David Steurer
Abstract:
For every $ε>0$, we give an $\exp(\tilde{O}(\sqrt{n}/ε^2))$-time algorithm for the $1$ vs $1-ε$ \emph{Best Separable State (BSS)} problem of distinguishing, given an $n^2\times n^2$ matrix $\mathcal{M}$ corresponding to a quantum measurement, between the case that there is a separable (i.e., non-entangled) state $ρ$ that $\mathcal{M}$ accepts with probability $1$, and the case that every separable…
▽ More
For every $ε>0$, we give an $\exp(\tilde{O}(\sqrt{n}/ε^2))$-time algorithm for the $1$ vs $1-ε$ \emph{Best Separable State (BSS)} problem of distinguishing, given an $n^2\times n^2$ matrix $\mathcal{M}$ corresponding to a quantum measurement, between the case that there is a separable (i.e., non-entangled) state $ρ$ that $\mathcal{M}$ accepts with probability $1$, and the case that every separable state is accepted with probability at most $1-ε$. Equivalently, our algorithm takes the description of a subspace $\mathcal{W} \subseteq \mathbb{F}^{n^2}$ (where $\mathbb{F}$ can be either the real or complex field) and distinguishes between the case that $\mathcal{W}$ contains a rank one matrix, and the case that every rank one matrix is at least $ε$ far (in $\ell_2$ distance) from $\mathcal{W}$.
To the best of our knowledge, this is the first improvement over the brute-force $\exp(n)$-time algorithm for this problem. Our algorithm is based on the \emph{sum-of-squares} hierarchy and its analysis is inspired by Lovett's proof (STOC '14, JACM '16) that the communication complexity of every rank-$n$ Boolean matrix is bounded by $\tilde{O}(\sqrt{n})$.
△ Less
Submitted 9 July, 2017; v1 submitted 23 January, 2017;
originally announced January 2017.
-
A Nearly Tight Sum-of-Squares Lower Bound for the Planted Clique Problem
Authors:
Boaz Barak,
Samuel B. Hopkins,
Jonathan Kelner,
Pravesh K. Kothari,
Ankur Moitra,
Aaron Potechin
Abstract:
We prove that with high probability over the choice of a random graph $G$ from the Erdős-Rényi distribution $G(n,1/2)$, the $n^{O(d)}$-time degree $d$ Sum-of-Squares semidefinite programming relaxation for the clique problem will give a value of at least $n^{1/2-c(d/\log n)^{1/2}}$ for some constant $c>0$. This yields a nearly tight $n^{1/2 - o(1)}$ bound on the value of this program for any degre…
▽ More
We prove that with high probability over the choice of a random graph $G$ from the Erdős-Rényi distribution $G(n,1/2)$, the $n^{O(d)}$-time degree $d$ Sum-of-Squares semidefinite programming relaxation for the clique problem will give a value of at least $n^{1/2-c(d/\log n)^{1/2}}$ for some constant $c>0$. This yields a nearly tight $n^{1/2 - o(1)}$ bound on the value of this program for any degree $d = o(\log n)$. Moreover we introduce a new framework that we call \emph{pseudo-calibration} to construct Sum of Squares lower bounds. This framework is inspired by taking a computational analog of Bayesian probability theory. It yields a general recipe for constructing good pseudo-distributions (i.e., dual certificates for the Sum-of-Squares semidefinite program), and sheds further light on the ways in which this hierarchy differs from others.
△ Less
Submitted 12 April, 2016; v1 submitted 11 April, 2016;
originally announced April 2016.
-
Beating the random assignment on constraint satisfaction problems of bounded degree
Authors:
Boaz Barak,
Ankur Moitra,
Ryan O'Donnell,
Prasad Raghavendra,
Oded Regev,
David Steurer,
Luca Trevisan,
Aravindan Vijayaraghavan,
David Witmer,
John Wright
Abstract:
We show that for any odd $k$ and any instance of the Max-kXOR constraint satisfaction problem, there is an efficient algorithm that finds an assignment satisfying at least a $\frac{1}{2} + Ω(1/\sqrt{D})$ fraction of constraints, where $D$ is a bound on the number of constraints that each variable occurs in. This improves both qualitatively and quantitatively on the recent work of Farhi, Goldstone,…
▽ More
We show that for any odd $k$ and any instance of the Max-kXOR constraint satisfaction problem, there is an efficient algorithm that finds an assignment satisfying at least a $\frac{1}{2} + Ω(1/\sqrt{D})$ fraction of constraints, where $D$ is a bound on the number of constraints that each variable occurs in. This improves both qualitatively and quantitatively on the recent work of Farhi, Goldstone, and Gutmann (2014), which gave a \emph{quantum} algorithm to find an assignment satisfying a $\frac{1}{2} + Ω(D^{-3/4})$ fraction of the equations.
For arbitrary constraint satisfaction problems, we give a similar result for "triangle-free" instances; i.e., an efficient algorithm that finds an assignment satisfying at least a $μ+ Ω(1/\sqrt{D})$ fraction of constraints, where $μ$ is the fraction that would be satisfied by a uniformly random assignment.
△ Less
Submitted 11 August, 2015; v1 submitted 13 May, 2015;
originally announced May 2015.
-
Noisy Tensor Completion via the Sum-of-Squares Hierarchy
Authors:
Boaz Barak,
Ankur Moitra
Abstract:
In the noisy tensor completion problem we observe $m$ entries (whose location is chosen uniformly at random) from an unknown $n_1 \times n_2 \times n_3$ tensor $T$. We assume that $T$ is entry-wise close to being rank $r$. Our goal is to fill in its missing entries using as few observations as possible. Let $n = \max(n_1, n_2, n_3)$. We show that if $m = n^{3/2} r$ then there is a polynomial time…
▽ More
In the noisy tensor completion problem we observe $m$ entries (whose location is chosen uniformly at random) from an unknown $n_1 \times n_2 \times n_3$ tensor $T$. We assume that $T$ is entry-wise close to being rank $r$. Our goal is to fill in its missing entries using as few observations as possible. Let $n = \max(n_1, n_2, n_3)$. We show that if $m = n^{3/2} r$ then there is a polynomial time algorithm based on the sixth level of the sum-of-squares hierarchy for completing it. Our estimate agrees with almost all of $T$'s entries almost exactly and works even when our observations are corrupted by noise. This is also the first algorithm for tensor completion that works in the overcomplete case when $r > n$, and in fact it works all the way up to $r = n^{3/2-ε}$.
Our proofs are short and simple and are based on establishing a new connection between noisy tensor completion (through the language of Rademacher complexity) and the task of refuting random constant satisfaction problems. This connection seems to have gone unnoticed even in the context of matrix completion. Furthermore, we use this connection to show matching lower bounds. Our main technical result is in characterizing the Rademacher complexity of the sequence of norms that arise in the sum-of-squares relaxations to the tensor nuclear norm. These results point to an interesting new direction: Can we explore computational vs. sample complexity tradeoffs through the sum-of-squares hierarchy?
△ Less
Submitted 18 February, 2016; v1 submitted 26 January, 2015;
originally announced January 2015.
-
Sum of Squares Lower Bounds from Pairwise Independence
Authors:
Boaz Barak,
Siu On Chan,
Pravesh Kothari
Abstract:
We prove that for every $ε>0$ and predicate $P:\{0,1\}^k\rightarrow \{0,1\}$ that supports a pairwise independent distribution, there exists an instance $\mathcal{I}$ of the $\mathsf{Max}P$ constraint satisfaction problem on $n$ variables such that no assignment can satisfy more than a $\tfrac{|P^{-1}(1)|}{2^k}+ε$ fraction of $\mathcal{I}$'s constraints but the degree $Ω(n)$ Sum of Squares semidef…
▽ More
We prove that for every $ε>0$ and predicate $P:\{0,1\}^k\rightarrow \{0,1\}$ that supports a pairwise independent distribution, there exists an instance $\mathcal{I}$ of the $\mathsf{Max}P$ constraint satisfaction problem on $n$ variables such that no assignment can satisfy more than a $\tfrac{|P^{-1}(1)|}{2^k}+ε$ fraction of $\mathcal{I}$'s constraints but the degree $Ω(n)$ Sum of Squares semidefinite programming hierarchy cannot certify that $\mathcal{I}$ is unsatisfiable. Similar results were previously only known for weaker hierarchies.
△ Less
Submitted 26 March, 2015; v1 submitted 4 January, 2015;
originally announced January 2015.
-
Dictionary Learning and Tensor Decomposition via the Sum-of-Squares Method
Authors:
Boaz Barak,
Jonathan A. Kelner,
David Steurer
Abstract:
We give a new approach to the dictionary learning (also known as "sparse coding") problem of recovering an unknown $n\times m$ matrix $A$ (for $m \geq n$) from examples of the form \[ y = Ax + e, \] where $x$ is a random vector in $\mathbb R^m$ with at most $τm$ nonzero coordinates, and $e$ is a random noise vector in $\mathbb R^n$ with bounded magnitude. For the case $m=O(n)$, our algorithm recov…
▽ More
We give a new approach to the dictionary learning (also known as "sparse coding") problem of recovering an unknown $n\times m$ matrix $A$ (for $m \geq n$) from examples of the form \[ y = Ax + e, \] where $x$ is a random vector in $\mathbb R^m$ with at most $τm$ nonzero coordinates, and $e$ is a random noise vector in $\mathbb R^n$ with bounded magnitude. For the case $m=O(n)$, our algorithm recovers every column of $A$ within arbitrarily good constant accuracy in time $m^{O(\log m/\log(τ^{-1}))}$, in particular achieving polynomial time if $τ= m^{-δ}$ for any $δ>0$, and time $m^{O(\log m)}$ if $τ$ is (a sufficiently small) constant. Prior algorithms with comparable assumptions on the distribution required the vector $x$ to be much sparser---at most $\sqrt{n}$ nonzero coordinates---and there were intrinsic barriers preventing these algorithms from applying for denser $x$.
We achieve this by designing an algorithm for noisy tensor decomposition that can recover, under quite general conditions, an approximate rank-one decomposition of a tensor $T$, given access to a tensor $T'$ that is $τ$-close to $T$ in the spectral norm (when considered as a matrix). To our knowledge, this is the first algorithm for tensor decomposition that works in the constant spectral-norm noise regime, where there is no guarantee that the local optima of $T$ and $T'$ have similar structures.
Our algorithm is based on a novel approach to using and analyzing the Sum of Squares semidefinite programming hierarchy (Parrilo 2000, Lasserre 2001), and it can be viewed as an indication of the utility of this very general and powerful tool for unsupervised learning problems.
△ Less
Submitted 7 November, 2014; v1 submitted 6 July, 2014;
originally announced July 2014.
-
Sum-of-squares proofs and the quest toward optimal algorithms
Authors:
Boaz Barak,
David Steurer
Abstract:
In order to obtain the best-known guarantees, algorithms are traditionally tailored to the particular problem we want to solve. Two recent developments, the Unique Games Conjecture (UGC) and the Sum-of-Squares (SOS) method, surprisingly suggest that this tailoring is not necessary and that a single efficient algorithm could achieve best possible guarantees for a wide range of different problems.…
▽ More
In order to obtain the best-known guarantees, algorithms are traditionally tailored to the particular problem we want to solve. Two recent developments, the Unique Games Conjecture (UGC) and the Sum-of-Squares (SOS) method, surprisingly suggest that this tailoring is not necessary and that a single efficient algorithm could achieve best possible guarantees for a wide range of different problems.
The Unique Games Conjecture (UGC) is a tantalizing conjecture in computational complexity, which, if true, will shed light on the complexity of a great many problems. In particular this conjecture predicts that a single concrete algorithm provides optimal guarantees among all efficient algorithms for a large class of computational problems.
The Sum-of-Squares (SOS) method is a general approach for solving systems of polynomial constraints. This approach is studied in several scientific disciplines, including real algebraic geometry, proof complexity, control theory, and mathematical programming, and has found applications in fields as diverse as quantum information theory, formal verification, game theory and many others.
We survey some connections that were recently uncovered between the Unique Games Conjecture and the Sum-of-Squares method. In particular, we discuss new tools to rigorously bound the running time of the SOS method for obtaining approximate solutions to hard optimization problems, and how these tools give the potential for the sum-of-squares method to provide new guarantees for many problems of interest, and possibly to even refute the UGC.
△ Less
Submitted 27 May, 2014; v1 submitted 21 April, 2014;
originally announced April 2014.
-
Rounding Sum-of-Squares Relaxations
Authors:
Boaz Barak,
Jonathan Kelner,
David Steurer
Abstract:
We present a general approach to rounding semidefinite programming relaxations obtained by the Sum-of-Squares method (Lasserre hierarchy). Our approach is based on using the connection between these relaxations and the Sum-of-Squares proof system to transform a *combining algorithm* -- an algorithm that maps a distribution over solutions into a (possibly weaker) solution -- into a *rounding algori…
▽ More
We present a general approach to rounding semidefinite programming relaxations obtained by the Sum-of-Squares method (Lasserre hierarchy). Our approach is based on using the connection between these relaxations and the Sum-of-Squares proof system to transform a *combining algorithm* -- an algorithm that maps a distribution over solutions into a (possibly weaker) solution -- into a *rounding algorithm* that maps a solution of the relaxation to a solution of the original problem.
Using this approach, we obtain algorithms that yield improved results for natural variants of three well-known problems:
1) We give a quasipolynomial-time algorithm that approximates the maximum of a low degree multivariate polynomial with non-negative coefficients over the Euclidean unit sphere. Beyond being of interest in its own right, this is related to an open question in quantum information theory, and our techniques have already led to improved results in this area (Brandão and Harrow, STOC '13).
2) We give a polynomial-time algorithm that, given a d dimensional subspace of R^n that (almost) contains the characteristic function of a set of size n/k, finds a vector $v$ in the subspace satisfying $|v|_4^4 > c(k/d^{1/3}) |v|_2^2$, where $|v|_p = (E_i v_i^p)^{1/p}$. Aside from being a natural relaxation, this is also motivated by a connection to the Small Set Expansion problem shown by Barak et al. (STOC 2012) and our results yield a certain improvement for that problem.
3) We use this notion of L_4 vs. L_2 sparsity to obtain a polynomial-time algorithm with substantially improved guarantees for recovering a planted $μ$-sparse vector v in a random d-dimensional subspace of R^n. If v has mu n nonzero coordinates, we can recover it with high probability whenever $μ< O(\min(1,n/d^2))$, improving for $d < n^{2/3}$ prior methods which intrinsically required $μ< O(1/\sqrt(d))$.
△ Less
Submitted 23 December, 2013;
originally announced December 2013.
-
Hypercontractivity, Sum-of-Squares Proofs, and their Applications
Authors:
Boaz Barak,
Fernando G. S. L. Brandão,
Aram W. Harrow,
Jonathan A. Kelner,
David Steurer,
Yuan Zhou
Abstract:
We study the computational complexity of approximating the 2->q norm of linear operators (defined as ||A||_{2->q} = sup_v ||Av||_q/||v||_2), as well as connections between this question and issues arising in quantum information theory and the study of Khot's Unique Games Conjecture (UGC). We show the following:
1. For any constant even integer q>=4, a graph $G$ is a "small-set expander" if and o…
▽ More
We study the computational complexity of approximating the 2->q norm of linear operators (defined as ||A||_{2->q} = sup_v ||Av||_q/||v||_2), as well as connections between this question and issues arising in quantum information theory and the study of Khot's Unique Games Conjecture (UGC). We show the following:
1. For any constant even integer q>=4, a graph $G$ is a "small-set expander" if and only if the projector into the span of the top eigenvectors of G's adjacency matrix has bounded 2->q norm. As a corollary, a good approximation to the 2->q norm will refute the Small-Set Expansion Conjecture--a close variant of the UGC. We also show that such a good approximation can be obtained in exp(n^(2/q)) time, thus obtaining a different proof of the known subexponential algorithm for Small Set Expansion.
2. Constant rounds of the "Sum of Squares" semidefinite programing hierarchy certify an upper bound on the 2->4 norm of the projector to low-degree polynomials over the Boolean cube, as well certify the unsatisfiability of the "noisy cube" and "short code" based instances of Unique Games considered by prior works. This improves on the previous upper bound of exp(poly log n) rounds (for the "short code"), as well as separates the "Sum of Squares"/"Lasserre" hierarchy from weaker hierarchies that were known to require omega(1) rounds.
3. We show reductions between computing the 2->4 norm and computing the injective tensor norm of a tensor, a problem with connections to quantum information theory. Three corollaries are: (i) the 2->4 norm is NP-hard to approximate to precision inverse-polynomial in the dimension, (ii) the 2->4 norm does not have a good approximation (in the sense above) unless 3-SAT can be solved in time exp(sqrt(n) polylog(n)), and (iii) known algorithms for the quantum separability problem imply a non-trivial additive approximation for the 2->4 norm.
△ Less
Submitted 16 November, 2014; v1 submitted 21 May, 2012;
originally announced May 2012.
-
Making the long code shorter, with applications to the Unique Games Conjecture
Authors:
Boaz Barak,
Parikshit Gopalan,
Johan Hastad,
Raghu Meka,
Prasad Raghavendra,
David Steurer
Abstract:
The long code is a central tool in hardness of approximation, especially in questions related to the unique games conjecture. We construct a new code that is exponentially more efficient, but can still be used in many of these applications. Using the new code we obtain exponential improvements over several known results, including the following:
1. For any eps > 0, we show the existence of an n…
▽ More
The long code is a central tool in hardness of approximation, especially in questions related to the unique games conjecture. We construct a new code that is exponentially more efficient, but can still be used in many of these applications. Using the new code we obtain exponential improvements over several known results, including the following:
1. For any eps > 0, we show the existence of an n vertex graph G where every set of o(n) vertices has expansion 1 - eps, but G's adjacency matrix has more than exp(log^delta n) eigenvalues larger than 1 - eps, where delta depends only on eps. This answers an open question of Arora, Barak and Steurer (FOCS 2010) who asked whether one can improve over the noise graph on the Boolean hypercube that has poly(log n) such eigenvalues.
2. A gadget that reduces unique games instances with linear constraints modulo K into instances with alphabet k with a blowup of K^polylog(K), improving over the previously known gadget with blowup of 2^K.
3. An n variable integrality gap for Unique Games that that survives exp(poly(log log n)) rounds of the SDP + Sherali Adams hierarchy, improving on the previously known bound of poly(log log n).
We show a connection between the local testability of linear codes and small set expansion in certain related Cayley graphs, and use this connection to derandomize the noise graph on the Boolean hypercube.
△ Less
Submitted 2 November, 2011;
originally announced November 2011.
-
Rounding Semidefinite Programming Hierarchies via Global Correlation
Authors:
Boaz Barak,
Prasad Raghavendra,
David Steurer
Abstract:
We show a new way to round vector solutions of semidefinite programming (SDP) hierarchies into integral solutions, based on a connection between these hierarchies and the spectrum of the input graph. We demonstrate the utility of our method by providing a new SDP-hierarchy based algorithm for constraint satisfaction problems with 2-variable constraints (2-CSP's).
More concretely, we show for eve…
▽ More
We show a new way to round vector solutions of semidefinite programming (SDP) hierarchies into integral solutions, based on a connection between these hierarchies and the spectrum of the input graph. We demonstrate the utility of our method by providing a new SDP-hierarchy based algorithm for constraint satisfaction problems with 2-variable constraints (2-CSP's).
More concretely, we show for every 2-CSP instance I a rounding algorithm for r rounds of the Lasserre SDP hierarchy for I that obtains an integral solution that is at most \eps worse than the relaxation's value (normalized to lie in [0,1]), as long as r > k\cdot\rank_{\geq θ}(\Ins)/\poly(\e) \;, where k is the alphabet size of I, $θ=\poly(\e/k)$, and $\rank_{\geq θ}(\Ins)$ denotes the number of eigenvalues larger than $θ$ in the normalized adjacency matrix of the constraint graph of $\Ins$.
In the case that $\Ins$ is a \uniquegames instance, the threshold $θ$ is only a polynomial in $\e$, and is independent of the alphabet size. Also in this case, we can give a non-trivial bound on the number of rounds for \emph{every} instance. In particular our result yields an SDP-hierarchy based algorithm that matches the performance of the recent subexponential algorithm of Arora, Barak and Steurer (FOCS 2010) in the worst case, but runs faster on a natural family of instances, thus further restricting the set of possible hard instances for Khot's Unique Games Conjecture.
Our algorithm actually requires less than the $n^{O(r)}$ constraints specified by the $r^{th}$ level of the Lasserre hierarchy, and in some cases $r$ rounds of our program can be evaluated in time $2^{O(r)}\poly(n)$.
△ Less
Submitted 25 April, 2011;
originally announced April 2011.
-
Rank Bounds for Design Matrices with Applications to Combinatorial Geometry and Locally Correctable Codes
Authors:
Boaz Barak,
Zeev Dvir,
Avi Wigderson,
Amir Yehudayoff
Abstract:
A (q,k,t)-design matrix is an m x n matrix whose pattern of zeros/non-zeros satisfies the following design-like condition: each row has at most q non-zeros, each column has at least k non-zeros and the supports of every two columns intersect in at most t rows. We prove that the rank of any (q,k,t)-design matrix over a field of characteristic zero (or sufficiently large finite characteristic) is at…
▽ More
A (q,k,t)-design matrix is an m x n matrix whose pattern of zeros/non-zeros satisfies the following design-like condition: each row has at most q non-zeros, each column has at least k non-zeros and the supports of every two columns intersect in at most t rows. We prove that the rank of any (q,k,t)-design matrix over a field of characteristic zero (or sufficiently large finite characteristic) is at least n - (qtn/2k)^2 . Using this result we derive the following applications:
(1) Impossibility results for 2-query LCCs over the complex numbers: A 2-query locally correctable code (LCC) is an error correcting code in which every codeword coordinate can be recovered, probabilistically, by reading at most two other code positions. Such codes have numerous applications and constructions (with exponential encoding length) are known over finite fields of small characteristic. We show that infinite families of such linear 2-query LCCs do not exist over the complex numbers.
(2) Generalization of results in combinatorial geometry: We prove a quantitative analog of the Sylvester-Gallai theorem: Let $v_1,...,v_m$ be a set of points in $\C^d$ such that for every $i \in [m]$ there exists at least $δm$ values of $j \in [m]$ such that the line through $v_i,v_j$ contains a third point in the set. We show that the dimension of $\{v_1,...,v_m \}$ is at most $O(1/δ^2)$. Our results generalize to the high dimensional case (replacing lines with planes, etc.) and to the case where the points are colored (as in the Motzkin-Rabin Theorem).
△ Less
Submitted 10 March, 2011; v1 submitted 22 September, 2010;
originally announced September 2010.
-
Subsampling Mathematical Relaxations and Average-case Complexity
Authors:
Boaz Barak,
Moritz Hardt,
Thomas Holenstein,
David Steurer
Abstract:
We initiate a study of when the value of mathematical relaxations such as linear and semidefinite programs for constraint satisfaction problems (CSPs) is approximately preserved when restricting the instance to a sub-instance induced by a small random subsample of the variables. Let $C$ be a family of CSPs such as 3SAT, Max-Cut, etc., and let $Π$ be a relaxation for $C$, in the sense that for eve…
▽ More
We initiate a study of when the value of mathematical relaxations such as linear and semidefinite programs for constraint satisfaction problems (CSPs) is approximately preserved when restricting the instance to a sub-instance induced by a small random subsample of the variables. Let $C$ be a family of CSPs such as 3SAT, Max-Cut, etc., and let $Π$ be a relaxation for $C$, in the sense that for every instance $P\in C$, $Π(P)$ is an upper bound the maximum fraction of satisfiable constraints of $P$. Loosely speaking, we say that subsampling holds for $C$ and $Π$ if for every sufficiently dense instance $P \in C$ and every $ε>0$, if we let $P'$ be the instance obtained by restricting $P$ to a sufficiently large constant number of variables, then $Π(P') \in (1\pm ε)Π(P)$. We say that weak subsampling holds if the above guarantee is replaced with $Π(P')=1-Θ(γ)$ whenever $Π(P)=1-γ$. We show: 1. Subsampling holds for the BasicLP and BasicSDP programs. BasicSDP is a variant of the relaxation considered by Raghavendra (2008), who showed it gives an optimal approximation factor for every CSP under the unique games conjecture. BasicLP is the linear programming analog of BasicSDP. 2. For tighter versions of BasicSDP obtained by adding additional constraints from the Lasserre hierarchy, weak subsampling holds for CSPs of unique games type. 3. There are non-unique CSPs for which even weak subsampling fails for the above tighter semidefinite programs. Also there are unique CSPs for which subsampling fails for the Sherali-Adams linear programming hierarchy. As a corollary of our weak subsampling for strong semidefinite programs, we obtain a polynomial-time algorithm to certify that random geometric graphs (of the type considered by Feige and Schechtman, 2002) of max-cut value $1-γ$ have a cut value at most $1-γ/10$.
△ Less
Submitted 29 April, 2010; v1 submitted 29 November, 2009;
originally announced November 2009.
-
Lower Bounds on Signatures from Symmetric Primitives
Authors:
Boaz Barak,
Mohammad Mahmoody
Abstract:
We show that every construction of one-time signature schemes from a random oracle achieves black-box security at most $2^{(1+o(1))q}$, where $q$ is the total number of oracle queries asked by the key generation, signing, and verification algorithms. That is, any such scheme can be broken with probability close to $1$ by a (computationally unbounded) adversary making $2^{(1+o(1))q}$ queries to the…
▽ More
We show that every construction of one-time signature schemes from a random oracle achieves black-box security at most $2^{(1+o(1))q}$, where $q$ is the total number of oracle queries asked by the key generation, signing, and verification algorithms. That is, any such scheme can be broken with probability close to $1$ by a (computationally unbounded) adversary making $2^{(1+o(1))q}$ queries to the oracle. This is tight up to a constant factor in the number of queries, since a simple modification of Lamport's one-time signatures (Lamport '79) achieves $2^{(0.812-o(1))q}$ black-box security using $q$ queries to the oracle.
Our result extends (with a loss of a constant factor in the number of queries) also to the random permutation and ideal-cipher oracles. Since the symmetric primitives (e.g. block ciphers, hash functions, and message authentication codes) can be constructed by a constant number of queries to the mentioned oracles, as corollary we get lower bounds on the efficiency of signature schemes from symmetric primitives when the construction is black-box. This can be taken as evidence of an inherent efficiency gap between signature schemes and symmetric primitives.
△ Less
Submitted 30 March, 2019; v1 submitted 23 January, 2008;
originally announced January 2008.
-
Merkle's Key Agreement Protocol is Optimal: An $O(n^2)$ Attack on any Key Agreement from Random Oracles
Authors:
Boaz Barak,
Mohammad Mahmoody
Abstract:
We prove that every key agreement protocol in the random oracle model in which the honest users make at most $n$ queries to the oracle can be broken by an adversary who makes $O(n^2)$ queries to the oracle. This improves on the previous $\widetildeΩ(n^6)$ query attack given by Impagliazzo and Rudich (STOC '89) and resolves an open question posed by them.
Our bound is optimal up to a constant fac…
▽ More
We prove that every key agreement protocol in the random oracle model in which the honest users make at most $n$ queries to the oracle can be broken by an adversary who makes $O(n^2)$ queries to the oracle. This improves on the previous $\widetildeΩ(n^6)$ query attack given by Impagliazzo and Rudich (STOC '89) and resolves an open question posed by them.
Our bound is optimal up to a constant factor since Merkle proposed a key agreement protocol in 1974 that can be easily implemented with $n$ queries to a random oracle and cannot be broken by any adversary who asks $o(n^2)$ queries.
△ Less
Submitted 30 March, 2019; v1 submitted 23 January, 2008;
originally announced January 2008.
-
A Simple Explicit Construction of an $n^{\Tilde{O}(\log n)}$-Ramsey Graph
Authors:
Boaz Barak
Abstract:
We show a simple explicit construction of an $2^{\Tilde{O}(\sqrt{\log n})}$ Ramsey graph. That is, we provide a $\poly(n)$-time algorithm to output the adjacency matrix of an undirected $n$-vertex graph with no clique or independent set of size $2^{\e \sqrt{\log n}\log\log n}$ for every $\e>0$.
Our construction has the very serious disadvantage over the well-known construction of Frankl and Wi…
▽ More
We show a simple explicit construction of an $2^{\Tilde{O}(\sqrt{\log n})}$ Ramsey graph. That is, we provide a $\poly(n)$-time algorithm to output the adjacency matrix of an undirected $n$-vertex graph with no clique or independent set of size $2^{\e \sqrt{\log n}\log\log n}$ for every $\e>0$.
Our construction has the very serious disadvantage over the well-known construction of Frankl and Wilson \cite{FranklWi81} that it is only explicit and not very explicit, in the sense that we do \emph{not} provide a poly-logarithmic time algorithm to compute the neighborhood relation. The main advantage of this construction is its extreme simplicity. It is also somewhat surprising that even though we use a completely different approach we get a bound which essentially equals the bound of \cite{FranklWi81}. This construction is quite simple and was obtained independently by others as well\footnote{P.~Pudlak, personal communications, July 2004.} but as far as we know has not been published elsewhere.
△ Less
Submitted 26 January, 2006;
originally announced January 2006.
