-
Trustworthiness of Children Stories Generated by Large Language Models
Authors:
Prabin Bhandari,
Hannah Marie Brennan
Abstract:
Large Language Models (LLMs) have shown a tremendous capacity for generating literary text. However, their effectiveness in generating children's stories has yet to be thoroughly examined. In this study, we evaluate the trustworthiness of children's stories generated by LLMs using various measures, and we compare and contrast our results with both old and new children's stories to better assess th…
▽ More
Large Language Models (LLMs) have shown a tremendous capacity for generating literary text. However, their effectiveness in generating children's stories has yet to be thoroughly examined. In this study, we evaluate the trustworthiness of children's stories generated by LLMs using various measures, and we compare and contrast our results with both old and new children's stories to better assess their significance. Our findings suggest that LLMs still struggle to generate children's stories at the level of quality and nuance found in actual stories
△ Less
Submitted 25 July, 2023;
originally announced August 2023.
-
Threshold for Detecting High Dimensional Geometry in Anisotropic Random Geometric Graphs
Authors:
Matthew Brennan,
Guy Bresler,
Brice Huang
Abstract:
In the anisotropic random geometric graph model, vertices correspond to points drawn from a high-dimensional Gaussian distribution and two vertices are connected if their distance is smaller than a specified threshold. We study when it is possible to hypothesis test between such a graph and an Erdős-Rényi graph with the same edge probability. If $n$ is the number of vertices and $α$ is the vector…
▽ More
In the anisotropic random geometric graph model, vertices correspond to points drawn from a high-dimensional Gaussian distribution and two vertices are connected if their distance is smaller than a specified threshold. We study when it is possible to hypothesis test between such a graph and an Erdős-Rényi graph with the same edge probability. If $n$ is the number of vertices and $α$ is the vector of eigenvalues, Eldan and Mikulincer show that detection is possible when $n^3 \gg (\|α\|_2/\|α\|_3)^6$ and impossible when $n^3 \ll (\|α\|_2/\|α\|_4)^4$. We show detection is impossible when $n^3 \ll (\|α\|_2/\|α\|_3)^6$, closing this gap and affirmatively resolving the conjecture of Eldan and Mikulincer.
△ Less
Submitted 29 June, 2022;
originally announced June 2022.
-
The staircase property: How hierarchical structure can guide deep learning
Authors:
Emmanuel Abbe,
Enric Boix-Adsera,
Matthew Brennan,
Guy Bresler,
Dheeraj Nagaraj
Abstract:
This paper identifies a structural property of data distributions that enables deep neural networks to learn hierarchically. We define the "staircase" property for functions over the Boolean hypercube, which posits that high-order Fourier coefficients are reachable from lower-order Fourier coefficients along increasing chains. We prove that functions satisfying this property can be learned in poly…
▽ More
This paper identifies a structural property of data distributions that enables deep neural networks to learn hierarchically. We define the "staircase" property for functions over the Boolean hypercube, which posits that high-order Fourier coefficients are reachable from lower-order Fourier coefficients along increasing chains. We prove that functions satisfying this property can be learned in polynomial time using layerwise stochastic coordinate descent on regular neural networks -- a class of network architectures and initializations that have homogeneity properties. Our analysis shows that for such staircase functions and neural networks, the gradient-based algorithm learns high-level features by greedily combining lower-level features along the depth of the network. We further back our theoretical results with experiments showing that staircase functions are also learnable by more standard ResNet architectures with stochastic gradient descent. Both the theoretical and experimental results support the fact that staircase properties have a role to play in understanding the capabilities of gradient-based learning on regular networks, in contrast to general polynomial-size networks that can emulate any SQ or PAC algorithms as recently shown.
△ Less
Submitted 23 November, 2021; v1 submitted 24 August, 2021;
originally announced August 2021.
-
Interpretable Machine Learning Classifiers for Brain Tumour Survival Prediction
Authors:
Colleen E. Charlton,
Michael Tin Chung Poon,
Paul M. Brennan,
Jacques D. Fleuriot
Abstract:
Prediction of survival in patients diagnosed with a brain tumour is challenging because of heterogeneous tumour behaviours and responses to treatment. Better estimations of prognosis would support treatment planning and patient support. Advances in machine learning have informed development of clinical predictive models, but their integration into clinical practice is almost non-existent. One reas…
▽ More
Prediction of survival in patients diagnosed with a brain tumour is challenging because of heterogeneous tumour behaviours and responses to treatment. Better estimations of prognosis would support treatment planning and patient support. Advances in machine learning have informed development of clinical predictive models, but their integration into clinical practice is almost non-existent. One reasons for this is the lack of interpretability of models. In this paper, we use a novel brain tumour dataset to compare two interpretable rule list models against popular machine learning approaches for brain tumour survival prediction. All models are quantitatively evaluated using standard performance metrics. The rule lists are also qualitatively assessed for their interpretability and clinical utility. The interpretability of the black box machine learning models is evaluated using two post-hoc explanation techniques, LIME and SHAP. Our results show that the rule lists were only slightly outperformed by the black box models. We demonstrate that rule list algorithms produced simple decision lists that align with clinical expertise. By comparison, post-hoc interpretability methods applied to black box models may produce unreliable explanations of local model predictions. Model interpretability is essential for understanding differences in predictive performance and for integration into clinical practice.
△ Less
Submitted 17 June, 2021;
originally announced June 2021.
-
De Finetti-Style Results for Wishart Matrices: Combinatorial Structure and Phase Transitions
Authors:
Matthew Brennan,
Guy Bresler,
Brice Huang
Abstract:
A recent line of work has studied the relationship between the Wishart matrix $X^\top X$, where $X\in \mathbb{R}^{d\times n}$ has i.i.d. standard Gaussian entries, and the corresponding Gaussian matrix with independent entries above the diagonal. Jiang and Li (2015) and Bubeck et al. (2016) showed that these two matrix ensembles converge in total variation whenever $d/n^3\to \infty$, and Bubeck et…
▽ More
A recent line of work has studied the relationship between the Wishart matrix $X^\top X$, where $X\in \mathbb{R}^{d\times n}$ has i.i.d. standard Gaussian entries, and the corresponding Gaussian matrix with independent entries above the diagonal. Jiang and Li (2015) and Bubeck et al. (2016) showed that these two matrix ensembles converge in total variation whenever $d/n^3\to \infty$, and Bubeck et al. (2016) showed this to be sharp. In this paper we aim to identify the precise threshold for $d$ in terms of $n$ for subsets of Wishart matrices to converge in total variation to independent Gaussians. It turns out that the combinatorial structure of the revealed entries, viewed as the adjacency matrix of a graph $G$, characterizes the distance from fully independent. Specifically, we show that the threshold for $d$ depends on the number of various small subgraphs in $G$. So, even when the number of revealed entries is fixed, the threshold can vary wildly depending on their configuration. Convergence of masked Wishart to independent Gaussians thus inherently involves an interplay between both probabilistic and combinatorial phenomena. Our results determine the sharp threshold for a large family of $G$, including Erdős-Rényi $G\sim \mathcal{G}(n,p)$ at all values $p\gtrsim n^{-2}\mathrm{polylog}(n)$. Our proof techniques are both combinatorial and information theoretic, which together allow us to carefully unravel the dependencies in the masked Wishart ensemble.
△ Less
Submitted 25 March, 2021;
originally announced March 2021.
-
Statistical Query Algorithms and Low-Degree Tests Are Almost Equivalent
Authors:
Matthew Brennan,
Guy Bresler,
Samuel B. Hopkins,
Jerry Li,
Tselil Schramm
Abstract:
Researchers currently use a number of approaches to predict and substantiate information-computation gaps in high-dimensional statistical estimation problems. A prominent approach is to characterize the limits of restricted models of computation, which on the one hand yields strong computational lower bounds for powerful classes of algorithms and on the other hand helps guide the development of ef…
▽ More
Researchers currently use a number of approaches to predict and substantiate information-computation gaps in high-dimensional statistical estimation problems. A prominent approach is to characterize the limits of restricted models of computation, which on the one hand yields strong computational lower bounds for powerful classes of algorithms and on the other hand helps guide the development of efficient algorithms. In this paper, we study two of the most popular restricted computational models, the statistical query framework and low-degree polynomials, in the context of high-dimensional hypothesis testing. Our main result is that under mild conditions on the testing problem, the two classes of algorithms are essentially equivalent in power. As corollaries, we obtain new statistical query lower bounds for sparse PCA, tensor PCA and several variants of the planted clique problem.
△ Less
Submitted 26 June, 2021; v1 submitted 13 September, 2020;
originally announced September 2020.
-
Reducibility and Statistical-Computational Gaps from Secret Leakage
Authors:
Matthew Brennan,
Guy Bresler
Abstract:
Inference problems with conjectured statistical-computational gaps are ubiquitous throughout modern statistics, computer science and statistical physics. While there has been success evidencing these gaps from the failure of restricted classes of algorithms, progress towards a more traditional reduction-based approach to computational complexity in statistical inference has been limited. Existing…
▽ More
Inference problems with conjectured statistical-computational gaps are ubiquitous throughout modern statistics, computer science and statistical physics. While there has been success evidencing these gaps from the failure of restricted classes of algorithms, progress towards a more traditional reduction-based approach to computational complexity in statistical inference has been limited. Existing reductions have largely been limited to inference problems with similar structure -- primarily map** among problems representable as a sparse submatrix signal plus a noise matrix, which are similar to the common hardness assumption of planted clique.
The insight in this work is that a slight generalization of the planted clique conjecture -- secret leakage planted clique -- gives rise to a variety of new average-case reduction techniques, yielding a web of reductions among problems with very different structure. Using variants of the planted clique conjecture for specific forms of secret leakage planted clique, we deduce tight statistical-computational tradeoffs for a diverse range of problems including robust sparse mean estimation, mixtures of sparse linear regressions, robust sparse linear regression, tensor PCA, variants of dense $k$-block stochastic block models, negatively correlated sparse PCA, semirandom planted dense subgraph, detection in hidden partition models and a universality principle for learning sparse mixtures. In particular, a $k$-partite hypergraph variant of the planted clique conjecture is sufficient to establish all of our computational lower bounds. Our techniques also reveal novel connections to combinatorial designs and to random matrix theory. This work gives the first evidence that an expanded set of hardness assumptions, such as for secret leakage planted clique, may be a key first step towards a more complete theory of reductions among statistical problems.
△ Less
Submitted 28 June, 2020; v1 submitted 16 May, 2020;
originally announced May 2020.
-
Phase Transitions for Detecting Latent Geometry in Random Graphs
Authors:
Matthew Brennan,
Guy Bresler,
Dheeraj Nagaraj
Abstract:
Random graphs with latent geometric structure are popular models of social and biological networks, with applications ranging from network user profiling to circuit design. These graphs are also of purely theoretical interest within computer science, probability and statistics. A fundamental initial question regarding these models is: when are these random graphs affected by their latent geometry…
▽ More
Random graphs with latent geometric structure are popular models of social and biological networks, with applications ranging from network user profiling to circuit design. These graphs are also of purely theoretical interest within computer science, probability and statistics. A fundamental initial question regarding these models is: when are these random graphs affected by their latent geometry and when are they indistinguishable from simpler models without latent structure, such as the Erdős-Rényi graph $\mathcal{G}(n, p)$? We address this question for two of the most well-studied models of random graphs with latent geometry -- the random intersection and random geometric graph.
Our results are as follows: (1) we prove that the random intersection graph converges in total variation to $\mathcal{G}(n, p)$ when $d = \tildeω(n^3)$, and does not if $d = o(n^3)$, resolving an open problem in Fill et al. (2000), Rybarczyk (2011) and Kim et al. (2018); (2) we provide conditions under which the matrix of intersection sizes of random family of sets converges in total variation to a symmetric matrix with independent Poisson entries, yielding the first total variation convergence result for $τ$-random intersection graphs to $\mathcal{G}(n, p)$; and (3) we show that the random geometric graph on $\mathbb{S}^{d - 1}$ with edge density $p$ converges in total variation to $\mathcal{G}(n, p)$ when $d = \tildeω\left(\min\{ pn^3, p^2 n^{7/2} \} \right)$, yielding the first progress towards a conjecture of Bubeck et al. (2016). The first of these three results was obtained simultaneously and independently by Bubeck, Racz and Richey.
△ Less
Submitted 2 August, 2020; v1 submitted 30 October, 2019;
originally announced October 2019.
-
Average-Case Lower Bounds for Learning Sparse Mixtures, Robust Estimation and Semirandom Adversaries
Authors:
Matthew Brennan,
Guy Bresler
Abstract:
This paper develops several average-case reduction techniques to show new hardness results for three central high-dimensional statistics problems, implying a statistical-computational gap induced by robustness, a detection-recovery gap and a universality principle for these gaps. A main feature of our approach is to map to these problems via a common intermediate problem that we introduce, which w…
▽ More
This paper develops several average-case reduction techniques to show new hardness results for three central high-dimensional statistics problems, implying a statistical-computational gap induced by robustness, a detection-recovery gap and a universality principle for these gaps. A main feature of our approach is to map to these problems via a common intermediate problem that we introduce, which we call Imbalanced Sparse Gaussian Mixtures. We assume the planted clique conjecture for a version of the planted clique problem where the position of the planted clique is mildly constrained, and from this obtain the following computational lower bounds: (1) a $k$-to-$k^2$ statistical-computational gap for robust sparse mean estimation, providing the first average-case evidence for a conjecture of Li (2017) and Balakrishnan et al. (2017); (2) a tight lower bound for semirandom planted dense subgraph, which shows that a semirandom adversary shifts the detection threshold in planted dense subgraph to the conjectured recovery threshold; and (3) a universality principle for $k$-to-$k^2$ gaps in a broad class of sparse mixture problems that includes many natural formulations such as the spiked covariance model.
Our main approach is to introduce several average-case techniques to produce structured and Gaussianized versions of an input graph problem, and then to rotate these high-dimensional Gaussians by matrices carefully constructed from hyperplanes in $\mathbb{F}_r^t$. For our universality result, we introduce a new method to perform an algorithmic change of measure tailored to sparse mixtures. We also provide evidence that the mild promise in our variant of planted clique does not change the complexity of the problem.
△ Less
Submitted 18 May, 2020; v1 submitted 8 August, 2019;
originally announced August 2019.
-
The Average-Case Complexity of Counting Cliques in Erdos-Renyi Hypergraphs
Authors:
Enric Boix-Adserà,
Matthew Brennan,
Guy Bresler
Abstract:
We consider the problem of counting $k$-cliques in $s$-uniform Erdos-Renyi hypergraphs $G(n,c,s)$ with edge density $c$, and show that its fine-grained average-case complexity can be based on its worst-case complexity. We prove the following:
1. Dense Erdos-Renyi graphs and hypergraphs: Counting $k$-cliques on $G(n,c,s)$ with $k$ and $c$ constant matches its worst-case time complexity up to a…
▽ More
We consider the problem of counting $k$-cliques in $s$-uniform Erdos-Renyi hypergraphs $G(n,c,s)$ with edge density $c$, and show that its fine-grained average-case complexity can be based on its worst-case complexity. We prove the following:
1. Dense Erdos-Renyi graphs and hypergraphs: Counting $k$-cliques on $G(n,c,s)$ with $k$ and $c$ constant matches its worst-case time complexity up to a $\mathrm{polylog}(n)$ factor. Assuming randomized ETH, it takes $n^{Ω(k)}$ time to count $k$-cliques in $G(n,c,s)$ if $k$ and $c$ are constant.
2. Sparse Erdos-Renyi graphs and hypergraphs: When $c = Θ(n^{-α})$, we give several algorithms exploiting the sparsity of $G(n, c, s)$ that are faster than the best known worst-case algorithms. Complementing this, based on a fine-grained worst-case assumption, our results imply a different average-case phase diagram for each fixed $α$ depicting a tradeoff between a runtime lower bound and $k$. Surprisingly, in the hypergraph case ($s \ge 3$), these lower bounds are tight against our algorithms exactly when $c$ is above the Erdős-Rényi $k$-clique percolation threshold.
This is the first worst-case-to-average-case hardness reduction for a problem on Erdős-Rényi hypergraphs that we are aware of. We also give a variant of our result for computing the parity of the $k$-clique count that tolerates higher error probability.
△ Less
Submitted 21 July, 2021; v1 submitted 19 March, 2019;
originally announced March 2019.
-
Optimal Average-Case Reductions to Sparse PCA: From Weak Assumptions to Strong Hardness
Authors:
Matthew Brennan,
Guy Bresler
Abstract:
In the past decade, sparse principal component analysis has emerged as an archetypal problem for illustrating statistical-computational tradeoffs. This trend has largely been driven by a line of research aiming to characterize the average-case complexity of sparse PCA through reductions from the planted clique (PC) conjecture - which conjectures that there is no polynomial-time algorithm to detect…
▽ More
In the past decade, sparse principal component analysis has emerged as an archetypal problem for illustrating statistical-computational tradeoffs. This trend has largely been driven by a line of research aiming to characterize the average-case complexity of sparse PCA through reductions from the planted clique (PC) conjecture - which conjectures that there is no polynomial-time algorithm to detect a planted clique of size $K = o(N^{1/2})$ in $\mathcal{G}(N, \frac{1}{2})$. All previous reductions to sparse PCA either fail to show tight computational lower bounds matching existing algorithms or show lower bounds for formulations of sparse PCA other than its canonical generative model, the spiked covariance model. Also, these lower bounds all quickly degrade with the exponent in the PC conjecture. Specifically, when only given the PC conjecture up to $K = o(N^α)$ where $α< 1/2$, there is no sparsity level $k$ at which these lower bounds remain tight. If $α\le 1/3$ these reductions fail to even show the existence of a statistical-computational tradeoff at any sparsity $k$. We give a reduction from PC that yields the first full characterization of the computational barrier in the spiked covariance model, providing tight lower bounds at all sparsities $k$. We also show the surprising result that weaker forms of the PC conjecture up to clique size $K = o(N^α)$ for any given $α\in (0, 1/2]$ imply tight computational lower bounds for sparse PCA at sparsities $k = o(n^{α/3})$. This shows that even a mild improvement in the signal strength needed by the best known polynomial-time sparse PCA algorithms would imply that the hardness threshold for PC is subpolynomial. This is the first instance of a suboptimal hardness assumption implying optimal lower bounds for another problem in unsupervised learning.
△ Less
Submitted 19 February, 2019;
originally announced February 2019.
-
Universality of Computational Lower Bounds for Submatrix Detection
Authors:
Matthew Brennan,
Guy Bresler,
Wasim Huleihel
Abstract:
In the general submatrix detection problem, the task is to detect the presence of a small $k \times k$ submatrix with entries sampled from a distribution $\mathcal{P}$ in an $n \times n$ matrix of samples from $\mathcal{Q}$. This formulation includes a number of well-studied problems, such as biclustering when $\mathcal{P}$ and $\mathcal{Q}$ are Gaussians and the planted dense subgraph formulation…
▽ More
In the general submatrix detection problem, the task is to detect the presence of a small $k \times k$ submatrix with entries sampled from a distribution $\mathcal{P}$ in an $n \times n$ matrix of samples from $\mathcal{Q}$. This formulation includes a number of well-studied problems, such as biclustering when $\mathcal{P}$ and $\mathcal{Q}$ are Gaussians and the planted dense subgraph formulation of community detection when the submatrix is a principal minor and $\mathcal{P}$ and $\mathcal{Q}$ are Bernoulli random variables. These problems all seem to exhibit a universal phenomenon: there is a statistical-computational gap depending on $\mathcal{P}$ and $\mathcal{Q}$ between the minimum $k$ at which this task can be solved and the minimum $k$ at which it can be solved in polynomial time. Our main result is to tightly characterize this computational barrier as a tradeoff between $k$ and the KL divergences between $\mathcal{P}$ and $\mathcal{Q}$ through average-case reductions from the planted clique conjecture. These computational lower bounds hold given mild assumptions on $\mathcal{P}$ and $\mathcal{Q}$ arising naturally from classical binary hypothesis testing. Our results recover and generalize the planted clique lower bounds for Gaussian biclustering in Ma-Wu (2015) and Brennan et al. (2018) and for the sparse and general regimes of planted dense subgraph in Hajek et al. (2015) and Brennan et al. (2018). This yields the first universality principle for computational lower bounds obtained through average-case reductions.
△ Less
Submitted 1 June, 2019; v1 submitted 19 February, 2019;
originally announced February 2019.
-
Reducibility and Computational Lower Bounds for Problems with Planted Sparse Structure
Authors:
Matthew Brennan,
Guy Bresler,
Wasim Huleihel
Abstract:
The prototypical high-dimensional statistics problem entails finding a structured signal in noise. Many of these problems exhibit an intriguing phenomenon: the amount of data needed by all known computationally efficient algorithms far exceeds what is needed for inefficient algorithms that search over all possible structures. A line of work initiated by Berthet and Rigollet in 2013 has aimed to ex…
▽ More
The prototypical high-dimensional statistics problem entails finding a structured signal in noise. Many of these problems exhibit an intriguing phenomenon: the amount of data needed by all known computationally efficient algorithms far exceeds what is needed for inefficient algorithms that search over all possible structures. A line of work initiated by Berthet and Rigollet in 2013 has aimed to explain these statistical-computational gaps by reducing from conjecturally hard average-case problems in computer science. However, the delicate nature of average-case reductions has limited the applicability of this approach. In this work we introduce several new techniques to give a web of average-case reductions showing strong computational lower bounds based on the planted clique conjecture using natural problems as intermediates. These include tight lower bounds for Planted Independent Set, Planted Dense Subgraph, Sparse Spiked Wigner, Sparse PCA, a subgraph variant of the Stochastic Block Model and a biased variant of Sparse PCA. We also give algorithms matching our lower bounds and identify the information-theoretic limits of the models we consider.
△ Less
Submitted 18 November, 2019; v1 submitted 19 June, 2018;
originally announced June 2018.
-
Comparing Clinical Judgment with MySurgeryRisk Algorithm for Preoperative Risk Assessment: A Pilot Study
Authors:
Meghan Brennan,
Sahil Puri,
Tezcan Ozrazgat-Baslanti,
Rajendra Bhat,
Zheng Feng,
Petar Momcilovic,
Xiaolin Li,
Daisy Zhe Wang,
Azra Bihorac
Abstract:
Background: Major postoperative complications are associated with increased short and long-term mortality, increased healthcare cost, and adverse long-term consequences. The large amount of data contained in the electronic health record (EHR) creates barriers for physicians to recognize patients most at risk. We hypothesize, if presented in an optimal format, information from data-driven predictiv…
▽ More
Background: Major postoperative complications are associated with increased short and long-term mortality, increased healthcare cost, and adverse long-term consequences. The large amount of data contained in the electronic health record (EHR) creates barriers for physicians to recognize patients most at risk. We hypothesize, if presented in an optimal format, information from data-driven predictive risk algorithms for postoperative complications can improve physician risk assessment. Methods: Prospective, non-randomized, interventional pilot study of twenty perioperative physicians at a quarterly academic medical center. Using 150 clinical cases we compared physicians' risk assessment before and after interaction with MySurgeryRisk, a validated machine-learning algorithm predicting preoperative risk for six major postoperative complications using EHR data. Results: The area under the curve (AUC) of MySurgeryRisk algorithm ranged between 0.73 and 0.85 and was significantly higher than physicians' risk assessments (AUC between 0.47 and 0.69) for all postoperative complications except cardiovascular complications. The AUC for repeated physician's risk assessment improved by 2% to 5% for all complications with the exception of thirty-day mortality. Physicians' risk assessment for acute kidney injury and intensive care unit admission longer than 48 hours significantly improved after knowledge exchange, resulting in net reclassification improvement of 12.4% and 16%, respectively. Conclusions: The validated MySurgeryRisk algorithm predicted postoperative complications with equal or higher accuracy than pilot cohort of physicians using available clinical preoperative data. The interaction with algorithm significantly improved physicians' risk assessment.
△ Less
Submitted 9 April, 2018;
originally announced April 2018.
