Showing 1–50 of 50 results for author: Courtade, T

Stability of the Poincaré-Korn inequality

Authors: Thomas A. Courtade, Max Fathi

Abstract: We resolve a question of Carrapatoso et al. on Gaussian optimality for the sharp constant in Poincaré-Korn inequalities, under a moment constraint. We also prove stability, showing that measures with near-optimal constant are quantitatively close to standard Gaussian. We resolve a question of Carrapatoso et al. on Gaussian optimality for the sharp constant in Poincaré-Korn inequalities, under a moment constraint. We also prove stability, showing that measures with near-optimal constant are quantitatively close to standard Gaussian. △ Less

Submitted 2 May, 2024; originally announced May 2024.

Stability of Klartag's improved Lichnerowicz inequality

Authors: Thomas A. Courtade, Max Fathi

Abstract: In a recent work, Klartag gave an improved version of Lichnerowicz' spectral gap bound for uniformly log-concave measures, which improves on the classical estimate by taking into account the covariance matrix. We analyze the equality cases in Klartag's bound, showing that it can be further improved whenever the measure has no Gaussian factor. Additionally, we give a quantitative improvement for lo… ▽ More In a recent work, Klartag gave an improved version of Lichnerowicz' spectral gap bound for uniformly log-concave measures, which improves on the classical estimate by taking into account the covariance matrix. We analyze the equality cases in Klartag's bound, showing that it can be further improved whenever the measure has no Gaussian factor. Additionally, we give a quantitative improvement for log-concave measures with finite Fisher information. △ Less

Submitted 18 April, 2024; originally announced April 2024.

Rigid characterizations of probability measures through independence, with applications

Authors: Thomas A. Courtade

Abstract: Three equivalent characterizations of probability measures through independence criteria are given. These characterizations lead to a family of Brascamp--Lieb-type inequalities for relative entropy, determine equilibrium states and sharp rates of convergence for certain linear Boltzmann-type dynamics, and unify an assortment of $L^2$ inequalities in probability. Three equivalent characterizations of probability measures through independence criteria are given. These characterizations lead to a family of Brascamp--Lieb-type inequalities for relative entropy, determine equilibrium states and sharp rates of convergence for certain linear Boltzmann-type dynamics, and unify an assortment of $L^2$ inequalities in probability. △ Less

Submitted 11 March, 2024; originally announced March 2024.

Comments: Comments welcome!

HWI inequalities in discrete spaces via couplings

Authors: Thomas A. Courtade, Max Fathi

Abstract: HWI inequalities are interpolation inequalities relating entropy, Fisher information and optimal transport distances. We adapt an argument of Y. Wu for proving the Gaussian HWI inequality via a coupling argument to the discrete setting, establishing new interpolation inequalities for the discrete hypercube and the discrete torus. In particular, we obtain an improvement of the modified logarithmic… ▽ More HWI inequalities are interpolation inequalities relating entropy, Fisher information and optimal transport distances. We adapt an argument of Y. Wu for proving the Gaussian HWI inequality via a coupling argument to the discrete setting, establishing new interpolation inequalities for the discrete hypercube and the discrete torus. In particular, we obtain an improvement of the modified logarithmic Sobolev inequality for the discrete hypercube of Bobkov and Tetali. △ Less

Submitted 3 December, 2023; originally announced December 2023.

Comments: 11 pages, comments welcome

Mean Estimation Under Heterogeneous Privacy Demands

Authors: Syomantak Chaudhuri, Konstantin Miagkov, Thomas A. Courtade

Abstract: Differential Privacy (DP) is a well-established framework to quantify privacy loss incurred by any algorithm. Traditional formulations impose a uniform privacy requirement for all users, which is often inconsistent with real-world scenarios in which users dictate their privacy preferences individually. This work considers the problem of mean estimation, where each user can impose their own distinc… ▽ More Differential Privacy (DP) is a well-established framework to quantify privacy loss incurred by any algorithm. Traditional formulations impose a uniform privacy requirement for all users, which is often inconsistent with real-world scenarios in which users dictate their privacy preferences individually. This work considers the problem of mean estimation, where each user can impose their own distinct privacy level. The algorithm we propose is shown to be minimax optimal and has a near-linear run-time. Our results elicit an interesting saturation phenomenon that occurs. Namely, the privacy requirements of the most stringent users dictate the overall error rates. As a consequence, users with less but differing privacy requirements are all given more privacy than they require, in equal amounts. In other words, these privacy-indifferent users are given a nontrivial degree of privacy for free, without any sacrifice in the performance of the estimator. △ Less

Submitted 19 October, 2023; originally announced October 2023.

Comments: A preliminary conference version was published at ISIT 2023 and uploaded to arxiv (arXiv:2305.09668). This version significantly expands on the previous article and is being submitted to a journal

Mean Estimation Under Heterogeneous Privacy: Some Privacy Can Be Free

Authors: Syomantak Chaudhuri, Thomas A. Courtade

Abstract: Differential Privacy (DP) is a well-established framework to quantify privacy loss incurred by any algorithm. Traditional DP formulations impose a uniform privacy requirement for all users, which is often inconsistent with real-world scenarios in which users dictate their privacy preferences individually. This work considers the problem of mean estimation under heterogeneous DP constraints, where… ▽ More Differential Privacy (DP) is a well-established framework to quantify privacy loss incurred by any algorithm. Traditional DP formulations impose a uniform privacy requirement for all users, which is often inconsistent with real-world scenarios in which users dictate their privacy preferences individually. This work considers the problem of mean estimation under heterogeneous DP constraints, where each user can impose their own distinct privacy level. The algorithm we propose is shown to be minimax optimal when there are two groups of users with distinct privacy levels. Our results elicit an interesting saturation phenomenon that occurs as one group's privacy level is relaxed, while the other group's privacy level remains constant. Namely, after a certain point, further relaxing the privacy requirement of the former group does not improve the performance of the minimax optimal mean estimator. Thus, the central server can offer a certain degree of privacy without any sacrifice in performance. △ Less

Submitted 27 April, 2023; originally announced May 2023.

Comments: To appear at ISIT 2023

Entropy Inequalities and Gaussian Comparisons

Authors: Efe Aras, Thomas A. Courtade

Abstract: We establish a general class of entropy inequalities that take the concise form of Gaussian comparisons. The main result unifies many classical and recent results, including the Shannon-Stam inequality, the Brunn-Minkowski inequality, the Zamir-Feder inequality, the Brascamp-Lieb and Barthe inequalities, the Anantharam-Jog-Nair inequality, and others. We establish a general class of entropy inequalities that take the concise form of Gaussian comparisons. The main result unifies many classical and recent results, including the Shannon-Stam inequality, the Brunn-Minkowski inequality, the Zamir-Feder inequality, the Brascamp-Lieb and Barthe inequalities, the Anantharam-Jog-Nair inequality, and others. △ Less

Submitted 28 June, 2022; originally announced June 2022.

Comments: 23 pages. Comments welcome

Equality cases in the Anantharam-Jog-Nair inequality

Authors: Efe Aras, Thomas A. Courtade, Albert Zhang

Abstract: Anantharam, Jog and Nair recently unified the Shannon-Stam inequality and the entropic form of the Brascamp-Lieb inequalities under a common inequality. They left open the problems of extremizability and characterization of extremizers. Both questions are resolved in the present paper. Anantharam, Jog and Nair recently unified the Shannon-Stam inequality and the entropic form of the Brascamp-Lieb inequalities under a common inequality. They left open the problems of extremizability and characterization of extremizers. Both questions are resolved in the present paper. △ Less

Submitted 23 June, 2022; originally announced June 2022.

Comments: 21 pages. Comments welcome

Utility-based Resource Allocation and Pricing for Serverless Computing

Authors: Vipul Gupta, Soham Phade, Thomas Courtade, Kannan Ramchandran

Abstract: Serverless computing platforms currently rely on basic pricing schemes that are static and do not reflect customer feedback. This leads to significant inefficiencies from a total utility perspective. As one of the fastest-growing cloud services, serverless computing provides an opportunity to better serve both users and providers through the incorporation of market-based strategies for pricing and… ▽ More Serverless computing platforms currently rely on basic pricing schemes that are static and do not reflect customer feedback. This leads to significant inefficiencies from a total utility perspective. As one of the fastest-growing cloud services, serverless computing provides an opportunity to better serve both users and providers through the incorporation of market-based strategies for pricing and resource allocation. With the help of utility functions to model the delay-sensitivity of customers, we propose a novel scheduler to allocate resources for serverless computing. The resulting resource allocation scheme is optimal in the sense that it maximizes the aggregate utility of all users across the system, thus maximizing social welfare. Our approach gives rise to a natural dynamic pricing scheme that is obtained by solving an optimization problem in its dual form. We further develop feedback mechanisms that allow the cloud provider to converge to optimal resource allocation, even when the users' utilities are private and unknown to the service provider. Simulations show that our approach can track market demand and achieve significantly higher social welfare (or, equivalently, cost savings for customers) compared to existing schemes. △ Less

Submitted 24 January, 2022; v1 submitted 18 August, 2020; originally announced August 2020.

Comments: 31 pages, 10 figures

Linear Models are Most Favorable among Generalized Linear Models

Authors: Kuan-Yun Lee, Thomas A. Courtade

Abstract: We establish a nonasymptotic lower bound on the $L_2$ minimax risk for a class of generalized linear models. It is further shown that the minimax risk for the canonical linear model matches this lower bound up to a universal constant. Therefore, the canonical linear model may be regarded as most favorable among the considered class of generalized linear models (in terms of minimax risk). The proof… ▽ More We establish a nonasymptotic lower bound on the $L_2$ minimax risk for a class of generalized linear models. It is further shown that the minimax risk for the canonical linear model matches this lower bound up to a universal constant. Therefore, the canonical linear model may be regarded as most favorable among the considered class of generalized linear models (in terms of minimax risk). The proof makes use of an information-theoretic Bayesian Cramér-Rao bound for log-concave priors, established by Aras et al. (2019). △ Less

Submitted 9 June, 2020; originally announced June 2020.

Comments: To appear in the 2020 IEEE International Symposium on Information Theory

Serverless Straggler Mitigation using Local Error-Correcting Codes

Authors: Vipul Gupta, Dominic Carrano, Yaoqing Yang, Vaishaal Shankar, Thomas Courtade, Kannan Ramchandran

Abstract: Inexpensive cloud services, such as serverless computing, are often vulnerable to straggling nodes that increase end-to-end latency for distributed computation. We propose and implement simple yet principled approaches for straggler mitigation in serverless systems for matrix multiplication and evaluate them on several common applications from machine learning and high-performance computing. The p… ▽ More Inexpensive cloud services, such as serverless computing, are often vulnerable to straggling nodes that increase end-to-end latency for distributed computation. We propose and implement simple yet principled approaches for straggler mitigation in serverless systems for matrix multiplication and evaluate them on several common applications from machine learning and high-performance computing. The proposed schemes are inspired by error-correcting codes and employ parallel encoding and decoding over the data stored in the cloud using serverless workers. This creates a fully distributed computing framework without using a master node to conduct encoding or decoding, which removes the computation, communication and storage bottleneck at the master. On the theory side, we establish that our proposed scheme is asymptotically optimal in terms of decoding time and provide a lower bound on the number of stragglers it can tolerate with high probability. Through extensive experiments, we show that our scheme outperforms existing schemes such as speculative execution and other coding theoretic methods by at least 25%. △ Less

Submitted 21 January, 2020; originally announced January 2020.

Euclidean Forward-Reverse Brascamp-Lieb Inequalities: Finiteness, Structure and Extremals

Authors: Thomas A. Courtade, **gbo Liu

Abstract: A new proof is given for the fact that centered gaussian functions saturate the Euclidean forward-reverse Brascamp-Lieb inequalities, extending the Brascamp-Lieb and Barthe theorems. A duality principle for best constants is also developed, which generalizes the fact that the best constants in the Brascamp-Lieb and Barthe inequalities are equal. Finally, as the title hints, the main results concer… ▽ More A new proof is given for the fact that centered gaussian functions saturate the Euclidean forward-reverse Brascamp-Lieb inequalities, extending the Brascamp-Lieb and Barthe theorems. A duality principle for best constants is also developed, which generalizes the fact that the best constants in the Brascamp-Lieb and Barthe inequalities are equal. Finally, as the title hints, the main results concerning finiteness, structure and gaussian-extremizability for the Brascamp-Lieb inequality due to Bennett, Carbery, Christ and Tao are generalized to the setting of the forward-reverse Brascamp-Lieb inequality. △ Less

Submitted 29 August, 2019; v1 submitted 29 July, 2019; originally announced July 2019.

Comments: 37 pages, no figures. v2 includes added examples and minor updates to simplify presentation. Comments welcome

OverSketched Newton: Fast Convex Optimization for Serverless Systems

Authors: Vipul Gupta, Swanand Kadhe, Thomas Courtade, Michael W. Mahoney, Kannan Ramchandran

Abstract: Motivated by recent developments in serverless systems for large-scale computation as well as improvements in scalable randomized matrix algorithms, we develop OverSketched Newton, a randomized Hessian-based optimization algorithm to solve large-scale convex optimization problems in serverless systems. OverSketched Newton leverages matrix sketching ideas from Randomized Numerical Linear Algebra to… ▽ More Motivated by recent developments in serverless systems for large-scale computation as well as improvements in scalable randomized matrix algorithms, we develop OverSketched Newton, a randomized Hessian-based optimization algorithm to solve large-scale convex optimization problems in serverless systems. OverSketched Newton leverages matrix sketching ideas from Randomized Numerical Linear Algebra to compute the Hessian approximately. These sketching methods lead to inbuilt resiliency against stragglers that are a characteristic of serverless architectures. Depending on whether the problem is strongly convex or not, we propose different iteration updates using the approximate Hessian. For both cases, we establish convergence guarantees for OverSketched Newton and empirically validate our results by solving large-scale supervised learning problems on real-world datasets. Experiments demonstrate a reduction of ~50% in total running time on AWS Lambda, compared to state-of-the-art distributed optimization schemes. △ Less

Submitted 27 August, 2020; v1 submitted 21 March, 2019; originally announced March 2019.

Comments: 37 pages, 12 figures

A Family of Bayesian Cramér-Rao Bounds, and Consequences for Log-Concave Priors

Authors: Efe Aras, Kuan-Yun Lee, Ashwin Pananjady, Thomas A. Courtade

Abstract: Under minimal regularity assumptions, we establish a family of information-theoretic Bayesian Cramér-Rao bounds, indexed by probability measures that satisfy a logarithmic Sobolev inequality. This family includes as a special case the known Bayesian Cramér-Rao bound (or van Trees inequality), and its less widely known entropic improvement due to Efroimovich. For the setting of a log-concave prior,… ▽ More Under minimal regularity assumptions, we establish a family of information-theoretic Bayesian Cramér-Rao bounds, indexed by probability measures that satisfy a logarithmic Sobolev inequality. This family includes as a special case the known Bayesian Cramér-Rao bound (or van Trees inequality), and its less widely known entropic improvement due to Efroimovich. For the setting of a log-concave prior, we obtain a Bayesian Cramér-Rao bound which holds for any (possibly biased) estimator and, unlike the van Trees inequality, does not depend on the Fisher information of the prior. △ Less

Submitted 22 February, 2019; originally announced February 2019.

Comments: 15 pages, no figures. Comments welcome

Transportation Proof of an inequality by Anantharam, Jog and Nair

Authors: Thomas A. Courtade

Abstract: Anantharam, Jog and Nair recently put forth an entropic inequality which simultaneously generalizes the Shannon-Stam entropy power inequality and the Brascamp-Lieb inequality in entropic form. We give a brief proof of their result based on optimal transport. Anantharam, Jog and Nair recently put forth an entropic inequality which simultaneously generalizes the Shannon-Stam entropy power inequality and the Brascamp-Lieb inequality in entropic form. We give a brief proof of their result based on optimal transport. △ Less

Submitted 31 January, 2019; v1 submitted 30 January, 2019; originally announced January 2019.

Comments: 4 pages; no figures

OverSketch: Approximate Matrix Multiplication for the Cloud

Authors: Vipul Gupta, Shusen Wang, Thomas Courtade, Kannan Ramchandran

Abstract: We propose OverSketch, an approximate algorithm for distributed matrix multiplication in serverless computing. OverSketch leverages ideas from matrix sketching and high-performance computing to enable cost-efficient multiplication that is resilient to faults and straggling nodes pervasive in low-cost serverless architectures. We establish statistical guarantees on the accuracy of OverSketch and em… ▽ More We propose OverSketch, an approximate algorithm for distributed matrix multiplication in serverless computing. OverSketch leverages ideas from matrix sketching and high-performance computing to enable cost-efficient multiplication that is resilient to faults and straggling nodes pervasive in low-cost serverless architectures. We establish statistical guarantees on the accuracy of OverSketch and empirically validate our results by solving a large-scale linear program using interior-point methods and demonstrate a 34% reduction in compute time on AWS Lambda. △ Less

Submitted 21 February, 2019; v1 submitted 6 November, 2018; originally announced November 2018.

Comments: Published in Proc. IEEE Big Data 2018. Updated version provides details of distributed sketching and highlights other advantages of OverSketch

Journal ref: 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA, 2018, pp. 298-304

Stability of the Bakry-Emery theorem on $\mathbb{R}^n$

Authors: Thomas A. Courtade, Max Fathi

Abstract: We prove stability estimates for the Bakry-Emery bound on Poincaré and logarithmic Sobolev constants of uniformly log-concave measures. In particular, we improve the quantitative bound in a result of De Philippis and Figalli asserting that if a $1$-uniformly log-concave measure has almost the same Poincaré constant as the standard Gaussian measure, then it almost splits off a Gaussian factor, and… ▽ More We prove stability estimates for the Bakry-Emery bound on Poincaré and logarithmic Sobolev constants of uniformly log-concave measures. In particular, we improve the quantitative bound in a result of De Philippis and Figalli asserting that if a $1$-uniformly log-concave measure has almost the same Poincaré constant as the standard Gaussian measure, then it almost splits off a Gaussian factor, and establish similar new results for logarithmic Sobolev inequalities. As a consequence, we obtain dimension-free stability estimates for Gaussian concentration of Lipschitz functions. The proofs are based on Stein's method, optimal transport, and an approximate integration by parts identity relating measures and approximate optimizers in the associated functional inequality. △ Less

Submitted 14 September, 2018; v1 submitted 25 July, 2018; originally announced July 2018.

Comments: 25 pages. V2 includes new results on logarithmic Sobolev inequalities and Gaussian concentration. Comments are welcome

Bounds on the Poincaré constant for convolution measures

Authors: Thomas A. Courtade

Abstract: We establish a Shearer-type inequality for the Poincaré constant, showing that the Poincaré constant corresponding to the convolution of a collection of measures can be nontrivially controlled by the Poincaré constants corresponding to convolutions of subsets of measures. This implies, for example, that the Poincaré constant is non-increasing along the central limit theorem. We also establish a di… ▽ More We establish a Shearer-type inequality for the Poincaré constant, showing that the Poincaré constant corresponding to the convolution of a collection of measures can be nontrivially controlled by the Poincaré constants corresponding to convolutions of subsets of measures. This implies, for example, that the Poincaré constant is non-increasing along the central limit theorem. We also establish a dimension-free stability estimate for subadditivity of the Poincaré constant on convolutions which uniformly improves an earlier one-dimensional estimate of a similar nature by Johnson (2004). As a byproduct of our arguments, we find that the monotone properties of entropy, Fisher information and the Poincaré constant along the CLT find a common root in Shearer's inequality. △ Less

Submitted 29 June, 2018; originally announced July 2018.

Comments: comments welcome

Worst-case vs Average-case Design for Estimation from Fixed Pairwise Comparisons

Authors: Ashwin Pananjady, Cheng Mao, Vidya Muthukumar, Martin J. Wainwright, Thomas A. Courtade

Abstract: Pairwise comparison data arises in many domains, including tournament rankings, web search, and preference elicitation. Given noisy comparisons of a fixed subset of pairs of items, we study the problem of estimating the underlying comparison probabilities under the assumption of strong stochastic transitivity (SST). We also consider the noisy sorting subclass of the SST model. We show that when th… ▽ More Pairwise comparison data arises in many domains, including tournament rankings, web search, and preference elicitation. Given noisy comparisons of a fixed subset of pairs of items, we study the problem of estimating the underlying comparison probabilities under the assumption of strong stochastic transitivity (SST). We also consider the noisy sorting subclass of the SST model. We show that when the assignment of items to the topology is arbitrary, these permutation-based models, unlike their parametric counterparts, do not admit consistent estimation for most comparison topologies used in practice. We then demonstrate that consistent estimation is possible when the assignment of items to the topology is randomized, thus establishing a dichotomy between worst-case and average-case designs. We propose two estimators in the average-case setting and analyze their risk, showing that it depends on the comparison topology only through the degree sequence of the topology. The rates achieved by these estimators are shown to be optimal for a large class of graphs. Our results are corroborated by simulations on multiple comparison topologies. △ Less

Submitted 19 July, 2017; originally announced July 2017.

Denoising Linear Models with Permuted Data

Authors: Ashwin Pananjady, Martin J. Wainwright, Thomas A. Courtade

Abstract: The multivariate linear regression model with shuffled data and additive Gaussian noise arises in various correspondence estimation and matching problems. Focusing on the denoising aspect of this problem, we provide a characterization the minimax error rate that is sharp up to logarithmic factors. We also analyze the performance of two versions of a computationally efficient estimator, and establi… ▽ More The multivariate linear regression model with shuffled data and additive Gaussian noise arises in various correspondence estimation and matching problems. Focusing on the denoising aspect of this problem, we provide a characterization the minimax error rate that is sharp up to logarithmic factors. We also analyze the performance of two versions of a computationally efficient estimator, and establish their consistency for a large range of input parameters. Finally, we provide an exact algorithm for the noiseless problem and demonstrate its performance on an image point-cloud matching task. Our analysis also extends to datasets with outliers. △ Less

Submitted 24 April, 2017; originally announced April 2017.

Comments: To appear in part at ISIT 2017, Aachen

A Counterexample to the Vector Generalization of Costa's EPI, and Partial Resolution

Authors: Thomas A. Courtade, Guangyue Han, Yaochen Wu

Abstract: We give a counterexample to the vector generalization of Costa's entropy power inequality (EPI) due to Liu, Liu, Poor and Shamai. In particular, the claimed inequality can fail if the matix-valued parameter in the convex combination does not commute with the covariance of the additive Gaussian noise. Conversely, the inequality holds if these two matrices commute. We give a counterexample to the vector generalization of Costa's entropy power inequality (EPI) due to Liu, Liu, Poor and Shamai. In particular, the claimed inequality can fail if the matix-valued parameter in the convex combination does not commute with the covariance of the additive Gaussian noise. Conversely, the inequality holds if these two matrices commute. △ Less

Submitted 20 April, 2017; originally announced April 2017.

Comments: 3 pages

Existence of Stein Kernels under a Spectral Gap, and Discrepancy Bound

Authors: Thomas A. Courtade, Max Fathi, Ashwin Pananjady

Abstract: We establish existence of Stein kernels for probability measures on $\mathbb{R}^d$ satisfying a Poincaré inequality, and obtain bounds on the Stein discrepancy of such measures. Applications to quantitative central limit theorems are discussed, including a new CLT in Wasserstein distance $W_2$ with optimal rate and dependence on the dimension. As a byproduct, we obtain a stability version of an es… ▽ More We establish existence of Stein kernels for probability measures on $\mathbb{R}^d$ satisfying a Poincaré inequality, and obtain bounds on the Stein discrepancy of such measures. Applications to quantitative central limit theorems are discussed, including a new CLT in Wasserstein distance $W_2$ with optimal rate and dependence on the dimension. As a byproduct, we obtain a stability version of an estimate of the Poincaré constant of probability measures under a second moment constraint. The results extend more generally to the setting of converse weighted Poincaré inequalities. The proof is based on simple arguments of calculus of variations. Further, we establish two general properties enjoyed by the Stein discrepancy, holding whenever a Stein kernel exists: Stein discrepancy is strictly decreasing along the CLT, and it controls the skewness of a random vector. △ Less

Submitted 8 March, 2018; v1 submitted 22 March, 2017; originally announced March 2017.

Comments: revised version, comments are welcome

Information-Theoretic Perspectives on Brascamp-Lieb Inequality and Its Reverse

Authors: **gbo Liu, Thomas A. Courtade, Paul Cuff, Sergio Verdu

Abstract: We introduce an inequality which may be viewed as a generalization of both the Brascamp-Lieb inequality and its reverse (Barthe's inequality), and prove its information-theoretic (i.e.\ entropic) formulation. This result leads to a unified approach to functional inequalities such as the variational formula of Rényi entropy, hypercontractivity and its reverse, strong data processing inequalities, a… ▽ More We introduce an inequality which may be viewed as a generalization of both the Brascamp-Lieb inequality and its reverse (Barthe's inequality), and prove its information-theoretic (i.e.\ entropic) formulation. This result leads to a unified approach to functional inequalities such as the variational formula of Rényi entropy, hypercontractivity and its reverse, strong data processing inequalities, and transportation-cost inequalities, whose utility in the proofs of various coding theorems has gained growing popularity recently. We show that our information-theoretic setting is convenient for proving properties such as data processing, tensorization, convexity (Riesz-Thorin interpolation) and Gaussian optimality. In particular, we elaborate on a "doubling trick" used by Lieb and Geng-Nair to prove several results on Gaussian optimality. Several applications are discussed, including a generalization of the Brascamp-Lieb inequality involving Gaussian random transformations, the determination of Wyner's common information of vector Gaussian sources, and the achievable rate region of certain key generation problems in the case of vector Gaussian sources. △ Less

Submitted 3 December, 2017; v1 submitted 20 February, 2017; originally announced February 2017.

Comments: Corrected some typos in the previous version

Wasserstein Stability of the Entropy Power Inequality for Log-Concave Densities

Authors: Thomas A. Courtade, Max Fathi, Ashwin Pananjady

Abstract: We establish quantitative stability results for the entropy power inequality (EPI). Specifically, we show that if uniformly log-concave densities nearly saturate the EPI, then they must be close to Gaussian densities in the quadratic Wasserstein distance. Further, if one of the densities is log-concave and the other is Gaussian, then the deficit in the EPI can be controlled in terms of the $L^1$-W… ▽ More We establish quantitative stability results for the entropy power inequality (EPI). Specifically, we show that if uniformly log-concave densities nearly saturate the EPI, then they must be close to Gaussian densities in the quadratic Wasserstein distance. Further, if one of the densities is log-concave and the other is Gaussian, then the deficit in the EPI can be controlled in terms of the $L^1$-Wasserstein distance. As a counterpoint, an example shows that the EPI can be unstable with respect to the quadratic Wasserstein distance when densities are uniformly log-concave on sets of measure arbitrarily close to one. Our stability results can be extended to non-log-concave densities, provided certain regularity conditions are met. The proofs are based on optimal transportation. △ Less

Submitted 25 October, 2016; originally announced October 2016.

Comments: 19 pages

Novel probabilistic models of spatial genetic ancestry with applications to stratification correction in genome-wide association studies

Authors: Anand Bhaskar, Adel Javanmard, Thomas A. Courtade, David Tse

Abstract: Genetic variation in human populations is influenced by geographic ancestry due to spatial locality in historical mating and migration patterns. Spatial population structure in genetic datasets has been traditionally analyzed using either model-free algorithms, such as principal components analysis (PCA) and multidimensional scaling, or using explicit spatial probabilistic models of allele frequen… ▽ More Genetic variation in human populations is influenced by geographic ancestry due to spatial locality in historical mating and migration patterns. Spatial population structure in genetic datasets has been traditionally analyzed using either model-free algorithms, such as principal components analysis (PCA) and multidimensional scaling, or using explicit spatial probabilistic models of allele frequency evolution. We develop a general probabilistic model and an associated inference algorithm that unify the model-based and data-driven approaches to visualizing and inferring population structure. Our algorithm, Geographic Ancestry Positioning (GAP), relates local genetic distances between samples to their spatial distances, and can be used for visually discerning population structure as well as accurately inferring the spatial origin of individuals on a two-dimensional continuum. On both simulated and several real datasets from diverse human populations, GAP exhibits substantially lower error in reconstructing spatial ancestry coordinates compared to PCA. Our spatial inference algorithm can also be effectively applied to the problem of population stratification in genome-wide association studies (GWAS), where hidden population structure can create fictitious associations when population ancestry is correlated with both the genotype and the trait. We develop an association test that uses the ancestry coordinates inferred by GAP to accurately account for ancestry-induced correlations in GWAS. Based on simulations and analysis of a dataset of 10 metabolic traits measured in a Northern Finland cohort, which is known to exhibit significant population structure, we find that our method has superior power to current approaches. △ Less

Submitted 25 October, 2016; v1 submitted 24 October, 2016; originally announced October 2016.

Comments: Supplementary information included to the main text

Monotonicity of Entropy and Fisher Information: A Quick Proof via Maximal Correlation

Authors: Thomas A. Courtade

Abstract: A simple proof is given for the monotonicity of entropy and Fisher information associated to sums of i.i.d. random variables. The proof relies on a characterization of maximal correlation for partial sums due to Dembo, Kagan and Shepp. A simple proof is given for the monotonicity of entropy and Fisher information associated to sums of i.i.d. random variables. The proof relies on a characterization of maximal correlation for partial sums due to Dembo, Kagan and Shepp. △ Less

Submitted 13 October, 2016; originally announced October 2016.

Comments: To appear, Communications on Information and Systems

Links between the Logarithmic Sobolev Inequality and the convolution inequalities for Entropy and Fisher Information

Authors: Thomas A. Courtade

Abstract: Relative to the Gaussian measure on $\mathbb{R}^d$, entropy and Fisher information are famously related via Gross' logarithmic Sobolev inequality (LSI). These same functionals also separately satisfy convolution inequalities, as proved by Stam. We establish a dimension-free inequality that interpolates among these relations. Several interesting corollaries follow: (i) the deficit in the LSI satisf… ▽ More Relative to the Gaussian measure on $\mathbb{R}^d$, entropy and Fisher information are famously related via Gross' logarithmic Sobolev inequality (LSI). These same functionals also separately satisfy convolution inequalities, as proved by Stam. We establish a dimension-free inequality that interpolates among these relations. Several interesting corollaries follow: (i) the deficit in the LSI satisfies a convolution inequality itself; (ii) the deficit in the LSI controls convergence in the entropic and Fisher information central limit theorems; and (iii) the LSI is stable with respect to HWI jumps (i.e., a jump in any of the convolution inequalities associated to the HWI functionals). Another consequence is that the convolution inequalities for Fisher information and entropy powers are reversible in general, up to a factor depending on the Stam defect. An improved form of Nelson's hypercontractivity estimate also follows. Finally, we speculate on the possibility of an analogous reverse Brunn-Minkowski inequality and a related upper bound on surface area associated to Minkowski sums. △ Less

Submitted 18 August, 2016; originally announced August 2016.

Entropy Jumps for Radially Symmetric Random Vectors

Authors: Thomas A. Courtade

Abstract: We establish a quantitative bound on the entropy jump associated to the sum of independent, identically distributed (IID) radially symmetric random vectors having dimension greater than one. Following the usual approach, we first consider the analogous problem of Fisher information dissipation, and then integrate along the Ornstein-Uhlenbeck semigroup to obtain an entropic inequality. In a departu… ▽ More We establish a quantitative bound on the entropy jump associated to the sum of independent, identically distributed (IID) radially symmetric random vectors having dimension greater than one. Following the usual approach, we first consider the analogous problem of Fisher information dissipation, and then integrate along the Ornstein-Uhlenbeck semigroup to obtain an entropic inequality. In a departure from previous work, we appeal to a result by Desvillettes and Villani on entropy production associated to the Landau equation. This obviates strong regularity assumptions, such as presence of a spectral gap and log-concavity of densities, but comes at the expense of radial symmetry. As an application, we give a quantitative estimate of the deficit in the Gaussian logarithmic Sobolev inequality for radially symmetric functions. △ Less

Submitted 3 November, 2016; v1 submitted 18 August, 2016; originally announced August 2016.

Comments: 19 pages. Updates relative to v1: Fixed a reference and added another

Linear Regression with an Unknown Permutation: Statistical and Computational Limits

Authors: Ashwin Pananjady, Martin J. Wainwright, Thomas A. Courtade

Abstract: Consider a noisy linear observation model with an unknown permutation, based on observing $y = Π^* A x^* + w$, where $x^* \in \mathbb{R}^d$ is an unknown vector, $Π^*$ is an unknown $n \times n$ permutation matrix, and $w \in \mathbb{R}^n$ is additive Gaussian noise. We analyze the problem of permutation recovery in a random design setting in which the entries of the matrix $A$ are drawn i.i.d. fr… ▽ More Consider a noisy linear observation model with an unknown permutation, based on observing $y = Π^* A x^* + w$, where $x^* \in \mathbb{R}^d$ is an unknown vector, $Π^*$ is an unknown $n \times n$ permutation matrix, and $w \in \mathbb{R}^n$ is additive Gaussian noise. We analyze the problem of permutation recovery in a random design setting in which the entries of the matrix $A$ are drawn i.i.d. from a standard Gaussian distribution, and establish sharp conditions on the SNR, sample size $n$, and dimension $d$ under which $Π^*$ is exactly and approximately recoverable. On the computational front, we show that the maximum likelihood estimate of $Π^*$ is NP-hard to compute, while also providing a polynomial time algorithm when $d =1$. △ Less

Submitted 9 August, 2016; originally announced August 2016.

Comments: To appear in part at the 2016 Allerton Conference on Control, Communication and Computing

Brascamp-Lieb Inequality and Its Reverse: An Information Theoretic View

Authors: **gbo Liu, Thomas A. Courtade, Paul Cuff, Sergio Verdu

Abstract: We generalize a result by Carlen and Cordero-Erausquin on the equivalence between the Brascamp-Lieb inequality and the subadditivity of relative entropy by allowing for random transformations (a broadcast channel). This leads to a unified perspective on several functional inequalities that have been gaining popularity in the context of proving impossibility results. We demonstrate that the informa… ▽ More We generalize a result by Carlen and Cordero-Erausquin on the equivalence between the Brascamp-Lieb inequality and the subadditivity of relative entropy by allowing for random transformations (a broadcast channel). This leads to a unified perspective on several functional inequalities that have been gaining popularity in the context of proving impossibility results. We demonstrate that the information theoretic dual of the Brascamp-Lieb inequality is a convenient setting for proving properties such as data processing, tensorization, convexity and Gaussian optimality. Consequences of the latter include an extension of the Brascamp-Lieb inequality allowing for Gaussian random transformations, the determination of the multivariate Wyner common information for Gaussian sources, and a multivariate version of Nelson's hypercontractivity theorem. Finally we present an information theoretic characterization of a reverse Brascamp-Lieb inequality involving a random transformation (a multiple access channel). △ Less

Submitted 9 May, 2016; originally announced May 2016.

Comments: 5 pages; to be presented at ISIT 2016

Partial DNA Assembly: A Rate-Distortion Perspective

Authors: Ilan Shomorony, Govinda M. Kamath, Fei Xia, Thomas A. Courtade, David N. Tse

Abstract: Earlier formulations of the DNA assembly problem were all in the context of perfect assembly; i.e., given a set of reads from a long genome sequence, is it possible to perfectly reconstruct the original sequence? In practice, however, it is very often the case that the read data is not sufficiently rich to permit unambiguous reconstruction of the original sequence. While a natural generalization o… ▽ More Earlier formulations of the DNA assembly problem were all in the context of perfect assembly; i.e., given a set of reads from a long genome sequence, is it possible to perfectly reconstruct the original sequence? In practice, however, it is very often the case that the read data is not sufficiently rich to permit unambiguous reconstruction of the original sequence. While a natural generalization of the perfect assembly formulation to these cases would be to consider a rate-distortion framework, partial assemblies are usually represented in terms of an assembly graph, making the definition of a distortion measure challenging. In this work, we introduce a distortion function for assembly graphs that can be understood as the logarithm of the number of Eulerian cycles in the assembly graph, each of which correspond to a candidate assembly that could have generated the observed reads. We also introduce an algorithm for the construction of an assembly graph and analyze its performance on real genomes. △ Less

Submitted 6 May, 2016; originally announced May 2016.

Comments: To be published at ISIT-2016. 11 pages, 10 figures

Strengthening the Entropy Power Inequality

Authors: Thomas A. Courtade

Abstract: We tighten the Entropy Power Inequality (EPI) when one of the random summands is Gaussian. Our strengthening is closely connected to the concept of strong data processing for Gaussian channels and generalizes the (vector extension of) Costa's EPI. This leads to a new reverse entropy power inequality and, as a corollary, sharpens Stam's inequality relating entropy power and Fisher information. Appl… ▽ More We tighten the Entropy Power Inequality (EPI) when one of the random summands is Gaussian. Our strengthening is closely connected to the concept of strong data processing for Gaussian channels and generalizes the (vector extension of) Costa's EPI. This leads to a new reverse entropy power inequality and, as a corollary, sharpens Stam's inequality relating entropy power and Fisher information. Applications to network information theory are given, including a short self-contained proof of the rate region for the two-encoder quadratic Gaussian source coding problem. Our argument is based on weak convergence and a technique employed by Geng and Nair for establishing Gaussian optimality via rotational-invariance, which traces its roots to a `doubling trick' that has been successfully used in the study of functional inequalities. △ Less

Submitted 9 February, 2016; originally announced February 2016.

Comments: 23 pages. Full version of submission to 2016 International Symposium on Information Theory. Presented in part at Institut Henri Poincaré Feb 10, 2016

Smoothing Brascamp-Lieb Inequalities and Strong Converses for Common Randomness Generation

Authors: **gbo Liu, Thomas A. Courtade, Paul Cuff, Sergio Verdu

Abstract: We study the infimum of the best constant in a functional inequality, the Brascamp-Lieb-like inequality, over auxiliary measures within a neighborhood of a product distribution. In the finite alphabet and the Gaussian cases, such an infimum converges to the best constant in a mutual information inequality. Implications for strong converse properties of two common randomness (CR) generation problem… ▽ More We study the infimum of the best constant in a functional inequality, the Brascamp-Lieb-like inequality, over auxiliary measures within a neighborhood of a product distribution. In the finite alphabet and the Gaussian cases, such an infimum converges to the best constant in a mutual information inequality. Implications for strong converse properties of two common randomness (CR) generation problems are discussed. In particular, we prove the strong converse property of the rate region for the omniscient helper CR generation problem in the discrete and the Gaussian cases. The latter case is perhaps the first instance of a strong converse for a continuous source when the rate region involves auxiliary random variables. △ Less

Submitted 6 February, 2016; originally announced February 2016.

Comments: 7 pages; first 5 pages submitted to ISIT 2016

Compressing Sparse Sequences under Local Decodability Constraints

Authors: Ashwin Pananjady, Thomas A. Courtade

Abstract: We consider a variable-length source coding problem subject to local decodability constraints. In particular, we investigate the blocklength scaling behavior attainable by encodings of $r$-sparse binary sequences, under the constraint that any source bit can be correctly decoded upon probing at most $d$ codeword bits. We consider both adaptive and non-adaptive access models, and derive upper and l… ▽ More We consider a variable-length source coding problem subject to local decodability constraints. In particular, we investigate the blocklength scaling behavior attainable by encodings of $r$-sparse binary sequences, under the constraint that any source bit can be correctly decoded upon probing at most $d$ codeword bits. We consider both adaptive and non-adaptive access models, and derive upper and lower bounds that often coincide up to constant factors. Notably, such a characterization for the fixed-blocklength analog of our problem remains unknown, despite considerable research over the last three decades. Connections to communication complexity are also briefly discussed. △ Less

Submitted 8 April, 2015; originally announced April 2015.

Comments: 8 pages, 1 figure. First five pages to appear in 2015 International Symposium on Information Theory. This version contains supplementary material

Do Read Errors Matter for Genome Assembly?

Authors: Ilan Shomorony, Thomas Courtade, David Tse

Abstract: While most current high-throughput DNA sequencing technologies generate short reads with low error rates, emerging sequencing technologies generate long reads with high error rates. A basic question of interest is the tradeoff between read length and error rate in terms of the information needed for the perfect assembly of the genome. Using an adversarial erasure error model, we make progress on t… ▽ More While most current high-throughput DNA sequencing technologies generate short reads with low error rates, emerging sequencing technologies generate long reads with high error rates. A basic question of interest is the tradeoff between read length and error rate in terms of the information needed for the perfect assembly of the genome. Using an adversarial erasure error model, we make progress on this problem by establishing a critical read length, as a function of the genome and the error rate, above which perfect assembly is guaranteed. For several real genomes, including those from the GAGE dataset, we verify that this critical read length is not significantly greater than the read length required for perfect assembly from reads without errors. △ Less

Submitted 25 January, 2015; originally announced January 2015.

Comments: Submitted to ISIT 2015

Coded Cooperative Data Exchange for a Secret Key

Authors: Thomas A. Courtade, Thomas R. Halford

Abstract: We consider a coded cooperative data exchange problem with the goal of generating a secret key. Specifically, we investigate the number of public transmissions required for a set of clients to agree on a secret key with probability one, subject to the constraint that it remains private from an eavesdropper. Although the problems are closely related, we prove that secret key generation with fewes… ▽ More We consider a coded cooperative data exchange problem with the goal of generating a secret key. Specifically, we investigate the number of public transmissions required for a set of clients to agree on a secret key with probability one, subject to the constraint that it remains private from an eavesdropper. Although the problems are closely related, we prove that secret key generation with fewest number of linear transmissions is NP-hard, while it is known that the analogous problem in traditional cooperative data exchange can be solved in polynomial time. In doing this, we completely characterize the best possible performance of linear coding schemes, and also prove that linear codes can be strictly suboptimal. Finally, we extend the single-key results to characterize the minimum number of public transmissions required to generate a desired integer number of statistically independent secret keys. △ Less

Submitted 1 July, 2014; originally announced July 2014.

Comments: Full version of a paper that appeared at ISIT 2014. 19 pages, 2 figures

An Extremal Inequality for Long Markov Chains

Authors: Thomas Courtade, Jiantao Jiao

Abstract: Let $X,Y$ be jointly Gaussian vectors, and consider random variables $U,V$ that satisfy the Markov constraint $U-X-Y-V$. We prove an extremal inequality relating the mutual informations between all ${4 \choose 2}$ pairs of random variables from the set $(U,X,Y,V)$. As a first application, we show that the rate region for the two-encoder quadratic Gaussian source coding problem follows as an immedi… ▽ More Let $X,Y$ be jointly Gaussian vectors, and consider random variables $U,V$ that satisfy the Markov constraint $U-X-Y-V$. We prove an extremal inequality relating the mutual informations between all ${4 \choose 2}$ pairs of random variables from the set $(U,X,Y,V)$. As a first application, we show that the rate region for the two-encoder quadratic Gaussian source coding problem follows as an immediate corollary of the the extremal inequality. In a second application, we establish the rate region for a vector-Gaussian source coding problem where Löwner-John ellipsoids are approximated based on rate-constrained descriptions of the data. △ Less

Submitted 28 April, 2014; originally announced April 2014.

Comments: 18 pages, 1 figure. Submitted to Transactions on Information Theory

Information Measures: the Curious Case of the Binary Alphabet

Authors: Jiantao Jiao, Thomas Courtade, Albert No, Kartik Venkat, Tsachy Weissman

Abstract: Four problems related to information divergence measures defined on finite alphabets are considered. In three of the cases we consider, we illustrate a contrast which arises between the binary-alphabet and larger-alphabet settings. This is surprising in some instances, since characterizations for the larger-alphabet settings do not generalize their binary-alphabet counterparts. Specifically, we sh… ▽ More Four problems related to information divergence measures defined on finite alphabets are considered. In three of the cases we consider, we illustrate a contrast which arises between the binary-alphabet and larger-alphabet settings. This is surprising in some instances, since characterizations for the larger-alphabet settings do not generalize their binary-alphabet counterparts. Specifically, we show that $f$-divergences are not the unique decomposable divergences on binary alphabets that satisfy the data processing inequality, thereby clarifying claims that have previously appeared in the literature. We also show that KL divergence is the unique Bregman divergence which is also an $f$-divergence for any alphabet size. We show that KL divergence is the unique Bregman divergence which is invariant to statistically sufficient transformations of the data, even when non-decomposable divergences are considered. Like some of the problems we consider, this result holds only when the alphabet size is at least three. △ Less

Submitted 28 November, 2014; v1 submitted 27 April, 2014; originally announced April 2014.

Comments: to appear in IEEE Transactions on Information Theory

Justification of Logarithmic Loss via the Benefit of Side Information

Authors: Jiantao Jiao, Thomas Courtade, Kartik Venkat, Tsachy Weissman

Abstract: We consider a natural measure of relevance: the reduction in optimal prediction risk in the presence of side information. For any given loss function, this relevance measure captures the benefit of side information for performing inference on a random variable under this loss function. When such a measure satisfies a natural data processing property, and the random variable of interest has alphabe… ▽ More We consider a natural measure of relevance: the reduction in optimal prediction risk in the presence of side information. For any given loss function, this relevance measure captures the benefit of side information for performing inference on a random variable under this loss function. When such a measure satisfies a natural data processing property, and the random variable of interest has alphabet size greater than two, we show that it is uniquely characterized by the mutual information, and the corresponding loss function coincides with logarithmic loss. In doing so, our work provides a new characterization of mutual information, and justifies its use as a measure of relevance. When the alphabet is binary, we characterize the only admissible forms the measure of relevance can assume while obeying the specified data processing property. Our results naturally extend to measuring causal influence between stochastic processes, where we unify different causal-inference measures in the literature as instantiations of directed information. △ Less

Submitted 22 December, 2015; v1 submitted 18 March, 2014; originally announced March 2014.

Journal ref: IEEE Transactions on Information Theory, Vol.61, Issue 10, pp 5357 - 5365, Oct. 2015

Enhanced Precision Through Multiple Reads for LDPC Decoding in Flash Memories

Authors: Jiadong Wang, Kasra Vakilinia, Tsung-Yi Chen, Thomas Courtade, Guiqiang Dong, Tong Zhang, Hari Shankar, Richard Wesel

Abstract: Multiple reads of the same Flash memory cell with distinct word-line voltages provide enhanced precision for LDPC decoding. In this paper, the word-line voltages are optimized by maximizing the mutual information (MI) of the quantized channel. The enhanced precision from a few additional reads allows FER performance to approach that of full-precision soft information and enables an LDPC code to si… ▽ More Multiple reads of the same Flash memory cell with distinct word-line voltages provide enhanced precision for LDPC decoding. In this paper, the word-line voltages are optimized by maximizing the mutual information (MI) of the quantized channel. The enhanced precision from a few additional reads allows FER performance to approach that of full-precision soft information and enables an LDPC code to significantly outperform a BCH code. A constant-ratio constraint provides a significant simplification in the optimization with no noticeable loss in performance. For a well-designed LDPC code, the quantization that maximizes the mutual information also minimizes the frame error rate in our simulations. However, for an example LDPC code with a high error floor caused by small absorbing sets, the MMI quantization does not provide the lowest frame error rate. The best quantization in this case introduces more erasures than would be optimal for the channel MI in order to mitigate the absorbing sets of the poorly designed code. The paper also identifies a trade-off in LDPC code design when decoding is performed with multiple precision levels; the best code at one level of precision will typically not be the best code at a different level of precision. △ Less

Submitted 18 February, 2014; v1 submitted 2 September, 2013; originally announced September 2013.

Comments: arXiv admin note: substantial text overlap with arXiv:1210.0149

Compression for Quadratic Similarity Queries

Authors: Amir Ingber, Thomas Courtade, Tsachy Weissman

Abstract: The problem of performing similarity queries on compressed data is considered. We focus on the quadratic similarity measure, and study the fundamental tradeoff between compression rate, sequence length, and reliability of queries performed on compressed data. For a Gaussian source, we show that queries can be answered reliably if and only if the compression rate exceeds a given threshold - the ide… ▽ More The problem of performing similarity queries on compressed data is considered. We focus on the quadratic similarity measure, and study the fundamental tradeoff between compression rate, sequence length, and reliability of queries performed on compressed data. For a Gaussian source, we show that queries can be answered reliably if and only if the compression rate exceeds a given threshold - the identification rate - which we explicitly characterize. Moreover, when compression is performed at a rate greater than the identification rate, responses to queries on the compressed data can be made exponentially reliable. We give a complete characterization of this exponent, which is analogous to the error and excess-distortion exponents in channel and source coding, respectively. For a general source we prove that, as with classical compression, the Gaussian source requires the largest compression rate among sources with a given variance. Moreover, a robust scheme is described that attains this maximal rate for any source distribution. △ Less

Submitted 24 July, 2013; originally announced July 2013.

Comments: 39 pages, 6 figures, submitted to IEEE Trans. on Information Theory

Outer Bounds for Multiterminal Source Coding via a Strong Data Processing Inequality

Authors: Thomas A. Courtade

Abstract: An intuitive outer bound for the multiterminal source coding problem is given. The proposed bound explicitly couples the rate distortion functions for each source and correlation measures which derive from a "strong" data processing inequality. Unlike many standard outer bounds, the proposed bound is not parameterized by a continuous family of auxiliary random variables, but instead only requires… ▽ More An intuitive outer bound for the multiterminal source coding problem is given. The proposed bound explicitly couples the rate distortion functions for each source and correlation measures which derive from a "strong" data processing inequality. Unlike many standard outer bounds, the proposed bound is not parameterized by a continuous family of auxiliary random variables, but instead only requires maximizing two ratios of divergences which do not depend on the distortion functions under consideration. △ Less

Submitted 15 July, 2013; v1 submitted 14 February, 2013; originally announced February 2013.

Comments: 5 pages, 2 figures. Presented at ISIT 2013 in Istanbul, Turkey. (NOTE: v2 corrects an error in v1 due to its use of the Erkip-Cover strong data processing inequality. This inequality has recently been corrected by Anantharam et al. in arxiv/1304.6133v1 [cs.IT])

Which Boolean Functions are Most Informative?

Authors: Gowtham R. Kumar, Thomas A. Courtade

Abstract: We introduce a simply stated conjecture regarding the maximum mutual information a Boolean function can reveal about noisy inputs. Specifically, let $X^n$ be i.i.d. Bernoulli(1/2), and let $Y^n$ be the result of passing $X^n$ through a memoryless binary symmetric channel with crossover probability $α$. For any Boolean function $b:\{0,1\}^n\rightarrow \{0,1\}$, we conjecture that… ▽ More We introduce a simply stated conjecture regarding the maximum mutual information a Boolean function can reveal about noisy inputs. Specifically, let $X^n$ be i.i.d. Bernoulli(1/2), and let $Y^n$ be the result of passing $X^n$ through a memoryless binary symmetric channel with crossover probability $α$. For any Boolean function $b:\{0,1\}^n\rightarrow \{0,1\}$, we conjecture that $I(b(X^n);Y^n)\leq 1-H(α)$. While the conjecture remains open, we provide substantial evidence supporting its validity. △ Less

Submitted 15 July, 2013; v1 submitted 11 February, 2013; originally announced February 2013.

Comments: 5 pages, 1 figure. Presented at ISIT 2013 in Istanbul, Turkey. (v2 corrects minor typos present in v1)

LDPC Decoding with Limited-Precision Soft Information in Flash Memories

Authors: Jiadong Wang, Guiqiang Dong, Thomas Courtade, Hari Shankar, Tong Zhang, Richard Wesel

Abstract: This paper investigates the application of low-density parity-check (LDPC) codes to Flash memories. Multiple cell reads with distinct word-line voltages provide limited-precision soft information for the LDPC decoder. The values of the word-line voltages (also called reference voltages) are optimized by maximizing the mutual information (MI) between the input and output of the multiple-read channe… ▽ More This paper investigates the application of low-density parity-check (LDPC) codes to Flash memories. Multiple cell reads with distinct word-line voltages provide limited-precision soft information for the LDPC decoder. The values of the word-line voltages (also called reference voltages) are optimized by maximizing the mutual information (MI) between the input and output of the multiple-read channel. Constraining the maximum mutual-information (MMI) quantization to enforce a constant-ratio constraint provides a significant simplification with no noticeable loss in performance. Our simulation results suggest that for a well-designed LDPC code, the quantization that maximizes the mutual information will also minimize the frame error rate. However, care must be taken to design the code to perform well in the quantized channel. An LDPC code designed for a full-precision Gaussian channel may perform poorly in the quantized setting. Our LDPC code designs provide an example where quantization increases the importance of absorbing sets thus changing how the LDPC code should be optimized. Simulation results show that small increases in precision enable the LDPC code to significantly outperform a BCH code with comparable rate and block length (but without the benefit of the soft information) over a range of frame error rates. △ Less

Submitted 29 September, 2012; originally announced October 2012.

Information Masking and Amplification: The Source Coding Setting

Authors: Thomas Courtade

Abstract: The complementary problems of masking and amplifying channel state information in the Gel'fand-Pinsker channel have recently been solved by Merhav and Shamai, and Kim et al., respectively. In this paper, we study a related source coding problem. Specifically, we consider the two-encoder source coding setting where one source is to be amplified, while the other source is to be masked. In general, t… ▽ More The complementary problems of masking and amplifying channel state information in the Gel'fand-Pinsker channel have recently been solved by Merhav and Shamai, and Kim et al., respectively. In this paper, we study a related source coding problem. Specifically, we consider the two-encoder source coding setting where one source is to be amplified, while the other source is to be masked. In general, there is a tension between these two objectives which is characterized by the amplification-masking tradeoff. In this paper, we give a single-letter description of this tradeoff. We apply this result, together with a recent theorem by Courtade and Weissman on multiterminal source coding, to solve a fundamental entropy characterization problem. △ Less

Submitted 25 April, 2012; originally announced April 2012.

Comments: 6 pages, 1 figure, to appear at the IEEE 2012 International Symposium on Information Theory (ISIT 2012)

Coded Cooperative Data Exchange in Multihop Networks

Authors: Thomas A. Courtade, Richard D. Wesel

Abstract: Consider a connected network of n nodes that all wish to recover k desired packets. Each node begins with a subset of the desired packets and exchanges coded packets with its neighbors. This paper provides necessary and sufficient conditions which characterize the set of all transmission schemes that permit every node to ultimately learn (recover) all k packets. When the network satisfies certain… ▽ More Consider a connected network of n nodes that all wish to recover k desired packets. Each node begins with a subset of the desired packets and exchanges coded packets with its neighbors. This paper provides necessary and sufficient conditions which characterize the set of all transmission schemes that permit every node to ultimately learn (recover) all k packets. When the network satisfies certain regularity conditions and packets are randomly distributed, this paper provides tight concentration results on the number of transmissions required to achieve universal recovery. For the case of a fully connected network, a polynomial-time algorithm for computing an optimal transmission scheme is derived. An application to secrecy generation is discussed. △ Less

Submitted 15 March, 2012; originally announced March 2012.

Comments: 49 pages, 6 figures, submitted to Transactions on Information Theory

Multiterminal Source Coding under Logarithmic Loss

Authors: Thomas Courtade, Tsachy Weissman

Abstract: We consider the classical two-encoder multiterminal source coding problem where distortion is measured under logarithmic loss. We provide a single-letter characterization of the achievable rate distortion region for arbitrarily correlated sources with finite alphabets. In doing so, we also give the rate distortion region for the $m$-encoder CEO problem (also under logarithmic loss). Several applic… ▽ More We consider the classical two-encoder multiterminal source coding problem where distortion is measured under logarithmic loss. We provide a single-letter characterization of the achievable rate distortion region for arbitrarily correlated sources with finite alphabets. In doing so, we also give the rate distortion region for the $m$-encoder CEO problem (also under logarithmic loss). Several applications and examples are given. △ Less

Submitted 11 July, 2012; v1 submitted 13 October, 2011; originally announced October 2011.

Comments: 52 pages, 5 figures. Submitted to IEEE Transactions on Information Theory, partial version appeared in IEEE International Symposium on Information Theory (ISIT) 2012. Version 3 contains minor revisions and corrects the proof of the strengthened converse of Theorem 6

Designing Nonlinear Turbo Codes with a Target Ones Density

Authors: Jiadong Wang, Thomas Courtade, Tsung-Yi Chen, Bike Xie, Richard Wesel

Abstract: Certain binary asymmetric channels, such as Z-channels in which one of the two crossover probabilities is zero, demand optimal ones densities different from 50%. Some broadcast channels, such as broadcast binary symmetric channels (BBSC) where each component channel is a binary symmetric channel, also require a non-uniform input distribution due to the superposition coding scheme, which is known t… ▽ More Certain binary asymmetric channels, such as Z-channels in which one of the two crossover probabilities is zero, demand optimal ones densities different from 50%. Some broadcast channels, such as broadcast binary symmetric channels (BBSC) where each component channel is a binary symmetric channel, also require a non-uniform input distribution due to the superposition coding scheme, which is known to achieve the boundary of capacity region. This paper presents a systematic technique for designing nonlinear turbo codes that are able to support ones densities different from 50%. To demonstrate the effectiveness of our design technique, we design and simulate nonlinear turbo codes for the Z-channel and the BBSC. The best nonlinear turbo code is less than 0.02 bits from capacity. △ Less

Submitted 8 July, 2011; originally announced July 2011.

Multiterminal Source Coding with an Entropy-Based Distortion Measure

Authors: Thomas Courtade, Richard Wesel

Abstract: In this paper, we consider a class of multiterminal source coding problems, each subject to distortion constraints computed using a specific, entropy-based, distortion measure. We provide the achievable rate distortion region for two cases and, in so doing, we demonstrate a relationship between the lossy multiterminal source coding problems with our specific distortion measure and (1) the canonica… ▽ More In this paper, we consider a class of multiterminal source coding problems, each subject to distortion constraints computed using a specific, entropy-based, distortion measure. We provide the achievable rate distortion region for two cases and, in so doing, we demonstrate a relationship between the lossy multiterminal source coding problems with our specific distortion measure and (1) the canonical Slepian-Wolf lossless distributed source coding network, and (2) the Ahlswede-Körner-Wyner source coding with side information problem in which only one of the sources is recovered losslessly. △ Less

Submitted 31 May, 2011; originally announced June 2011.

Comments: 6 pages, to appear at IEEE International Symposium on Information Theory 2011 (ISIT 2011), Saint-Petersburg, Russia

Optimal Encoding Schemes for Several Classes of Discrete Degraded Broadcast Channels

Authors: Bike Xie, Thomas Courtade, Richard D. Wesel

Abstract: Consider a memoryless degraded broadcast channel (DBC) in which the channel output is a single-letter function of the channel input and the channel noise. As examples, for the Gaussian broadcast channel (BC) this single-letter function is regular Euclidian addition and for the binary-symmetric BC this single-letter function is Galois-Field-two addition. This paper identifies several classes of dis… ▽ More Consider a memoryless degraded broadcast channel (DBC) in which the channel output is a single-letter function of the channel input and the channel noise. As examples, for the Gaussian broadcast channel (BC) this single-letter function is regular Euclidian addition and for the binary-symmetric BC this single-letter function is Galois-Field-two addition. This paper identifies several classes of discrete memoryless DBCs for which a relatively simple encoding scheme, which we call natural encoding, achieves capacity. Natural Encoding (NE) combines symbols from independent codebooks (one for each receiver) using the same single-letter function that adds distortion to the channel. The alphabet size of each NE codebook is bounded by that of the channel input. Inspired by Witsenhausen and Wyner, this paper defines the conditional entropy bound function $F^*$, studies its properties, and applies them to show that NE achieves the boundary of the capacity region for the multi-receiver broadcast Z channel. Then, this paper defines the input-symmetric DBC, introduces permutation encoding for the input-symmetric DBC, and proves its optimality. Because it is a special case of permutation encoding, NE is capacity achieving for the two-receiver group-operation DBC. Combining the broadcast Z channel and group-operation DBC results yields a proof that NE is also optimal for the discrete multiplication DBC. Along the way, the paper also provides explicit parametric expressions for the two-receiver binary-symmetric DBC and broadcast Z channel. △ Less

Submitted 23 October, 2011; v1 submitted 25 November, 2008; originally announced November 2008.

Comments: 50 pages, 18 figures