Showing 1–29 of 29 results for author: Madiman, M

The entropic doubling constant and robustness of Gaussian codebooks for additive-noise channels

Authors: Lampros Gavalakis, Ioannis Kontoyiannis, Mokshay Madiman

Abstract: Entropy comparison inequalities are obtained for the differential entropy $h(X+Y)$ of the sum of two independent random vectors $X,Y$, when one is replaced by a Gaussian. For identically distributed random vectors $X,Y$, these are closely related to bounds on the entropic doubling constant, which quantifies the entropy increase when adding an independent copy of a random vector to itself. Conseque… ▽ More Entropy comparison inequalities are obtained for the differential entropy $h(X+Y)$ of the sum of two independent random vectors $X,Y$, when one is replaced by a Gaussian. For identically distributed random vectors $X,Y$, these are closely related to bounds on the entropic doubling constant, which quantifies the entropy increase when adding an independent copy of a random vector to itself. Consequences of both large and small doubling are explored. For the former, lower bounds are deduced on the entropy increase when adding an independent Gaussian, while for the latter, a qualitative stability result for the entropy power inequality is obtained. In the more general case of non-identically distributed random vectors $X,Y$, a Gaussian comparison inequality with interesting implications for channel coding is established: For additive-noise channels with a power constraint, Gaussian codebooks come within a $\frac{\sf snr}{3{\sf snr}+2}$ factor of capacity. In the low-SNR regime this improves the half-a-bit additive bound of Zamir and Erez (2004). Analogous results are obtained for additive-noise multiple access channels, and for linear, additive-noise MIMO channels. △ Less

Submitted 11 March, 2024; originally announced March 2024.

Comments: 23 pages, no figures

A Closed-Form EVSI Expression for a Multinomial Data-Generating Process

Authors: Adam Fleischhacker, Pak-Wing Fok, Mokshay Madiman, Nan Wu

Abstract: This paper derives analytic expressions for the expected value of sample information (EVSI), the expected value of distribution information (EVDI), and the optimal sample size when data consists of independent draws from a bounded sequence of integers. Due to challenges of creating tractable EVSI expressions, most existing work valuing data does so in one of three ways: 1) analytically through clo… ▽ More This paper derives analytic expressions for the expected value of sample information (EVSI), the expected value of distribution information (EVDI), and the optimal sample size when data consists of independent draws from a bounded sequence of integers. Due to challenges of creating tractable EVSI expressions, most existing work valuing data does so in one of three ways: 1) analytically through closed-form expressions on the upper bound of the value of data, 2) calculating the expected value of data using numerical comparisons of decisions made using simulated data to optimal decisions where the underlying data distribution is known, or 3) using variance reduction as proxy for the uncertainty reduction that accompanies more data. For the very flexible case of modelling integer-valued observations using a multinomial data-generating process with Dirichlet prior, this paper develops expressions that 1) generalize existing beta-Binomial computations, 2) do not require prior knowledge of some underlying "true" distribution, and 3) can be computed prior to the collection of any sample data. △ Less

Submitted 1 December, 2022; originally announced January 2023.

Journal ref: Decision Analysis, Vol. 20, No. 1, pp. 73--84, March 2023

Bernoulli sums and Rényi entropy inequalities

Authors: Mokshay Madiman, James Melbourne, Cyril Roberto

Abstract: We investigate the Rényi entropy of independent sums of integer valued random variables through Fourier theoretic means, and give sharp comparisons between the variance and the Rényi entropy, for Poisson-Bernoulli variables. As applications we prove that a discrete ``min-entropy power'' is super additive on independent variables up to a universal constant, and give new bounds on an entropic genera… ▽ More We investigate the Rényi entropy of independent sums of integer valued random variables through Fourier theoretic means, and give sharp comparisons between the variance and the Rényi entropy, for Poisson-Bernoulli variables. As applications we prove that a discrete ``min-entropy power'' is super additive on independent variables up to a universal constant, and give new bounds on an entropic generalization of the Littlewood-Offord problem that are sharp in the ``Poisson regime''. △ Less

Submitted 15 March, 2024; v1 submitted 1 March, 2021; originally announced March 2021.

Journal ref: Bernoulli, Vol. 29, No. 2, pp. 1578--1599, May 2023

Conditional Rényi entropy and the relationships between Rényi capacities

Authors: Gautam Aishwarya, Mokshay Madiman

Abstract: The analogues of Arimoto's definition of conditional Rényi entropy and Rényi mutual information are explored for abstract alphabets. These quantities, although dependent on the reference measure, have some useful properties similar to those known in the discrete setting. In addition to laying out some such basic properties and the relations to Rényi divergences, the relationships between the famil… ▽ More The analogues of Arimoto's definition of conditional Rényi entropy and Rényi mutual information are explored for abstract alphabets. These quantities, although dependent on the reference measure, have some useful properties similar to those known in the discrete setting. In addition to laying out some such basic properties and the relations to Rényi divergences, the relationships between the families of mutual informations defined by Sibson, Augustin-Csiszár, and Lapidoth-Pfister, as well as the corresponding capacities, are explored. △ Less

Submitted 2 May, 2020; originally announced May 2020.

Comments: 17 pages, 1 figure

Journal ref: Entropy (Special issue on "Information Measures with Applications" edited by A. Lapidoth), vol. 22, no. 5, 2020

Two remarks on generalized entropy power inequalities

Authors: Mokshay Madiman, Piotr Nayar, Tomasz Tkocz

Abstract: This note contributes to the understanding of generalized entropy power inequalities. Our main goal is to construct a counter-example regarding monotonicity and entropy comparison of weighted sums of independent identically distributed log-concave random variables. We also present a complex analogue of a recent dependent entropy power inequality of Hao and Jog, and give a very simple proof. This note contributes to the understanding of generalized entropy power inequalities. Our main goal is to construct a counter-example regarding monotonicity and entropy comparison of weighted sums of independent identically distributed log-concave random variables. We also present a complex analogue of a recent dependent entropy power inequality of Hao and Jog, and give a very simple proof. △ Less

Submitted 3 April, 2019; originally announced April 2019.

Comments: arXiv:1811.00345 is split into 2 papers, with this being one

MSC Class: 94A17

Journal ref: Geometric Aspects of Functional Analysis: Israel Seminar (GAFA) 2017-2019 Volume II, B. Klartag and E. Milman (eds.), Lecture Notes in Mathematics, vol. 2266, Springer, 2020

Sharp moment-entropy inequalities and capacity bounds for log-concave distributions

Authors: Mokshay Madiman, Piotr Nayar, Tomasz Tkocz

Abstract: We show that the uniform distribution minimizes entropy among all one-dimensional symmetric log-concave distributions with fixed variance, as well as various generalizations of this fact to Rényi entropies of orders less than 1 and with moment constraints involving $p$-th absolute moments with $p\leq 2$. As consequences, we give new capacity bounds for additive noise channels with symmetric log-co… ▽ More We show that the uniform distribution minimizes entropy among all one-dimensional symmetric log-concave distributions with fixed variance, as well as various generalizations of this fact to Rényi entropies of orders less than 1 and with moment constraints involving $p$-th absolute moments with $p\leq 2$. As consequences, we give new capacity bounds for additive noise channels with symmetric log-concave noises, as well as for timing channels involving positive signal and noise where the noise has a decreasing log-concave density. In particular, we show that the capacity of an additive noise channel with symmetric, log-concave noise under an average power constraint is at most 0.254 bits per channel use greater than the capacity of an additive Gaussian noise channel with the same noise power. Consequences for reverse entropy power inequalities and connections to the slicing problem in convex geometry are also discussed. △ Less

Submitted 12 October, 2020; v1 submitted 1 November, 2018; originally announced November 2018.

Comments: the original paper split into two: this one and arXiv:1904.02314; additional results added to this part

MSC Class: 94A17 (Primary); 60E15 (Secondary)

Journal ref: IEEE Trans. Inform. Theory, vol. 67, no. 1, pp. 81-94, 2021

An Exact Upper Bound on the $L^p$ Lebesgue Constant and The $\infty$-Rényi Entropy Power Inequality for Integer Valued Random Variables

Authors: Peng Xu, Mokshay Madiman, James Melbourne

Abstract: In this paper, we proved an exact asymptotically sharp upper bound of the $L^p$ Lebesgue Constant (i.e. the $L^p$ norm of Dirichlet kernel) for $p\ge 2$. As an application, we also verified the implication of a new $\infty $-Rényi entropy power inequality for integer valued random variables. In this paper, we proved an exact asymptotically sharp upper bound of the $L^p$ Lebesgue Constant (i.e. the $L^p$ norm of Dirichlet kernel) for $p\ge 2$. As an application, we also verified the implication of a new $\infty $-Rényi entropy power inequality for integer valued random variables. △ Less

Submitted 23 August, 2018; originally announced August 2018.

Comments: 13 pages, 2 figures

MSC Class: 26D15 (Primary) 42A16; 46B99; 60B99 (Secondary)

The Differential Entropy of Mixtures: New Bounds and Applications

Authors: James Melbourne, Saurav Talukdar, Shreyas Bhaban, Mokshay Madiman, Murti V. Salapaka

Abstract: Mixture distributions are extensively used as a modeling tool in diverse areas from machine learning to communications engineering to physics, and obtaining bounds on the entropy of probability distributions is of fundamental importance in many of these applications. This article provides sharp bounds on the entropy concavity deficit, which is the difference between the entropy of the mixture and… ▽ More Mixture distributions are extensively used as a modeling tool in diverse areas from machine learning to communications engineering to physics, and obtaining bounds on the entropy of probability distributions is of fundamental importance in many of these applications. This article provides sharp bounds on the entropy concavity deficit, which is the difference between the entropy of the mixture and the weighted sum of entropies of constituent components. Toward establishing lower and upper bounds on the concavity deficit, results that are of importance in their own right are obtained. In order to obtain nontrivial upper bounds, properties of the skew-divergence are developed and notions of "skew" $f$-divergences are introduced; a reverse Pinsker inequality and a bound on Jensen-Shannon divergence are obtained along the way. Complementary lower bounds are derived with special attention paid to the case that corresponds to independent summation of a continuous and a discrete random variable. Several applications of the bounds are delineated, including to mutual information of additive noise channels, thermodynamics of computation, and functional inequalities. △ Less

Submitted 22 April, 2020; v1 submitted 29 May, 2018; originally announced May 2018.

Journal ref: IEEE Transactions on Information Theory, vol. 68, no. 4, pp. 2123-2146, April 2022

Majorization and Rényi Entropy Inequalities via Sperner Theory

Authors: Mokshay Madiman, Liyao Wang, Jae Oh Woo

Abstract: A natural link between the notions of majorization and strongly Sperner posets is elucidated. It is then used to obtain a variety of consequences, including new Rényi entropy inequalities for sums of independent, integer-valued random variables. A natural link between the notions of majorization and strongly Sperner posets is elucidated. It is then used to obtain a variety of consequences, including new Rényi entropy inequalities for sums of independent, integer-valued random variables. △ Less

Submitted 13 November, 2018; v1 submitted 4 December, 2017; originally announced December 2017.

Comments: Introduction was completely rewritten and there are numerous corrections. Expansion of background on Sperner theory, and several references are added

Journal ref: Discrete Mathematics (AEGT 2017 Special issue edited by S. Cioaba, R. Coulter, E. Fiorini, Q. Xiang, F. Pfender), vol. 342, no. 10, pp. 2911-2923, October 2019

Entropy Inequalities for Sums in Prime Cyclic Groups

Authors: Mokshay Madiman, Liyao Wang, Jae Oh Woo

Abstract: Lower bounds for the Rényi entropies of sums of independent random variables taking values in cyclic groups of prime order under permutations are established. The main ingredients of our approach are extended rearrangement inequalities in prime cyclic groups building on Lev (2001), and notions of stochastic ordering. Several applications are developed, including to discrete entropy power inequalit… ▽ More Lower bounds for the Rényi entropies of sums of independent random variables taking values in cyclic groups of prime order under permutations are established. The main ingredients of our approach are extended rearrangement inequalities in prime cyclic groups building on Lev (2001), and notions of stochastic ordering. Several applications are developed, including to discrete entropy power inequalities, the Littlewood-Offord problem, and counting solutions of certain linear systems. △ Less

Submitted 26 November, 2020; v1 submitted 2 October, 2017; originally announced October 2017.

Comments: 25 pages

Journal ref: SIAM J. Discrete Math., 35(3), pp. 1628-1649, 2021

Combinatorial Entropy Power Inequalities: A Preliminary Study of the Stam region

Authors: Mokshay Madiman, Farhad Ghassemi

Abstract: We initiate the study of the Stam region, defined as the subset of the positive orthant in $\mathbb{R}^{2^n-1}$ that arises from considering entropy powers of subset sums of $n$ independent random vectors in a Euclidean space of finite dimension. We show that the class of fractionally superadditive set functions provides an outer bound to the Stam region, resolving a conjecture of A. R. Barron and… ▽ More We initiate the study of the Stam region, defined as the subset of the positive orthant in $\mathbb{R}^{2^n-1}$ that arises from considering entropy powers of subset sums of $n$ independent random vectors in a Euclidean space of finite dimension. We show that the class of fractionally superadditive set functions provides an outer bound to the Stam region, resolving a conjecture of A. R. Barron and the first author. On the other hand, the entropy power of a sum of independent random vectors is not supermodular in any dimension. We also develop some qualitative properties of the Stam region, showing for instance that its closure is a logarithmically convex cone. △ Less

Submitted 29 June, 2018; v1 submitted 4 April, 2017; originally announced April 2017.

Comments: 16 pages

Journal ref: IEEE Transactions on Information Theory, vol. 65, no. 3, pp. 1375-1386, March 2019

The norm of the Fourier transform on compact or discrete abelian groups

Authors: Mokshay Madiman, Peng Xu

Abstract: We calculate the norm of the Fourier operator from $L^p(X)$ to $L^q(\hat{X})$ when $X$ is an infinite locally compact abelian group that is, furthermore, compact or discrete. This subsumes the sharp Hausdorff-Young inequality on such groups. In particular, we identify the region in $(p,q)$-space where the norm is infinite, generalizing a result of Fournier, and setting up a contrast with the case… ▽ More We calculate the norm of the Fourier operator from $L^p(X)$ to $L^q(\hat{X})$ when $X$ is an infinite locally compact abelian group that is, furthermore, compact or discrete. This subsumes the sharp Hausdorff-Young inequality on such groups. In particular, we identify the region in $(p,q)$-space where the norm is infinite, generalizing a result of Fournier, and setting up a contrast with the case of finite abelian groups, where the norm was determined by Gilbert and Rzeszotnik. As an application, uncertainty principles on such groups expressed in terms of Rényi entropies are discussed. △ Less

Submitted 14 November, 2016; originally announced November 2016.

Comments: 36 pages

Journal ref: Journal of Fourier Analysis and Applications, vol. 26, article no. 37, 2020

Forward and Reverse Entropy Power Inequalities in Convex Geometry

Authors: Mokshay Madiman, James Melbourne, Peng Xu

Abstract: The entropy power inequality, which plays a fundamental role in information theory and probability, may be seen as an analogue of the Brunn-Minkowski inequality. Motivated by this connection to Convex Geometry, we survey various recent developments on forward and reverse entropy power inequalities not just for the Shannon-Boltzmann entropy but also more generally for Rényi entropy. In the process,… ▽ More The entropy power inequality, which plays a fundamental role in information theory and probability, may be seen as an analogue of the Brunn-Minkowski inequality. Motivated by this connection to Convex Geometry, we survey various recent developments on forward and reverse entropy power inequalities not just for the Shannon-Boltzmann entropy but also more generally for Rényi entropy. In the process, we discuss connections between the so-called functional (or integral) and probabilistic (or entropic) analogues of some classical inequalities in geometric functional analysis △ Less

Submitted 24 December, 2016; v1 submitted 14 April, 2016; originally announced April 2016.

Comments: 54 pages. Changes in v2: improved organization, cleaned up exposition, and numerous references added

Journal ref: Book chapter in "Convexity and Concentration", E. Carlen, M. Madiman, E. Werner (eds.), The IMA Volumes in Mathematics and its Applications, Vol. 161, pp. 427-485, Springer, 2017

Entropies of weighted sums in cyclic groups and an application to polar codes

Authors: Emmanuel Abbe, Jiange Li, Mokshay Madiman

Abstract: In this note, the following basic question is explored: in a cyclic group, how are the Shannon entropies of the sum and difference of i.i.d. random variables related to each other? For the integer group, we show that they can differ by any real number additively, but not too much multiplicatively; on the other hand, for $\mathbb{Z}/3\mathbb{Z}$, the entropy of the difference is always at least as… ▽ More In this note, the following basic question is explored: in a cyclic group, how are the Shannon entropies of the sum and difference of i.i.d. random variables related to each other? For the integer group, we show that they can differ by any real number additively, but not too much multiplicatively; on the other hand, for $\mathbb{Z}/3\mathbb{Z}$, the entropy of the difference is always at least as large as that of the sum. These results are closely related to the study of more-sum-than-difference (i.e. MSTD) sets in additive combinatorics. We also investigate polar codes for $q$-ary input channels using non-canonical kernels to construct the generator matrix, and present applications of our results to constructing polar codes with significantly improved error probability compared to the canonical construction. △ Less

Submitted 21 April, 2016; v1 submitted 30 November, 2015; originally announced December 2015.

Journal ref: Entropy, 19(9), 235, 2017

Optimal Concentration of Information Content For Log-Concave Densities

Authors: Matthieu Fradelizi, Mokshay Madiman, Liyao Wang

Abstract: An elementary proof is provided of sharp bounds for the varentropy of random vectors with log-concave densities, as well as for deviations of the information content from its mean. These bounds significantly improve on the bounds obtained by Bobkov and Madiman ({\it Ann. Probab.}, 39(4):1528--1543, 2011). An elementary proof is provided of sharp bounds for the varentropy of random vectors with log-concave densities, as well as for deviations of the information content from its mean. These bounds significantly improve on the bounds obtained by Bobkov and Madiman ({\it Ann. Probab.}, 39(4):1528--1543, 2011). △ Less

Submitted 26 October, 2015; v1 submitted 17 August, 2015; originally announced August 2015.

Comments: 15 pages. Changes in v2: Remark 2.5 (due to C. Saroglou) added with more general sufficient conditions for equality in Theorem 2.3. Also some minor corrections and added references

Journal ref: C. Houdré et al. (eds.), High Dimensional Probability VII, Progress in Probability 71, Springer, 2016

Entropy bounds on abelian groups and the Ruzsa divergence

Authors: Mokshay Madiman, Ioannis Kontoyiannis

Abstract: Over the past few years, a family of interesting new inequalities for the entropies of sums and differences of random variables has been developed by Ruzsa, Tao and others, motivated by analogous results in additive combinatorics. The present work extends these earlier results to the case of random variables taking values in $\mathbb{R}^n$ or, more generally, in arbitrary locally compact and Polis… ▽ More Over the past few years, a family of interesting new inequalities for the entropies of sums and differences of random variables has been developed by Ruzsa, Tao and others, motivated by analogous results in additive combinatorics. The present work extends these earlier results to the case of random variables taking values in $\mathbb{R}^n$ or, more generally, in arbitrary locally compact and Polish abelian groups. We isolate and study a key quantity, the Ruzsa divergence between two probability distributions, and we show that its properties can be used to extend the earlier inequalities to the present general setting. The new results established include several variations on the theme that the entropies of the sum and the difference of two independent random variables severely constrain each other. Although the setting is quite general, the result are already of interest (and new) for random vectors in $\mathbb{R}^n$. In that special case, quantitative bounds are provided for the stability of the equality conditions in the entropy power inequality; a reverse entropy power inequality for log-concave random vectors is proved; an information-theoretic analog of the Rogers-Shephard inequality for convex bodies is established; and it is observed that some of these results lead to new inequalities for the determinants of positive-definite matrices. Moreover, by considering the multiplicative subgroups of the complex plane, one obtains new inequalities for the differential entropies of products and ratios of nonzero, complex-valued random variables. △ Less

Submitted 26 October, 2015; v1 submitted 17 August, 2015; originally announced August 2015.

Comments: 26 pages. Changes in v2: Added Section V on entropies of products of random variables, corrected several typos, added some references

Journal ref: IEEE Transactions on Information Theory, vol. 64, no. 1, pp. 77-92, January 2018

Beyond the entropy power inequality, via rearrangements

Authors: Liyao Wang, Mokshay Madiman

Abstract: A lower bound on the Rényi differential entropy of a sum of independent random vectors is demonstrated in terms of rearrangements. For the special case of Boltzmann-Shannon entropy, this lower bound is better than that given by the entropy power inequality. Several applications are discussed, including a new proof of the classical entropy power inequality and an entropy inequality involving symmet… ▽ More A lower bound on the Rényi differential entropy of a sum of independent random vectors is demonstrated in terms of rearrangements. For the special case of Boltzmann-Shannon entropy, this lower bound is better than that given by the entropy power inequality. Several applications are discussed, including a new proof of the classical entropy power inequality and an entropy inequality involving symmetrization of Lévy processes. △ Less

Submitted 19 May, 2014; v1 submitted 23 July, 2013; originally announced July 2013.

Comments: 32 pages. v2: Several minor edits + Section X added

Journal ref: IEEE Transactions on Information Theory, vol. 60, no. 9, pp. 5116-5137, September 2014

Sumset and Inverse Sumset Inequalities for Differential Entropy and Mutual Information

Authors: Ioannis Kontoyiannis, Mokshay Madiman

Abstract: The sumset and inverse sumset theories of Freiman, Plünnecke and Ruzsa, give bounds connecting the cardinality of the sumset $A+B=\{a+b\;;\;a\in A,\,b\in B\}$ of two discrete sets $A,B$, to the cardinalities (or the finer structure) of the original sets $A,B$. For example, the sum-difference bound of Ruzsa states that, $|A+B|\,|A|\,|B|\leq|A-B|^3$, where the difference set… ▽ More The sumset and inverse sumset theories of Freiman, Plünnecke and Ruzsa, give bounds connecting the cardinality of the sumset $A+B=\{a+b\;;\;a\in A,\,b\in B\}$ of two discrete sets $A,B$, to the cardinalities (or the finer structure) of the original sets $A,B$. For example, the sum-difference bound of Ruzsa states that, $|A+B|\,|A|\,|B|\leq|A-B|^3$, where the difference set $A-B= \{a-b\;;\;a\in A,\,b\in B\}$. Interpreting the differential entropy $h(X)$ of a continuous random variable $X$ as (the logarithm of) the size of the effective support of $X$, the main contribution of this paper is a series of natural information-theoretic analogs for these results. For example, the Ruzsa sum-difference bound becomes the new inequality, $h(X+Y)+h(X)+h(Y)\leq 3h(X-Y)$, for any pair of independent continuous random variables $X$ and $Y$. Our results include differential-entropy versions of Ruzsa's triangle inequality, the Plünnecke-Ruzsa inequality, and the Balog-Szemerédi-Gowers lemma. Also we give a differential entropy version of the Freiman-Green-Ruzsa inverse-sumset theorem, which can be seen as a quantitative converse to the entropy power inequality. Versions of most of these results for the discrete entropy $H(X)$ were recently proved by Tao, relying heavily on a strong, functional form of the submodularity property of $H(X)$. Since differential entropy is {\em not} functionally submodular, in the continuous case many of the corresponding discrete proofs fail, in many cases requiring substantially new proof strategies. We find that the basic property that naturally replaces the discrete functional submodularity, is the data processing property of mutual information. △ Less

Submitted 3 June, 2012; originally announced June 2012.

Comments: 23 pages

Journal ref: IEEE Transactions on Information Theory, vol. 60, no. 8, pp. 4503-4514, August 2014

On the problem of reversibility of the entropy power inequality

Authors: Sergey G. Bobkov, Mokshay M. Madiman

Abstract: As was shown recently by the authors, the entropy power inequality can be reversed for independent summands with sufficiently concave densities, when the distributions of the summands are put in a special position. In this note it is proved that reversibility is impossible over the whole class of convex probability distributions. Related phenomena for identically distributed summands are also disc… ▽ More As was shown recently by the authors, the entropy power inequality can be reversed for independent summands with sufficiently concave densities, when the distributions of the summands are put in a special position. In this note it is proved that reversibility is impossible over the whole class of convex probability distributions. Related phenomena for identically distributed summands are also discussed. △ Less

Submitted 29 November, 2011; originally announced November 2011.

Comments: 13 pages

Journal ref: "Limit Theorems in Probability, Statistics and Number Theory (in honor of Friedrich Götze)", P. Eichelsbacher et al. (ed.), Springer Proceedings in Mathematics and Statistics 42, pp. 61-74, Springer-Verlag, 2013

Dimensional behaviour of entropy and information

Authors: Sergey Bobkov, Mokshay Madiman

Abstract: We develop an information-theoretic perspective on some questions in convex geometry, providing for instance a new equipartition property for log-concave probability measures, some Gaussian comparison results for log-concave measures, an entropic formulation of the hyperplane conjecture, and a new reverse entropy power inequality for log-concave measures analogous to V. Milman's reverse Brunn-Mink… ▽ More We develop an information-theoretic perspective on some questions in convex geometry, providing for instance a new equipartition property for log-concave probability measures, some Gaussian comparison results for log-concave measures, an entropic formulation of the hyperplane conjecture, and a new reverse entropy power inequality for log-concave measures analogous to V. Milman's reverse Brunn-Minkowski inequality. △ Less

Submitted 17 January, 2011; originally announced January 2011.

Comments: 6 pages

Journal ref: C. R. Acad. Sci. Paris, Ser. I 349 (2011), pp. 201--204

Fractional generalizations of Young and Brunn-Minkowski inequalities

Authors: Sergey Bobkov, Mokshay Madiman, Liyao Wang

Abstract: A generalization of Young's inequality for convolution with sharp constant is conjectured for scenarios where more than two functions are being convolved, and it is proven for certain parameter ranges. The conjecture would provide a unified proof of recent entropy power inequalities of Barron and Madiman, as well as of a (conjectured) generalization of the Brunn-Minkowski inequality. It is shown t… ▽ More A generalization of Young's inequality for convolution with sharp constant is conjectured for scenarios where more than two functions are being convolved, and it is proven for certain parameter ranges. The conjecture would provide a unified proof of recent entropy power inequalities of Barron and Madiman, as well as of a (conjectured) generalization of the Brunn-Minkowski inequality. It is shown that the generalized Brunn-Minkowski conjecture is true for convex sets; an application of this to the law of large numbers for random sets is described. △ Less

Submitted 1 January, 2011; v1 submitted 15 June, 2010; originally announced June 2010.

Comments: 19 pages, numerous typos corrected, exposition improved, and references added, but no other substantial changes

Journal ref: Contemporary Mathematics 545, American Mathematical Society, 2011, pp. 35-53

The entropy per coordinate of a random vector is highly constrained under convexity conditions

Authors: Sergey Bobkov, Mokshay Madiman

Abstract: The entropy per coordinate in a log-concave random vector of any dimension with given density at the mode is shown to have a range of just 1. Uniform distributions on convex bodies are at the lower end of this range, the distribution with i.i.d. exponentially distributed coordinates is at the upper end, and the normal is exactly in the middle. Thus in terms of the amount of randomness as measured… ▽ More The entropy per coordinate in a log-concave random vector of any dimension with given density at the mode is shown to have a range of just 1. Uniform distributions on convex bodies are at the lower end of this range, the distribution with i.i.d. exponentially distributed coordinates is at the upper end, and the normal is exactly in the middle. Thus in terms of the amount of randomness as measured by entropy per coordinate, any log-concave random vector of any dimension contains randomness that differs from that in the normal random variable with the same maximal density value by at most 1/2. As applications, we obtain an information-theoretic formulation of the famous hyperplane conjecture in convex geometry, entropy bounds for certain infinitely divisible distributions, and quantitative estimates for the behavior of the density at the mode on convolution. More generally, one may consider so-called convex or hyperbolic probability measures on Euclidean spaces; we give new constraints on entropy per coordinate for this class of measures, which generalize our results under the log-concavity assumption, expose the extremal role of multivariate Pareto-type distributions, and give some applications. △ Less

Submitted 24 December, 2010; v1 submitted 15 June, 2010; originally announced June 2010.

Comments: 15 pages, revised for IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, Vol. 57, no. 8, pp. 4940-4954, August 2011

Compound Poisson Approximation via Information Functionals

Authors: A. D. Barbour, Oliver Johnson, Ioannis Kontoyiannis, Mokshay Madiman

Abstract: An information-theoretic development is given for the problem of compound Poisson approximation, which parallels earlier treatments for Gaussian and Poisson approximation. Let $P_{S_n}$ be the distribution of a sum $S_n=\Sumn Y_i$ of independent integer-valued random variables $Y_i$. Nonasymptotic bounds are derived for the distance between $P_{S_n}$ and an appropriately chosen compound Poisson la… ▽ More An information-theoretic development is given for the problem of compound Poisson approximation, which parallels earlier treatments for Gaussian and Poisson approximation. Let $P_{S_n}$ be the distribution of a sum $S_n=\Sumn Y_i$ of independent integer-valued random variables $Y_i$. Nonasymptotic bounds are derived for the distance between $P_{S_n}$ and an appropriately chosen compound Poisson law. In the case where all $Y_i$ have the same conditional distribution given $\{Y_i\neq 0\}$, a bound on the relative entropy distance between $P_{S_n}$ and the compound Poisson distribution is derived, based on the data-processing property of relative entropy and earlier Poisson approximation results. When the $Y_i$ have arbitrary distributions, corresponding bounds are derived in terms of the total variation distance. The main technical ingredient is the introduction of two "information functionals," and the analysis of their properties. These information functionals play a role analogous to that of the classical Fisher information in normal approximation. Detailed comparisons are made between the resulting inequalities and related bounds. △ Less

Submitted 21 April, 2010; originally announced April 2010.

Comments: 27 pages

Journal ref: Electronic Journal of Probability, Vol 15, Paper no. 42, pages 1344-1369, 2010

Log-concavity, ultra-log-concavity, and a maximum entropy property of discrete compound Poisson measures

Authors: Oliver Johnson, Ioannis Kontoyiannis, Mokshay Madiman

Abstract: Sufficient conditions are developed, under which the compound Poisson distribution has maximal entropy within a natural class of probability measures on the nonnegative integers. Recently, one of the authors [O. Johnson, {\em Stoch. Proc. Appl.}, 2007] used a semigroup approach to show that the Poisson has maximal entropy among all ultra-log-concave distributions with fixed mean. We show via a non… ▽ More Sufficient conditions are developed, under which the compound Poisson distribution has maximal entropy within a natural class of probability measures on the nonnegative integers. Recently, one of the authors [O. Johnson, {\em Stoch. Proc. Appl.}, 2007] used a semigroup approach to show that the Poisson has maximal entropy among all ultra-log-concave distributions with fixed mean. We show via a non-trivial extension of this semigroup approach that the natural analog of the Poisson maximum entropy property remains valid if the compound Poisson distributions under consideration are log-concave, but that it fails in general. A parallel maximum entropy result is established for the family of compound binomial measures. Sufficient conditions for compound distributions to be log-concave are discussed and applications to combinatorics are examined; new bounds are derived on the entropy of the cardinality of a random independent set in a claw-free graph, and a connection is drawn to Mason's conjecture for matroids. The present results are primarily motivated by the desire to provide an information-theoretic foundation for compound Poisson approximation and associated limit theorems, analogous to the corresponding developments for the central limit theorem and for Poisson approximation. Our results also demonstrate new links between some probabilistic methods and the combinatorial notions of log-concavity and ultra-log-concavity, and they add to the growing body of work exploring the applications of maximum entropy characterizations to problems in discrete mathematics. △ Less

Submitted 27 September, 2011; v1 submitted 3 December, 2009; originally announced December 2009.

Comments: 30 pages. This submission supersedes arXiv:0805.4112v1. Changes in v2: Updated references, typos corrected

MSC Class: 94A17; 60E07; 60E15

Journal ref: Discrete Applied Mathematics, vol 161/9, pages 1232-1250, 2013

Cores of Cooperative Games in Information Theory

Authors: Mokshay Madiman

Abstract: Cores of cooperative games are ubiquitous in information theory, and arise most frequently in the characterization of fundamental limits in various scenarios involving multiple users. Examples include classical settings in network information theory such as Slepian-Wolf source coding and multiple access channels, classical settings in statistics such as robust hypothesis testing, and new setting… ▽ More Cores of cooperative games are ubiquitous in information theory, and arise most frequently in the characterization of fundamental limits in various scenarios involving multiple users. Examples include classical settings in network information theory such as Slepian-Wolf source coding and multiple access channels, classical settings in statistics such as robust hypothesis testing, and new settings at the intersection of networking and statistics such as distributed estimation problems for sensor networks. Cooperative game theory allows one to understand aspects of all of these problems from a fresh and unifying perspective that treats users as players in a game, sometimes leading to new insights. At the heart of these analyses are fundamental dualities that have been long studied in the context of cooperative games; for information theoretic purposes, these are dualities between information inequalities on the one hand and properties of rate, capacity or other resource allocation regions on the other. △ Less

Submitted 31 December, 2008; originally announced January 2009.

Comments: 12 pages, published at http://www.hindawi.com/GetArticle.aspx?doi=10.1155/2008/318704 in EURASIP Journal on Wireless Communications and Networking, Special Issue on "Theory and Applications in Multiuser/Multiterminal Communications", April 2008

Journal ref: EURASIP Journal on Wireless Communications and Networking, Volume 2008, Article ID 318704

Entropy and set cardinality inequalities for partition-determined functions

Authors: Mokshay Madiman, Adam Marcus, Prasad Tetali

Abstract: A new notion of partition-determined functions is introduced, and several basic inequalities are developed for the entropy of such functions of independent random variables, as well as for cardinalities of compound sets obtained using these functions. Here a compound set means a set obtained by varying each argument of a function of several variables over a set associated with that argument, where… ▽ More A new notion of partition-determined functions is introduced, and several basic inequalities are developed for the entropy of such functions of independent random variables, as well as for cardinalities of compound sets obtained using these functions. Here a compound set means a set obtained by varying each argument of a function of several variables over a set associated with that argument, where all the sets are subsets of an appropriate algebraic structure so that the function is well defined. On the one hand, the entropy inequalities developed for partition-determined functions imply entropic analogues of general inequalities of Plünnecke-Ruzsa type. On the other hand, the cardinality inequalities developed for compound sets imply several inequalities for sumsets, including for instance a generalization of inequalities proved by Gyarmati, Matolcsi and Ruzsa (2010). We also provide partial progress towards a conjecture of Ruzsa (2007) for sumsets in nonabelian groups. All proofs are elementary and rely on properly develo** certain information-theoretic inequalities. △ Less

Submitted 9 August, 2011; v1 submitted 30 December, 2008; originally announced January 2009.

Comments: 26 pages. v2: Revised version incorporating referee feedback plus inclusion of some additional corollaries and discussion. v3: Final version with minor corrections. To appear in Random Structures and Algorithms

Journal ref: Random Structures and Algorithms, Vol. 40, pp. 399-424, 2012

Information Inequalities for Joint Distributions, with Interpretations and Applications

Authors: Mokshay Madiman, Prasad Tetali

Abstract: Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as inequalities of Han, Fujishige and Shearer. A duality between the upper and lower bounds for joint entropy is developed. All of these results are shown to be special c… ▽ More Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as inequalities of Han, Fujishige and Shearer. A duality between the upper and lower bounds for joint entropy is developed. All of these results are shown to be special cases of general, new results for submodular functions-- thus, the inequalities presented constitute a richly structured class of Shannon-type inequalities. The new inequalities are applied to obtain new results in combinatorics, such as bounds on the number of independent sets in an arbitrary graph and the number of zero-error source-channel codes, as well as new determinantal inequalities in matrix theory. A new inequality for relative entropies is also developed, along with interpretations in terms of hypothesis testing. Finally, revealing connections of the results to literature in economics, computer science, and physics are explored. △ Less

Submitted 3 August, 2009; v1 submitted 30 December, 2008; originally announced January 2009.

Comments: 15 pages, 1 figure. Originally submitted to the IEEE Transactions on Information Theory in May 2007, the current version incorporates reviewer comments including elimination of an error

Journal ref: IEEE Transactions on Information Theory, Vol. 56(6), pp. 2699-2713, June 2010

On the entropy and log-concavity of compound Poisson measures

Authors: Oliver Johnson, Ioannis Kontoyiannis, Mokshay Madiman

Abstract: Motivated, in part, by the desire to develop an information-theoretic foundation for compound Poisson approximation limit theorems (analogous to the corresponding developments for the central limit theorem and for simple Poisson approximation), this work examines sufficient conditions under which the compound Poisson distribution has maximal entropy within a natural class of probability measures… ▽ More Motivated, in part, by the desire to develop an information-theoretic foundation for compound Poisson approximation limit theorems (analogous to the corresponding developments for the central limit theorem and for simple Poisson approximation), this work examines sufficient conditions under which the compound Poisson distribution has maximal entropy within a natural class of probability measures on the nonnegative integers. We show that the natural analog of the Poisson maximum entropy property remains valid if the measures under consideration are log-concave, but that it fails in general. A parallel maximum entropy result is established for the family of compound binomial measures. The proofs are largely based on ideas related to the semigroup approach introduced in recent work by Johnson for the Poisson family. Sufficient conditions are given for compound distributions to be log-concave, and specific examples are presented illustrating all the above results. △ Less

Submitted 27 May, 2008; originally announced May 2008.

Report number: Superceded by arXiv:0912.0581 MSC Class: 62B10; 94A17

Generalized Entropy Power Inequalities and Monotonicity Properties of Information

Authors: Mokshay Madiman, Andrew Barron

Abstract: New families of Fisher information and entropy power inequalities for sums of independent random variables are presented. These inequalities relate the information in the sum of $n$ independent random variables to the information contained in sums over subsets of the random variables, for an arbitrary collection of subsets. As a consequence, a simple proof of the monotonicity of information in c… ▽ More New families of Fisher information and entropy power inequalities for sums of independent random variables are presented. These inequalities relate the information in the sum of $n$ independent random variables to the information contained in sums over subsets of the random variables, for an arbitrary collection of subsets. As a consequence, a simple proof of the monotonicity of information in central limit theorems is obtained, both in the setting of i.i.d. summands as well as in the more general setting of independent summands with variance-standardized sums. △ Less

Submitted 2 July, 2007; v1 submitted 10 May, 2006; originally announced May 2006.

Comments: 13 pages. Many minor modifications from first version, plus a section on refined results. This is almost but not exactly identical to the published version

ACM Class: E.4; G.3

Journal ref: IEEE Transactions on Information Theory, Vol. 53(7), pp. 2317-2329, July 2007