Showing 1–15 of 15 results for author: Melbourne, J

Causal Structure Recovery of Linear Dynamical Systems: An FFT based Approach

Authors: Mishfad Shaikh Veedu, James Melbourne, Murti V. Salapaka

Abstract: Learning causal effects from data is a fundamental and well-studied problem across science, especially when the cause-effect relationship is static in nature. However, causal effect is less explored when there are dynamical dependencies, i.e., when dependencies exist between entities across time. Identifying dynamic causal effects from time-series observations is computationally expensive when com… ▽ More Learning causal effects from data is a fundamental and well-studied problem across science, especially when the cause-effect relationship is static in nature. However, causal effect is less explored when there are dynamical dependencies, i.e., when dependencies exist between entities across time. Identifying dynamic causal effects from time-series observations is computationally expensive when compared to the static scenario. We demonstrate that the computational complexity of recovering the causation structure for the vector auto-regressive (VAR) model is $O(Tn^3N^2)$, where $n$ is the number of nodes, $T$ is the number of samples, and $N$ is the largest time-lag in the dependency between entities. We report a method, with a reduced complexity of $O(Tn^3 \log N)$, to recover the causation structure to obtain frequency-domain (FD) representations of time-series. Since FFT accumulates all the time dependencies on every frequency, causal inference can be performed efficiently by considering the state variables as random variables at any given frequency. We additionally show that, for systems with interactions that are LTI, do-calculus machinery can be realized in the FD resulting in versions of the classical single-door (with cycles), front and backdoor criteria. We demonstrate, for a large class of problems, graph reconstruction using multivariate Wiener projections results in a significant computational advantage with $O(n)$ complexity over reconstruction algorithms such as the PC algorithm which has $O(n^q)$ complexity, where $q$ is the maximum neighborhood size. This advantage accrues due to some remarkable properties of the phase response of the frequency-dependent Wiener coefficients which is not present in any time-domain approach. △ Less

Submitted 5 September, 2023; originally announced September 2023.

Comments: 34 pages

Minimum entropy of a log-concave variable for fixed variance

Authors: James Melbourne, Piotr Nayar, Cyril Roberto

Abstract: We show that for log-concave real random variables with fixed variance the Shannon differential entropy is minimized for an exponential random variable. We apply this result to derive upper bounds on capacities of additive noise channels with log-concave noise. We also improve constants in the reverse entropy power inequalities for log-concave random variables. We show that for log-concave real random variables with fixed variance the Shannon differential entropy is minimized for an exponential random variable. We apply this result to derive upper bounds on capacities of additive noise channels with log-concave noise. We also improve constants in the reverse entropy power inequalities for log-concave random variables. △ Less

Submitted 15 March, 2024; v1 submitted 4 September, 2023; originally announced September 2023.

Comments: A simpler proof of the "Three-point inequality'' is given

Moments, Concentration, and Entropy of Log-Concave Distributions

Authors: Arnaud Marsiglietti, James Melbourne

Abstract: We utilize and extend a simple and classical mechanism, combining log-concavity and majorization in the convex order to derive moments, concentration, and entropy inequalities for certain classes of log-concave distributions. We utilize and extend a simple and classical mechanism, combining log-concavity and majorization in the convex order to derive moments, concentration, and entropy inequalities for certain classes of log-concave distributions. △ Less

Submitted 1 September, 2022; v1 submitted 17 May, 2022; originally announced May 2022.

Comments: 19 pages

A discrete complement of Lyapunov's inequality and its information theoretic consequences

Authors: James Melbourne, Gerardo Palafox-Castillo

Abstract: We establish a reversal of Lyapunov's inequality for monotone log-concave sequences, settling a conjecture of Havrilla-Tkocz and Melbourne-Tkocz. A strengthened version of the same conjecture is disproved through counter example. We also derive several information theoretic inequalities as consequences. In particular sharp bounds are derived for the varentropy, Rényi entropies, and the concentrati… ▽ More We establish a reversal of Lyapunov's inequality for monotone log-concave sequences, settling a conjecture of Havrilla-Tkocz and Melbourne-Tkocz. A strengthened version of the same conjecture is disproved through counter example. We also derive several information theoretic inequalities as consequences. In particular sharp bounds are derived for the varentropy, Rényi entropies, and the concentration of information of monotone log-concave random variables. Moreover, the majorization approach utilized in the proof of the main theorem, is applied to derive analogous information theoretic results in the symmetric setting, where the Lyapunov reversal is known to fail. △ Less

Submitted 12 November, 2021; originally announced November 2021.

Comments: 16 pages

Quantitative form of Ball's Cube slicing in $\mathbb{R}^n$ and equality cases in the min-entropy power inequality

Authors: James Melbourne, Cyril Roberto

Abstract: We prove a quantitative form of the celebrated Ball's theorem on cube slicing in $\mathbb{R}^n$ and obtain, as a consequence, equality cases in the min-entropy power inequality. Independently, we also give a quantitative form of Khintchine's inequality in the special case $p=1$. We prove a quantitative form of the celebrated Ball's theorem on cube slicing in $\mathbb{R}^n$ and obtain, as a consequence, equality cases in the min-entropy power inequality. Independently, we also give a quantitative form of Khintchine's inequality in the special case $p=1$. △ Less

Submitted 8 September, 2021; originally announced September 2021.

Comments: 14 pages

Bernoulli sums and Rényi entropy inequalities

Authors: Mokshay Madiman, James Melbourne, Cyril Roberto

Abstract: We investigate the Rényi entropy of independent sums of integer valued random variables through Fourier theoretic means, and give sharp comparisons between the variance and the Rényi entropy, for Poisson-Bernoulli variables. As applications we prove that a discrete ``min-entropy power'' is super additive on independent variables up to a universal constant, and give new bounds on an entropic genera… ▽ More We investigate the Rényi entropy of independent sums of integer valued random variables through Fourier theoretic means, and give sharp comparisons between the variance and the Rényi entropy, for Poisson-Bernoulli variables. As applications we prove that a discrete ``min-entropy power'' is super additive on independent variables up to a universal constant, and give new bounds on an entropic generalization of the Littlewood-Offord problem that are sharp in the ``Poisson regime''. △ Less

Submitted 15 March, 2024; v1 submitted 1 March, 2021; originally announced March 2021.

Journal ref: Bernoulli, Vol. 29, No. 2, pp. 1578--1599, May 2023

Strongly Convex Divergences

Authors: James Melbourne

Abstract: We consider a sub-class of the $f$-divergences satisfying a stronger convexity property, which we refer to as strongly convex, or $κ$-convex divergences. We derive new and old relationships, based on convexity arguments, between popular $f$-divergences. We consider a sub-class of the $f$-divergences satisfying a stronger convexity property, which we refer to as strongly convex, or $κ$-convex divergences. We derive new and old relationships, based on convexity arguments, between popular $f$-divergences. △ Less

Submitted 22 September, 2020; originally announced September 2020.

Comments: Submitted to Entropy, special issue, Divergence Measures: Mathematical Foundations and Applications in Information-Theoretic and Statistical Problems, edited by Igal Sason

Convex Decreasing Algorithms: Distributed Synthesis and Finite-time Termination in Higher Dimension

Authors: James Melbourne, Govind Saraswat, Vivek Khatana, Sourav Patel, Murti V. Salapaka

Abstract: We introduce a general mathematical framework for distributed algorithms, and a monotonicity property frequently satisfied in application. These properties are leveraged to provide finite-time guarantees for converging algorithms, suited for use in the absence of a central authority. A central application is to consensus algorithms in higher dimension. These pursuits motivate a new peer to peer co… ▽ More We introduce a general mathematical framework for distributed algorithms, and a monotonicity property frequently satisfied in application. These properties are leveraged to provide finite-time guarantees for converging algorithms, suited for use in the absence of a central authority. A central application is to consensus algorithms in higher dimension. These pursuits motivate a new peer to peer convex hull algorithm which we demonstrate to be an instantiation of the described theory. To address the diversity of convex sets and the potential computation and communication costs of knowing such sets in high dimension, a lightweight norm based stop** criteria is developed. More explicitly, we give a distributed algorithm that terminates in finite time when applied to consensus problems in higher dimensions and guarantees the convergence of the consensus algorithm in norm, within any given tolerance. Applications to consensus least squared estimation and distributed function determination are developed. The practical utility of the algorithm is illustrated through MATLAB simulations. △ Less

Submitted 30 July, 2020; v1 submitted 26 July, 2020; originally announced July 2020.

Reversals of Rényi Entropy Inequalities under Log-Concavity

Authors: James Melbourne, Tomasz Tkocz

Abstract: We establish a discrete analog of the Rényi entropy comparison due to Bobkov and Madiman. For log-concave variables on the integers, the min entropy is within log e of the usual Shannon entropy. Additionally we investigate the entropic Rogers-Shephard inequality studied by Madiman and Kontoyannis, and establish a sharp Rényi version for certain parameters in both the continuous and discrete cases We establish a discrete analog of the Rényi entropy comparison due to Bobkov and Madiman. For log-concave variables on the integers, the min entropy is within log e of the usual Shannon entropy. Additionally we investigate the entropic Rogers-Shephard inequality studied by Madiman and Kontoyannis, and establish a sharp Rényi version for certain parameters in both the continuous and discrete cases △ Less

Submitted 21 May, 2020; originally announced May 2020.

Journal ref: IEEE Trans. Inform. Theory 67 (2021), no. 1, 45-51

An Exact Upper Bound on the $L^p$ Lebesgue Constant and The $\infty$-Rényi Entropy Power Inequality for Integer Valued Random Variables

Authors: Peng Xu, Mokshay Madiman, James Melbourne

Abstract: In this paper, we proved an exact asymptotically sharp upper bound of the $L^p$ Lebesgue Constant (i.e. the $L^p$ norm of Dirichlet kernel) for $p\ge 2$. As an application, we also verified the implication of a new $\infty $-Rényi entropy power inequality for integer valued random variables. In this paper, we proved an exact asymptotically sharp upper bound of the $L^p$ Lebesgue Constant (i.e. the $L^p$ norm of Dirichlet kernel) for $p\ge 2$. As an application, we also verified the implication of a new $\infty $-Rényi entropy power inequality for integer valued random variables. △ Less

Submitted 23 August, 2018; originally announced August 2018.

Comments: 13 pages, 2 figures

MSC Class: 26D15 (Primary) 42A16; 46B99; 60B99 (Secondary)

Rearrangement and Prekopa-Leindler type inequalities

Authors: James Melbourne

Abstract: We investigate the interactions of functional rearrangements with Prekopa-Leindler type inequalities. It is shown that that a general class of integral inequalities tighten on rearrangement to "isoperimetric" sets with respect to a relevant measure. Applications to the Borell-Brascamp-Lieb, Borell-Ehrhart, and the recent polar Prekopa-Leindler inequalities are demonstrated. It is also proven that… ▽ More We investigate the interactions of functional rearrangements with Prekopa-Leindler type inequalities. It is shown that that a general class of integral inequalities tighten on rearrangement to "isoperimetric" sets with respect to a relevant measure. Applications to the Borell-Brascamp-Lieb, Borell-Ehrhart, and the recent polar Prekopa-Leindler inequalities are demonstrated. It is also proven that an integrated form of the Gaussian log-Sobolev inequality decreases on half-space rearrangement. △ Less

Submitted 22 May, 2019; v1 submitted 22 June, 2018; originally announced June 2018.

Comments: 22 pages

Error Bounds on a Mixed Entropy Inequality

Authors: James Melbourne, Saurav Talukdar, Shreyas Bhaban, Murti V. Salapaka

Abstract: Motivated by the entropy computations relevant to the evaluation of decrease in entropy in bit reset operations, the authors investigate the deficit in an entropic inequality involving two independent random variables, one continuous and the other discrete. In the case where the continuous random variable is Gaussian, we derive strong quantitative bounds on the deficit in the inequality. More expl… ▽ More Motivated by the entropy computations relevant to the evaluation of decrease in entropy in bit reset operations, the authors investigate the deficit in an entropic inequality involving two independent random variables, one continuous and the other discrete. In the case where the continuous random variable is Gaussian, we derive strong quantitative bounds on the deficit in the inequality. More explicitly it is shown that the decay of the deficit is sub-Gaussian with respect to the reciprocal of the standard deviation of the Gaussian variable. What is more, up to rational terms these results are shown to be sharp. △ Less

Submitted 29 May, 2018; originally announced May 2018.

The Differential Entropy of Mixtures: New Bounds and Applications

Authors: James Melbourne, Saurav Talukdar, Shreyas Bhaban, Mokshay Madiman, Murti V. Salapaka

Abstract: Mixture distributions are extensively used as a modeling tool in diverse areas from machine learning to communications engineering to physics, and obtaining bounds on the entropy of probability distributions is of fundamental importance in many of these applications. This article provides sharp bounds on the entropy concavity deficit, which is the difference between the entropy of the mixture and… ▽ More Mixture distributions are extensively used as a modeling tool in diverse areas from machine learning to communications engineering to physics, and obtaining bounds on the entropy of probability distributions is of fundamental importance in many of these applications. This article provides sharp bounds on the entropy concavity deficit, which is the difference between the entropy of the mixture and the weighted sum of entropies of constituent components. Toward establishing lower and upper bounds on the concavity deficit, results that are of importance in their own right are obtained. In order to obtain nontrivial upper bounds, properties of the skew-divergence are developed and notions of "skew" $f$-divergences are introduced; a reverse Pinsker inequality and a bound on Jensen-Shannon divergence are obtained along the way. Complementary lower bounds are derived with special attention paid to the case that corresponds to independent summation of a continuous and a discrete random variable. Several applications of the bounds are delineated, including to mutual information of additive noise channels, thermodynamics of computation, and functional inequalities. △ Less

Submitted 22 April, 2020; v1 submitted 29 May, 2018; originally announced May 2018.

Journal ref: IEEE Transactions on Information Theory, vol. 68, no. 4, pp. 2123-2146, April 2022

On the entropy power inequality for the Rényi entropy of order [0,1]

Authors: Arnaud Marsiglietti, James Melbourne

Abstract: Using a sharp version of the reverse Young inequality, and a Rényi entropy comparison result due to Fradelizi, Madiman, and Wang, the authors are able to derive Rényi entropy power inequalities for log-concave random vectors when Rényi parameters belong to $(0,1)$. Furthermore, the estimates are shown to be sharp up to absolute constants. Using a sharp version of the reverse Young inequality, and a Rényi entropy comparison result due to Fradelizi, Madiman, and Wang, the authors are able to derive Rényi entropy power inequalities for log-concave random vectors when Rényi parameters belong to $(0,1)$. Furthermore, the estimates are shown to be sharp up to absolute constants. △ Less

Submitted 21 July, 2018; v1 submitted 2 October, 2017; originally announced October 2017.

Forward and Reverse Entropy Power Inequalities in Convex Geometry

Authors: Mokshay Madiman, James Melbourne, Peng Xu

Abstract: The entropy power inequality, which plays a fundamental role in information theory and probability, may be seen as an analogue of the Brunn-Minkowski inequality. Motivated by this connection to Convex Geometry, we survey various recent developments on forward and reverse entropy power inequalities not just for the Shannon-Boltzmann entropy but also more generally for Rényi entropy. In the process,… ▽ More The entropy power inequality, which plays a fundamental role in information theory and probability, may be seen as an analogue of the Brunn-Minkowski inequality. Motivated by this connection to Convex Geometry, we survey various recent developments on forward and reverse entropy power inequalities not just for the Shannon-Boltzmann entropy but also more generally for Rényi entropy. In the process, we discuss connections between the so-called functional (or integral) and probabilistic (or entropic) analogues of some classical inequalities in geometric functional analysis △ Less

Submitted 24 December, 2016; v1 submitted 14 April, 2016; originally announced April 2016.

Comments: 54 pages. Changes in v2: improved organization, cleaned up exposition, and numerous references added

Journal ref: Book chapter in "Convexity and Concentration", E. Carlen, M. Madiman, E. Werner (eds.), The IMA Volumes in Mathematics and its Applications, Vol. 161, pp. 427-485, Springer, 2017