Showing 1–2 of 2 results for author: Benger, E

The Cardinality Bound on the Information Bottleneck Representations is Tight

Authors: Etam Benger, Shahab Asoodeh, Jun Chen

Abstract: The information bottleneck (IB) method aims to find compressed representations of a variable $X$ that retain the most relevant information about a target variable $Y$. We show that for a wide family of distributions -- namely, when $Y$ is generated by $X$ through a Hamming channel, under mild conditions -- the optimal IB representations require an alphabet strictly larger than that of $X$. This im… ▽ More The information bottleneck (IB) method aims to find compressed representations of a variable $X$ that retain the most relevant information about a target variable $Y$. We show that for a wide family of distributions -- namely, when $Y$ is generated by $X$ through a Hamming channel, under mild conditions -- the optimal IB representations require an alphabet strictly larger than that of $X$. This implies that, despite several recent works, the cardinality bound first identified by Witsenhausen and Wyner in 1975 is tight. At the core of our finding is the observation that the IB function in this setting is not strictly concave, similar to the deterministic case, even though the joint distribution of $X$ and $Y$ is of full support. Finally, we provide a complete characterization of the IB function, as well as of the optimal representations for the Hamming case. △ Less

Submitted 12 May, 2023; v1 submitted 11 May, 2023; originally announced May 2023.

Comments: 7 pages, 2 figures, ISIT 2023 submission

Journal ref: 2023 IEEE International Symposium on Information Theory (ISIT), Taipei, Taiwan, 2023, pp. 1478-1483

Critical Slowing Down Near Topological Transitions in Rate-Distortion Problems

Authors: Shlomi Agmon, Etam Benger, Or Ordentlich, Naftali Tishby

Abstract: In rate-distortion (RD) problems one seeks reduced representations of a source that meet a target distortion constraint. Such optimal representations undergo topological transitions at some critical rate values, when their cardinality or dimensionality change. We study the convergence time of the Arimoto-Blahut alternating projection algorithms, used to solve such problems, near those critical poi… ▽ More In rate-distortion (RD) problems one seeks reduced representations of a source that meet a target distortion constraint. Such optimal representations undergo topological transitions at some critical rate values, when their cardinality or dimensionality change. We study the convergence time of the Arimoto-Blahut alternating projection algorithms, used to solve such problems, near those critical points, both for the rate-distortion and information bottleneck settings. We argue that they suffer from critical slowing down -- a diverging number of iterations for convergence -- near the critical points. This phenomenon can have theoretical and practical implications for both machine learning and data compression problems. △ Less

Submitted 9 May, 2021; v1 submitted 3 March, 2021; originally announced March 2021.

Comments: 10 pages, 2 figures, ISIT 2021 submission

Journal ref: 2021 IEEE International Symposium on Information Theory (ISIT), Melbourne, Australia, 2021, pp. 2625-2630