Showing 1–39 of 39 results for author: Verdú, S

Sharp Bounds for Mutual Covering

Authors: **gbo Liu, Mohammad H. Yassaee, Sergio Verdú

Abstract: A fundamental tool in network information theory is the covering lemma, which lower bounds the probability that there exists a pair of random variables, among a give number of independently generated candidates, falling within a given set. We use a weighted sum trick and Talagrand's concentration inequality to prove new mutual covering bounds. We identify two interesting applications: 1) When the… ▽ More A fundamental tool in network information theory is the covering lemma, which lower bounds the probability that there exists a pair of random variables, among a give number of independently generated candidates, falling within a given set. We use a weighted sum trick and Talagrand's concentration inequality to prove new mutual covering bounds. We identify two interesting applications: 1) When the probability of the set under the given joint distribution is bounded away from 0 and 1, the covering probability converges to 1 \emph{doubly} exponentially fast in the blocklength, which implies that the covering lemma does not induce penalties on the error exponents in the applications to coding theorems. 2) Using Hall's marriage lemma, we show that the maximum difference between the probability of the set under the joint distribution and the covering probability equals half the minimum total variation distance between the joint distribution and any distribution that can be simulated by selecting a pair from the candidates. Thus we use the mutual covering bound to derive the exact error exponent in the joint distribution simulation problem. In both applications, the determination of the exact exponential (or doubly exponential) behavior relies crucially on the sharp concentration inequality used in the proof of the mutual covering lemma. △ Less

Submitted 16 April, 2019; v1 submitted 1 January, 2019; originally announced January 2019.

Comments: 33 pages, 1 figure, presented in part at ISIT 2017

Second-Order Converses via Reverse Hypercontractivity

Authors: **gbo Liu, Ramon van Handel, Sergio Verdú

Abstract: A strong converse shows that no procedure can beat the asymptotic (as blocklength $n\to\infty$) fundamental limit of a given information-theoretic problem for any fixed error probability. A second-order converse strengthens this conclusion by showing that the asymptotic fundamental limit cannot be exceeded by more than $O(\tfrac{1}{\sqrt{n}})$. While strong converses are achieved in a broad range… ▽ More A strong converse shows that no procedure can beat the asymptotic (as blocklength $n\to\infty$) fundamental limit of a given information-theoretic problem for any fixed error probability. A second-order converse strengthens this conclusion by showing that the asymptotic fundamental limit cannot be exceeded by more than $O(\tfrac{1}{\sqrt{n}})$. While strong converses are achieved in a broad range of information-theoretic problems by virtue of the "blowing-up method"---a powerful methodology due to Ahlswede, Gács and Körner (1976) based on concentration of measure---this method is fundamentally unable to attain second-order converses and is restricted to finite-alphabet settings. Capitalizing on reverse hypercontractivity of Markov semigroups and functional inequalities, this paper develops the "smoothing-out" method, an alternative to the blowing-up approach that does not rely on finite alphabets and that leads to second-order converses in a variety of information-theoretic problems that were out of reach of previous methods. △ Less

Submitted 14 November, 2019; v1 submitted 25 December, 2018; originally announced December 2018.

Comments: 61 pages, 3 figures, short version presented at ISIT 2017

Chaining Mutual Information and Tightening Generalization Bounds

Authors: Amir R. Asadi, Emmanuel Abbe, Sergio Verdú

Abstract: Bounding the generalization error of learning algorithms has a long history, which yet falls short in explaining various generalization successes including those of deep learning. Two important difficulties are (i) exploiting the dependencies between the hypotheses, (ii) exploiting the dependence between the algorithm's input and output. Progress on the first point was made with the chaining metho… ▽ More Bounding the generalization error of learning algorithms has a long history, which yet falls short in explaining various generalization successes including those of deep learning. Two important difficulties are (i) exploiting the dependencies between the hypotheses, (ii) exploiting the dependence between the algorithm's input and output. Progress on the first point was made with the chaining method, originating from the work of Kolmogorov, and used in the VC-dimension bound. More recently, progress on the second point was made with the mutual information method by Russo and Zou '15. Yet, these two methods are currently disjoint. In this paper, we introduce a technique to combine the chaining and mutual information methods, to obtain a generalization bound that is both algorithm-dependent and that exploits the dependencies between the hypotheses. We provide an example in which our bound significantly outperforms both the chaining and the mutual information bounds. As a corollary, we tighten Dudley's inequality when the learning algorithm chooses its output from a small subset of hypotheses with high probability. △ Less

Submitted 1 July, 2019; v1 submitted 11 June, 2018; originally announced June 2018.

Comments: 20 pages, 1 figure; published at the NeurIPS 2018 conference

The Error Probability of Generalized Perfect Codes via the Meta-Converse

Authors: Gonzalo Vazquez-Vilar, Albert Guillén i Fàbregas, Sergio Verdú

Abstract: We introduce a definition of perfect and quasi-perfect codes for symmetric channels parametrized by an auxiliary output distribution. This notion generalizes previous definitions of perfect and quasi-perfect codes and encompasses maximum distance separable codes. The error probability of these codes, whenever they exist, is shown to coincide with the estimate provided by the meta-converse lower bo… ▽ More We introduce a definition of perfect and quasi-perfect codes for symmetric channels parametrized by an auxiliary output distribution. This notion generalizes previous definitions of perfect and quasi-perfect codes and encompasses maximum distance separable codes. The error probability of these codes, whenever they exist, is shown to coincide with the estimate provided by the meta-converse lower bound. We illustrate how the proposed definition naturally extends to cover almost-lossless source-channel coding and lossy compression. △ Less

Submitted 25 May, 2018; originally announced May 2018.

Comments: Submitted to IEEE Transactions on Information Theory

Improved Bounds on Lossless Source Coding and Guessing Moments via Rényi Measures

Authors: Igal Sason, Sergio Verdú

Abstract: This paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available. These moments quantify the number of guesses required for correctly identifying the unknown object and, similarly to Arikan's bounds, they are expressed in terms of the Arimoto-Rényi conditional entropy. Although Arikan's bounds are a… ▽ More This paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available. These moments quantify the number of guesses required for correctly identifying the unknown object and, similarly to Arikan's bounds, they are expressed in terms of the Arimoto-Rényi conditional entropy. Although Arikan's bounds are asymptotically tight, the improvement of the bounds in this paper is significant in the non-asymptotic regime. Relationships between moments of the optimal guessing function and the MAP error probability are also established, characterizing the exact locus of their attainable values. The bounds on optimal guessing moments serve to improve non-asymptotic bounds on the cumulant generating function of the codeword lengths for fixed-to-variable optimal lossless source coding without prefix constraints. Non-asymptotic bounds on the reliability function of discrete memoryless sources are derived as well. Relying on these techniques, lower bounds on the cumulant generating function of the codeword lengths are derived, by means of the smooth Rényi entropy, for source codes that allow decoding errors. △ Less

Submitted 21 June, 2018; v1 submitted 4 January, 2018; originally announced January 2018.

Comments: This paper was published in the IEEE Trans. on Information Theory, vol. 64, no. 6, pp. 4323-4346, June 2018. The current version is a post-print which includes few corrections of printing typos

Optimal Universal Lossless Compression with Side Information

Authors: Yeohee Im, Sergio Verdú

Abstract: This paper presents conditional versions of Lempel-Ziv (LZ) algorithm for settings where compressor and decompressor have access to the same side information. We propose a fixed-length-parsing LZ algorithm with side information, motivated by the Willems algorithm, and prove the optimality for any stationary processes. In addition, we suggest strategies to improve the algorithm which lower the data… ▽ More This paper presents conditional versions of Lempel-Ziv (LZ) algorithm for settings where compressor and decompressor have access to the same side information. We propose a fixed-length-parsing LZ algorithm with side information, motivated by the Willems algorithm, and prove the optimality for any stationary processes. In addition, we suggest strategies to improve the algorithm which lower the data compression rate. A modification of a variable-length-parsing LZ algorithm with side information is proposed and proved to be asymptotically optimal for any stationary and ergodic processes. △ Less

Submitted 17 July, 2017; originally announced July 2017.

Comments: 20 Pages, Submitted to IEEE Transactions on Information Theory, Part of this work was presented in ISIT '17

Information-Theoretic Perspectives on Brascamp-Lieb Inequality and Its Reverse

Authors: **gbo Liu, Thomas A. Courtade, Paul Cuff, Sergio Verdu

Abstract: We introduce an inequality which may be viewed as a generalization of both the Brascamp-Lieb inequality and its reverse (Barthe's inequality), and prove its information-theoretic (i.e.\ entropic) formulation. This result leads to a unified approach to functional inequalities such as the variational formula of Rényi entropy, hypercontractivity and its reverse, strong data processing inequalities, a… ▽ More We introduce an inequality which may be viewed as a generalization of both the Brascamp-Lieb inequality and its reverse (Barthe's inequality), and prove its information-theoretic (i.e.\ entropic) formulation. This result leads to a unified approach to functional inequalities such as the variational formula of Rényi entropy, hypercontractivity and its reverse, strong data processing inequalities, and transportation-cost inequalities, whose utility in the proofs of various coding theorems has gained growing popularity recently. We show that our information-theoretic setting is convenient for proving properties such as data processing, tensorization, convexity (Riesz-Thorin interpolation) and Gaussian optimality. In particular, we elaborate on a "doubling trick" used by Lieb and Geng-Nair to prove several results on Gaussian optimality. Several applications are discussed, including a generalization of the Brascamp-Lieb inequality involving Gaussian random transformations, the determination of Wyner's common information of vector Gaussian sources, and the achievable rate region of certain key generation problems in the case of vector Gaussian sources. △ Less

Submitted 3 December, 2017; v1 submitted 20 February, 2017; originally announced February 2017.

Comments: Corrected some typos in the previous version

Arimoto-Rényi Conditional Entropy and Bayesian $M$-ary Hypothesis Testing

Authors: Igal Sason, Sergio Verdú

Abstract: This paper gives upper and lower bounds on the minimum error probability of Bayesian $M$-ary hypothesis testing in terms of the Arimoto-Rényi conditional entropy of an arbitrary order $α$. The improved tightness of these bounds over their specialized versions with the Shannon conditional entropy ($α=1$) is demonstrated. In particular, in the case where $M$ is finite, we show how to generalize Fano… ▽ More This paper gives upper and lower bounds on the minimum error probability of Bayesian $M$-ary hypothesis testing in terms of the Arimoto-Rényi conditional entropy of an arbitrary order $α$. The improved tightness of these bounds over their specialized versions with the Shannon conditional entropy ($α=1$) is demonstrated. In particular, in the case where $M$ is finite, we show how to generalize Fano's inequality under both the conventional and list-decision settings. As a counterpart to the generalized Fano's inequality, allowing $M$ to be infinite, a lower bound on the Arimoto-Rényi conditional entropy is derived as a function of the minimum error probability. Explicit upper and lower bounds on the minimum error probability are obtained as a function of the Arimoto-Rényi conditional entropy for both positive and negative $α$. Furthermore, we give upper bounds on the minimum error probability as functions of the Rényi divergence. In the setup of discrete memoryless channels, we analyze the exponentially vanishing decay of the Arimoto-Rényi conditional entropy of the transmitted codeword given the channel output when averaged over a random coding ensemble. △ Less

Submitted 5 December, 2017; v1 submitted 8 January, 2017; originally announced January 2017.

Comments: To appear in the IEEE Trans. on Information Theory, vol. 64, no. 1, January 2018 (22 pages in two columns)

Minimax Rényi Redundancy

Authors: Semih Yagli, Yücel Altuğ, Sergio Verdú

Abstract: The redundancy for universal lossless compression of discrete memoryless sources in Campbell's setting is characterized as a minimax Rényi divergence, which is shown to be equal to the maximal $α$-mutual information via a generalized redundancy-capacity theorem. Special attention is placed on the analysis of the asymptotics of minimax Rényi divergence, which is determined up to a term vanishing in… ▽ More The redundancy for universal lossless compression of discrete memoryless sources in Campbell's setting is characterized as a minimax Rényi divergence, which is shown to be equal to the maximal $α$-mutual information via a generalized redundancy-capacity theorem. Special attention is placed on the analysis of the asymptotics of minimax Rényi divergence, which is determined up to a term vanishing in blocklength. △ Less

Submitted 24 January, 2018; v1 submitted 4 January, 2017; originally announced January 2017.

Comments: Double-column, 19 pages, submitted for publication to IEEE Transactions on Information Theory. Disclaimer: the published version may have slight variations. Part of this work appeared in ISIT '17

$f$-Divergence Inequalities via Functional Domination

Authors: Igal Sason, Sergio Verdú

Abstract: This paper considers derivation of $f$-divergence inequalities via the approach of functional domination. Bounds on an $f$-divergence based on one or several other $f$-divergences are introduced, dealing with pairs of probability measures defined on arbitrary alphabets. In addition, a variety of bounds are shown to hold under boundedness assumptions on the relative information. The journal paper,… ▽ More This paper considers derivation of $f$-divergence inequalities via the approach of functional domination. Bounds on an $f$-divergence based on one or several other $f$-divergences are introduced, dealing with pairs of probability measures defined on arbitrary alphabets. In addition, a variety of bounds are shown to hold under boundedness assumptions on the relative information. The journal paper, which includes more approaches for the derivation of f-divergence inequalities and proofs, is available on the arXiv at https://arxiv.longhoe.net/abs/1508.00335, and it has been published in the IEEE Trans. on Information Theory, vol. 62, no. 11, pp. 5973-6006, November 2016. △ Less

Submitted 28 October, 2016; originally announced October 2016.

Comments: A conference paper, 5 pages. To be presented in the 2016 ICSEE International Conference on the Science of Electrical Engineering, Nov. 16--18, Eilat, Israel. See https://arxiv.longhoe.net/abs/1508.00335 for the full paper version, published as a journal paper in the IEEE Trans. on Information Theory, vol. 62, no. 11, pp. 5973-6006, November 2016

Brascamp-Lieb Inequality and Its Reverse: An Information Theoretic View

Authors: **gbo Liu, Thomas A. Courtade, Paul Cuff, Sergio Verdu

Abstract: We generalize a result by Carlen and Cordero-Erausquin on the equivalence between the Brascamp-Lieb inequality and the subadditivity of relative entropy by allowing for random transformations (a broadcast channel). This leads to a unified perspective on several functional inequalities that have been gaining popularity in the context of proving impossibility results. We demonstrate that the informa… ▽ More We generalize a result by Carlen and Cordero-Erausquin on the equivalence between the Brascamp-Lieb inequality and the subadditivity of relative entropy by allowing for random transformations (a broadcast channel). This leads to a unified perspective on several functional inequalities that have been gaining popularity in the context of proving impossibility results. We demonstrate that the information theoretic dual of the Brascamp-Lieb inequality is a convenient setting for proving properties such as data processing, tensorization, convexity and Gaussian optimality. Consequences of the latter include an extension of the Brascamp-Lieb inequality allowing for Gaussian random transformations, the determination of the multivariate Wyner common information for Gaussian sources, and a multivariate version of Nelson's hypercontractivity theorem. Finally we present an information theoretic characterization of a reverse Brascamp-Lieb inequality involving a random transformation (a multiple access channel). △ Less

Submitted 9 May, 2016; originally announced May 2016.

Comments: 5 pages; to be presented at ISIT 2016

Smoothing Brascamp-Lieb Inequalities and Strong Converses for Common Randomness Generation

Authors: **gbo Liu, Thomas A. Courtade, Paul Cuff, Sergio Verdu

Abstract: We study the infimum of the best constant in a functional inequality, the Brascamp-Lieb-like inequality, over auxiliary measures within a neighborhood of a product distribution. In the finite alphabet and the Gaussian cases, such an infimum converges to the best constant in a mutual information inequality. Implications for strong converse properties of two common randomness (CR) generation problem… ▽ More We study the infimum of the best constant in a functional inequality, the Brascamp-Lieb-like inequality, over auxiliary measures within a neighborhood of a product distribution. In the finite alphabet and the Gaussian cases, such an infimum converges to the best constant in a mutual information inequality. Implications for strong converse properties of two common randomness (CR) generation problems are discussed. In particular, we prove the strong converse property of the rate region for the omniscient helper CR generation problem in the discrete and the Gaussian cases. The latter case is perhaps the first instance of a strong converse for a continuous source when the rate region involves auxiliary random variables. △ Less

Submitted 6 February, 2016; originally announced February 2016.

Comments: 7 pages; first 5 pages submitted to ISIT 2016

Secret Key Generation with Limited Interaction

Authors: **gbo Liu, Paul Cuff, Sergio Verdú

Abstract: A basic two-terminal secret key generation model is considered, where the interactive communication rate between the terminals may be limited, and in particular may not be enough to achieve the maximum key rate. We first prove a multi-letter characterization of the key-communication rate region (where the number of auxiliary random variables depend on the number of rounds of the communication), an… ▽ More A basic two-terminal secret key generation model is considered, where the interactive communication rate between the terminals may be limited, and in particular may not be enough to achieve the maximum key rate. We first prove a multi-letter characterization of the key-communication rate region (where the number of auxiliary random variables depend on the number of rounds of the communication), and then provide an equivalent but simpler characterization in terms of concave envelopes in the case of unlimited number of rounds. Two extreme cases are given special attention. First, in the regime of very low communication rates, the \emph{key bits per interaction bit} (KBIB) is expressed with a new "symmetric strong data processing constant", which has a concave envelope characterization analogous to that of the conventional strong data processing constant. The symmetric strong data processing constant can be upper bounded by the supremum of the maximal correlation coefficient over a set of distributions, which allows us to determine the KBIB for binary symmetric sources, and conclude, in particular, that the interactive scheme is not more efficient than the one-way scheme at least in the low communication-rate regime. Second, a new characterization of the \emph{minimum interaction rate needed for achieving the maximum key rate} (MIMK) is given, and we resolve a conjecture by Tyagi regarding the MIMK for (possibly nonsymmetric) binary sources. We also propose a new conjecture for binary symmetric sources that the interactive scheme is not more efficient than the one-way scheme at any communication rate. △ Less

Submitted 27 March, 2017; v1 submitted 5 January, 2016; originally announced January 2016.

Comments: 50 pages; short version appeared in ISIT 2016. Removed an incorrect result in the previous arXiv version (see acknowledgement)

$E_γ$-Resolvability

Authors: **gbo Liu, Paul Cuff, Sergio Verdú

Abstract: The conventional channel resolvability refers to the minimum rate needed for an input process to approximate the channel output distribution in total variation distance. In this paper we study $E_γ$-resolvability, in which total variation is replaced by the more general $E_γ$ distance. A general one-shot achievability bound for the precision of such an approximation is developed. Let… ▽ More The conventional channel resolvability refers to the minimum rate needed for an input process to approximate the channel output distribution in total variation distance. In this paper we study $E_γ$-resolvability, in which total variation is replaced by the more general $E_γ$ distance. A general one-shot achievability bound for the precision of such an approximation is developed. Let $Q_{\sf X|U}$ be a random transformation, $n$ be an integer, and $E\in(0,+\infty)$. We show that in the asymptotic setting where $γ=\exp(nE)$, a (nonnegative) randomness rate above $\inf_{Q_{\sf U}: D(Q_{\sf X}\|{π}_{\sf X})\le E} \{D(Q_{\sf X}\|{π}_{\sf X})+I(Q_{\sf U},Q_{\sf X|U})-E\}$ is sufficient to approximate the output distribution ${π}_{\sf X}^{\otimes n}$ using the channel $Q_{\sf X|U}^{\otimes n}$, where $Q_{\sf U}\to Q_{\sf X|U}\to Q_{\sf X}$, and is also necessary in the case of finite $\mathcal{U}$ and $\mathcal{X}$. In particular, a randomness rate of $\inf_{Q_{\sf U}}I(Q_{\sf U},Q_{\sf X|U})-E$ is always sufficient. We also study the convergence of the approximation error under the high probability criteria in the case of random codebooks. Moreover, by develo** simple bounds relating $E_γ$ and other distance measures, we are able to determine the exact linear growth rate of the approximation errors measured in relative entropy and smooth Rényi divergences for a fixed-input randomness rate. The new resolvability result is then used to derive 1) a one-shot upper bound on the probability of excess distortion in lossy compression, which is exponentially tight in the i.i.d.~setting, 2) a one-shot version of the mutual covering lemma, and 3) a lower bound on the size of the eavesdropper list to include the actual message and a lower bound on the eavesdropper false-alarm probability in the wiretap channel problem, which is (asymptotically) ensemble-tight. △ Less

Submitted 20 July, 2017; v1 submitted 24 November, 2015; originally announced November 2015.

Comments: 30 pages, 5 figures, presented in part at 2015 IEEE International Symposium on Information Theory (ISIT)

$f$-divergence Inequalities

Authors: Igal Sason, Sergio Verdú

Abstract: This paper develops systematic approaches to obtain $f$-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of $f$-divergences. Another approach used for the derivation of bounds among $f$-divergences relies on mom… ▽ More This paper develops systematic approaches to obtain $f$-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of $f$-divergences. Another approach used for the derivation of bounds among $f$-divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of $χ^2$ divergences. A rich variety of bounds are shown to hold under boundedness assumptions on the relative information. Special attention is devoted to the total variation distance and its relation to the relative information and relative entropy, including "reverse Pinsker inequalities," as well as on the $E_γ$ divergence, which generalizes the total variation distance. Pinsker's inequality is extended for this type of $f$-divergence, a result which leads to an inequality linking the relative entropy and relative information spectrum. Integral expressions of the Rényi divergence in terms of the relative information spectrum are derived, leading to bounds on the Rényi divergence in terms of either the variational distance or relative entropy. △ Less

Submitted 4 December, 2016; v1 submitted 3 August, 2015; originally announced August 2015.

Comments: IEEE Trans. on Information Theory, vol. 62, no. 11, pp. 5973--6006, November 2016. This manuscript is identical to the journal paper, apart of some additional material which includes Sections III-C and IV-F, and three technical proofs

Resolvability in Eγ with Applications to Lossy Compression and Wiretap Channels

Authors: **gbo Liu, Paul Cuff, Sergio Verdú

Abstract: We study the amount of randomness needed for an input process to approximate a given output distribution of a channel in the $E_γ$ distance. A general one-shot achievability bound for the precision of such an approximation is developed. In the i.i.d.~setting where $γ=\exp(nE)$, a (nonnegative) randomness rate above… ▽ More We study the amount of randomness needed for an input process to approximate a given output distribution of a channel in the $E_γ$ distance. A general one-shot achievability bound for the precision of such an approximation is developed. In the i.i.d.~setting where $γ=\exp(nE)$, a (nonnegative) randomness rate above $\inf_{Q_{\sf U}: D(Q_{\sf X}||π_{\sf X})\le E} \{D(Q_{\sf X}||π_{\sf X})+I(Q_{\sf U},Q_{\sf X|U})-E\}$ is necessary and sufficient to asymptotically approximate the output distribution $π_{\sf X}^{\otimes n}$ using the channel $Q_{\sf X|U}^{\otimes n}$, where $Q_{\sf U}\to Q_{\sf X|U}\to Q_{\sf X}$. The new resolvability result is then used to derive a one-shot upper bound on the error probability in the rate distortion problem, and a lower bound on the size of the eavesdropper list to include the actual message in the wiretap channel problem. Both bounds are asymptotically tight in i.i.d.~settings. △ Less

Submitted 30 May, 2015; originally announced June 2015.

Comments: ISIT 2015

Journal ref: 2015 IEEE International Symposium on Information Theory (ISIT), 14-19 June 2015, pages: 755 - 759, Hong Kong

Secret Key Generation with One Communicator and a One-Shot Converse via Hypercontractivity

Authors: **gbo Liu, Paul Cuff, Sergio Verdu

Abstract: A new model of multi-party secret key agreement is proposed, in which one terminal called the communicator can transmit public messages to other terminals before all terminals agree on a secret key. A single-letter characterization of the achievable region is derived in the stationary memoryless case. The new model generalizes some other (old and new) models of key agreement. In particular, key ge… ▽ More A new model of multi-party secret key agreement is proposed, in which one terminal called the communicator can transmit public messages to other terminals before all terminals agree on a secret key. A single-letter characterization of the achievable region is derived in the stationary memoryless case. The new model generalizes some other (old and new) models of key agreement. In particular, key generation with an omniscient helper is the special case where the communicator knows all sources, for which we derive a zero-rate one-shot converse for the secret key per bit of communication. △ Less

Submitted 22 April, 2015; v1 submitted 21 April, 2015; originally announced April 2015.

Comments: slightly extended version of ISIT 2015 paper

Journal ref: Proc. 2015 IEEE Int. Symposium on Information Theory, Hong-Kong, June 15--19, 2015, pp 1457--1461

One-Shot Mutual Covering Lemma and Marton's Inner Bound with a Common Message

Authors: **gbo Liu, Paul Cuff, Sergio Verdu

Abstract: By develo** one-shot mutual covering lemmas, we derive a one-shot achievability bound for broadcast with a common message which recovers Marton's inner bound (with three auxiliary random variables) in the i.i.d.~case. The encoder employed is deterministic. Relationship between the mutual covering lemma and a new type of channel resolvability problem is discussed. By develo** one-shot mutual covering lemmas, we derive a one-shot achievability bound for broadcast with a common message which recovers Marton's inner bound (with three auxiliary random variables) in the i.i.d.~case. The encoder employed is deterministic. Relationship between the mutual covering lemma and a new type of channel resolvability problem is discussed. △ Less

Submitted 6 June, 2015; v1 submitted 15 April, 2015; originally announced April 2015.

Comments: 6 pages; extended version of ISIT paper

Journal ref: 2015 IEEE International Symposium on Information Theory (ISIT), 14-19 June 2015, pages: 1457 - 1461, Hong Kong

Upper Bounds on the Relative Entropy and Rényi Divergence as a Function of Total Variation Distance for Finite Alphabets

Authors: Igal Sason, Sergio Verdu

Abstract: A new upper bound on the relative entropy is derived as a function of the total variation distance for probability measures defined on a common finite alphabet. The bound improves a previously reported bound by Csiszár and Talata. It is further extended to an upper bound on the Rényi divergence of an arbitrary non-negative order (including $\infty$) as a function of the total variation distance. A new upper bound on the relative entropy is derived as a function of the total variation distance for probability measures defined on a common finite alphabet. The bound improves a previously reported bound by Csiszár and Talata. It is further extended to an upper bound on the Rényi divergence of an arbitrary non-negative order (including $\infty$) as a function of the total variation distance. △ Less

Submitted 17 October, 2015; v1 submitted 11 March, 2015; originally announced March 2015.

Comments: Proceedings of the 2015 IEEE Information Theory Workshop, pp. 214-218, Jeju, Korea, October 2015. This presents in part the work which is available at arXiv:1508.00335

Joint source-channel coding with feedback

Authors: Victoria Kostina, Yury Polyanskiy, Sergio Verdú

Abstract: This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion and guaranteed distortion ($d$-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-lengt… ▽ More This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion and guaranteed distortion ($d$-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-length codes and feedback leads to a sizable improvement in the fundamental delay-distortion tradeoff. In addition, we investigate the minimum energy required to reproduce $k$ source samples with a given fidelity after transmission over a memoryless Gaussian channel, and we show that the required minimum energy is reduced with feedback and an average (rather than maximal) power constraint. △ Less

Submitted 25 February, 2017; v1 submitted 29 January, 2015; originally announced January 2015.

Comments: To appear in IEEE Transactions on Information Theory

Key Capacity for Product Sources with Application to Stationary Gaussian Processes

Authors: **gbo Liu, Paul Cuff, Sergio Verdú

Abstract: We show that for product sources, rate splitting is optimal for secret key agreement using limited one-way communication between two terminals. This yields an alternative information-theoretic-converse-style proof of the tensorization property of a strong data processing inequality originally studied by Erkip and Cover and amended recently by Anantharam et al. We derive a water-filling solution of… ▽ More We show that for product sources, rate splitting is optimal for secret key agreement using limited one-way communication between two terminals. This yields an alternative information-theoretic-converse-style proof of the tensorization property of a strong data processing inequality originally studied by Erkip and Cover and amended recently by Anantharam et al. We derive a water-filling solution of the communication-rate--key-rate tradeoff for a wide class of discrete memoryless vector Gaussian sources which subsumes the case without an eavesdropper. Moreover, we derive an explicit formula for the maximum secret key per bit of communication for all discrete memoryless vector Gaussian sources using a tensorization property and a variation on the enhanced channel technique of Weingarten et al. Finally, a one-shot information spectrum achievability bound for key generation is proved from which we characterize the communication-rate--key-rate tradeoff for stationary Gaussian processes. △ Less

Submitted 24 April, 2016; v1 submitted 19 September, 2014; originally announced September 2014.

Comments: Presented in part at 2014 IEEE International Symposium on Information Theory (ISIT)

Journal ref: IEEE Trans. on Information Theory, vol. 62, no. 2, pp. 984--1005, Feb. 2016

Key Capacity with Limited One-Way Communication for Product Sources

Authors: **gbo Liu, Paul Cuff, Sergio Verdú

Abstract: We show that for product sources, rate splitting is optimal for secret key agreement using limited one-way communication at two terminals. This yields an alternative proof of the tensorization property of a strong data processing inequality originally studied by Erkip and Cover and amended recently by Anantharam et al. We derive a `water-filling' solution of the communication-rate--key-rate tradeo… ▽ More We show that for product sources, rate splitting is optimal for secret key agreement using limited one-way communication at two terminals. This yields an alternative proof of the tensorization property of a strong data processing inequality originally studied by Erkip and Cover and amended recently by Anantharam et al. We derive a `water-filling' solution of the communication-rate--key-rate tradeoff for two arbitrarily correlated vector Gaussian sources, for the case with an eavesdropper, and for stationary Gaussian processes. △ Less

Submitted 28 May, 2014; originally announced May 2014.

Comments: 5 pages, ISIT 2014

Journal ref: 2014 IEEE International Symposium on Information Theory (ISIT), pages: 1146 - 1150, June 29-July 4 2014, Honolulu, HI

Variable-length compression allowing errors

Authors: Victoria Kostina, Yury Polyanskiy, Sergio Verdú

Abstract: This paper studies the fundamental limits of the minimum average length of lossless and lossy variable-length compression, allowing a nonzero error probability $ε$, for lossless compression. We give non-asymptotic bounds on the minimum average length in terms of Erokhin's rate-distortion function and we use those bounds to obtain a Gaussian approximation on the speed of approach to the limit which… ▽ More This paper studies the fundamental limits of the minimum average length of lossless and lossy variable-length compression, allowing a nonzero error probability $ε$, for lossless compression. We give non-asymptotic bounds on the minimum average length in terms of Erokhin's rate-distortion function and we use those bounds to obtain a Gaussian approximation on the speed of approach to the limit which is quite accurate for all but small blocklengths: $$(1 - ε) k H(\mathsf S) - \sqrt{\frac{k V(\mathsf S)}{2 π} } e^{- \frac {(Q^{-1}(ε))^2} 2 }$$ where $Q^{-1}(\cdot)$ is the functional inverse of the standard Gaussian complementary cdf, and $V(\mathsf S)$ is the source dispersion. A nonzero error probability thus not only reduces the asymptotically achievable rate by a factor of $1 - ε$, but this asymptotic limit is approached from below, i.e. larger source dispersions and shorter blocklengths are beneficial. Variable-length lossy compression under an excess distortion constraint is shown to exhibit similar properties. △ Less

Submitted 7 October, 2015; v1 submitted 3 February, 2014; originally announced February 2014.

Journal ref: IEEE Transactions on Information Theory, vol. 61, no. 9, pp. 4316-4330, Aug. 2015

Nonasymptotic noisy lossy source coding

Authors: Victoria Kostina, Sergio Verdú

Abstract: This paper shows new general nonasymptotic achievability and converse bounds and performs their dispersion analysis for the lossy compression problem in which the compressor observes the source through a noisy channel. While this problem is asymptotically equivalent to a noiseless lossy source coding problem with a modified distortion function, nonasymptotically there is a noticeable gap in how fa… ▽ More This paper shows new general nonasymptotic achievability and converse bounds and performs their dispersion analysis for the lossy compression problem in which the compressor observes the source through a noisy channel. While this problem is asymptotically equivalent to a noiseless lossy source coding problem with a modified distortion function, nonasymptotically there is a noticeable gap in how fast their minimum achievable coding rates approach the common rate-distortion function, as evidenced both by the refined asymptotic analysis (dispersion) and the numerical results. The size of the gap between the dispersions of the noisy problem and the asymptotically equivalent noiseless problem depends on the stochastic variability of the channel through which the compressor observes the source. △ Less

Submitted 15 September, 2016; v1 submitted 20 January, 2014; originally announced January 2014.

Comments: IEEE Transactions on Information Theory, 2016

Channels with cost constraints: strong converse and dispersion

Authors: Victoria Kostina, Sergio Verdú

Abstract: This paper shows the strong converse and the dispersion of memoryless channels with cost constraints and performs refined analysis of the third order term in the asymptotic expansion of the maximum achievable channel coding rate, showing that it is equal to $\frac 1 2 \frac {\log n}{n}$ in most cases of interest. The analysis is based on a non-asymptotic converse bound expressed in terms of the di… ▽ More This paper shows the strong converse and the dispersion of memoryless channels with cost constraints and performs refined analysis of the third order term in the asymptotic expansion of the maximum achievable channel coding rate, showing that it is equal to $\frac 1 2 \frac {\log n}{n}$ in most cases of interest. The analysis is based on a non-asymptotic converse bound expressed in terms of the distribution of a random variable termed the $\mathsf b$-tilted information density, which plays a role similar to that of the $\mathsf d$-tilted information in lossy source coding. We also analyze the fundamental limits of lossy joint-source-channel coding over channels with cost constraints. △ Less

Submitted 7 October, 2015; v1 submitted 20 January, 2014; originally announced January 2014.

Journal ref: IEEE Transactions on Information Theory, vol. 61, no. 5, pp. 2415-2429, May 2015

Empirical distribution of good channel codes with non-vanishing error probability (extended version)

Authors: Yury Polyanskiy, Sergio Verdu

Abstract: This paper studies several properties of channel codes that approach the fundamental limits of a given (discrete or Gaussian) memoryless channel with a non-vanishing probability of error. The output distribution induced by an $ε$-capacity-achieving code is shown to be close in a strong sense to the capacity achieving output distribution. Relying on the concentration of measure (isoperimetry) prope… ▽ More This paper studies several properties of channel codes that approach the fundamental limits of a given (discrete or Gaussian) memoryless channel with a non-vanishing probability of error. The output distribution induced by an $ε$-capacity-achieving code is shown to be close in a strong sense to the capacity achieving output distribution. Relying on the concentration of measure (isoperimetry) property enjoyed by the latter, it is shown that regular (Lipschitz) functions of channel outputs can be precisely estimated and turn out to be essentially non-random and independent of the actual code. It is also shown that the output distribution of a good code and the capacity achieving one cannot be distinguished with exponential reliability. The random process produced at the output of the channel is shown to satisfy the asymptotic equipartition property. Using related methods it is shown that quadratic forms and sums of $q$-th powers when evaluated at codewords of good AWGN codes approach the values obtained from a randomly generated Gaussian codeword. △ Less

Submitted 31 August, 2013; originally announced September 2013.

Comments: Extended version of the article submitted to IEEE Trans. Information Theory

Lossless Data Compression at Finite Blocklengths

Authors: Ioannis Kontoyiannis, Sergio Verdu

Abstract: This paper provides an extensive study of the behavior of the best achievable rate (and other related fundamental limits) in variable-length lossless compression. In the non-asymptotic regime, the fundamental limits of fixed-to-variable lossless compression with and without prefix constraints are shown to be tightly coupled. Several precise, quantitative bounds are derived, connecting the distribu… ▽ More This paper provides an extensive study of the behavior of the best achievable rate (and other related fundamental limits) in variable-length lossless compression. In the non-asymptotic regime, the fundamental limits of fixed-to-variable lossless compression with and without prefix constraints are shown to be tightly coupled. Several precise, quantitative bounds are derived, connecting the distribution of the optimal codelengths to the source information spectrum, and an exact analysis of the best achievable rate for arbitrary sources is given. Fine asymptotic results are proved for arbitrary (not necessarily prefix) compressors on general mixing sources. Non-asymptotic, explicit Gaussian approximation bounds are established for the best achievable rate on Markov sources. The source dispersion and the source varentropy rate are defined and characterized. Together with the entropy rate, the varentropy rate serves to tightly approximate the fundamental non-asymptotic limits of fixed-to-variable compression for all but very small blocklengths. △ Less

Submitted 11 December, 2012; originally announced December 2012.

Lossy joint source-channel coding in the finite blocklength regime

Authors: Victoria Kostina, Sergio Verdú

Abstract: This paper finds new tight finite-blocklength bounds for the best achievable lossy joint source-channel code rate, and demonstrates that joint source-channel code design brings considerable performance advantage over a separate one in the non-asymptotic regime. A joint source-channel code maps a block of $k$ source symbols onto a length$-n$ channel codeword, and the fidelity of reproduction at the… ▽ More This paper finds new tight finite-blocklength bounds for the best achievable lossy joint source-channel code rate, and demonstrates that joint source-channel code design brings considerable performance advantage over a separate one in the non-asymptotic regime. A joint source-channel code maps a block of $k$ source symbols onto a length$-n$ channel codeword, and the fidelity of reproduction at the receiver end is measured by the probability $ε$ that the distortion exceeds a given threshold $d$. For memoryless sources and channels, it is demonstrated that the parameters of the best joint source-channel code must satisfy $nC - kR(d) \approx \sqrt{nV + k \mathcal V(d)} Q(ε)$, where $C$ and $V$ are the channel capacity and channel dispersion, respectively; $R(d)$ and $\mathcal V(d)$ are the source rate-distortion and rate-dispersion functions; and $Q$ is the standard Gaussian complementary cdf. Symbol-by-symbol (uncoded) transmission is known to achieve the Shannon limit when the source and channel satisfy a certain probabilistic matching condition. In this paper we show that even when this condition is not satisfied, symbol-by-symbol transmission is, in some cases, the best known strategy in the non-asymptotic regime. △ Less

Submitted 3 February, 2014; v1 submitted 6 September, 2012; originally announced September 2012.

Journal ref: IEEE Transactions on Information Theory, vol. 59, no. 5, pp. 2545-2575, May 2013

Optimal Phase Transitions in Compressed Sensing

Authors: Yihong Wu, Sergio Verdú

Abstract: Compressed sensing deals with efficient recovery of analog signals from linear encodings. This paper presents a statistical study of compressed sensing by modeling the input signal as an i.i.d. process with known distribution. Three classes of encoders are considered, namely optimal nonlinear, optimal linear and random linear encoders. Focusing on optimal decoders, we investigate the fundamental t… ▽ More Compressed sensing deals with efficient recovery of analog signals from linear encodings. This paper presents a statistical study of compressed sensing by modeling the input signal as an i.i.d. process with known distribution. Three classes of encoders are considered, namely optimal nonlinear, optimal linear and random linear encoders. Focusing on optimal decoders, we investigate the fundamental tradeoff between measurement rate and reconstruction fidelity gauged by error probability and noise sensitivity in the absence and presence of measurement noise, respectively. The optimal phase transition threshold is determined as a functional of the input distribution and compared to suboptimal thresholds achieved by popular reconstruction algorithms. In particular, we show that Gaussian sensing matrices incur no penalty on the phase transition threshold with respect to optimal nonlinear encoding. Our results also provide a rigorous justification of previous results based on replica heuristics in the weak-noise regime. △ Less

Submitted 10 July, 2012; v1 submitted 29 November, 2011; originally announced November 2011.

Comments: to appear in IEEE Transactions of Information Theory

Fixed-length lossy compression in the finite blocklength regime

Authors: Victoria Kostina, Sergio Verdú

Abstract: This paper studies the minimum achievable source coding rate as a function of blocklength $n$ and probability $ε$ that the distortion exceeds a given level $d$. Tight general achievability and converse bounds are derived that hold at arbitrary fixed blocklength. For stationary memoryless sources with separable distortion, the minimum rate achievable is shown to be closely approximated by… ▽ More This paper studies the minimum achievable source coding rate as a function of blocklength $n$ and probability $ε$ that the distortion exceeds a given level $d$. Tight general achievability and converse bounds are derived that hold at arbitrary fixed blocklength. For stationary memoryless sources with separable distortion, the minimum rate achievable is shown to be closely approximated by $R(d) + \sqrt{\frac{V(d)}{n}} Q^{-1}(ε)$, where $R(d)$ is the rate-distortion function, $V(d)$ is the rate dispersion, a characteristic of the source which measures its stochastic variability, and $Q^{-1}(ε)$ is the inverse of the standard Gaussian complementary cdf. △ Less

Submitted 3 February, 2014; v1 submitted 18 February, 2011; originally announced February 2011.

Journal ref: IEEE Transactions on Information Theory, vol. 58, no. 6, pp. 3309-3338, June 2012

Estimation in Gaussian Noise: Properties of the Minimum Mean-Square Error

Authors: Dongning Guo, Yihong Wu, Shlomo Shamai, Sergio Verdu

Abstract: Consider the minimum mean-square error (MMSE) of estimating an arbitrary random variable from its observation contaminated by Gaussian noise. The MMSE can be regarded as a function of the signal-to-noise ratio (SNR) as well as a functional of the input distribution (of the random variable to be estimated). It is shown that the MMSE is concave in the input distribution at any given SNR. For a given… ▽ More Consider the minimum mean-square error (MMSE) of estimating an arbitrary random variable from its observation contaminated by Gaussian noise. The MMSE can be regarded as a function of the signal-to-noise ratio (SNR) as well as a functional of the input distribution (of the random variable to be estimated). It is shown that the MMSE is concave in the input distribution at any given SNR. For a given input distribution, the MMSE is found to be infinitely differentiable at all positive SNR, and in fact a real analytic function in SNR under mild conditions. The key to these regularity results is that the posterior distribution conditioned on the observation through Gaussian channels always decays at least as quickly as some Gaussian density. Furthermore, simple expressions for the first three derivatives of the MMSE with respect to the SNR are obtained. It is also shown that, as functions of the SNR, the curves for the MMSE of a Gaussian input and that of a non-Gaussian input cross at most once over all SNRs. These properties lead to simple proofs of the facts that Gaussian inputs achieve both the secrecy capacity of scalar Gaussian wiretap channels and the capacity of scalar Gaussian broadcast channels, as well as a simple proof of the entropy power inequality in the special case where one of the variables is Gaussian. △ Less

Submitted 19 April, 2010; originally announced April 2010.

Comments: Submitted to IEEE Transactions on Information Theory, revised.

Multicasting in Large Wireless Networks: Bounds on the Minimum Energy per Bit

Authors: Aman Jain, Sanjeev R. Kulkarni, Sergio Verdu

Abstract: We consider scaling laws for maximal energy efficiency of communicating a message to all the nodes in a wireless network, as the number of nodes in the network becomes large. Two cases of large wireless networks are studied -- dense random networks and constant density (extended) random networks. In addition, we also study finite size regular networks in order to understand how regularity in nod… ▽ More We consider scaling laws for maximal energy efficiency of communicating a message to all the nodes in a wireless network, as the number of nodes in the network becomes large. Two cases of large wireless networks are studied -- dense random networks and constant density (extended) random networks. In addition, we also study finite size regular networks in order to understand how regularity in node placement affects energy consumption. We first establish an information-theoretic lower bound on the minimum energy per bit for multicasting in arbitrary wireless networks when the channel state information is not available at the transmitters. Upper bounds are obtained by constructing a simple flooding scheme that requires no information at the receivers about the channel states or the locations and identities of the nodes. The gap between the upper and lower bounds is only a constant factor for dense random networks and regular networks, and differs by a poly-logarithmic factor for extended random networks. Furthermore, we show that the proposed upper and lower bounds for random networks hold almost surely in the node locations as the number of nodes approaches infinity. △ Less

Submitted 23 May, 2009; originally announced May 2009.

Comments: 40 pages, 4 figures, submitted to IEEE Transactions on Information Theory

Cognitive Interference Channels with Confidential Messages

Authors: Yingbin Liang, Anelia Somekh-Baruch, H. Vincent Poor, Shlomo Shamai, Sergio Verdu

Abstract: The cognitive interference channel with confidential messages is studied. Similarly to the classical two-user interference channel, the cognitive interference channel consists of two transmitters whose signals interfere at the two receivers. It is assumed that there is a common message source (message 1) known to both transmitters, and an additional independent message source (message 2) known o… ▽ More The cognitive interference channel with confidential messages is studied. Similarly to the classical two-user interference channel, the cognitive interference channel consists of two transmitters whose signals interfere at the two receivers. It is assumed that there is a common message source (message 1) known to both transmitters, and an additional independent message source (message 2) known only to the cognitive transmitter (transmitter 2). The cognitive receiver (receiver 2) needs to decode both messages, while the non-cognitive receiver (receiver 1) should decode only the common message. Furthermore, message 2 is assumed to be a confidential message which needs to be kept as secret as possible from receiver 1, which is viewed as an eavesdropper with regard to message 2. The level of secrecy is measured by the equivocation rate. A single-letter expression for the capacity-equivocation region of the discrete memoryless cognitive interference channel is established and is further explicitly derived for the Gaussian case. Moreover, particularizing the capacity-equivocation region to the case without a secrecy constraint, establishes a new capacity theorem for a class of interference channels, by providing a converse theorem. △ Less

Submitted 10 October, 2007; originally announced October 2007.

Comments: To appear in Proc. of forty-fifth annual Allerton conference on communication, control, and computing, Allerton house, Monticello, IL, USA

Canonical time-frequency, time-scale, and frequency-scale representations of time-varying channels

Authors: Scott T. Rickard, Radu V. Balan, H. Vincent Poor, Sergio Verdu

Abstract: Mobile communication channels are often modeled as linear time-varying filters or, equivalently, as time-frequency integral operators with finite support in time and frequency. Such a characterization inherently assumes the signals are narrowband and may not be appropriate for wideband signals. In this paper time-scale characterizations are examined that are useful in wideband time-varying chann… ▽ More Mobile communication channels are often modeled as linear time-varying filters or, equivalently, as time-frequency integral operators with finite support in time and frequency. Such a characterization inherently assumes the signals are narrowband and may not be appropriate for wideband signals. In this paper time-scale characterizations are examined that are useful in wideband time-varying channels, for which a time-scale integral operator is physically justifiable. A review of these time-frequency and time-scale characterizations is presented. Both the time-frequency and time-scale integral operators have a two-dimensional discrete characterization which motivates the design of time-frequency or time-scale rake receivers. These receivers have taps for both time and frequency (or time and scale) shifts of the transmitted signal. A general theory of these characterizations which generates, as specific cases, the discrete time-frequency and time-scale models is presented here. The interpretation of these models, namely, that they can be seen to arise from processing assumptions on the transmit and receive waveforms is discussed. Out of this discussion a third model arises: a frequency-scale continuous channel model with an associated discrete frequency-scale characterization. △ Less

Submitted 26 October, 2005; originally announced October 2005.

Comments: To appear in Communications in Information and Systems - special issue in honor of Thomas Kailath's seventieth birthday

ACM Class: E.4; C.2.1

On-Off Frequency-Shift-Keying for Wideband Fading Channels

Authors: Mustafa Cenk Gursoy, Sergio Verdu, H. Vincent Poor

Abstract: M-ary On-Off Frequency-Shift-Keying (OOFSK) is a digital modulation format in which M-ary FSK signaling is overlaid on On/Off keying. This paper investigates the potential of this modulation format in the context of wideband fading channels. First it is assumed that the receiver uses energy detection for the reception of OOFSK signals. Capacity expressions are obtained for the cases in which the… ▽ More M-ary On-Off Frequency-Shift-Keying (OOFSK) is a digital modulation format in which M-ary FSK signaling is overlaid on On/Off keying. This paper investigates the potential of this modulation format in the context of wideband fading channels. First it is assumed that the receiver uses energy detection for the reception of OOFSK signals. Capacity expressions are obtained for the cases in which the receiver has perfect and imperfect fading side information. Power efficiency is investigated when the transmitter is subject to a peak-to-average power ratio (PAR) limitation or a peak power limitation. It is shown that under a PAR limitation, it is extremely power inefficient to operate in the very low SNR regime. On the other hand, if there is only a peak power limitation, it is demonstrated that power efficiency improves as one operates with smaller SNR and vanishing duty factor. Also studied are the capacity improvements that accrue when the receiver can track phase shifts in the channel or if the received signal has a specular component. To take advantage of those features, the phase of the modulation is also allowed to carry information. △ Less

Submitted 24 October, 2005; originally announced October 2005.

Comments: To appear in the EURASIP Journal on Wireless Communications and Networking

Randomly Spread CDMA: Asymptotics via Statistical Physics

Authors: Dongning Guo, Sergio Verdu

Abstract: This paper studies randomly spread code-division multiple access (CDMA) and multiuser detection in the large-system limit using the replica method developed in statistical physics. Arbitrary input distributions and flat fading are considered. A generic multiuser detector in the form of the posterior mean estimator is applied before single-user decoding. The generic detector can be particularized… ▽ More This paper studies randomly spread code-division multiple access (CDMA) and multiuser detection in the large-system limit using the replica method developed in statistical physics. Arbitrary input distributions and flat fading are considered. A generic multiuser detector in the form of the posterior mean estimator is applied before single-user decoding. The generic detector can be particularized to the matched filter, decorrelator, linear MMSE detector, the jointly or the individually optimal detector, and others. It is found that the detection output for each user, although in general asymptotically non-Gaussian conditioned on the transmitted symbol, converges as the number of users go to infinity to a deterministic function of a "hidden" Gaussian statistic independent of the interferers. Thus the multiuser channel can be decoupled: Each user experiences an equivalent single-user Gaussian channel, whose signal-to-noise ratio suffers a degradation due to the multiple-access interference. The uncoded error performance (e.g., symbol-error-rate) and the mutual information can then be fully characterized using the degradation factor, also known as the multiuser efficiency, which can be obtained by solving a pair of coupled fixed-point equations identified in this paper. Based on a general linear vector channel model, the results are also applicable to MIMO channels such as in multiantenna systems. △ Less

Submitted 23 March, 2005; originally announced March 2005.

Comments: To be published in IEEE Transactions on Information Theory

The Noncoherent Rician Fading Channel -- Part II : Spectral Efficiency in the Low-Power Regime

Authors: Mustafa Cenk Gursoy, H. Vincent Poor, Sergio Verdu

Abstract: Transmission of information over a discrete-time memoryless Rician fading channel is considered where neither the receiver nor the transmitter knows the fading coefficients. The spectral-efficiency/bit-energy tradeoff in the low-power regime is examined when the input has limited peakedness. It is shown that if a fourth moment input constraint is imposed or the input peak-to-average power ratio… ▽ More Transmission of information over a discrete-time memoryless Rician fading channel is considered where neither the receiver nor the transmitter knows the fading coefficients. The spectral-efficiency/bit-energy tradeoff in the low-power regime is examined when the input has limited peakedness. It is shown that if a fourth moment input constraint is imposed or the input peak-to-average power ratio is limited, then in contrast to the behavior observed in average power limited channels, the minimum bit energy is not always achieved at zero spectral efficiency. The low-power performance is also characterized when there is a fixed peak limit that does not vary with the average power. A new signaling scheme that overlays phase-shift keying on on-off keying is proposed and shown to be optimally efficient in the low-power regime. △ Less

Submitted 24 January, 2005; originally announced January 2005.

Comments: To appear in the IEEE Transactions on Wireless Communications

The Noncoherent Rician Fading Channel -- Part I : Structure of the Capacity-Achieving Input

Authors: Mustafa Cenk Gursoy, H. Vincent Poor, Sergio Verdu

Abstract: Transmission of information over a discrete-time memoryless Rician fading channel is considered where neither the receiver nor the transmitter knows the fading coefficients. First the structure of the capacity-achieving input signals is investigated when the input is constrained to have limited peakedness by imposing either a fourth moment or a peak constraint. When the input is subject to secon… ▽ More Transmission of information over a discrete-time memoryless Rician fading channel is considered where neither the receiver nor the transmitter knows the fading coefficients. First the structure of the capacity-achieving input signals is investigated when the input is constrained to have limited peakedness by imposing either a fourth moment or a peak constraint. When the input is subject to second and fourth moment limitations, it is shown that the capacity-achieving input amplitude distribution is discrete with a finite number of mass points in the low-power regime. A similar discrete structure for the optimal amplitude is proven over the entire SNR range when there is only a peak power constraint. The Rician fading with phase-noise channel model, where there is phase uncertainty in the specular component, is analyzed. For this model it is shown that, with only an average power constraint, the capacity-achieving input amplitude is discrete with a finite number of levels. For the classical average power limited Rician fading channel, it is proven that the optimal input amplitude distribution has bounded support. △ Less

Submitted 24 January, 2005; originally announced January 2005.

Comments: To appear in the IEEE Transactions on Wireless Communications

Mutual Information and Minimum Mean-square Error in Gaussian Channels

Authors: Dongning Guo, Shlomo Shamai, Sergio Verdu

Abstract: This paper deals with arbitrarily distributed finite-power input signals observed through an additive Gaussian noise channel. It shows a new formula that connects the input-output mutual information and the minimum mean-square error (MMSE) achievable by optimal estimation of the input given the output. That is, the derivative of the mutual information (nats) with respect to the signal-to-noise r… ▽ More This paper deals with arbitrarily distributed finite-power input signals observed through an additive Gaussian noise channel. It shows a new formula that connects the input-output mutual information and the minimum mean-square error (MMSE) achievable by optimal estimation of the input given the output. That is, the derivative of the mutual information (nats) with respect to the signal-to-noise ratio (SNR) is equal to half the MMSE, regardless of the input statistics. This relationship holds for both scalar and vector signals, as well as for discrete-time and continuous-time noncausal MMSE estimation. This fundamental information-theoretic result has an unexpected consequence in continuous-time nonlinear estimation: For any input signal with finite power, the causal filtering MMSE achieved at SNR is equal to the average value of the noncausal smoothing MMSE achieved with a channel whose signal-to-noise ratio is chosen uniformly distributed between 0 and SNR. △ Less

Submitted 23 December, 2004; originally announced December 2004.