Showing 1–50 of 51 results for author: Charalambous, C D

Characterization of the Gray-Wyner Rate Region for Multivariate Gaussian Sources: Optimality of Gaussian Auxiliary RV

Authors: Evagoras Stylianou, Charalambos D. Charalambous, Jan H. van Schuppen

Abstract: Examined in this paper, is the Gray and Wyner achievable lossy rate region for a tuple of correlated multivariate Gaussian random variables (RVs) $X_1 : Ω\rightarrow {\mathbb R}^{p_1}$ and $X_2 : Ω\rightarrow {\mathbb R}^{p_2}$ with respect to square-error distortions at the two decoders. It is shown that among all joint distributions induced by a triple of RVs $(X_1,X_2, W)$, such that… ▽ More Examined in this paper, is the Gray and Wyner achievable lossy rate region for a tuple of correlated multivariate Gaussian random variables (RVs) $X_1 : Ω\rightarrow {\mathbb R}^{p_1}$ and $X_2 : Ω\rightarrow {\mathbb R}^{p_2}$ with respect to square-error distortions at the two decoders. It is shown that among all joint distributions induced by a triple of RVs $(X_1,X_2, W)$, such that $W : Ω\rightarrow {\mathbb W} $ is the auxiliary RV taking continuous, countable, or finite values, the Gray and Wyner achievable rate region is characterized by jointly Gaussian RVs $(X_1,X_2, W)$ such that $W $ is an $n$-dimensional Gaussian RV. It then follows that the achievable rate region is parametrized by the three conditional covariances $Q_{X_1,X_2|W}, Q_{X_1|W}, Q_{X_2|W}$ of the jointly Gaussian RVs. Furthermore, if the RV $W$ makes $X_1$ and $X_2$ conditionally independent, then the corresponding subset of the achievable rate region, is simpler, and parametrized by only the two conditional covariances $Q_{X_1|W}, Q_{X_2|W}$. The paper also includes the characterization of the Pangloss plane of the Gray-Wyner rate region along with the characterizations of the corresponding rate distortion functions, their test-channel distributions, and structural properties of the realizations which induce these distributions. △ Less

Submitted 16 May, 2022; originally announced May 2022.

On the Capacity of Gaussian MIMO Channels with Memory

Authors: Sergey Loyka, Charalambos D. Charalambous

Abstract: The operational capacity of Gaussian MIMO channels with memory was obtained by Brandenburg and Wyner in [9] under certain mild assumptions on the channel impulse response and its noise covariance matrix, which essentuially require channel memory to be not too strong. This channel was also considered by Tsybakov in [10] and its information capacity was obtained in some cases. It was further conject… ▽ More The operational capacity of Gaussian MIMO channels with memory was obtained by Brandenburg and Wyner in [9] under certain mild assumptions on the channel impulse response and its noise covariance matrix, which essentuially require channel memory to be not too strong. This channel was also considered by Tsybakov in [10] and its information capacity was obtained in some cases. It was further conjectured, based on numerical evidence, that these capacities are the same in all cases. This conjecture is proved here. An explicit closed-form expression for the optimal input power spectral density matrix is also given. The obtained result is further extended to the case of joint constraints, including per-antenna and interference power constraints as well as energy harvesting constraints. These results imply the information-theoretic optimality of OFDM-type transmission systems for such channels with memory. △ Less

Submitted 10 May, 2022; originally announced May 2022.

Comments: accepted by IEEE Comm. Letters

A Riccati-Lyapunov Approach to Nonfeedback Capacity of MIMO Gaussian Channels Driven by Stable and Unstable Noise

Authors: Charalambos D. Charalambous, Stelios Louka

Abstract: In this paper it is shown that the nonfeedback capacity of multiple-input multiple-output (MIMO) additive Gaussian noise (AGN) channels, when the noise is nonstationary and unstable, is characterized by an asymptotic optimization problem that involves, a generalized matrix algebraic Riccati equation (ARE) of filtering theory, and a matrix Lyapunov equation of stability theory of Gaussian systems.… ▽ More In this paper it is shown that the nonfeedback capacity of multiple-input multiple-output (MIMO) additive Gaussian noise (AGN) channels, when the noise is nonstationary and unstable, is characterized by an asymptotic optimization problem that involves, a generalized matrix algebraic Riccati equation (ARE) of filtering theory, and a matrix Lyapunov equation of stability theory of Gaussian systems. Furthermore, conditions are identified such that, the characterization of nonfeedback capacity corresponds to the uniform asymptotic per unit time limit, over all initial distributions, of the characterization of a finite block or transmission without feedback information (FTwFI) capacity, which involves, two generalized matrix difference Riccati equations (DREs) and a matrix difference Lyapunov equation. △ Less

Submitted 3 May, 2022; originally announced May 2022.

Comments: 6 pages

Structural Properties of Optimal Test Channels for Gaussian Multivariate Partially Observable Distributed Sources

Authors: Michail Gkagkos, Evagoras Stylianou, Charalambos D. Charalambous

Abstract: In this paper, we analyze the operational information rate distortion function (RDF) ${R}_{S;Z|Y}(Δ_X)$, introduced by Draper and Wornell, for a triple of jointly independent and identically distributed, multivariate Gaussian random variables (RVs), $(X^n, S^n, Y^n)= \{(X_{t}, S_t, Y_{t}): t=1,2, \ldots,n\}$, where $X^n$ is the source, $S^n$ is a measurement of $X^n$, available to the encoder,… ▽ More In this paper, we analyze the operational information rate distortion function (RDF) ${R}_{S;Z|Y}(Δ_X)$, introduced by Draper and Wornell, for a triple of jointly independent and identically distributed, multivariate Gaussian random variables (RVs), $(X^n, S^n, Y^n)= \{(X_{t}, S_t, Y_{t}): t=1,2, \ldots,n\}$, where $X^n$ is the source, $S^n$ is a measurement of $X^n$, available to the encoder, $Y^n$ is side information available to the decoder only, $Z^n$ is the auxiliary RV available to the decoder, with respect to the square-error fidelity, between the source $X^n$ and its reconstruction $\widehat{X}^n$. We also analyze the RDF ${R}_{S;\widehat{X}|Y}(Δ_X)$ that corresponds to the above set up, when side information $Y^n$ is available to the encoder and decoder. The main results include, (1) Structural properties of test channel realizations that induce distributions, which achieve the two RDFs, (2) Water-filling solutions of the two RDFs, based on parallel channel realizations of test channels, (3) A proof of equality ${R}_{S;Z|Y}(Δ_X) = {R}_{S;\widehat{X}|Y}(Δ_X)$, i.e., side information $Y^n$ at both the encoder and decoder does not incur smaller compression, and (4) Relations to other RDFs, as degenerate cases, which show past literature, contain oversights related to the optimal test channel realizations and value of the RDF ${R}_{S;Z|Y}(Δ_X)$. △ Less

Submitted 31 August, 2021; v1 submitted 30 August, 2021; originally announced August 2021.

On the Fragile Rates of Linear Feedback Coding Schemes of Gaussian Channels with Memory

Authors: Charalambos D. Charalambous, Christos Kourtellaris, Themistoklis Charalambous

Abstract: In \cite{butman1976} the linear coding scheme is applied, $X_t =g_t\Big(Θ- {\bf E}\Big\{Θ\Big|Y^{t-1}, V_0=v_0\Big\}\Big)$, $t=2,\ldots,n$, $X_1=g_1Θ$, with $Θ: Ω\to {\mathbb R}$, a Gaussian random variable, to derive a lower bound on the feedback rate, for additive Gaussian noise (AGN) channels, $Y_t=X_t+V_t, t=1, \ldots, n$, where $V_t$ is a Gaussian autoregressive (AR) noise, and… ▽ More In \cite{butman1976} the linear coding scheme is applied, $X_t =g_t\Big(Θ- {\bf E}\Big\{Θ\Big|Y^{t-1}, V_0=v_0\Big\}\Big)$, $t=2,\ldots,n$, $X_1=g_1Θ$, with $Θ: Ω\to {\mathbb R}$, a Gaussian random variable, to derive a lower bound on the feedback rate, for additive Gaussian noise (AGN) channels, $Y_t=X_t+V_t, t=1, \ldots, n$, where $V_t$ is a Gaussian autoregressive (AR) noise, and $κ\in [0,\infty)$ is the total transmitter power. For the unit memory AR noise, with parameters $(c, K_W)$, where $c\in [-1,1]$ is the pole and $K_W$ is the variance of the Gaussian noise, the lower bound is $C^{L,B} =\frac{1}{2} \log χ^2$, where $χ=\lim_{n\longrightarrow \infty} χ_n$ is the positive root of $χ^2=1+\Big(1+ \frac{|c|}χ\Big)^2 \fracκ{K_W}$, and the sequence $χ_n \triangleq \Big|\frac{g_n}{g_{n-1}}\Big|, n=2, 3, \ldots,$ satisfies a certain recursion, and conjectured that $C^{L,B}$ is the feedback capacity. In this correspondence, it is observed that the nontrivial lower bound $C^{L,B}=\frac{1}{2} \log χ^2$ such that $χ>1$, necessarily implies the scaling coefficients of the feedback code, $g_n$, $n=1,2, \ldots$, grow unbounded, in the sense that, $\lim_{n\longrightarrow\infty}|g_n| =+\infty$. The unbounded behaviour of $g_n$ follows from the ratio limit theorem of a sequence of real numbers, and it is verified by simulations. It is then concluded that such linear codes are not practical, and fragile with respect to a mismatch between the statistics of the mathematical model of the channel and the real statistics of the channel. In particular, if the error is perturbed by $ε_n>0$ no matter how small, then $X_n =g_t\Big(Θ- {\bf E}\Big\{Θ\Big|Y^{t-1}, V_0=v_0\Big\}\Big)+g_n ε_n$, and $|g_n|ε_n \longrightarrow \infty$, as $n \longrightarrow \infty$. △ Less

Submitted 16 June, 2021; originally announced June 2021.

Comments: 18 pages, 3 figures

On the Cover and Pombra Gaussian Feedback Capacity: Complete Sequential Characterizations via a Sufficient Statistic

Authors: Charalambos D. Charalambous, Christos Kourtellaris, Stelios Louka

Abstract: The main objective of this paper is to derive a new sequential characterization of the Cover and Pombra \cite{cover-pombra1989} characterization of the $n-$finite block or transmission feedback information ($n$-FTFI) capacity, which clarifies several issues of confusion and incorrect interpretation of results in literature. The optimal channel input processes of the new equivalent sequential chara… ▽ More The main objective of this paper is to derive a new sequential characterization of the Cover and Pombra \cite{cover-pombra1989} characterization of the $n-$finite block or transmission feedback information ($n$-FTFI) capacity, which clarifies several issues of confusion and incorrect interpretation of results in literature. The optimal channel input processes of the new equivalent sequential characterizations are expressed as functionals of a sufficient statistic and a Gaussian orthogonal innovations process. From the new representations follows that the Cover and Pombra characterization of the $n-$FTFI capacity is expressed as a functional of two generalized matrix difference Riccati equations (DRE) of filtering theory of Gaussian systems. This contradicts results which are redundant in the literature, and illustrates the fundamental complexity of the feedback capacity formula. △ Less

Submitted 9 June, 2021; originally announced June 2021.

Comments: 6 pages. arXiv admin note: substantial text overlap with arXiv:2010.06226

Feedback Capacity Formulas of AGN Channels Driven by Nonstationary Autoregressive Moving Average Noise

Authors: Stelios Louka, Christos Kourtellaris, Charalambos D. Charalambous

Abstract: In this paper we derive closed-form formulas of feedback capacity and nonfeedback achievable rates, for Additive Gaussian Noise (AGN) channels driven by nonstationary autoregressive moving average (ARMA) noise (with unstable one poles and zeros), based on time-invariant feedback codes and channel input distributions. From the analysis and simulations follows the surprising observations, (i) the us… ▽ More In this paper we derive closed-form formulas of feedback capacity and nonfeedback achievable rates, for Additive Gaussian Noise (AGN) channels driven by nonstationary autoregressive moving average (ARMA) noise (with unstable one poles and zeros), based on time-invariant feedback codes and channel input distributions. From the analysis and simulations follows the surprising observations, (i) the use of time-invariant channel input distributions gives rise to multiple regimes of capacity that depend on the parameters of the ARMA noise, which may or may not use feedback, (ii) the more unstable the pole (resp. zero) of the ARMA noise the higher (resp. lower) the feedback capacity, (iii) certain conditions, known as detectability and stabilizability are necessary and sufficient to ensure the feedback capacity formulas and nonfeedback achievable rates {\it are independent of the initial state of the ARMA noise}. Another surprizing observation is that Kim's \cite{kim2010} characterization of feedback capacity which is developed for stable ARMA noise, if applied to the unstable ARMA noise, gives a lower value of feedback capacity compared to our feedback capacity formula. △ Less

Submitted 9 June, 2021; originally announced June 2021.

Comments: 6 pages, 3 figures

Joint Nonanticipative Rate Distortion Function for a Tuple of Random Processes with Individual Fidelity Criteria

Authors: Charalambos D. Charalambous, Evagoras Stylianou

Abstract: The joint nonanticipative rate distortion function (NRDF) for a tuple of random processes with individual fidelity criteria is considered. Structural properties of optimal test channel distributions are derived. Further, for the application example of the joint NRDF of a tuple of jointly multivariate Gaussian Markov processes with individual square-error fidelity criteria, a realization of the rep… ▽ More The joint nonanticipative rate distortion function (NRDF) for a tuple of random processes with individual fidelity criteria is considered. Structural properties of optimal test channel distributions are derived. Further, for the application example of the joint NRDF of a tuple of jointly multivariate Gaussian Markov processes with individual square-error fidelity criteria, a realization of the reproduction processes which induces the optimal test channel distribution is derived, and the corresponding joint NRDF is characterized. The analysis of the simplest example, of a tuple of scalar correlated Markov processes, illustrates many of the challenging aspects of such problems. △ Less

Submitted 29 March, 2021; originally announced March 2021.

Joint Rate Distortion Function of a Tuple of Correlated Multivariate Gaussian Sources with Individual Fidelity Criteria

Authors: Evagoras Stylianou, Charalambos D. Charalambous, Themistoklis Charalambous

Abstract: In this paper we analyze the joint rate distortion function (RDF), for a tuple of correlated sources taking values in abstract alphabet spaces (i.e., continuous) subject to two individual distortion criteria. First, we derive structural properties of the realizations of the reproduction Random Variables (RVs), which induce the corresponding optimal test channel distributions of the joint RDF. Seco… ▽ More In this paper we analyze the joint rate distortion function (RDF), for a tuple of correlated sources taking values in abstract alphabet spaces (i.e., continuous) subject to two individual distortion criteria. First, we derive structural properties of the realizations of the reproduction Random Variables (RVs), which induce the corresponding optimal test channel distributions of the joint RDF. Second, we consider a tuple of correlated multivariate jointly Gaussian RVs, $X_1 : Ω\rightarrow {\mathbb R}^{p_1}, X_2 : Ω\rightarrow {\mathbb R}^{p_2}$ with two square-error fidelity criteria, and we derive additional structural properties of the optimal realizations, and use these to characterize the RDF as a convex optimization problem with respect to the parameters of the realizations. We show that the computation of the joint RDF can be performed by semidefinite programming. Further, we derive closed-form expressions of the joint RDF, such that Gray's [1] lower bounds hold with equality, and verify their consistency with the semidefinite programming computations. △ Less

Submitted 7 May, 2021; v1 submitted 14 February, 2021; originally announced February 2021.

Structural Properties of Optimal Test Channels for Distributed Source Coding with Decoder Side Information for Multivariate Gaussian Sources with Square-Error Fidelity

Authors: Michail Gkagkos, Charalambos D. Charalambous

Abstract: This paper focuses on the structural properties of test channels, of Wyner's operational information rate distortion function (RDF), $\overline{R}(Δ_X)$, of a tuple of multivariate correlated, jointly independent and identically distributed Gaussian random variables (RVs), $\{X_t, Y_t\}_{t=1}^\infty$, $X_t: Ω\rightarrow {\mathbb R}^{n_x}$, $Y_t: Ω\rightarrow {\mathbb R}^{n_y}$, with average mean-s… ▽ More This paper focuses on the structural properties of test channels, of Wyner's operational information rate distortion function (RDF), $\overline{R}(Δ_X)$, of a tuple of multivariate correlated, jointly independent and identically distributed Gaussian random variables (RVs), $\{X_t, Y_t\}_{t=1}^\infty$, $X_t: Ω\rightarrow {\mathbb R}^{n_x}$, $Y_t: Ω\rightarrow {\mathbb R}^{n_y}$, with average mean-square error at the decoder, $\frac{1}{n} {\bf E}\sum_{t=1}^n||X_t - \widehat{X}_t||^2\leq Δ_X$, when $\{Y_t\}_{t=1}^\infty$ is the side information available to the decoder only. We construct optimal test channel realizations, which achieve the informational RDF, $\overline{R}(Δ_X) \triangleq\inf_{{\cal M}(Δ_X)} I(X;Z|Y)$, where ${\cal M}(Δ_X)$ is the set of auxiliary RVs $Z$ such that, ${\bf P}_{Z|X,Y}={\bf P}_{Z|X}$, $\widehat{X}=f(Y,Z)$, and ${\bf E}\{||X-\widehat{X}||^2\}\leq Δ_X$. We show the fundamental structural properties: (1) Optimal test channel realizations that achieve the RDF, $\overline{R}(Δ_X)$, satisfy conditional independence, $ {\bf P}_{X|\widehat{X}, Y, Z}={\bf P}_{X|\widehat{X},Y}={\bf P}_{X|\widehat{X}}, \hspace{.2in} {\bf E}\Big\{X\Big|\widehat{X}, Y, Z\Big\}={\bf E}\Big\{X\Big|\widehat{X}\Big\}=\widehat{X} $ and (2) similarly for the conditional RDF, ${R}_{X|Y}(Δ_X) \triangleq \inf_{{\bf P}_{\widehat{X}|X,Y}:{\bf E}\{||X-\widehat{X}||^2\} \leq Δ_X} I(X; \widehat{X}|Y)$, when $\{Y_t\}_{t=1}^\infty$ is available to both the encoder and decoder, and the equality $\overline{R}(Δ_X)={R}_{X|Y}(Δ_X)$. △ Less

Submitted 22 November, 2020; originally announced November 2020.

New Formulas of Feedback Capacity for AGN Channels with Memory: A Time-Domain Sufficient Statistic Approach

Authors: Charalambos D. Charalambous, Christos Kourtellaris, Stelios Louka

Abstract: In the recent paper [1] it is shown, via an application example, that the Cover and Pombra [2] "characterization of the $n-$block or transmission" feedback capacity formula, of additive Gaussian noise (AGN) channels, is the subject of much confusion in the literature, with redundant incorrect results. The main objective of this paper is to clarify the main points of confusion and remove any furthe… ▽ More In the recent paper [1] it is shown, via an application example, that the Cover and Pombra [2] "characterization of the $n-$block or transmission" feedback capacity formula, of additive Gaussian noise (AGN) channels, is the subject of much confusion in the literature, with redundant incorrect results. The main objective of this paper is to clarify the main points of confusion and remove any further ambiguity. The first part of the paper applies time-domain methods, to derive for a first time, equivalent sequential characterizations of the Cover and Pombra characterization of feedback capacity of AGN channels driven by nonstationary and nonergodic Gaussian noise. The optimal channel input processes of the new equivalent sequential characterizations are expressed as functionals of a sufficient statistic and a Gaussian orthogonal innovations process. From the new representations follows that the Cover and Pombra $n-$block capacity formula is expressed as a functional of two generalized matrix difference Riccati equations (DRE) of filtering theory of Gaussian systems, contrary to results that appeared in the literature. In the second part of the paper the existence of the asymptotic limit of the $n-$block feedback capacity formula is shown to be equivalent to the convergence properties of solutions of the two generalized DREs. Further, necessary and or sufficient conditions are identified for existence of the asymptotic limits, for stable and unstable Gaussian noise, when the optimal input distributions are time-invariant, but not necessarily stationary. The paper contains an in depth analysis, with examples, of the specific technical issues, which are overlooked in past literature [3-7], that studied the AGN channel of [2], for stationary noises. △ Less

Submitted 13 October, 2020; originally announced October 2020.

MSC Class: 68P30; 94A15; 94A40 ACM Class: E.4

Characterization of Conditional Independence and Weak Realizations of Multivariate Gaussian Random Variables: Applications to Networks

Authors: Charalambos D. Charalambous, Jan H. van Schuppen

Abstract: The Gray and Wyner lossy source coding for a simple network for sources that generate a tuple of jointly Gaussian random variables (RVs) $X_1 : Ω\rightarrow {\mathbb R}^{p_1}$ and $X_2 : Ω\rightarrow {\mathbb R}^{p_2}$, with respect to square-error distortion at the two decoders is re-examined using (1) Hotelling's geometric approach of Gaussian RVs-the canonical variable form, and (2) van Putten'… ▽ More The Gray and Wyner lossy source coding for a simple network for sources that generate a tuple of jointly Gaussian random variables (RVs) $X_1 : Ω\rightarrow {\mathbb R}^{p_1}$ and $X_2 : Ω\rightarrow {\mathbb R}^{p_2}$, with respect to square-error distortion at the two decoders is re-examined using (1) Hotelling's geometric approach of Gaussian RVs-the canonical variable form, and (2) van Putten's and van Schuppen's parametrization of joint distributions ${\bf P}_{X_1, X_2, W}$ by Gaussian RVs $W : Ω\rightarrow {\mathbb R}^n $ which make $(X_1,X_2)$ conditionally independent, and the weak stochastic realization of $(X_1, X_2)$. Item (2) is used to parametrize the lossy rate region of the Gray and Wyner source coding problem for joint decoding with mean-square error distortions ${\bf E}\big\{||X_i-\hat{X}_i||_{{\mathbb R}^{p_i}}^2 \big\}\leq Δ_i \in [0,\infty], i=1,2$, by the covariance matrix of RV $W$. From this then follows Wyner's common information $C_W(X_1,X_2)$ (information definition) is achieved by $W$ with identity covariance matrix, while a formula for Wyner's lossy common information (operational definition) is derived, given by $C_{WL}(X_1,X_2)=C_W(X_1,X_2) = \frac{1}{2} \sum_{j=1}^n \ln \left( \frac{1+d_j}{1-d_j} \right),$ for the distortion region $ 0\leq Δ_1 \leq \sum_{j=1}^n(1-d_j)$, $0\leq Δ_2 \leq \sum_{j=1}^n(1-d_j)$, and where $1 > d_1 \geq d_2 \geq \ldots \geq d_n>0$ in $(0,1)$ are {\em the canonical correlation coefficients} computed from the canonical variable form of the tuple $(X_1, X_2)$. The methods are of fundamental importance to other problems of multi-user communication, where conditional independence is imposed as a constraint. △ Less

Submitted 19 January, 2020; originally announced January 2020.

Comments: 6 pages

Time-Invariant Feedback Strategies Do Not Increase Capacity of AGN Channels Driven by Stable and Certain Unstable Autoregressive Noise

Authors: Charalambos D. Charalambous, Christos Kourtellaris, Sergey Loyka

Abstract: The capacity of additive Gaussian noise (AGN) channels, $Y_t=X_t+V_t, t=1, \ldots, n$, $\frac{1}{n} {\bf E}\big\{\sum_{t=1}^n |X_t|^2 \big\}\leq κ, κ\in [0,\infty)$, with time-invariant channel input feedback strategies, is characterized and conditions are identified for entropy rates, and limit of average power to exist, when the noise is described by {\it stable and unstable} autoregressive mode… ▽ More The capacity of additive Gaussian noise (AGN) channels, $Y_t=X_t+V_t, t=1, \ldots, n$, $\frac{1}{n} {\bf E}\big\{\sum_{t=1}^n |X_t|^2 \big\}\leq κ, κ\in [0,\infty)$, with time-invariant channel input feedback strategies, is characterized and conditions are identified for entropy rates, and limit of average power to exist, when the noise is described by {\it stable and unstable} autoregressive models, AR$(c)$, $V_t=cV_{t-1}+ W_t, V_0=v_0, t=1, \ldots, n$, where $c\in (-\infty,\infty)$, $W_t, t=1,\ldots, n$, is a zero mean, variance $K_W$, independent Gaussian sequence, independent of $V_0$. For stable AR$(c), c\in (-1,1)$ the conditions are necessary and sufficient for asymptotic stationarity of the processes $(X_t, Y_t), t=1, 2, \ldots$. New closed form capacity formulas and lower bounds are derived, for the AR$(c), c\in (-\infty,\infty)$ noise, which are fundamentally different from existing formulas in the literature, and illustrate multiple regimes of capacity, as a function of the parameters $(c,K_W,κ)$, as follows.\\ 1) feedback increases capacity for the regime, $c^2 \in (1, \infty),$ for $κ> \frac{K_W\big(1+\sqrt{4c^2-3}\big)}{2\big(c^2-1\big)^2}$, \\ 2) feedback does not increase capacity for the regime $c^2 \in (1, \infty)$, for $κ\leq \frac{K_W\big(1+\sqrt{4c^2-3}\big)}{2\big(c^2-1\big)^2}$, and \\ 3) feedback does not increase capacity for the regime $c \in [-1,1]$, for $ κ\in [0,\infty)$. △ Less

Submitted 13 August, 2021; v1 submitted 25 July, 2019; originally announced July 2019.

Comments: submitted to IEEE Transactions on Information Theory (IT-19-0520)

A New Approach to Lossy Network Compression of a Tuple of Correlated Multivariate Gaussian RVs

Authors: Charalambos D. Charalambous, Jan H. van Schuppen

Abstract: The classical Gray and Wyner source coding for a simple network for sources that generate a tuple of multivariate, correlated Gaussian random variables $(Y_1,Y_2)$ is re-examined using the geometric approach of Gaussian random variables, and the weak stochastic realization of correlated Gaussian random variables. New results are: (1) The formulation, methods and algorithms to parametrize all rando… ▽ More The classical Gray and Wyner source coding for a simple network for sources that generate a tuple of multivariate, correlated Gaussian random variables $(Y_1,Y_2)$ is re-examined using the geometric approach of Gaussian random variables, and the weak stochastic realization of correlated Gaussian random variables. New results are: (1) The formulation, methods and algorithms to parametrize all random variables $W : Ω\rightarrow {\mathbb R}^n $ which make the two components of the tuple $(Y_1,Y_2)$ conditionally independent, according to the weak stochastic realization of $(Y_1, Y_2)$. (2) The transformation of random variables $(Y_1,Y_2)$ via non-singular transformations $(S_1,S_2)$, into their canonical variable form. (3) A formula for Wyner's lossy common information for joint decoding with mean-square error distortions. (4) The methods are shown to be of fundamental importance to the parametrization of the lossy rate region of the Gray and Wyner source coding problem, and the calculation of the smallest common message rate $R_0$ on the Gray and Wyner source problem, when the sum rate $R_0+R_1+R_2$ is arbitrary close to the joint rate distortion function $R_{Y_1, Y_2}(Δ_1, Δ_2)$ of joint decoding. The methods and algorithms may be applicable to other problems of multi-user communication, such as, the multiple access channel, etc. The discussion is largely self-contained and proceeds from first principles; basic concepts of weak stochastic realization theory of multivariate correlated Gaussian random variables are reviewed, while certain results are developed to meet the requirement of results (1)-(4). △ Less

Submitted 14 February, 2021; v1 submitted 29 May, 2019; originally announced May 2019.

Comments: 49 pages, 1 figure

MSC Class: 94-A15; 94-A34

Zero-Delay Rate Distortion via Filtering for Vector-Valued Gaussian Sources

Authors: Photios A. Stavrou, Jan Ostergaard, Charalambos D. Charalambous

Abstract: We deal with zero-delay source coding of a vector-valued Gauss-Markov source subject to a mean-squared error (MSE) fidelity criterion characterized by the operational zero-delay vector-valued Gaussian rate distortion function (RDF). We address this problem by considering the nonanticipative RDF (NRDF) which is a lower bound to the causal optimal performance theoretically attainable (OPTA) function… ▽ More We deal with zero-delay source coding of a vector-valued Gauss-Markov source subject to a mean-squared error (MSE) fidelity criterion characterized by the operational zero-delay vector-valued Gaussian rate distortion function (RDF). We address this problem by considering the nonanticipative RDF (NRDF) which is a lower bound to the causal optimal performance theoretically attainable (OPTA) function and operational zero-delay RDF. We recall the realization that corresponds to the optimal "test-channel" of the Gaussian NRDF, when considering a vector Gauss-Markov source subject to a MSE distortion in the finite time horizon. Then, we introduce sufficient conditions to show existence of solution for this problem in the infinite time horizon. For the asymptotic regime, we use the asymptotic characterization of the Gaussian NRDF to provide a new equivalent realization scheme with feedback which is characterized by a resource allocation (reverse-waterfilling) problem across the dimension of the vector source. We leverage the new realization to derive a predictive coding scheme via lattice quantization with subtractive dither and joint memoryless entropy coding. This coding scheme offers an upper bound to the operational zero-delay vector-valued Gaussian RDF. When we use scalar quantization, then for "r" active dimensions of the vector Gauss-Markov source the gap between the obtained lower and theoretical upper bounds is less than or equal to 0.254r + 1 bits/vector. We further show that it is possible when we use vector quantization, and assume infinite dimensional Gauss-Markov sources to make the previous gap to be negligible, i.e., Gaussian NRDF approximates the operational zero-delay Gaussian RDF. We also extend our results to vector-valued Gaussian sources of any finite memory under mild conditions. Our theoretical framework is demonstrated with illustrative numerical experiments. △ Less

Submitted 29 September, 2018; originally announced October 2018.

Comments: 32 pages, 9 figures, published in IEEE Journal of Selected Topics in Signal Processing

An Upper Bound to Zero-Delay Rate Distortion via Kalman Filtering for Vector Gaussian Sources

Authors: Photios A. Stavrou, Jan Ostergaard, Charalambos D. Charalambous, Milan Derpich

Abstract: We deal with zero-delay source coding of a vector Gaussian autoregressive (AR) source subject to an average mean squared error (MSE) fidelity criterion. Toward this end, we consider the nonanticipative rate distortion function (NRDF) which is a lower bound to the causal and zero-delay rate distortion function (RDF). We use the realization scheme with feedback proposed in [1] to model the correspon… ▽ More We deal with zero-delay source coding of a vector Gaussian autoregressive (AR) source subject to an average mean squared error (MSE) fidelity criterion. Toward this end, we consider the nonanticipative rate distortion function (NRDF) which is a lower bound to the causal and zero-delay rate distortion function (RDF). We use the realization scheme with feedback proposed in [1] to model the corresponding optimal "test-channel" of the NRDF, when considering vector Gaussian AR(1) sources subject to an average MSE distortion. We give conditions on the vector Gaussian AR(1) source to ensure asymptotic stationarity of the realization scheme (bounded performance). Then, we encode the vector innovations due to Kalman filtering via lattice quantization with subtractive dither and memoryless entropy coding. This coding scheme provides a tight upper bound to the zero-delay Gaussian RDF. We extend this result to vector Gaussian AR sources of any finite order. Further, we show that for infinite dimensional vector Gaussian AR sources of any finite order, the NRDF coincides with the zero-delay RDF. Our theoretical framework is corroborated with a simulation example. △ Less

Submitted 21 August, 2017; v1 submitted 23 January, 2017; originally announced January 2017.

Comments: 7 pages, 6 figures, accepted for publication in IEEE Information Theory Workshop (ITW)

Single Letter Expression of Capacity for a Class of Channels with Memory

Authors: Christos K. Kourtellaris, Charalambos D. Charalambous, Ioannis Tzortzis

Abstract: We study finite alphabet channels with Unit Memory on the previous Channel Outputs called UMCO channels. We identify necessary and sufficient conditions, to test whether the capacity achieving channel input distributions with feedback are time-invariant, and whether feedback capacity is characterized by single letter, expressions, similar to that of memoryless channels. The method is based on show… ▽ More We study finite alphabet channels with Unit Memory on the previous Channel Outputs called UMCO channels. We identify necessary and sufficient conditions, to test whether the capacity achieving channel input distributions with feedback are time-invariant, and whether feedback capacity is characterized by single letter, expressions, similar to that of memoryless channels. The method is based on showing that a certain dynamic programming equation, which in general, is a nested optimization problem over the sequence of channel input distributions, reduces to a non-nested optimization problem. Moreover, for UMCO channels, we give a simple expression for the ML error exponent, and we identify sufficient conditions to test whether feedback does not increase capacity. We derive similar results, when transmission cost constraints are imposed. We apply the results to a special class of the UMCO channels, the Binary State Symmetric Channel (BSSC) with and without transmission cost constraints, to show that the optimization problem of feedback capacity is non-nested, the capacity achieving channel input distribution and the corresponding channel output transition probability distribution are time-invariant, and feedback capacity is characterized by a single letter formulae, precisely as Shannon's single letter characterization of capacity of memoryless channels. Then we derive closed form expressions for the capacity achieving channel input distribution and feedback capacity. We use the closed form expressions to evaluate an error exponent for ML decoding. △ Less

Submitted 4 January, 2017; originally announced January 2017.

Comments: submitted to IEEE Transactions on Information Theory, Paper no. IT-16-0909

Feedback Does Not Increase the Capacity of Compound Channels with Additive Noise

Authors: Sergey Loyka, Charalambos D. Charalambous

Abstract: A discrete compound channel with memory is considered, where no stationarity, ergodicity or information stability is required, and where the uncertainty set can be arbitrary. When the discrete noise is additive but otherwise arbitrary and there is no cost constraint on the input, it is shown that the causal feedback does not increase the capacity. This extends the earlier result obtained for gener… ▽ More A discrete compound channel with memory is considered, where no stationarity, ergodicity or information stability is required, and where the uncertainty set can be arbitrary. When the discrete noise is additive but otherwise arbitrary and there is no cost constraint on the input, it is shown that the causal feedback does not increase the capacity. This extends the earlier result obtained for general single-state channels with full transmitter (Tx) channel state information (CSI) to the compound setting. It is further shown that, for this compound setting and under a mild technical condition on the additive noise, the addition of the full Tx CSI does not increase the capacity either, so that the worst-case and compound channel capacities are the same. This can also be expressed as a saddle-point in the information-theoretic game between the transmitter (who selects the input distribution) and the nature (who selects the channel state), even though the objective function (the inf-information rate) is not convex/concave in the right way. Cases where the Tx CSI does increase the capacity are identified. Conditions under which the strong converse holds for this channel are studied. The ergodic behaviour of the worst-case noise in otherwise information-unstable channel is shown to be both sufficient and necessary for the strong converse to hold, including feedback and no feedback cases. △ Less

Submitted 20 December, 2016; originally announced December 2016.

Comments: submitted to IEEE Info. Theory Transactions

Optimal Signaling for Secure Communications over Gaussian MIMO Wiretap Channels

Authors: Sergey Loyka, Charalambos D. Charalambous

Abstract: Optimal signalling over the Gaussian MIMO wire-tap channel is studied under the total transmit power constraint. A closed-form solution for an optimal transmit covariance matrix is obtained when the channel is strictly degraded. In combination with the rank-1 solution, this provides the complete characterization of the optimal covariance for the case of two transmit antennas. The cases of weak eav… ▽ More Optimal signalling over the Gaussian MIMO wire-tap channel is studied under the total transmit power constraint. A closed-form solution for an optimal transmit covariance matrix is obtained when the channel is strictly degraded. In combination with the rank-1 solution, this provides the complete characterization of the optimal covariance for the case of two transmit antennas. The cases of weak eavesdropper and high SNR are considered. It is shown that the optimal covariance does not converge to a scaled identity in the high-SNR regime. Necessary optimality conditions and a tight upper bound on the rank of an optimal covariance matrix are established for the general case, along with a lower bound to the secrecy capacity, which is tight in a number of scenarios. △ Less

Submitted 31 October, 2016; originally announced November 2016.

Comments: accepted by IEEE Trans. Info. Theory

Rank-Deficient Solutions for Optimal Signaling over Wiretap MIMO Channels

Authors: Sergey Loyka, Charalambos D. Charalambous

Abstract: Capacity-achieving signaling strategies for the Gaussian wiretap MIMO channel are investigated without the degradedness assumption. In addition to known solutions, a number of new rank-deficient solutions for the optimal transmit covariance matrix are obtained. The case of a weak eavesdropper is considered in detail and the optimal covariance is established in an explicit, closed form with no extr… ▽ More Capacity-achieving signaling strategies for the Gaussian wiretap MIMO channel are investigated without the degradedness assumption. In addition to known solutions, a number of new rank-deficient solutions for the optimal transmit covariance matrix are obtained. The case of a weak eavesdropper is considered in detail and the optimal covariance is established in an explicit, closed form with no extra assumptions. This provides lower and upper bounds to the secrecy capacity in the general case with a bounded gap, which are tight for a weak eavesdropper or/and low SNR. Closed form solutions are also obtained for isotropic and omnidirectional eavesdroppers, based on which lower and upper bounds to the secrecy capacity are established in the general case. Sufficient and necessary conditions for optimality of 3 popular transmission techniques, namely the zero-forcing (ZF), the standard water-filling (WF) over the channel eigenmodes and the isotropic signaling (IS), are established for the MIMO wiretap channel. These solutions are appealing due to their lower complexity. In particular, no wiretap codes are needed for the ZF transmission, and no precoding or feedback is needed for the isotropic signaling. △ Less

Submitted 22 April, 2016; originally announced April 2016.

Comments: accepted by IEEE Trans. Communications

Sequential Necessary and Sufficient Conditions for Capacity Achieving Distributions of Channels with Memory and Feedback

Authors: Photios A. Stavrou, Charalambos D. Charalambous, Christos K. Kourtellaris

Abstract: We derive sequential necessary and sufficient conditions for any channel input conditional distribution ${\cal P}_{0,n}\triangleq\{P_{X_t|X^{t-1},Y^{t-1}}:~t=0,\ldots,n\}$ to maximize the finite-time horizon directed information defined by $$C^{FB}_{X^n \rightarrow Y^n} \triangleq \sup_{{\cal P}_{0,n}} I(X^n\rightarrow{Y^n}),~~~ I(X^n \rightarrow Y^n) =\sum_{t=0}^n{I}(X^t;Y_t|Y^{t-1})$$ for channe… ▽ More We derive sequential necessary and sufficient conditions for any channel input conditional distribution ${\cal P}_{0,n}\triangleq\{P_{X_t|X^{t-1},Y^{t-1}}:~t=0,\ldots,n\}$ to maximize the finite-time horizon directed information defined by $$C^{FB}_{X^n \rightarrow Y^n} \triangleq \sup_{{\cal P}_{0,n}} I(X^n\rightarrow{Y^n}),~~~ I(X^n \rightarrow Y^n) =\sum_{t=0}^n{I}(X^t;Y_t|Y^{t-1})$$ for channel distributions $\{P_{Y_t|Y^{t-1},X_t}:~t=0,\ldots,n\}$ and $\{P_{Y_t|Y_{t-M}^{t-1},X_t}:~t=0,\ldots,n\}$, where $Y^t\triangleq\{Y_0,\ldots,Y_t\}$ and $X^t\triangleq\{X_0,\ldots,X_t\}$ are the channel input and output random processes, and $M$ is a finite nonnegative integer. \noi We apply the necessary and sufficient conditions to application examples of time-varying channels with memory and we derive recursive closed form expressions of the optimal distributions, which maximize the finite-time horizon directed information. Further, we derive the feedback capacity from the asymptotic properties of the optimal distributions by investigating the limit $$C_{X^\infty \rightarrow Y^\infty}^{FB} \triangleq \lim_{n \longrightarrow \infty} \frac{1}{n+1} C_{X^n \rightarrow Y^n}^{FB}$$ without any á priori assumptions, such as, stationarity, ergodicity or irreducibility of the channel distribution. The necessary and sufficient conditions can be easily extended to a variety of channels with memory, beyond the ones considered in this paper. △ Less

Submitted 18 April, 2016; v1 submitted 10 April, 2016; originally announced April 2016.

Comments: 57 pages, 9 figures, part of the paper was accepted for publication in the proceedings of the IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain 10-15 July, 2016 (Date of submission of the conference paper: 25/1/2016)

A General Formula for Compound Channel Capacity

Authors: Sergey Loyka, Charalambos D. Charalambous

Abstract: A general formula for the capacity of arbitrary compound channels with the receiver channel state information is obtained using the information density approach. No assumptions of ergodicity, stationarity or information stability are made and the channel state set is arbitrary. A direct (constructive) proof is given. To prove achievability, we generalize Feinstein Lemma to the compound channel set… ▽ More A general formula for the capacity of arbitrary compound channels with the receiver channel state information is obtained using the information density approach. No assumptions of ergodicity, stationarity or information stability are made and the channel state set is arbitrary. A direct (constructive) proof is given. To prove achievability, we generalize Feinstein Lemma to the compound channel setting, and to prove converse, we generalize Verdu-Han Lemma to the same compound setting. A notion of a uniform compound channel is introduced and the general formula is shown to reduce to the familiar $\sup-\inf$ expression for such channels. As a by-product, the arbitrary varying channel capacity is established under maximum error probability and deterministic coding. Conditions are established under which the worst-case and compound channel capacities are equal so that the full channel state information at the transmitter brings in no advantage. The compound inf-information rate plays a prominent role in the general formula. Its properties are studied and a link between information-unstable and information-stable regimes of a compound channel is established. The results are extended to include $\varepsilon$-capacity of compound channels. Sufficient and necessary conditions for the strong converse to hold are given. △ Less

Submitted 5 April, 2016; originally announced April 2016.

Comments: accepted by IEEE Transactions on Information Theory

Information Structures of Maximizing Distributions of Feedback Capacity for General Channels with Memory & Applications

Authors: Charalambos D. Charalambous, Christos K. Kourtellaris

Abstract: For any class of channel conditional distributions, with finite memory dependence on channel input RVs $A^n {\stackrel{\triangle}{=}} \{A_i: i=0, \ldots, n\}$ or channel output RVs $B^n {\stackrel{\triangle}{=}} \{B_i: i=0, \ldots, n\}$ or both, we characterize the sets of channel input distributions, which maximize directed information defined by… ▽ More For any class of channel conditional distributions, with finite memory dependence on channel input RVs $A^n {\stackrel{\triangle}{=}} \{A_i: i=0, \ldots, n\}$ or channel output RVs $B^n {\stackrel{\triangle}{=}} \{B_i: i=0, \ldots, n\}$ or both, we characterize the sets of channel input distributions, which maximize directed information defined by $ I(A^n \rightarrow B^n) {\stackrel{\triangle}{=}} \sum_{i=0}^n I(A^i;B_i|B^{i-1}) $ and we derive the corresponding expressions, called "characterizations of Finite Transmission Feedback Information (FTFI) capacity". The main theorems state that optimal channel input distributions occur in subsets ${\cal P}_{[0,n]}^{CI}\subseteq {\cal P}_{[0,n]} {\stackrel{\triangle}{=}}\big\{ {\bf P}_{A_i|A^{i-1}, B^{i-1}}: i=0, \ldots, n\big\}$, which satisfy conditional independence on past information. We derive similar characterizations, when general transmission cost constraints are imposed. Moreover, we also show that the structural properties apply to general nonlinear and linear autoregressive channel models defined by discrete-time recursions on general alphabet spaces, and driven by arbitrary distributed noise processes. We derive these structural properties by invoking stochastic optimal control theory and variational equalities of directed information, to identify tight upper bounds on $I(A^n \rightarrow B^n)$, which are achievable over subsets of conditional distributions ${\cal P}_{[0,n]}^{CI} \subseteq {\cal P}_{[0,n]}$, which satisfy conditional independence and they are specified by the dependence of channel distributions and transmission cost functions on inputs and output symbols. We apply the characterizations to recursive Multiple Input Multiple Output Gaussian Linear Channel Models with limited memory and we show a separation principle between the computation of the elements of the optimal strategies. △ Less

Submitted 19 August, 2016; v1 submitted 4 April, 2016; originally announced April 2016.

Comments: Submitted to the IEEE Transactions on Information Theory,Feb. 2016, paper IT-16-0120

Capacity Achieving Distributions & Information Lossless Randomized Strategies for Feedback Channels with Memory: The LQG Theory of Directed Information-Part II

Authors: Charalambos D. Charalambous, Christos K. Kourtellaris, Sergey Loyka

Abstract: A methodology is developed to realized optimal channel input conditional distributions, which maximize the finite-time horizon directed information, for channels with memory and feedback, by information lossless randomized strategies. The methodology is applied to general Time-Varying Multiple Input Multiple Output (MIMO) Gaussian Linear Channel Models (G-LCMs) with memory, subject to average tran… ▽ More A methodology is developed to realized optimal channel input conditional distributions, which maximize the finite-time horizon directed information, for channels with memory and feedback, by information lossless randomized strategies. The methodology is applied to general Time-Varying Multiple Input Multiple Output (MIMO) Gaussian Linear Channel Models (G-LCMs) with memory, subject to average transmission cost constraints of quadratic form. The realizations of optimal distributions by randomized strategies are shown to exhibit a decomposion into a deterministic part and a random part. The decomposition reveals the dual role of randomized strategies, to control the channel output process and to transmit new information over the channels. Moreover, a separation principle is shown between the computation of the optimal deterministic part and the random part of the randomized strategies. The dual role of randomized strategies generalizes the Linear-Quadratic-Gaussian (LQG) stochastic optimal control theory to directed information pay-offs. The characterizations of feedback capacity are obtained from the per unit time limits of finite-time horizon directed information, without imposing á priori assumptions, such as, stability of channel models or ergodicity of channel input and output processes. For time-invariant MIMO G-LCMs with memory, it is shown that whether feedback increases capacity, is directly related to the channel parameters and the transmission cost function, through the solutions of Riccati matrix equations, and moreover for unstable channels, feedback capacity is non-zero, provided the power exceeds a critical level. △ Less

Submitted 4 April, 2016; originally announced April 2016.

Comments: Submitted to the IEEE Transactions on Information Theory, Mar. 2016, paper IT-16-0166

Optimal Estimation via Nonanticipative Rate Distortion Function and Applications to Time-Varying Gauss-Markov Processes

Authors: Photios A. Stavrou, Themistoklis Charalambous, Charalambos D. Charalambous, Sergey Loyka

Abstract: In this paper, we develop {finite-time horizon} causal filters using the nonanticipative rate distortion theory. We apply the {developed} theory to {design optimal filters for} time-varying multidimensional Gauss-Markov processes, subject to a mean square error fidelity constraint. We show that such filters are equivalent to the design of an optimal \texttt{\{encoder, channel, decoder\}}, which en… ▽ More In this paper, we develop {finite-time horizon} causal filters using the nonanticipative rate distortion theory. We apply the {developed} theory to {design optimal filters for} time-varying multidimensional Gauss-Markov processes, subject to a mean square error fidelity constraint. We show that such filters are equivalent to the design of an optimal \texttt{\{encoder, channel, decoder\}}, which ensures that the error satisfies {a} fidelity constraint. Moreover, we derive a universal lower bound on the mean square error of any estimator of time-varying multidimensional Gauss-Markov processes in terms of conditional mutual information. Unlike classical Kalman filters, the filter developed is characterized by a reverse-waterfilling algorithm, which ensures {that} the fidelity constraint is satisfied. The theoretical results are demonstrated via illustrative examples. △ Less

Submitted 10 February, 2017; v1 submitted 14 March, 2016; originally announced March 2016.

Comments: 35 pages, 6 figures, submitted for publication in SIAM Journal on Control and Optimization (SICON)

Information Structures for Feedback Capacity of Channels with Memory and Transmission Cost: Stochastic Optimal Control & Variational Equalities-Part I

Authors: Christos K. Kourtellaris, Charalambos D. Charalambous

Abstract: The Finite Transmission Feedback Information (FTFI) capacity is characterized for any class of channel conditional distributions $\big\{{\bf P}_{B_i|B^{i-1}, A_i} :i=0, 1, \ldots, n\big\}$ and $\big\{ {\bf P}_{B_i|B_{i-M}^{i-1}, A_i} :i=0, 1, \ldots, n\big\}$, where $M$ is the memory of the channel, $B^n {\stackrel{\triangle}{=}} \{B_j: j=\ldots, 0,1, \ldots, n\}$ are the channel outputs and… ▽ More The Finite Transmission Feedback Information (FTFI) capacity is characterized for any class of channel conditional distributions $\big\{{\bf P}_{B_i|B^{i-1}, A_i} :i=0, 1, \ldots, n\big\}$ and $\big\{ {\bf P}_{B_i|B_{i-M}^{i-1}, A_i} :i=0, 1, \ldots, n\big\}$, where $M$ is the memory of the channel, $B^n {\stackrel{\triangle}{=}} \{B_j: j=\ldots, 0,1, \ldots, n\}$ are the channel outputs and $A^n{\stackrel{\triangle}{=}} \{A_j: j=\ldots, 0,1, \ldots, n\}$ are the channel inputs. The characterizations of FTFI capacity, are obtained by first identifying the information structures of the optimal channel input conditional distributions ${\cal P}_{[0,n]} {\stackrel{\triangle}{=}} \big\{ {\bf P}_{A_i|A^{i-1}, B^{i-1}}: i=0, \ldots, n\big\}$, which maximize directed information. The main theorem states, for any channel with memory $M$, the optimal channel input conditional distributions occur in the subset satisfying conditional independence $\stackrel{\circ}{\cal P}_{[0,n]}{\stackrel{\triangle}{=}} \big\{ {\bf P}_{A_i|A^{i-1}, B^{i-1}}= {\bf P}_{A_i|B_{i-M}^{i-1}}: i=1, \ldots, n\big\}$, and the characterization of FTFI capacity is given by $C_{A^n \rightarrow B^n}^{FB, M} {\stackrel{\triangle}{=}} \sup_{ \stackrel{\circ}{\cal P}_{[0,n]} } \sum_{i=0}^n I(A_i; B_i|B_{i-M}^{i-1}) $. The methodology utilizes stochastic optimal control theory and a variational equality of directed information, to derive upper bounds on $I(A^n \rightarrow B^n)$, which are achievable over specific subsets of channel input conditional distributions ${\cal P}_{[0,n]}$, which are characterized by conditional independence. For any of the above classes of channel distributions and transmission cost functions, a direct analogy, in terms of conditional independence, of the characterizations of FTFI capacity and Shannon's capacity formulae of Memoryless Channels is identified. △ Less

Submitted 18 August, 2016; v1 submitted 14 December, 2015; originally announced December 2015.

Comments: Submitted to IEEE Transactions on Information Theory, IT-15-0963

An Algorithm for Global Maximization of Secrecy Rates in Gaussian MIMO Wiretap Channels

Authors: Sergey Loyka, Charalambos D. Charalambous

Abstract: Optimal signaling for secrecy rate maximization in Gaussian MIMO wiretap channels is considered. While this channel has attracted a significant attention recently and a number of results have been obtained, including the proof of the optimality of Gaussian signalling, an optimal transmit covariance matrix is known for some special cases only and the general case remains an open problem. An iterati… ▽ More Optimal signaling for secrecy rate maximization in Gaussian MIMO wiretap channels is considered. While this channel has attracted a significant attention recently and a number of results have been obtained, including the proof of the optimality of Gaussian signalling, an optimal transmit covariance matrix is known for some special cases only and the general case remains an open problem. An iterative custom-made algorithm to find a globally-optimal transmit covariance matrix in the general case is developed in this paper, with guaranteed convergence to a \textit{global} optimum. While the original optimization problem is not convex and hence difficult to solve, its minimax reformulation can be solved via the convex optimization tools, which is exploited here. The proposed algorithm is based on the barrier method extended to deal with a minimax problem at hand. Its convergence to a global optimum is proved for the general case (degraded or not) and a bound for the optimality gap is given for each step of the barrier method. The performance of the algorithm is demonstrated via numerical examples. In particular, 20 to 40 Newton steps are already sufficient to solve the sufficient optimality conditions with very high precision (up to the machine precision level), even for large systems. Even fewer steps are required if the secrecy capacity is the only quantity of interest. The algorithm can be significantly simplified for the degraded channel case and can also be adopted to include the per-antenna power constraints (instead or in addition to the total power constraint). It also solves the dual problem of minimizing the total power subject to the secrecy rate constraint. △ Less

Submitted 14 April, 2015; originally announced April 2015.

Comments: accepted by IEEE Transactions on Communications

Information Nonanticipative Rate Distortion Function and Its Applications

Authors: Photios A. Stavrou, Christos K. Kourtellaris, C. D. Charalambous

Abstract: This paper investigates applications of nonanticipative Rate Distortion Function (RDF) in a) zero-delay Joint Source-Channel Coding (JSCC) design based on average and excess distortion probability, b) in bounding the Optimal Performance Theoretically Attainable (OPTA) by noncausal and causal codes, and computing the Rate Loss (RL) of zero-delay and causal codes with respect to noncausal codes. The… ▽ More This paper investigates applications of nonanticipative Rate Distortion Function (RDF) in a) zero-delay Joint Source-Channel Coding (JSCC) design based on average and excess distortion probability, b) in bounding the Optimal Performance Theoretically Attainable (OPTA) by noncausal and causal codes, and computing the Rate Loss (RL) of zero-delay and causal codes with respect to noncausal codes. These applications are described using two running examples, the Binary Symmetric Markov Source with parameter p, (BSMS(p)) and the multidimensional partially observed Gaussian-Markov source. For the multidimensional Gaussian-Markov source with square error distortion, the solution of the nonanticipative RDF is derived, its operational meaning using JSCC design via a noisy coding theorem is shown by providing the optimal encoding-decoding scheme over a vector Gaussian channel, and the RL of causal and zero-delay codes with respect to noncausal codes is computed. For the BSMS(p) with Hamming distortion, the solution of the nonanticipative RDF is derived, the RL of causal codes with respect to noncausal codes is computed, and an uncoded noisy coding theorem based on excess distortion probability is shown. The information nonanticipative RDF is shown to be equivalent to the nonanticipatory epsilon-entropy, which corresponds to the classical RDF with an additional causality or nonanticipative condition imposed on the optimal reproduction conditional distribution. △ Less

Submitted 18 November, 2015; v1 submitted 7 May, 2014; originally announced May 2014.

Comments: 34 pages, 12 figures, part of this paper was accepted for publication in IEEE International Symposium on Information Theory (ISIT), 2014 and in book Coordination Control of Distributed Systems of series Lecture Notes in Control and Information Sciences, 2015

Capacity of Binary State Symmetric Channel with and without Feedback and Transmission Cost

Authors: Christos K. Kourtellaris, Charalambos D. Charalambous

Abstract: We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the channel state is the modulo2 addition of the current channel input and the previous channel output. We derive closed form expressions for the capacity and corresponding channel input distribution, of this BSSC with and without feedback and transmission cost. We also show that the capacity of the BSSC is n… ▽ More We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the channel state is the modulo2 addition of the current channel input and the previous channel output. We derive closed form expressions for the capacity and corresponding channel input distribution, of this BSSC with and without feedback and transmission cost. We also show that the capacity of the BSSC is not increased by feedback, and it is achieved by a first order symmetric Markov process. △ Less

Submitted 15 February, 2015; v1 submitted 31 January, 2014; originally announced February 2014.

Applications of Information Nonanticipative Rate Distortion Function

Authors: Photios A. Stavrou, Christos K. Kourtellaris, Charalambos D. Charalambous

Abstract: The objective of this paper is to further investigate various applications of information Nonanticipative Rate Distortion Function (NRDF) by discussing two working examples, the Binary Symmetric Markov Source with parameter $p$ (BSMS($p$)) with Hamming distance distortion, and the multidimensional partially observed Gaussian-Markov source. For the BSMS($p$), we give the solution to the NRDF, and w… ▽ More The objective of this paper is to further investigate various applications of information Nonanticipative Rate Distortion Function (NRDF) by discussing two working examples, the Binary Symmetric Markov Source with parameter $p$ (BSMS($p$)) with Hamming distance distortion, and the multidimensional partially observed Gaussian-Markov source. For the BSMS($p$), we give the solution to the NRDF, and we use it to compute the Rate Loss (RL) of causal codes with respect to noncausal codes. For the multidimensional Gaussian-Markov source, we give the solution to the NRDF, we show its operational meaning via joint source-channel matching over a vector of parallel Gaussian channels, and we compute the RL of causal and zero-delay codes with respect to noncausal codes. △ Less

Submitted 29 April, 2014; v1 submitted 22 January, 2014; originally announced January 2014.

Comments: 5 pages, 3 figures, accepted for publication in IEEE International Symposium on Information Theory (ISIT) proceedings, 2014

Source-Channel Matching for Sources with Memory

Authors: Christos Kourtellaris, Charalambos D. Charalambous, Photios A. Stavrou

Abstract: In this paper we analyze the probabilistic matching of sources with memory to channels with memory so that symbol-by-symbol code with memory without anticipation are optimal, with respect to an average distortion and excess distortion probability. We show achievability of such a symbolby- symbol code with memory without anticipation, and we show matching for the Binary Symmetric Markov source (BSM… ▽ More In this paper we analyze the probabilistic matching of sources with memory to channels with memory so that symbol-by-symbol code with memory without anticipation are optimal, with respect to an average distortion and excess distortion probability. We show achievability of such a symbolby- symbol code with memory without anticipation, and we show matching for the Binary Symmetric Markov source (BSMS(p)) over a first-order symmetric channel with a cost constraint. △ Less

Submitted 25 March, 2014; v1 submitted 11 October, 2013; originally announced October 2013.

Comments: 6 pages. arXiv admin note: substantial text overlap with arXiv:1304.6528

Rate Distortion Function for a Class of Relative Entropy Sources

Authors: Farzad Rezaei, Charalambos D. Charalambous, Photios A. Stavrou

Abstract: This paper deals with rate distortion or source coding with fidelity criterion, in measure spaces, for a class of source distributions. The class of source distributions is described by a relative entropy constraint set between the true and a nominal distribution. The rate distortion problem for the class is thus formulated and solved using minimax strategies, which result in robust source coding… ▽ More This paper deals with rate distortion or source coding with fidelity criterion, in measure spaces, for a class of source distributions. The class of source distributions is described by a relative entropy constraint set between the true and a nominal distribution. The rate distortion problem for the class is thus formulated and solved using minimax strategies, which result in robust source coding with fidelity criterion. It is shown that minimax and maxmin strategies can be computed explicitly, and they are generalizations of the classical solution. Finally, for discrete memoryless uncertain sources, the rate distortion theorem is stated for the class omitting the derivations while the converse is derived. △ Less

Submitted 6 May, 2013; originally announced May 2013.

Comments: 6 pages, final version presented at Internation Symposium on Mathematical Theory of Networks and Systems (MTNS), Budapest, Hungary, 2010, pp. 1853-1858

Nonanticipative Rate Distortion Function for General Source-Channel Matching

Authors: Christos Kourtellaris, Charalambos D. Charalambous, Photios A. Stavrou

Abstract: In this paper we invoke a nonanticipative information Rate Distortion Function (RDF) for sources with memory, and we analyze its importance in probabilistic matching of the source to the channel so that transmission of a symbol-by-symbol code with memory without anticipation is optimal, with respect to an average distortion and excess distortion probability. We show achievability of the symbol-by-… ▽ More In this paper we invoke a nonanticipative information Rate Distortion Function (RDF) for sources with memory, and we analyze its importance in probabilistic matching of the source to the channel so that transmission of a symbol-by-symbol code with memory without anticipation is optimal, with respect to an average distortion and excess distortion probability. We show achievability of the symbol-by-symbol code with memory without anticipation, and we evaluate the probabilistic performance of the code for a Markov source. △ Less

Submitted 25 April, 2013; v1 submitted 24 April, 2013; originally announced April 2013.

Comments: 5 pages, 3 figures

Directed Information on Abstract Spaces: Properties and Variational Equalities

Authors: Charalambos D. Charalambous, Photios A. Stavrou

Abstract: Directed information or its variants are utilized extensively in the characterization of the capacity of channels with memory and feedback, nonanticipative lossy data compression, and their generalizations to networks. In this paper, we derive several functional and topological properties of directed information for general abstract alphabets (complete separable metric spaces) using the topology o… ▽ More Directed information or its variants are utilized extensively in the characterization of the capacity of channels with memory and feedback, nonanticipative lossy data compression, and their generalizations to networks. In this paper, we derive several functional and topological properties of directed information for general abstract alphabets (complete separable metric spaces) using the topology of weak convergence of probability measures. These include convexity of the set of consistent distributions, which uniquely define causally conditioned distributions, convexity and concavity of directed information with respect to the sets of consistent distributions, weak compactness of these sets of distributions, their joint distributions and their marginals. Furthermore, we show lower semicontinuity of directed information, and under certain conditions we also establish continuity of directed information. Finally, we derive variational equalities for directed information, including sequential versions. These may be viewed as the analogue of the variational equalities of mutual information (utilized in Blahut-Arimoto algorithm). In summary, we extend the basic functional and topological properties of mutual information to directed information. These properties are discussed in the context of extremum problems of directed information. △ Less

Submitted 22 December, 2015; v1 submitted 16 February, 2013; originally announced February 2013.

Comments: 49 pages, 2 figures, Submitted for publication to IEEE Transactions on Information Theory

Centralized Versus Decentralized Team Games of Distributed Stochastic Differential Decision Systems with Noiseless Information Structures-Part I: General Theory

Authors: Charalambos D. Charalambous, Nasir U. Ahmed

Abstract: Decentralized optimization of distributed stochastic differential systems has been an active area of research for over half a century. Its formulation utilizing static team and person-by-person optimality criteria is well investigated. However, the results have not been generalized to nonlinear distributed stochastic differential systems possibly due to technical difficulties inherent with decentr… ▽ More Decentralized optimization of distributed stochastic differential systems has been an active area of research for over half a century. Its formulation utilizing static team and person-by-person optimality criteria is well investigated. However, the results have not been generalized to nonlinear distributed stochastic differential systems possibly due to technical difficulties inherent with decentralized decision strategies. In this first part of the two-part paper, we derive team optimality and person-by-person optimality conditions for distributed stochastic differential systems with different information structures. The optimality conditions are given in terms of a Hamiltonian system of equations described by a system of coupled backward and forward stochastic differential equations and a conditional Hamiltonian, under both regular and relaxed strategies. Our methodology is based on the semi martingale representation theorem and variational methods. Throughout the presentation we discuss similarities to optimality conditions of centralized decision making. △ Less

Submitted 14 February, 2013; originally announced February 2013.

Comments: 39 pages, 1 figure Submitted to IEEE Transactions on Automatic Control

MSC Class: 49J55; 49K45; 93E20

Centralized Versus Decentralized Team Games of Distributed Stochastic Differential Decision Systems with Noiseless Information Structures-Part II: Applications

Authors: Charalambos D. Charalambous, Nasir U. Ahmed

Abstract: In this second part of our two-part paper, we invoke the stochastic maximum principle, conditional Hamiltonian and the coupled backward-forward stochastic differential equations of the first part [1] to derive team optimal decentralized strategies for distributed stochastic differential systems with noiseless information structures. We present examples of such team games of nonlinear as well as li… ▽ More In this second part of our two-part paper, we invoke the stochastic maximum principle, conditional Hamiltonian and the coupled backward-forward stochastic differential equations of the first part [1] to derive team optimal decentralized strategies for distributed stochastic differential systems with noiseless information structures. We present examples of such team games of nonlinear as well as linear quadratic forms. In some cases we obtain closed form expressions of the optimal decentralized strategies. Through the examples, we illustrate the effect of information signaling among the decision makers in reducing the computational complexity of optimal decentralized decision strategies. △ Less

Submitted 14 February, 2013; originally announced February 2013.

Comments: 39 pages Submitted to IEEE Transaction on Automatic Control

MSC Class: 49J55; 49K45; 93E20

Optimal Nonstationary Reproduction Distribution for Nonanticipative RDF on Abstract Alphabets

Authors: Photios A. Stavrou, Charalambos D. Charalambous, Christos K. Kourtellaris

Abstract: In this paper we introduce a definition for nonanticipative Rate Distortion Function (RDF) on abstract alphabets, and we invoke weak convergence of probability measures to show various of its properties, such as, existence of the optimal reproduction conditional distribution, compactness of the fidelity set, lower semicontinuity of the RDF functional, etc. Further, we derive the closed form expres… ▽ More In this paper we introduce a definition for nonanticipative Rate Distortion Function (RDF) on abstract alphabets, and we invoke weak convergence of probability measures to show various of its properties, such as, existence of the optimal reproduction conditional distribution, compactness of the fidelity set, lower semicontinuity of the RDF functional, etc. Further, we derive the closed form expression of the optimal nonstationary reproduction distribution. This expression is computed recursively backward in time. Throughout the paper we point out an operational meaning of the nonanticipative RDF by recalling the coding theorem derive in \cite{tatikonda2000}, and we state relations to Gorbunov-Pinsker's nonanticipatory $ε-$entropy \cite{gorbunov-pinsker}. △ Less

Submitted 28 January, 2013; originally announced January 2013.

Comments: 5 pages, submitted to ISIT 2013

Variational Equalities of Directed Information and Applications

Authors: Photios A. Stavrou, Charalambos D. Charalambous

Abstract: In this paper we introduce two variational equalities of directed information, which are analogous to those of mutual information employed in the Blahut-Arimoto Algorithm (BAA). Subsequently, we introduce nonanticipative Rate Distortion Function (RDF) ${R}^{na}_{0,n}(D)$ defined via directed information introduced in [1], and we establish its equivalence to Gorbunov-Pinsker's nonanticipatory $ε$-e… ▽ More In this paper we introduce two variational equalities of directed information, which are analogous to those of mutual information employed in the Blahut-Arimoto Algorithm (BAA). Subsequently, we introduce nonanticipative Rate Distortion Function (RDF) ${R}^{na}_{0,n}(D)$ defined via directed information introduced in [1], and we establish its equivalence to Gorbunov-Pinsker's nonanticipatory $ε$-entropy $R^{\varepsilon}_{0,n}(D)$. By invoking certain results we first establish existence of the infimizing reproduction distribution for ${R}^{na}_{0,n}(D)$, and then we give its implicit form for the stationary case. Finally, we utilize one of the variational equalities and the closed form expression of the optimal reproduction distribution to provide an algorithm for the computation of ${R}^{na}_{0,n}(D)$. △ Less

Submitted 16 May, 2013; v1 submitted 28 January, 2013; originally announced January 2013.

Comments: 5 pages, to appear in proceedings of International Symposium on Information Theory (ISIT), 2013

Nonanticipative Rate Distortion Function and Filtering Theory: A weak Convergence Approach

Authors: Photios A. Stavrou, Charalambos D. Charalambous

Abstract: In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian filtering theory is further investigated on general Polish spaces. The relation is established via an optimization on the space of conditional distributions of the so-called directed information subject to fidelity constraints. Existence of the optimal reproduction distribution of the nonanticipative RDF… ▽ More In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian filtering theory is further investigated on general Polish spaces. The relation is established via an optimization on the space of conditional distributions of the so-called directed information subject to fidelity constraints. Existence of the optimal reproduction distribution of the nonanticipative RDF is shown using the topology of weak convergence of probability measures. Subsequently, we use the solution of the nonanticipative RDF to present the realization of a multidimensional partially observable source over a scalar Gaussian channel. We show that linear encoders are optimal, establishing joint source-channel coding in real-time. △ Less

Submitted 18 January, 2014; v1 submitted 29 December, 2012; originally announced December 2012.

Comments: 26 pages and 4 figures, part of this work is published in European Control Conference (ECC' 13)

On the relation of nonanticipative rate distortion function and filtering theory

Authors: Charalambos D. Charalambous, Photios A. Stavrou

Abstract: In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian filtering theory is investigated using the topology of weak convergence of probability measures on Polish spaces. The relation is established via an optimization on the space of conditional distributions of the so-called directed information subject to fidelity constraints. Existence of the optimal repro… ▽ More In this paper the relation between nonanticipative rate distortion function (RDF) and Bayesian filtering theory is investigated using the topology of weak convergence of probability measures on Polish spaces. The relation is established via an optimization on the space of conditional distributions of the so-called directed information subject to fidelity constraints. Existence of the optimal reproduction distribution of the nonanticipative RDF is shown, while the optimal nonanticipative reproduction conditional distribution for stationary processes is derived in closed form. The realization procedure of nonanticipative RDF which is equivalent to joint-source channel matching for symbol-by-symbol transmission is described, while an example is introduced to illustrate the concepts. △ Less

Submitted 25 April, 2013; v1 submitted 7 October, 2012; originally announced October 2012.

Comments: 6 pages, 4 figures, final version submitted for publication at 12th Biannual European Control Conference (ECC), 2013

MSC Class: 60B11; 37A50

Nonanticipative Rate Distortion Function and Relations to Filtering Theory

Authors: Charalambos D. Charalambous, Photios A. Stavrou, Nasir U. Ahmed

Abstract: The relation between nonanticipative Rate Distortion Function (RDF) and filtering theory is discussed on abstract spaces. The relation is established by imposing a realizability constraint on the reconstruction conditional distribution of the classical RDF. Existence of the extremum solution of the nonanticipative RDF is shown using weak$^*$-convergence on appropriate topology. The extremum recons… ▽ More The relation between nonanticipative Rate Distortion Function (RDF) and filtering theory is discussed on abstract spaces. The relation is established by imposing a realizability constraint on the reconstruction conditional distribution of the classical RDF. Existence of the extremum solution of the nonanticipative RDF is shown using weak$^*$-convergence on appropriate topology. The extremum reconstruction conditional distribution is derived in closed form, for the case of stationary processes. The realization of the reconstruction conditional distribution which achieves the infimum of the nonanticipative RDF is described. Finally, an example is presented to illustrate the concepts. △ Less

Submitted 18 September, 2013; v1 submitted 3 October, 2012; originally announced October 2012.

Comments: 41 pages,5 figures, Provisionally Accepted Version of the manuscript submitted to IEEE Transactions on Automatic Control; Part of this work was presented in 20th International Symposium on Mathematical Theory of Networks and Systems (MTNS '12)

MSC Class: 28A33; 28A35; 60B10

Realizable Rate Distortion Function and Bayesian FIltering Theory

Authors: Photios A. Stavrou, Charalambos D. Charalambous, Christos K. Kourtellaris

Abstract: The relation between rate distortion function (RDF) and Bayesian filtering theory is discussed. The relation is established by imposing a causal or realizability constraint on the reconstruction conditional distribution of the RDF, leading to the definition of a causal RDF. Existence of the optimal reconstruction distribution of the causal RDF is shown using the topology of weak convergence of pro… ▽ More The relation between rate distortion function (RDF) and Bayesian filtering theory is discussed. The relation is established by imposing a causal or realizability constraint on the reconstruction conditional distribution of the RDF, leading to the definition of a causal RDF. Existence of the optimal reconstruction distribution of the causal RDF is shown using the topology of weak convergence of probability measures. The optimal non-stationary causal reproduction conditional distribution of the causal RDF is derived in closed form; it is given by a set of recursive equations which are computed backward in time. The realization of causal RDF is described via the source-channel matching approach, while an example is briefly discussed to illustrate the concepts. △ Less

Submitted 13 April, 2012; originally announced April 2012.

Comments: 5 pages, 3 figures, 1 table, 1 graph, submitted to Information Theory Workshop 2012

MSC Class: 28A33

Directed Information on Abstract spaces: Properties and Extremum Problems

Authors: Charalambos D. Charalambous, Photios A. Stavrou

Abstract: This paper describes a framework in which directed information is defined on abstract spaces. The framework is employed to derive properties of directed information such as convexity, concavity, lower semicontinuity, by using the topology of weak convergence of probability measures on Polish spaces. Two extremum problems of directed information related to capacity of channels with memory and feedb… ▽ More This paper describes a framework in which directed information is defined on abstract spaces. The framework is employed to derive properties of directed information such as convexity, concavity, lower semicontinuity, by using the topology of weak convergence of probability measures on Polish spaces. Two extremum problems of directed information related to capacity of channels with memory and feedback, and non-anticipative and sequential rate distortion are analyzed showing existence of maximizing and minimizing distributions, respectively. △ Less

Submitted 19 May, 2012; v1 submitted 5 February, 2012; originally announced February 2012.

Comments: 5 pages, to appear in proceedings of International Symposium on Information Theory (ISIT), 2012

Causal Rate Distortion Function on Abstract Alphabets: Optimal Reconstruction and Properties

Authors: Photios A. Stavrou, Charalambos D. Charalambous, Christos K. Kourtellaris

Abstract: A causal rate distortion function with a general fidelity criterion is formulated on abstract alphabets and a coding theorem is derived. Existence of the minimizing kernel is shown using the topology of weak convergence of probability measures. The optimal reconstruction kernel is derived, which is causal, and certain properties of the causal rate distortion function are presented. A causal rate distortion function with a general fidelity criterion is formulated on abstract alphabets and a coding theorem is derived. Existence of the minimizing kernel is shown using the topology of weak convergence of probability measures. The optimal reconstruction kernel is derived, which is causal, and certain properties of the causal rate distortion function are presented. △ Less

Submitted 4 February, 2012; originally announced February 2012.

Comments: 5 pages, Submitted to Internation Symposium on Information Theory(ISIT) 2012

Variable Length Lossless Coding for Variational Distance Class: An Optimal Merging Algorithm

Authors: Themistoklis Charalambous, Charalambos D. Charalambous, Sergey Loyka

Abstract: In this paper we consider lossless source coding for a class of sources specified by the total variational distance ball centred at a fixed nominal probability distribution. The objective is to find a minimax average length source code, where the minimizers are the codeword lengths -- real numbers for arithmetic or Shannon codes -- while the maximizers are the source distributions from the total v… ▽ More In this paper we consider lossless source coding for a class of sources specified by the total variational distance ball centred at a fixed nominal probability distribution. The objective is to find a minimax average length source code, where the minimizers are the codeword lengths -- real numbers for arithmetic or Shannon codes -- while the maximizers are the source distributions from the total variational distance ball. Firstly, we examine the maximization of the average codeword length by converting it into an equivalent optimization problem, and we give the optimal codeword lenghts via a waterfilling solution. Secondly, we show that the equivalent optimization problem can be solved via an optimal partition of the source alphabet, and re-normalization and merging of the fixed nominal probabilities. For the computation of the optimal codeword lengths we also develop a fast algorithm with a computational complexity of order ${\cal O}(n)$. △ Less

Submitted 24 January, 2013; v1 submitted 1 February, 2012; originally announced February 2012.

Comments: 28 pages, 4 figures. arXiv admin note: substantial text overlap with arXiv:1112.1715

Optimal Merging Algorithms for Lossless Codes with Generalized Criteria

Authors: Themistoklis Charalambous, Charalambos D. Charalambous, Farzad Rezaei

Abstract: This paper presents lossless prefix codes optimized with respect to a pay-off criterion consisting of a convex combination of maximum codeword length and average codeword length. The optimal codeword lengths obtained are based on a new coding algorithm which transforms the initial source probability vector into a new probability vector according to a merging rule. The coding algorithm is equivalen… ▽ More This paper presents lossless prefix codes optimized with respect to a pay-off criterion consisting of a convex combination of maximum codeword length and average codeword length. The optimal codeword lengths obtained are based on a new coding algorithm which transforms the initial source probability vector into a new probability vector according to a merging rule. The coding algorithm is equivalent to a partition of the source alphabet into disjoint sets on which a new transformed probability vector is defined as a function of the initial source probability vector and a scalar parameter. The pay-off criterion considered encompasses a trade-off between maximum and average codeword length; it is related to a pay-off criterion consisting of a convex combination of average codeword length and average of an exponential function of the codeword length, and to an average codeword length pay-off criterion subject to a limited length constraint. A special case of the first related pay-off is connected to coding problems involving source probability uncertainty and codeword overflow probability, while the second related pay-off compliments limited length Huffman coding algorithms. △ Less

Submitted 6 August, 2012; v1 submitted 7 December, 2011; originally announced December 2011.

Comments: 40 pages long, arXiv admin note: text overlap with arXiv:1102.2207, arXiv:1202.0136

Causal Rate Distortion Function and Relations to Filtering Theory

Authors: Photios A. Stavrou, Charalambos D. Charalambous

Abstract: A causal rate distortion function is defined, its solution is described, and its relation to filtering theory is discussed. The relation to filtering is obtained via a causal constraint imposed on the reconstruction kernel to be realizable. A causal rate distortion function is defined, its solution is described, and its relation to filtering theory is discussed. The relation to filtering is obtained via a causal constraint imposed on the reconstruction kernel to be realizable. △ Less

Submitted 6 June, 2012; v1 submitted 5 April, 2011; originally announced April 2011.

Comments: 8 pages; 3 figures; Presented in 20th International Symposium on Mathematical Theory of Networks and Systems (MTNS 2012)

Causal Rate Distortion Function on Abstract Alphabets and Optimal Reconstruction Kernel

Authors: Charalambos D. Charalambous, Photios A. Stavrou, Christos K. Kourtellaris

Abstract: A Causal rate distortion function with a general fidelity criterion is formulated on abstract alphabets and the optimal reconstruction kernel is derived, which consists of a product of causal kernels. In the process, general abstract spaces are introduced to show existence of the minimizing kernel using weak*-convergence. Certain properties of the causal rate distortion function are presented. A Causal rate distortion function with a general fidelity criterion is formulated on abstract alphabets and the optimal reconstruction kernel is derived, which consists of a product of causal kernels. In the process, general abstract spaces are introduced to show existence of the minimizing kernel using weak*-convergence. Certain properties of the causal rate distortion function are presented. △ Less

Submitted 16 February, 2011; originally announced February 2011.

Minimum Redundancy Coding for Uncertain Sources

Authors: Michael B. Baer, Farzad Rezaei, Charalambos D. Charalambous

Abstract: Consider the set of source distributions within a fixed maximum relative entropy with respect to a given nominal distribution. Lossless source coding over this relative entropy ball can be approached in more than one way. A problem previously considered is finding a minimax average length source code. The minimizing players are the codeword lengths --- real numbers for arithmetic codes, integers f… ▽ More Consider the set of source distributions within a fixed maximum relative entropy with respect to a given nominal distribution. Lossless source coding over this relative entropy ball can be approached in more than one way. A problem previously considered is finding a minimax average length source code. The minimizing players are the codeword lengths --- real numbers for arithmetic codes, integers for prefix codes --- while the maximizing players are the uncertain source distributions. Another traditional minimizing objective is the first one considered here, maximum (average) redundancy. This problem reduces to an extension of an exponential Huffman objective treated in the literature but heretofore without direct practical application. In addition to these, this paper examines the related problem of maximal minimax pointwise redundancy and the problem considered by Gawrychowski and Gagie, which, for a sufficiently small relative entropy ball, is equivalent to minimax redundancy. One can consider both Shannon-like coding based on optimal real number ("ideal") codeword lengths and a Huffman-like optimal prefix coding. △ Less

Submitted 1 May, 2011; v1 submitted 14 February, 2011; originally announced February 2011.

Comments: 5 pages

ACM Class: E.4; H.1.1

Lossless Coding with Generalised Criteria

Authors: Charalambos D. Charalambous, Themistoklis Charalambous, Farzad Rezaei

Abstract: This paper presents prefix codes which minimize various criteria constructed as a convex combination of maximum codeword length and average codeword length or maximum redundancy and average redundancy, including a convex combination of the average of an exponential function of the codeword length and the average redundancy. This framework encompasses as a special case several criteria previously i… ▽ More This paper presents prefix codes which minimize various criteria constructed as a convex combination of maximum codeword length and average codeword length or maximum redundancy and average redundancy, including a convex combination of the average of an exponential function of the codeword length and the average redundancy. This framework encompasses as a special case several criteria previously investigated in the literature, while relations to universal coding is discussed. The coding algorithm derived is parametric resulting in re-adjusting the initial source probabilities via a weighted probability vector according to a merging rule. The level of desirable merging has implication in applications where the maximum codeword length is bounded. △ Less

Submitted 10 February, 2011; originally announced February 2011.

Comments: 6 pages, 2 figures