Query-Based Sampling of Heterogeneous CTMCs: Modeling and Optimization with Binary Freshness

Abstract

We study a remote monitoring system in which a mutually independent and heterogeneous collection of finite-state irreducible continuous time Markov chain (CTMC) based information sources is considered. In this system, a common remote monitor queries the instantaneous states of the individual CTMCs according to a Poisson process with possibly different intensities across the sources, in order to maintain accurate estimates of the original sources. Three information freshness models are considered to quantify the accuracy of the remote estimates: fresh when equal (FWE), fresh when sampled (FWS) and fresh when close (FWC). For each of these freshness models, closed-form expressions are derived for mean information freshness for a given source. Using these expressions, optimum sampling rates for all sources are obtained so as to maximize the weighted sum freshness of the monitoring system, subject to an overall sampling rate constraint. This optimization problem leads to a water-filling solution with quadratic worst case computational complexity in the number of information sources. Numerical examples are provided to validate the effectiveness of the optimum sampling policy in comparison to several baseline sampling policies.

I Introduction

Timely delivery of status update packets from a number of information sources for maintaining information freshness at a remote monitor (or destination) has recently gained significant attention for the development of applications requiring real-time monitoring and control. In such applications, information sources generate status update packets that contain the samples of an underlying random process, e.g., a sensor sampling a physical quantity such as temperature, humidity, etc., or an information item with time-varying content such as news, weather reports, etc., which are subsequently delivered to the monitor through a communication network. In the push-based communication paradigm, information sources decide when to sample and form the information packets, which are subsequently forwarded towards the destination [1, 2]. On the other hand, in query-based (or pull-based) communications, monitors proactively query the information sources upon which sampling takes place [3, 4].

For the design and optimization of status update systems, a need for quantifying information freshness is evident. For this purpose, age of information (AoI) is commonly used which is a continuous-valued continuous-time stochastic process maintained at the destination that keeps track of the time elapsed since the reception of the last status update packet received from a particular information source. The AoI process was first introduced in [5] for a single-source M/M/1 queuing model which resulted in substantial interest in the modeling of AoI and its optimization in very general settings including multiple sources [6, 7].

In this paper, the interest will be on the remote estimation of discrete-valued information sources, in particular CTMC based information sources, for which the status of the sources change at random instants, and the source dynamics (set of states and transition rates) is known at the monitor. In this setting, the monitor and the source are synchronized (in sync or fresh) when the monitor’s estimate overlaps with the state of the source. Otherwise, they are de-synchronized (out of sync or erroneous or stale). Although performance metrics derived from AoI have played an important role for the development of status update systems for continuous-valued information sources, these metrics are not as suitable for discrete-valued information sources with known dynamics, the latter being the focus of this paper. For example, even when the monitor has perfect knowledge of the source when its discrete status stays intact, its corresponding AoI process would continue to increase with time, leading to an undesired penalty [8].

On the basis of such shortcomings of AoI, alternative performance metrics have been proposed including age of synchronization (AoS) which is defined as the elapsed time since the content at the destination has de-synchronized with the source [9, 10, 11]. With AoS, the penalty of the monitor having an erroneous estimate of the source increases linearly with time as long as the de-synchronization condition stays. A more general process proposed recently is age of incorrect information (AoII) which is defined as the product of an increasing time penalty function and another penalty function chosen general enough to depend on both the current estimate at the monitor and the actual state of the process [8]. AoII covers AoS as a sub-case whereas other application-oriented choices for the two penalty functions are further elaborated in [8]. For certain applications including caching systems, dissatisfaction of being in an erroneous state does not necessarily increase with time but instead, with the number of versions the monitor lags with respect to the original source. For this purpose, version age (VA) has been introduced which keeps track of the number of status changes that have occurred at the source since the last time the content at the destination is de-synchronized with the source [12, 13, 14, 15]. In some applications, in contrast to the age-driven metrics, the information at the destination may not possess a value unless the content at the destination is in sync with the source irrespective of the duration of the erroneous state. In such applications, there is not a need for the dissatisfaction to grow with time or with outdated versions, which gives rise to the so-called binary freshness (BF) process which takes the value of one when the information at the destination is in sync with the source, and is zero otherwise. This simple BF metric was extensively used in the optimization of Web crawling systems by which local copies of remote Web pages are managed for Web search engines [16, 17]. BF has recently been used as a performance metric for freshness quantification in other computer and communication systems, such as in cache update systems [18, 19, 20, 21], gossi** networks [22, 23], infection tracking systems [24]. The focus of this study is applications for which the use of the BF metric is suitable for freshness quantification.

Figure 1: Query-based status update system in which the monitor queries the CTMC $X_{n}(t)$ associated with source-$n$ with intensity $\lambda_{n}$, to maintain an estimate $\tilde{X}_{n}(t)$ of $X_{n}(t)$.

In this paper, we consider the query-based communication system in Fig. 1 involving a number of CTMC-based information sources each of which is queried by a common remote monitor according to a Poisson process after which samples are taken and sent as status update packets towards the monitor, to maintain a remote estimate of the original process. Queries and status update packet transmissions are assumed to take place immediately. The individual CTMCs are finite-state and irreducible, mutually independent, and their heterogeneous dynamics (state-space and transition rates do not have to be the same across the sources) are known at the monitor. We employ a martingale estimator that is also used in similar studies such as [8, 25, 24], for which the remote estimate of the source between two consecutive samples, is the value of the first sample. The martingale estimator is easy to maintain, and moreover, we will show that it is possible to obtain closed-form expressions for several freshness metrics with this estimator, which subsequently leads to an algorithmic optimization procedure for obtaining the optimum sampling rates in a heterogeneous setting. Non-Poisson query arrivals, such as periodic queries, more advanced estimators such as the maximum likelihood estimator (MLE), and non-zero transmission times for query and status update packets, are left for future research.

We employ three non-age driven freshness models, two being binary, to quantify the accuracy of the martingale estimators. In fresh when equal (FWE), the information is fresh at the destination when the remote estimator is in sync with the state of the original source. In the fresh when sampled (FWS) freshness model (which is also binary), the information becomes stale when the state of the original source changes and it becomes fresh only when a new sample is taken. For FWS, when the freshness process is stale, it cannot be set until the remote monitor re-samples the process, which is in contrast with FWE. The advantage of the FWS model is that it represents certain applications where freshness can be regained only with an explicit user action once it has been lost, and further, closed form expressions for mean freshness can be obtained for FWS for a very general class of CTMCs. The non-binary fresh when close (FWC) model is a generalization of the binary FWE model where the level of freshness depends on the proximity of the original process and its estimator at the monitor, yielding more flexibility than FWE.

In this work, for the three freshness models of interest, we first find closed form expressions for mean freshness as a function of the sampling rate and subsequently we find the optimum sampling rate for each source that maximizes the weighted sum freshness for the system, under a total sampling rate constraint for the monitor. We show that this optimization problem possesses a water-filling structure which amounts to a procedure in which a limited volume of water is poured into a pool organized into a number of bins with different ground levels with the water level in each bin giving the desired optimum solution [26]. Water-filling based optimization has been successfully used in solving optimal resource allocation problems arising in wireless networks; see [27, 26] for two related surveys, and also references [28] and [29], that make use of water-filling techniques. Iterative methods are available for water-filling based optimization which will be shown to give rise to an algorithm for the optimum monitoring problem in Fig. 1 with quadratic worst-case computational complexity in the number of sources.

The contributions of our paper are as follows:

• To the best of our knowledge, optimum sampling of heterogeneous CTMCs under overall sampling rate constraints has not been explored in the literature. Our work initiates this line of study.

• We derive closed form expressions for mean freshness for FWE, FWC and FWS models for finite-state and irreducible CTMCs. For the FWS model, the obtained expression is in terms of the sum of first order rational functions of the sampling rate which is a strictly concave increasing function. For the FWE and FWC models, similar expressions are obtained for the sub-case of time-reversible CTMCs (using the real-valued eigenvalues, and eigenvectors of the corresponding generators) that cover the well-established birth-death Markov chains that arise frequently in the performance of computer and communication systems.

• The obtained expressions allow us to use computationally efficient water-filling algorithms to obtain optimum sampling policies.

The remainder of this paper is organized as follows. Related work is summarized in Section II. Preliminaries on CTMCs are given in Section III. In Section IV, the detailed system model is presented. Section V addresses the derivation of mean freshness expressions for the three freshness models. The water-filling based optimization algorithm is presented in Section VI. A comparative evaluation of the proposed optimum sampler and several baseline sampling policies are presented in Section VII. Conclusions, some open problems and potential future directions are given in Section VIII.

II Related Work

The majority of the remote estimation problems in the literature are in the discrete-time setting. In [30], the authors study the remote estimation of a linear time-invariant dynamic system while focusing on the trade-off between reliability and freshness. In [31], AoII is investigated in a status update system involving a multi-state Markovian information source, a monitor, and a channel susceptible to packet errors, and the communication goal is to minimize AoII subject to a power constraint. The authors of [32] study a transmitter monitoring the evolution of a two-state discrete Markov source and sending status updates to a destination over an unreliable wireless channel for the purpose of real-time source reconstruction for remote actuation. This work is then extended in [33] with more general discrete stochastic source processes and resource constraints. The work presented in [34] studies the trade-off between the sampling frequency and staleness in detecting the events through a freshness metric called age penalty which is defined as the time elapsed since the first transition out of the most recently observed state. The authors of [35] investigate a time-slotted communication system for tracking a discrete-time Markovian source with joint sampling and transmission over a wireless channel.

For the continuous-time setting, [36] investigates the problem of sampling a Wiener process with samples forwarded to a remote estimator over a channel that is modeled as a queue. The authors of [37] investigate the effect of AoI on the accuracy of a remote monitoring system which displays the latest state information from a CTMC and they develop a computational method for finding the conditional probability of the displayed state, given the actual current state of the information source. The authors of [38] obtain a push-based sampling policy for remote tracking of a CTMC source subject to a sampling rate constraint using constrained semi-Markov decision processes. A common feature of the above works is the existence of a single information source which gets to be sampled. On the other hand, [24] studies the sampling of a collection of heterogeneous two-state CTMC-based information sources each modeling whether an individual is infected with a virus or not, while using the binary freshness metric, and [14] studies sampling of multiple heterogeneous Poisson processes representing the citation indices of multiple researchers with the goal of kee** timely estimates (similar to version age) of all the random processes. In the current manuscript, we study a heterogeneous collection of general finite-state (not necessarily two-state) CTMC-based information sources with three different freshness models (including the binary freshness metric) where the sources need to be sampled for remote estimation with a constraint on the overall sampling rate.

III Preliminaries

The focus of the current paper is on irreducible, finite-state, time-homogeneous (transition rates do not depend on time) CTMCs with their main properties given in this section based on [39, 40]. We consider the CTMC $X(t),t\geq 0,X(t)\in\mathcal{K}=\{1,2,\ldots,K\}$, where $K\geq 1$ is the number of states. The process $X(t)$ has the infinitesimal generator $Q$ of size $K$ with its $(i,j)$th entry denoted by $q_{ij}$ which is the transition rate from state $i$ to state $j$ for $i\neq j$ and its diagonal entries are strictly negative satisfying $Qe=0$, where $e$ is a column vector of ones of appropriate size. The CTMC $X(t)$ is called irreducible if it is possible with some positive probability to get from any state to any other state in some finite time. An irreducible CTMC does not have any transient states. Hence, the limiting probability of $X(t)$ being in state $j$ conditioned on being in state $i$ at time zero, exists and does not depend on the initial state,

$$\lim_{t\rightarrow\infty}\mathbb{P}[X(t)=j|X(0)=i] =:\pi_{j}.$$ (1)

Moreover, the row vector $\pi=\{\pi_{j}\}$, known as the stationary (or invariant) distribution, satisfies $\pi Q=0$ and $\pi e=1$, which are known as the global balance equations (GBE). The GBEs for CTMCs are a set of equations, one for each state $s$ of the CTMC, which states that the total probability flux out of a state $s$ should be equal to the total probability flux from other states into the state $s$, in steady-state [39].

Irreducible, finite-state CTMCs are ergodic, i.e., for any function $f:\mathcal{K}\rightarrow\mathbb{R}$, the following holds,

$$\lim_{\tau\rightarrow\infty}\frac{1}{\tau}\int_{t=0}^{\tau}f_{X(t)}\differential{t} =\sum_{i=1}^{K}f_{i}\pi_{i},$$ (2)

where $f_{i}$ is the value of the function at state $i$, known as the Ergodic Theorem [40]. Note that the right hand side of (2) is an expectation with respect to the invariant distribution $\pi$. In particular, when $f_{i}=i$, then we have

$$\lim_{\tau\rightarrow\infty}\frac{1}{\tau}\int_{t=0}^{\tau}X(t)\differential{t}=\sum_{i=1}^{K}i\pi_{i}=:\mathbb{E}[X],$$ (3)

where the random variable $X$ has the same limiting distribution in (1), i.e., $\mathbb{P}[X=j]=\lim_{t\rightarrow\infty}\mathbb{P}[X(t)=j]=\pi_{j},j\in\mathcal{K}$. $X$ is called the random variable associated with the CTMC $X(t)$ in the steady-state.

The generator $Q$ has a left eigenvector $\pi$ and right eigenvector $e$, associated with the simple eigenvalue at zero and all other eigenvalues having strictly negative real parts [39]. $X(t)$ is called a time-reversible CTMC if its generator $Q$ satisfies the following detailed balance equations (DBE),

$$\pi_{i}q_{ij} =\pi_{j}q_{ji},\quad i\neq j.$$ (4)

The equations (4) are the GBEs for birth-death chains since the corresponding state transitions take place between neighboring states only. Therefore, birth-death chains are time-reversible. For a time-reversible CTMC $X(t)$, let

$$\Pi= {\rm diag}\{\pi_{1},\pi_{2},\ldots,\pi_{K}\},$$ (5)

be the diagonal matrix composed of the entries of $\pi$. Also let

$$S =\Pi^{1/2}Q\Pi^{-1/2},$$ (6)

which is a symmetric matrix from (4). Symmetric matrices have real eigenvalues and they are diagonalizable by orthogonal transformations [41]. Therefore, there exists an orthonormal matrix $U$ such that

$$U^{T}SU =D={\rm diag}\{-d_{1},-d_{2},\ldots,-d_{K-1},0\},$$ (7)

where $-d_{i}$, with $d_{i}>0$, are the corresponding real eigenvalues of the matrix $S$. Moreover, the matrix defined by $T=\Pi^{-1/2}U$ diagonalizes the original generator $Q$,

$$T^{-1}QT =D.$$ (8)

Next, we present several properties on the left and right eigenvectors of time-reversible CTMCs. Let the $(i,j)$th entries of $T$ and $\tilde{T}=T^{-1}$ be denoted by $t_{ij}$ and $\tilde{t}_{ij}$, respectively. The way the transformation matrix $T$ is defined, the $i$th row of ${\tilde{T}}$, i.e., the $i$th left eigenvector of $Q$, is obtained by post-multiplying by $\Pi$ the transpose of the $i$th column of $T$, i.e., the transpose of the $i$th right eigenvector of $Q$. Consequently, $\tilde{t}_{ji}=\pi_{i}t_{ij}$ for all $i,j$. Moreover, the row vector $\pi$ is the $K$th row of $\tilde{T}$ and $e$ is the $K$th column of $T$ in (8).

IV System Model

We consider the monitoring system in Fig. 1 with $N$ continuous-time information sources each of which is a finite-state, irreducible CTMC. The CTMC associated with source-$n$ is denoted by $X_{n}(t)$, $n\in\mathcal{N}=\{1,2,\ldots,N\}$, $t\geq 0$, and $X_{n}(t)\in\{1,2,\ldots,K_{n}\}$, where $K_{n}$ is the size of state space for $X_{n}(t)$. The process $X_{n}(t)$ has the infinitesimal generator ${Q_{n}}$ of size $K_{n}$ with its $(i,j)$th entry denoted by $q_{n,ij}$. The steady-state vector satisfies $\bm{\pi}_{n}{Q_{n}}=0$ and $\bm{\pi}_{n}e=1$ with $\bm{\pi}_{n}=\{\pi_{n,i}\}$. The transition rate out of state $i$ is denoted by $\sigma_{n,i}=\sum_{j\neq i}q_{n,ij}$. The average transition intensity of source-$n$ is denoted by $r_{n}$, i.e., $r_{n}=\sum_{i=1}^{K_{n}}\pi_{n,i}\sigma_{n,i}$ which is the long-term frequency of state transitions for the CTMC $X_{n}(t)$. We denote by $r$ the system transition intensity, $r=\sum_{i=1}^{N}r_{n}$.

The remote monitor in Fig. 1 samples the original process $X_{n}(t)$ according to a Poisson process with intensity $\lambda_{n}>0$ in order to maintain an estimate of the instantaneous state of the original information process. In this paper, we propose to use the martingale estimator $\tilde{X}_{n}(t)$ which is given as $\tilde{X}_{n}(t)=X_{n}(t^{\prime})$, where $t^{\prime}$ is the latest sampling time before $t$. The accuracy of the remote estimator is studied with three information freshness models described below. For the FWE information freshness model, the information is said to be fresh at the remote monitor only when the original process and its estimate are equal, i.e., $\tilde{X}_{n}(t)=X_{n}(t)$, and is otherwise stale. In the FWE model, there is no value at all in a sample, unless the original process and its estimator are synchronized. Hence, the binary freshness process $F_{n,e}(t)$ is defined as $F_{n,e}(t)=1$ when $\tilde{X}_{n}(t)=X_{n}(t)$, and zero otherwise. It is clear that the joint process $(\tilde{X}(t),X(t))$ is also an irreducible CTMC as the original CTMC $X(t)$, and using the Ergodic Theorem [40], the time-average of the freshness process $F_{n,e}(t)$ is given by,

$$\lim_{\tau\rightarrow\infty}\frac{1}{\tau}\int_{t=0}^{\tau}F_{n,e}(t)\differential{t}:=\mathbb{E}[F_{n,e}]$$ , (9)

where $F_{n,e}$ is the random variable associated with the random process $F_{n,e}(t)$ in steady-state, i.e.,

$$\mathbb{P}[F_{n,e}=1] =\lim_{t\rightarrow\infty}\mathbb{P}[F_{n,e}(t)=1],$$ (10)

$$=\lim_{t\rightarrow\infty}\mathbb{P}[X(t)=\tilde{X}(t)],$$ (11)

and similarly,

$$\mathbb{P}[F_{n,e}=0] =\lim_{t\rightarrow\infty}\mathbb{P}[X(t)\neq\tilde{X}(t)].$$ (12)

Figure 2: Sample paths of the processes $X(t)$, $\tilde{X}(t)$, $F_{e}(t)$, $F_{c}(t)$ and $F_{s}(t)$ for a source with states 1, 2 and 3, for an example scenario. Down arrows represent sampling operation.

In the FWC model, we assume that there may be a value when the original process and its estimator are close enough to each other despite being out of sync, from a certain semantic perspective. For this purpose, for FWC, we introduce a proximity matrix ${P_{n}}=\{p_{n,ij}\}$, $0\leq p_{n,ij}\leq 1$ for source-$n$, and subsequently define a non-binary freshness process $F_{n,c}(t)$ which takes the value $p_{n,ij}$ when $X_{n}(t)=i$ and $\tilde{X}_{n}(t)=j$. In particular, $p_{n,ii}=1$, representing perfect freshness when the original process and its estimator are synchronized. Close to unity values of $p_{n,ij}$ are representative of proximity between the states $i$ and $j$. When ${P_{n}}$ is taken as the identity matrix, FWC reduces to FWE. For the FWS model, the binary freshness process $F_{n,s}(t)$ is set whenever the process is sampled, and it stays set until $X_{n}(t)$ makes a transition at which instant $F_{n,s}(t)$ becomes zero. The mean freshness for FWC (resp. FWS) is denoted by $\mathbb{E}[F_{n,c}]$ (resp. $\mathbb{E}[F_{n,s}])$ similar to (9) where $F_{n,c}$ (resp. $F_{n,s}$) is the random variable associated with the random process $F_{n,c}(t)$ (resp. $F_{n,s}(t)$) in the steady-state.

When the index of the source is immaterial, the subscript $n$ is dropped for the source process $X(t)$ and its estimator $\tilde{X}(t)$ along with the corresponding freshness processes $F_{e}(t)$, $F_{c}(t)$, and $F_{s}(t)$ for FWE, FWC, and FWS, respectively, and the proximity matrix ${P}=\{p_{ij}\}$ for FWC. Fig. 2 depicts the sample paths of the processes $X(t)$, $\tilde{X}(t)$, $F_{e}(t)$, $F_{c}(t)$ and $F_{s}(t)$ for a source with three states $\{1,2,3\}$ for an example scenario where the proximity matrix $P$ is chosen such that $p_{ij}=1$ when $|i-j|=0$, $0.5$ when $|i-j|=1$, and zero when $|i-j|=2$. Note that when $F_{s}(t)=1$, then $F_{e}(t)=1$, but not otherwise. Therefore, $\mathbb{E}[F_{e}]\geq\mathbb{E}[F_{s}]$ for any choice of the sampling rate $\lambda$. Moreover, $\mathbb{E}[F_{c}]\geq\mathbb{E}[F_{e}]$ since $F_{c}(t)$ can be larger than zero when $F_{e}(t)=0$, stemming from the structure of the proximity matrix $P$. Thus, it always holds that $\mathbb{E}[F_{c}]\geq\mathbb{E}[F_{e}]\geq\mathbb{E}[F_{s}]$.

V Analytical Expressions for Mean Freshness

The subscript $n$ indicating the source index is dropped for convenience in the current section where the mean freshness is derived for a single irreducible CTMC $X(t)$ for the three freshness models.

V-A FWE Model

Theorem 1 provides an expression for $f(\lambda)=\mathbb{E}[F_{e}]$ for the FWE model for the CTMC $X(t)$.

Theorem 1.

Let the irreducible CTMC $X(t)\in\{1,2,\ldots,K\}$ with generator $Q$ and steady-state vector $\pi$, be Poisson sampled with sampling rate $\lambda$. Then, for the FWE model, the mean freshness $f(\lambda)=\mathbb{E}[F_{e}]$ is given by,

$$f(\lambda) =\lambda\pi\ {\bm{d}iag}[(\lambda I-Q)^{-1}],$$ (13)

where $\bm{diag}[\cdot]$ represents a column vector composed of the diagonal entries of its matrix argument.

Proof.

Let us consider the two-dimensional random process $Y(t)=(\tilde{X}(t),X(t))$, which is also Markov. To see this, note that, the transition intensity from state $(i,j)$ to $(i,j^{\prime})$ is $q_{jj^{\prime}}$ and from state $(i,j)$ to $(j,j)$ for $j\neq i$ is $\lambda$. Let the steady-state vector of the process $Y(t)$ be denoted by $y$, i.e., $y_{ij}=\lim_{t\rightarrow\infty}\mathbb{P}[Y(t)=(i,j)]$, $1\leq i,j\leq K$. Let $Y$ be a $K\times K$ matrix such that $Y=\{y_{ij}\}$. Applying the GBE for the state $(i,i)$ of $Y(t)$ provides the following equation for each $i$, $1\leq i\leq K$,

$$y_{ii}\sigma_{i} =\sum_{j\neq i}y_{ij}q_{ji}+\sum_{j\neq i}y_{ji}\lambda,$$ (14)

$$=\sum_{j\neq i}y_{ij}q_{ji}+\lambda(\pi_{i}-y_{ii}),$$ (15)

where the last equality stems from the identity $\pi_{i}=\sum_{j}y_{ji}$. On the other hand, when the GBE is applied for the state $(i,k)$, $k\neq i$, then we obtain the following,

$$y_{ik}(\sigma_{k}+\lambda)=\sum_{j\neq k}y_{ij}q_{jk},\quad 1\leq i\leq K,\ k\neq i.$$ (16)

Writing the equations (15) and (16) in a matrix form, we obtain for each $i$, $1\leq i\leq K$,

$$Y(i,:)(Q-\lambda I) =-\lambda\pi_{i}I(i,:),$$ (17)

where $Y(i,:)$ and $I(i,:)$ denote the $i$th row of $Y$ and the $i$th row of the identity matrix, respectively. In FWE, the freshness process $F_{e}(t)=1$ when the joint process $Y(t)$ is visiting state $(i,i)$ (in the steady-state) for some state $i$ and $F_{e}(t)=0$ otherwise. Therefore, $\mathbb{E}[F_{e}]=\sum_{i=1}^{K}y_{ii},$ which yields the identity (13). ∎

The following corollary gives a simplified expression for mean freshness $f(\lambda)$ in terms of the sum of first-order rational functions of the variable $\lambda$ for time-reversible CTMCs on the basis of Theorem 1.

Corollary 1.

Consider the process $X(t)$ of Theorem 1 with generator $Q$ which is time-reversible and with diagonalizing transformation matrix $T$ as given in (8). Then, the mean freshness $f(\lambda)=\mathbb{E}[F_{e}]$ is given for the FWE model by

$$f(\lambda) =\sum_{j=1}^{K}\pi_{j}^{2}+\lambda\sum_{j=1}^{K-1}\frac{b_{j}}{\lambda+d_{j}},$$ (18)

$$=1-\sum_{j=1}^{K-1}\frac{a_{j}}{\lambda+d_{j}},$$ (19)

where $a_{j}$ and $b_{j}$, for $1\leq j\leq K-1$, are given by

$$b_{j} =\sum_{i=1}^{K}{\pi}_{i}^{2}t_{ij}^{2}=\sum_{i=1}^{K}{\tilde{t}_{ji}^{2}},\quad a_{j}=b_{j}d_{j}.$$ (20)

Moreover, $f(\lambda)$ is increasing and strictly concave, and has a continuous derivative $f^{\prime}(\lambda)$ with $\lim_{\lambda\rightarrow 0^{+}}f(\lambda)=\sum_{j=1}^{K}\pi_{j}^{2}$.

Proof.

Using the diagonalization equation (8), we first write the term $A=(\lambda I-Q)^{-1}=T(\lambda I-D)^{-1}T$ appearing in (13) as follows,

$$A =T\matrixquantity(\frac{1}{\lambda+d_{1}}&&&\\ &\ddots&&\\ &&\frac{1}{\lambda+d_{K-1}}&\\ &&&\frac{1}{\lambda})T^{-1}.$$ (21)

Using (13) and (21), we have

$$\frac{f(\lambda)}{\lambda} =\sum_{i=1}^{K}{\pi}_{i}A_{ii},$$ (22)

$$=\sum_{i=1}^{K}{\pi}_{i}\sum_{j=1}^{K}t_{ij}\tilde{t}_{ji}\frac{1}{\lambda+d_{j}},$$ (23)

$$=\sum_{j=1}^{K-1}\underbrace{\sum_{i=1}^{K}{\pi}_{i}^{2}t_{ij}^{2}}_{b_{j}}\frac{1}{\lambda+d_{j}}+\frac{1}{\lambda}\sum_{i=1}^{K}\pi_{i}^{2},$$ (24)

$$=\sum_{j=1}^{K-1}\frac{b_{j}}{\lambda+d_{j}}+\frac{1}{\lambda}\sum_{i=1}^{K}\pi_{i}^{2},$$ (25)

since $\tilde{t}_{ji}=\pi_{i}t_{ij}$ and also $\sum_{i=1}^{K}{\pi}_{i}t_{iK}\tilde{t}_{Ki}=\sum_{i=1}^{K}\pi_{i}^{2}$ by observing that $t_{iK}=1$ and $\tilde{t}_{Ki}={\pi}_{i}$. The result in (25) gives the desired expression in (18). Then, (19) follows directly from (25) and also the fact that $\lim_{\lambda\rightarrow\infty}f(\lambda)=1$. Moreover, the coefficients $a_{j}$ and $b_{j}$ are strictly positive since $\pi_{i}>0$ and the entries of a column of $T$ cannot be all zero. A first-order rational function of $\lambda$ in the form $-a/(\lambda+d)$ is increasing and strictly concave for $a,d>0$ and sums of concave functions are also increasing and strictly concave, completing the proof. The expression pertaining to $\lim_{\lambda\rightarrow 0^{+}}f(\lambda)$ immediately follows from (18). ∎

For the special case $K=2$, $X(t)$ is a two-state time-reversible CTMC with generator $Q$ and the diagonal matrix $\Pi$ given as follows (5),

$$Q=\begin{pmatrix}-\alpha&\alpha\\ \beta&-\beta\end{pmatrix},\quad\Pi=\begin{pmatrix}\frac{\beta}{\alpha+\beta}&0\\ 0&\frac{\alpha}{\alpha+\beta}\end{pmatrix}.$$ (26)

The generator $Q$ has two eigenvalues: the first eigenvalue being $-d_{1}$ where $d_{1}=\alpha+\beta$ and the second one at the origin. Consequently, we obtain the symmetric matrix $S$ according to (6) and the orthogonal transformation matrix $U$ from (7),

$$S=\begin{pmatrix}-\beta&\sqrt{\alpha\beta}\\ \sqrt{\alpha\beta}&-\beta\end{pmatrix},\quad U=\begin{pmatrix}\sqrt{\frac{\alpha}{\alpha+\beta}}&\sqrt{\frac{\beta}{\alpha+\beta}}\\ \sqrt{\frac{\beta}{\alpha+\beta}}&\sqrt{\frac{\alpha}{\alpha+\beta}}\end{pmatrix},$$ (27)

which gives rise to the matrix $\tilde{T}$ which is the inverse of the diagonalizing transformation matrix $T$ (see (8)),

$$\tilde{T}=\begin{pmatrix}{\frac{\alpha\beta}{(\alpha+\beta)^{2}}}&{\frac{\alpha\beta}{(\alpha+\beta)^{2}}}\\ {\frac{\beta}{\alpha+\beta}}&{\frac{\alpha}{\alpha+\beta}}\end{pmatrix}.$$ (28)

From (19), the following closed-form solution exists for mean freshness,

$$f(\lambda) =1-\frac{a_{1}}{\lambda+d_{1}},$$ (29)

where $a_{1}=\frac{2\alpha\beta}{(\alpha+\beta)}$ since $a_{1}=b_{1}d_{1}$ and $b_{1}$ is the sum of the squares of the entries of the first row of $\tilde{T}$ which is evident from the equations (19) and (20). We note that expressions for expected staleness for this limited special case of 2-state sources have been obtained in [24] using a different method.

V-B FWC Model

Corollary 2 provides an expression for $f(\lambda)=\mathbb{E}[F_{c}]$ for the FWC model for time-reversible CTMCs on the basis of Theorem 1.

Corollary 2.

Consider the process $X(t)$ of Theorem 1 with generator $Q$ which is time-reversible and with diagonalizing transformation matrix $T$ as given in (8). Then, the mean freshness $f(\lambda)=\mathbb{E}[F_{c}]$ with proximity matrix $P$ is given for the FWC model by

$$f(\lambda) =\pi\sum_{i=1}^{K}\pi_{i}P(:,i)+\sum_{j=1}^{K-1}\frac{b_{j}\lambda}{\lambda+d_{j}},$$ (30)

$$=1-\sum_{j=1}^{K-1}\frac{a_{j}}{\lambda+d_{j}},$$ (31)

where $a_{j}$ and $b_{j}$, for $1\leq j\leq K-1$, are given by

$$b_{j} =\tilde{T}(j,:)\sum_{i=1}^{K}{\pi}_{i}t_{ij}P(:,i),\quad a_{j}=b_{j}d_{j},$$ (32)

and $P(:,i)$ denotes the $i$th column of $P$, and $\lim_{\lambda\rightarrow 0^{+}}f(\lambda)=\pi\sum_{i=1}^{K}\pi_{i}P(:,i)$.

Proof.

Recalling the definition of $y_{ij}$, we write

$$f(\lambda) =\sum_{i=1}^{K}\sum_{j=1}^{K}y_{ij}p_{ji}.$$ (33)

Recalling the definition of matrix $A$, we first obtain the following identity from (17),

$$\frac{f(\lambda)}{\lambda} =\sum_{i=1}^{K}{\pi}_{i}A(i,:)P(:,i).$$ (34)

Consequently,

$$\frac{f(\lambda)}{\lambda} =\sum_{i=1}^{K}{\pi}_{i}\left(\sum_{j=1}^{K}t_{ij}\tilde{T}(j,:)\frac{1}{\lambda+d_{j}}\right)P(:,i),$$ (35)

which is equal to the following expression,

$$\sum_{j=1}^{K-1}\underbrace{\tilde{T}(j,:)\sum_{i=1}^{K}{\pi}_{i}t_{ij}P(:,i)}_{b_{j}}\frac{1}{\lambda+d_{j}}+\pi\sum_{i=1}^{K}\frac{\pi_{i}P(:,i)}{\lambda},$$ (36)

since $t_{iK}=1$ and $\tilde{t}_{Ki}={\pi}_{i}$, giving the desired expression in (30). Then, (31) follows directly from (36) and also from $\lim_{\lambda\rightarrow\infty}f(\lambda)=1$. However, the coefficients $b_{j}$ (and hence $a_{j}$) are not guaranteed to be non-negative and some of these coefficients may indeed be negative. ∎

Remark.

Although FWE is a sub-case of FWC, we presented the results for FWE separately since in this case the expressions are slightly simpler and the coefficients $a_{j}$’s are shown to be non-negative ensuring concavity of the expression (19).

V-C FWS Model

Theorem 2 provides an expression for $f(\lambda)=\mathbb{E}[F_{s}]$ for the FWS model for the CTMC $X(t)$.

Theorem 2.

Let the irreducible CTMC $X(t)\in\{1,2,\ldots,K\}$ with generator $Q$ and steady-state vector $\pi$, be Poisson sampled with sampling rate $\lambda$. Then, for the FWS model, the mean freshness $f(\lambda)=\mathbb{E}[F_{s}]$ is given by,

$$f(\lambda) =1-\sum_{i=1}^{K}\frac{a_{i}}{\lambda+\sigma_{i}},$$ (37)

where $a_{i}=\pi_{i}\sigma_{i}$. Moreover, $f(\lambda)$ is increasing and strictly concave, and has a continuous derivative $f^{\prime}(\lambda)$ with $\lim_{\lambda\rightarrow 0^{+}}f(\lambda)=0$.

Proof.

Consider the two-dimensional process $Z(t)=(F_{s}(t),X(t))$, which is Markov. To see this, in FWS, the transition intensity from states $(1,j)$ and $(0,j)$ to the states $(1,j^{\prime})$ and $(0,j^{\prime})$, respectively, is $q_{jj^{\prime}}$. On the other hand, the transition intensity from state $(0,j)$ to $(1,j)$ is $\lambda$. Let the steady-state solution of the process $Z(t)$ be denoted by $z$, i.e., $z_{ij}=\lim_{t\rightarrow\infty}\mathbb{P}[F_{s}(t)=i,X(t)=j],0\leq i\leq 1,1\leq j\leq K$. We show that the following choice of $z$

$$z_{0i} =\frac{\pi_{i}\sigma_{i}}{\lambda+\sigma_{i}},\quad z_{1i}=\frac{\pi_{i}\lambda}{\lambda+\sigma_{i}},$$ (38)

satisfies the following GBE for the states $(1,i)$, $1\leq i\leq K$,

$$z_{1i}\sigma_{i} =z_{0i}\lambda,$$

which is a direct result of (38). In order to show that $z$ defined as in (38) satisfies the GBE for the states $(0,i)$, $1\leq i\leq K$, we write from (38),

$$z_{0i}(\sigma_{i}+\lambda) =\pi_{i}\sigma_{i}=\sum_{j\neq i}\pi_{j}q_{ji},$$ (39)

$$=\sum_{j\neq i}\left(\frac{\lambda}{\lambda+\sigma_{j}}+\frac{\sigma_{j}}{\lambda+\sigma_{j}}\right)\pi_{j}q_{ji},$$ (40)

$$=\sum_{j\neq i}(z_{0j}+z_{1j})q_{ji}.$$ (41)

Moreover, $\sum_{i=0}^{1}\sum_{j=1}^{K}z_{ij}=1$, and therefore, $z$ as given in (21) is the steady-state solution for the CTMC $Z(t)$. The mean freshness is finally expressed as, $f(\lambda)=1-\sum_{j=1}^{K}z_{0j},$ which yields (37). Since the form of expression is the same as in the FWE model for time-reversible CTMCs, $f(\lambda)$ is an increasing and strictly concave function of $\lambda$. Moreover, $\lim_{\lambda\rightarrow 0^{+}}f(\lambda)=1-\sum_{i}\pi_{i}=0$ from (37). ∎

VI Optimum Monitoring of Heterogeneous CTMCs

The monitor is resource-constrained, and therefore, there is a constraint $\Lambda$ on the overall sampling rate of the monitor, i.e., $\lambda=\sum_{n=1}^{N}\lambda_{n}\leq\Lambda$. Let us first focus our attention to the FWE freshness model for time-reversible CTMCs in which case we use the mean freshness metric $f_{n}(\lambda_{n})=\mathbb{E}[F_{n,e}]$ for source-$n$, and the weighted sum freshness (or the system freshness) $F_{S}=\sum_{n=1}^{N}w_{n}f_{n}(\lambda_{n})$, for the overall monitoring system where the normalized weights $w_{n},n=1,\ldots,N,\;\sum_{n}w_{n}=1,$ reflect the relative importance of the freshness of the information processes. Thus, we have the following optimization problem for weighted sum freshness maximization, λ_n ≥0∑_n=1^N w_n f_n(λ_n) = 1 - ∑_n=1^N ∑_j=1^K_n - 1 wnan,jλn+ dn,j ∑_n=1^N λ_n≤Λ In (VI), the coefficients $a_{n,j},d_{n,j}>0$, for $1\leq j\leq K_{n}-1$ are to be obtained for the CTMC $X_{n}(t)$ using the procedure described in Corollary 1 and the expression (19). The function $f_{n}(\lambda_{n})$ is increasing and strictly concave, and has a continuous first order derivative $f_{n}^{\prime}(\lambda_{n})$ that monotonically decreases from the value $\infty$ at $\lambda_{n}=-d_{n}^{\ast}$ to zero as $\lambda_{n}$ is increased to $\infty$, where $d_{n}^{\ast}=\min_{j}d_{n,j}$. This optimization problem is known to have a water-filling solution [27] on the basis of which Algorithm 1 provides an efficient solution to the optimization problem (VI) which requires at most $N-1$ iterations until termination. Step 2 of Algorithm 1 can be solved by using the two-dimensional bisection search algorithm detailed in [27].

The algorithm is outlined as follows. Initially, $I_{n}=1$ for $n=1,\ldots,N$. Then, for a given $\mu>0$ and for each $n$ such that $I_{n}=1$, we iteratively find the value of $\lambda_{n}\in(-d_{n}^{\ast},\infty)$ that satisfies $w_{n}f_{n}^{\prime}(\lambda_{n})=\mu$ using an inner bisection search algorithm. Once $\lambda_{n}$’s are obtained, we check whether $\sum_{n=1}^{N}\lambda_{n}I_{n}<\Lambda$ or not, and we vary the value of $\mu$ according to an outer bisection search algorithm, which iteratively finds the value of $\mu$ such that $\sum_{n=1}^{N}\lambda_{n}I_{n}=\Lambda$. If $\lambda_{n}\leq 0$ at this step, then $I_{n}$ and $\lambda_{n}$ are set to zero for all such $n$ and the procedure above is repeated.

For the special case of two-state CTMCs, i.e., $K_{n}=2$, a closed-form solution is available for the solution of the equations in Step 2 since the inverse function of $f_{n}^{\prime}(\cdot)$ can be written in closed form. In this case, it is not difficult to show that the choices of $\mu$ and $\lambda_{n}$ for sources with $I_{n}=1$,

$$\mu=\left(\frac{\sum_{n=1}^{N}\sqrt{w_{n,1}a_{n,1}I_{n}}}{\Lambda+\sum_{n=1}^{N}d_{n,1}I_{n}}\right)^{2},\;\lambda_{n}=\sqrt{\frac{w_{n}a_{n,1}}{\mu}}-d_{n,1},$$ (42)

provide a single-shot solution for Step 2 of Algorithm 1 without a requirement for bisection search for this step.

We note that, for the FWS model, Algorithm 1 can be used with the only difference being the upper limit of the inner summation changed to $K_{n}$ in (VI). For the FWC model, since the coefficients $a_{i}$’s in (31) can be negative, concavity of the freshness function is not proven. However, we propose to use the same water-filling algorithm also for the FWC model based on the observation that the expression (31) turned out to be concave in all the examples we studied.

Algorithm 1 Water-filling algorithm for the optimization problem (VI) based on [27].

  Step 1: Initialize $I_{n}=1$ for $n=1,\ldots,N$.

  Step 2: Solve the following equations for the water level $\mu>0$ and $\lambda_{n}$ when $I_{n}=1$,

$$w_{n}f_{n}^{\prime}(\lambda_{n})=\mu,\;\sum_{n=1}^{N}\lambda_{n}I_{n}=\Lambda.$$ (43)

  Step 3: If $\lambda_{n}>0$ for all $n$ such that $I_{n}=1$, then terminate while returning $\lambda_{n}$’s.

  Step 4: Otherwise, set $I_{n}=0$ and $\lambda_{n}=0$ for all $n$ such that $I_{n}=1$ and $\lambda_{n}\leq 0$, and go to Step 2.

VII Numerical Examples

The first numerical example is presented to validate the analytical expressions obtained for mean freshness in corollaries 1 and 2, and Theorem 2. A time-reversible birth-death CTMC with three states is considered with generator

$$Q=\begin{pmatrix}-1.95&1.95&0\\ 1&-2.95&1.95\\ 0&2&-2\end{pmatrix}.$$ (44)

For the FWC model, we use the proximity matrix of Fig. 2. The mean freshness is first obtained in Fig. 3 as a function of $\lambda$ for the three freshness models FWE, FWC, and FWS, using the analytical expressions in (19), (31), and (37), respectively, which is termed as the analytical (A) method. Note that in the analytical method, the expressions are first obtained once for the CTMC which are then used to obtain the metrics for any given sampling rate $\lambda$. On the other hand, the same freshness metrics can also be obtained by numerically solving the two-dimensional Markov chains (used in the proofs of theorems 1 and 2) constructed for each given $\lambda$. The results match perfectly, validating the analytical method.

In the second numerical example, we focus on FWE and FWS in a scenario of $N=50$ heterogeneous two-state Markov chains with $\pi_{n,1}=0.3$, $\pi_{n,2}=0.7$ and linearly spaced transition intensities, i.e., $r_{n}=r_{n-1}+\delta$, $n=2,\ldots,N$. In the numerical example, we set $r_{1}=0.01$ and the average transition intensity $\frac{1}{N}r=\frac{1}{N}\sum_{n=1}^{N}r_{n}$ is set to 10 which yields the choice of $\delta=0.4078$. We denote by $\kappa$ the ratio of the overall sampling rate $\Lambda$ to the system transition intensity, i.e., $\kappa=\frac{\Lambda}{r}$. Obviously, the sampling ratio $\kappa$ should be sufficiently large so as to keep the remote estimates of all the information sources fresh. The source weights are assumed to be the same with $w_{n}=\frac{1}{N}$. Algorithm 1 is used for both FWE and FWS models, whereas for FWE, (42) is employed for Step 2 of the algorithm to obtain the optimum sampling rates $\lambda_{n}$’s under an overall sampling rate constraint $\Lambda$ which is chosen to attain a given sampling ratio $\kappa$.

Figure 3: The mean freshness as a function of the sampling rate $\lambda$ for the FWE, FWC, and FWS freshness models using the numerical (N) and analytical (A) methods.

We compare our proposed water-willing solution, namely WF, with three baseline policies: i) UNIFORM policy samples each source-$n$ uniformly likely, i.e., $\lambda_{n}=\frac{\Lambda}{N}$, ii) PROP policy chooses the sampling rate $\lambda_{n}$ proportional with the source’s transition intensity $r_{n}$, i.e., $\lambda_{n}\propto r_{n}$, iii) INVPROP policy chooses the sampling rate $\lambda_{n}$ inversely proportional with the source’s transition intensity $r_{n}$, i.e., $\lambda_{n}\propto\frac{1}{r_{n}}$. Fig. 4 depicts the system freshness $F_{S}$ as a function of the sampling ratio $\kappa$ when the WF, UNIF, PROP, and INVPROP sampling policies are employed for both FWE and FWS freshness models. For both models, we observe that the WF policy outperforms all the other three baseline policies. The UNIFORM policy yields very close to optimum freshness performance when the sampling ratio increases. However, for low sampling ratios, it is substantially outperformed by the WF policy. The PROP and INVPROP sampling policies perform poorly for small and large sampling ratios, respectively, against all other policies.

Figure 4: The system freshness $F_{S}$ as a function of the sampling ratio $\kappa$ for the WF, UNIFORM, PROP, and INVPROP sampling policies: (a) FWE (b) FWS.

Fig. 5 depicts the optimum sampling rate $\lambda_{n}$ divided by the sampling ratio $\kappa$ as a function of the source index $n$ for different values of the sampling ratio $\kappa$ for both freshness models. We observe that, for low sampling ratios, the water-filling solution chooses not to sample at all, a portion of the sources with high transition intensities, for both models. However, when the sampling ratio is sufficiently high, e.g., $\kappa=10$, the optimum sampling rate for a given source appears to be monotonically increasing with the transition intensity of the source. Although the general behavior of the optimum sampling rate with respect to source index is quite similar for FWE and FWS models, in the latter model, the optimum sampling rate is more uniform across the sources for FWS than FWE.

Figure 5: The optimum sampling rate $\lambda_{n}$ divided by $\kappa$ as a function of the source index $n$ for four values of the sampling ratio $\kappa$: (a) FWE (b) FWS.

Figure 6: System freshness $F_{S}$ for the FWC model as a function of the number of users, $N$, for the WF and UNIFORM sampling policies for four different choices of the proximity matrix, i.e., $v\in\{0,1,2,3\}$.

In the final example, we focus on the FWC model and the choice of the proximity matrix $P$ in terms of a proximity parameter $v$ such that $p_{ij}=1$ when $|i-j|\leq v$, and is zero otherwise. It is clear that FWC with $v=0$ reduces to FWE. We consider an independent collection of $N$ CTMCs each of which corresponds to the number of active servers in a multi-server M/M/c/c queuing system with $c$ servers, with common service rate $\gamma$, and arrival rate $\xi_{n}$ for source-$n$. The load for source-$n$ is denoted by $\rho_{n}=\xi_{n}/c\gamma$. In this example, we assume linearly spaced loads, $\rho_{n}=\rho_{n-1}+\delta$, $n=2,\ldots,N$ and the parameter $\delta$ is chosen so that the average load is fixed to $\rho_{avg}=\frac{1}{N}\sum_{n=1}^{N}\rho_{n}$. In this example, we fix $\gamma=1$ and $\rho_{avg}=0.9$. The source weights are identical as in the previous examples. The overall sampling rate bound is fixed to $\Lambda=20$. The weighted sum freshness $F_{S}=\frac{1}{N}\sum_{n=1}^{N}\mathbb{E}[F_{n,c}(t)]$ is plotted in Fig. 6 as a function of the number of users $N$ for FWC with the proximity parameter $v\in\{0,1,2,3\}$ and for two values of $\rho_{1}$ for the WF and UNIFORM policies. We have the following observations: When $\rho_{1}$ is close to $\rho_{avg}$, then all the sources have similar statistical behaviors and therefore the performances of WF and UNIFORM policies should be similar which is evident from Fig. 6(b). However, WF substantially outperforms the UNIFORM sampling policy in Fig. 6(a) where the smallest load source-1 has a load $\rho_{1}=0.01$ and consequently the sources are statistically dissimilar from each other. We have observed similar outperformance behavior of WF over the UNIFORM policy for the four values of the proximity parameter $v$ we have investigated. The weighted sum freshness decreases with increased $N$ since the sampling rate parameter $\Lambda$ is fixed for any choice of $N$. Therefore, sources are sampled at a lower intensity, on the average, as $N$ is increased in our example.

VIII Conclusions

We investigated a remote monitoring system which samples a heterogeneous collection of finite-state irreducible CTMCs according to a Poisson process, under an overall sampling rate constraint, employing a remote martingale estimate of the states of each of the CTMCs. Three binary freshness models are studied and expressions for mean freshness are obtained for all the freshness models of interest. Subsequently, the optimum sampling rates for all CTMCs are obtained using water-filling based optimization while maximizing the weighted sum freshness. The worst case computational complexity of the proposed method is quadratic in the number of CTMCs making it possible to solve for scenarios even with very large numbers of CTMCs. The optimum monitoring policy is shown to outperform a number of heuristic baseline policies especially when there is diversity in the statistical characteristics of the underlying sources. Future work will consist of the study of estimators other than the martingale estimator, information sources other than CTMCs, and the case of partially known source dynamics. Study of optimization techniques other than water-filling, especially for more general information sources (not necessarily time-reversible), is an interesting research direction. Another possibility is to take into consideration the most recently taken sample values while making a decision on which source to sample.