Tiered Service Architecture for Remote Patient Monitoring

Abstract

We develop a remote patient monitoring (RPM) service architecture, which has two tiers of monitoring: ordinary and intensive. The patient’s health state improves or worsens in each time period according to certain probabilities, which depend on the monitoring tier. The patient incurs a “loss of quality of life” cost or an “invasiveness” cost, which is higher under intensive monitoring than under ordinary. On the other hand, their health improves faster under intensive monitoring than under ordinary. In each period, the service decides which monitoring tier to use, based on the health of the patient. We investigate the optimal policy for making that choice by formulating the problem using dynamic programming. We first provide analytic conditions for selecting ordinary vs intensive monitoring in the asymptotic regime where the number of health states is large. In the general case, we investigate the optimal policy numerically. We observe a threshold behavior, that is, when the patient’s health drops below a certain threshold the service switches them to intensive monitoring, while ordinary monitoring is used during adequately good health states of the patient. The modeling and analysis provides a general framework for managing RPM services for various health conditions with medically/clinically defined system parameters.

I Introduction

Remote Patient Monitoring (RPM) is increasingly receiving attention as a method for monitoring patients with certain medical conditions in their normal living/working environments to increase their quality of life and the level of delivered health care [1, 2, 3]. This is becoming feasible via advancements in wearable medical devices, for example, wearable glucose monitors [4], smart watches with vital sign monitoring capabilities (e.g., heart rate, ECG, pulse oximetry) [5, 6], and other such sensors. Further, such devices are increasingly networked and can transmit and receive data over the Internet and act as edge devices in communication with computation servers in the Cloud.

Studies have shown the effectiveness of RPM for various medical conditions. For example, continuous glucose monitoring has been shown to improve glycemic control in patients with diabetes [7, 8]. Smart watches have been effectively used to monitor stress, movement disorders, sleep patterns, blood pressure, heart disease, and COVID-19 [5]. Other RPM devices have also been used to manage and track cardiac conditions, such as heart failure, arrhythmia, and hypertension [3]. These studies highlight the potential of RPM to improve patient outcomes and quality of life by allowing for timely interventions and personalized care.

However, the question remains of how intensively to monitor patients. Intensive monitoring schemes could range from remotely collecting more data on the patient health state and administering more medical intervention remotely (e.g., alerting the patient to increase medication dosage) to calling the patient into an urgent care facility.

While aggressive monitoring may provide more comprehensive data, it can be resource intensive, draining the wearable device battery [6] faster and requiring the clinicians to review more RPM data. From the patient’s perspective, under intensive monitoring the patient naturally experiences a higher “loss of quality of life” cost or “invasiveness” cost, since intensive monitoring would normally be more invasive to their personal lifestyle. In contrast, intensive monitoring (and correspondingly elevated medical intervention) would enable early detection and intervention for adverse events, and hence the patient’s health is expected to improve faster than under ordinary one. Therefore, to account for this trade-off between the invasiveness cost and the possibility of an early intervention, there is an inherent need for a systematic approach to determining the appropriate level of monitoring, based on the patient’s health state.

In this paper we develop a RPM service architecture, where the patient is placed under less or more intensive levels or tiers of monitoring, based on their health state. We then study the optimal monitoring strategy for this model and how it varies with different parameters. One would intuitively expect that a patient would be placed under intensive one when their health state deteriorates; on the other hand, they would be returned to ordinary monitoring when their health state improves enough. The decision to switch from ordinary to intensive monitoring (and/or vice versa) requires a systematic analysis and depends heavily on the various parameters of the service in a rather subtle and complicated way, as analyzed in the following sections.

Of course, the design of an RPM service for a specific medical condition is highly dependent on the specifics of that condition and requires specialized medical knowledge. The point of this paper is not to design a particular RPM service but to provide a general framework and systematic methodology for RPM services based on tunable parameters (e.g., health improvement or deterioration probabilities, monitoring options and invasiveness costs) so as to make justifiable monitoring choices. The parameters will have to be decided on and tuned by medical/clinical experts for condition-specific RPM services. While we work with a simplified model here, the intuition gained from the analysis can help clinicians take more informed monitoring decisions.

In section II, we develop the model of evolution of the patient health state under ordinary and intensive monitoring and demonstrate how it can be managed, using the methodology of dynamic programming. In section III, we provide some analytical results on the optimal management policy in the asymptotic regime of a large number of health states and provide conditions on the parameters for choosing ordinary vs intensive monitoring. In section IV, we numerically investigate the structure of optimal monitoring policies and demonstrate that they place the patient under intensive monitoring when the health state deteriorates below a certain threshold; otherwise they use ordinary monitoring. Finally, in section V, we discuss some extensions. Sketches of some necessary proofs are presented in the Appendix.

II The RPM Service Model

Consider a patient who can be in a health state $h_{t}\in\{0,1,2,3,...,H\}$ in each service time period $t\in\{0,1,2,3,...\}$. The RPM service places the patient in a monitoring/intervention state $m_{t}\in\mathcal{M}=\{o,i\}$ in each time period $t$, abbreviated to monitoring state, where $o$ denotes ordinary monitoring and $i$ intensive monitoring. Thus, one can view the monitoring-patient joint state $s_{t}=(m_{t},h_{t})\in\mathcal{M}\times\mathcal{H}$ as the system or service state at time $t$.

A higher patient health state $h\in\{0,1,2,3,...,H\}$ corresponds to the patient having better health. In particular, the lowest health state $0$ is critical in the sense that, when the patient drifts into that state, they go beyond the scope of the current service model; at that point other emergency and/or more severe medical interventions are required, which are outside the scope of this service. Because of that, the states $(i,0)$ and $(o,0)$ are absorbing for the Markovian evolution of the health state, as explained below. Indeed, when the patient enters heath state $0$ under any monitoring state $i$ or $o$, the service evolution stops, as other medical measures/interventions are initiated.

As seen below, in defining costs incurred at the various states, we first take the patient’s quality of life point of view. Under ordinary monitoring, the patient incurs a constant cost $C_{o}\geq 0$ at any state $(o,h)$ with $h\in\{1,2,...,H\}$. Correspondingly, under intensive monitoring, the patient incurs a constant cost $C_{i}\geq 0$ at any state $(i,h)$ with $h\in\{1,2,..,H\}$. These costs reflect the invasiveness loss for the patient. One may argue about the costs in more elaborate ways, for example, including patient risk factors and operational considerations of the service. For simplicity, we focus here on the invasiveness argument mentioned above.

Of special interest is the critical health state $h=0$, where this model ceases to apply. On either state $(o,0)$ or $(i,0)$ where the patients health is critical a cost $C_{c}$ is incurred.

We finally define the transition costs $C_{o\to i}$ (and $C_{i\to o}$), which is associated with the service transitioning the patient from ordinary to intensive monitoring (and vice versa, respectively).

We model the system as a controlled Markov chain, trying to stay as simple as possible, yet still capture the essence of the problem and get insights into its solution. We discuss the more general case in the final section. At the beginning of every time period $t$, the service takes the decision/action (control) to either keep the monitoring state the same (as in the previous time period) or switch it to the alternate monitoring state. Formally, the decision/action space is $\mathcal{A}=\{o,i\}$ and each (state, action) pair is associated with a cost given by the function $c:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}^{+}$. The transition probabilities are given by $p(s^{\prime}|s,a)$ where $s^{\prime},s\in\mathcal{S}$ and $a\in\mathcal{A}$. The cost functions and transition probabilities are defined as follows.

1. At health state $\mathbf{h=0}$:

No action is taken with the the service ceasing operation. A cost of $C_{c}$ is incurred.

2. At health states $\mathbf{1\leq h\leq H}$:

1. Ordinary Monitoring ($m=o$), no Switching ($a=o$): Does not induce a monitoring change, and the system state transitions as follows:

   1. $(o,h)\xrightarrow{a=o}(o,\min\{h+1,H\})$ with prob. $\lambda_{o}$

   2. $(o,h)\xrightarrow{a=o}(o,h-1)$ with prob. $\mu_{o}=1-\lambda_{o}$,

   and a cost $c\big{(}(o,h),o\big{)}=C_{o}$ is incurred. Note that $\min\{h+1,H\}$ above is used to account for $(o,H)\to(o,H)$ with prob. $\lambda_{o}$ since $H$ is the highest health state.

2. Ordinary Monitoring ($m=o$) with Switching ($a=i$): Induces a switch to intensive monitoring $m=i$, and the system state transitions as follows:

   1. $(o,h)\xrightarrow{a=i}(i,\min\{h+1,H\})$ with prob. $\lambda_{i}$

   2. $(o,h)\xrightarrow{a=i}(i,h-1)$ with prob. $\mu_{i}=1-\lambda_{i}$,

   and a cost $c\big{(}(o,h),i\big{)}=C_{o\to i}+C_{i}$ is incurred.

3. Intensive Monitoring ($m=i$), no Switching ($a=i$): Does not induce a monitoring change, and the system state transitions as follows:

   1. $(i,h)\xrightarrow{a=i}(i,\min\{h+1,H\})$ with prob. $\lambda_{i}$

   2. $(i,h)\xrightarrow{a=i}(i,h-1)$ with prob. $\mu_{i}=1-\lambda_{i}$,

   and a cost $c\big{(}(i,h),i\big{)}=C_{i}$ is incurred.

4. Intensive Monitoring ($m=i$) with Switching ($a=o$): Induces a switch to ordinary monitoring $m=o$, and the system state transitions as follows:

   1. $(i,h)\xrightarrow{a=o}(o,\max\{h+1,H\})$ with prob. $\lambda_{o}$

   2. $(i,h)\xrightarrow{a=o}(o,h-1)$ with prob. $\mu_{o}=1-\lambda_{o}$,

   and a cost $c\big{(}(i,h),o\big{)}=C_{i\to o}+C_{o}$ is incurred.

We can easily incorporate health state dependent costs and transition probabilities, but for simplicity we assume constant ones here. We make the following natural assumptions.

Assumption 1.

1. The transition probabilities satisfy: $\lambda_{i}\geq\lambda_{o}$.

2. The costs satisfy: $0\leq C_{o}\leq C_{i}\leq C_{c}$.

The first assumption 1.a intuitively states that the patient’s health improves faster under intensive monitoring, rather than under ordinary. Regarding assumption 1.b, it is naturally expected that $C_{o}\leq C_{i}$, as the patient’s “annoyance” is higher under intensive monitoring/intervention than under ordinary. Further, given the severity of entering the critical state $h=0$, it is naturally expected that $C_{o}\leq C_{i}\leq C_{c}$, and practically $C_{c}$ may be much larger than $C_{i}$.

II-A Optimal Monitoring Control

We study this problem under the discounted cost setting of the dynamic programming methodology [9], hence, costs incurred $t$ time periods into the future (with respect to present) are discounted by a factor of $\gamma^{t}$ with $0<\gamma<1$. Starting from state $s_{0}=s\in\mathcal{M}\times\mathcal{H}$, the total expected (discounted) cost to be incurred is

$$\mathbb{E}\Big{[}\sum_{t=0}^{T-1}\gamma^{t}c(s_{t},a_{t})+\gamma^{T}C_{c}\Big{|}\ s_{o}=s\Big{]}$$

when control action $a_{t}$ is taken by the service, at cost $c(s_{t},a_{t})$ introduced above, when its state is $s_{t}=(m_{t},h_{t})$ at time $t$, until the patient enters the critical state $h=0$ at time $T$ and the service ceases operation. Hence, $T$ is the time the patient spends in the service, and at time $T$ the critical cost $C_{c}$ is incurred, however, discounted to $\gamma^{T}C_{C}$. Thus, discounting by $\gamma$ implicitly reflects the patient’s desire to stay longer in service, hence, incur the critical cost deeper in the future and discounted to $\gamma^{T}C_{c}$.

A (stationary) monitoring policy $\pi(s)$ is a rule map** each state $s=(m,h)\in\mathcal{M}\times\mathcal{H}$ to a control action $a\in\mathcal{A}=\{o,i\}$ to be taken at that state. The value function $V_{\pi}(s)$ of a policy $\pi(s)$ is the total expected (discounted) cost the system will incur until reaching the critical state $0$ and stop, when it starts from state $s$ at time $t=0$. That is,

$$V_{\pi}(s)=\mathbb{E}\left[\sum_{t=0}^{T-1}\gamma^{t}c\Big{(}s_{t},\pi(s_{t})\Big{)}+\gamma^{T}C_{c}\ \Big{|}\ s_{0}=s\right]$$

and satisfies the dynamic programming equation [9].

$$V_{\pi}(s)=c\Big{(}s,\pi(s)\Big{)}+\gamma\sum_{s^{\prime}\in\mathcal{S}}\mathbb{P}\Big{(}s^{\prime}\mid s,\pi(s)\Big{)}V_{\pi}(s^{\prime}),$$

for all $s\in\mathcal{S}=\mathcal{M}\times\mathcal{H}$, given the Markovian evolution dynamics of the system, specified by the state transition probabilities defined above.

The goal is to find an optimal policy $\pi^{*}$ minimizing the $V_{\pi}$ over all policies $\pi$, that is, $V_{\pi^{*}}(s)\leq V_{\pi}(s)$ for every $s\in\mathcal{S}$ over all policies $\pi$. For simplicity we write $V^{*}(s)=V_{\pi^{*}}(s)$ and the following dynamic programming equation is satisfied [9].

$$V^{*}(s)=\min_{a\in\{o,i\}}\Bigg{\{}c(s,a)+\gamma\sum_{s^{\prime}\in\mathcal{S}}\mathbb{P}\Big{(}s^{\prime}|s,a\Big{)}V^{*}(s^{\prime})\Bigg{\}},$$

and can be solved numerically to yield the optimal policy $\pi^{*}(s)=\pi^{*}(m,h)$, that is, what optimal decision to take when the patient is in health state $h$ under monitoring $m$. For the state transition probabilities and costs defined before, this dynamic programming equation unfolds into:

1. For health states $1\leq h\leq H-1$

   $$V^{*}(i,h)=\min\bigg{\{}C_{i}+\gamma\Big{[}\lambda_{i}V^{*}(i,h{+}1)+\mu_{i}V^{*}(i,h{-}1)\Big{]},$$

   $$\;\;\;\;\;\;\;\;C_{i\to o}+C_{o}+\gamma\Big{[}\lambda_{o}V^{*}(o,h{+}1)+\mu_{o}V^{*}(o,h{-}1)\Big{]}\bigg{\}},$$

   $$V^{*}(o,h)=\min\bigg{\{}C_{o}+\gamma\Big{[}\lambda_{o}V^{*}(o,h{+}1)+\mu_{o}V^{*}(o,h{-}1)\Big{]},$$

   $$\;\;\;\;\;\;\;\;C_{o\to i}+C_{i}+\gamma\Big{[}\lambda_{i}V^{*}(i,h{+}1)+\mu_{i}V^{*}(i,h{-}1)\Big{]}\bigg{\}}.$$

2. At health state $H$,

   $$V^{*}(i,H)=\min\bigg{\{}C_{i}+\gamma\Big{[}\lambda_{i}V^{*}(i,H)+\mu_{i}V^{*}(i,H{-}1)\Big{]},$$

   $$\;\;\;\;\;\;\;\;C_{i\to o}+C_{o}+\gamma\Big{[}\lambda_{o}V^{*}(o,H)+\mu_{o}V^{*}(o,H{-}1)\Big{]}\bigg{\}},$$

   $$V^{*}(o,H)=\min\bigg{\{}C_{o}+\gamma\Big{[}\lambda_{o}V^{*}(o,H)+\mu_{o}V^{*}(o,H{-}1)\Big{]},$$

   $$\;\;\;\;\;\;\;\;C_{o\to i}+C_{i}+\gamma\Big{[}\lambda_{i}V^{*}(i,H)+\mu_{i}V^{*}(i,H{-}1)\Big{]}\bigg{\}}.$$

3. At the critical health state $h=0$,

   $$V^{*}(i,0)=V^{*}(o,0)=C_{c}.$$

Note that in every $\min\{\cdot,\cdot\}$ above, the first term corresponds to the control kee** the existing monitoring state, while the second corresponds to the control switching to the alternate monitoring state and incurring the switching cost.

II-B Simplified RPM Service

In order to reduce the number of parameters for tractability of the analysis, we make the following simplification.

Definition 1.

Simplified RPM Service:.

1. The cost for ordinary monitoring is set to zero: $C_{o}=0$.

2. The switching costs are set to zero: $C_{i\to o}=C_{o\to i}=0$.

Assumption 1 is still satisfied.

Given the limited space of this short paper, we work with the simplified RPM service below (analysis and numerical results), where interesting insights emerge. The zero cost of invasiveness under ordinary monitoring has no significant impact on the results, and is merely a technical assumption to make our analysis easier. The non-zero transition cost is true in several applications where the intensive monitoring just involves a higher rate of collecting data about the patient’s health. We briefly comment on how non-zero transition costs affect our results in Section V.

The simplified RPM service is illustrated in Figure 1 below.

Figure 1: The Simplified RPM Service. The blue and red arrows represent the possible transitions from state $(o,3)$ for actions $o$ and $i$ respectively. The arrows are labeled with probability of transition and cost incurred.

The dynamic programming equations for $V^{*}$ given above reduce in this case to the following:

1. For health states $1\leq h\leq H-1$,

   $$\hskip-8.5359ptV^{*}(i,h)=V^{*}(o,h)\\ \hskip-8.5359pt=\min\Bigg{\{}C_{i}+\gamma\Big{[}\lambda_{i}V^{*}(i,h+1)+\mu_{i}V^{*}(i,h-1)\Big{]},\\ \hskip 38.41139ptC_{o}+\gamma\Big{[}\lambda_{o}V^{*}(o,h+1)+\mu_{o}V^{*}(o,h-1)\Big{]}\Bigg{\}}$$ (1)

2. At health state $H$,

   $$V^{*}(o,H)=V^{*}(i,H)\\ =\min\Bigg{\{}C_{i}+\gamma\Big{[}\lambda_{i}V^{*}(i,H)+\mu_{i}V^{*}(i,H-1)\Big{]},\\ C_{o}+\gamma\Big{[}\lambda_{o}V^{*}(o,H)+\mu_{o}V^{*}(o,H-1)\Big{]}\Bigg{\}}$$ (2)

3. At the critical health state $h=0$,

   $$V^{*}(i,0)=V^{*}(o,0)=C_{c},$$ (3)

Note that, in the absence of switching costs (that is, $C_{o\to i}=C_{i\to o}=0$), for any health state $h$ the $V^{*}$ is the same under both ordinary and intensive monitoring.

III Asymptotic Analysis

The optimal policy ${\pi^{*}}$ and value function $V^{*}$ can be computed numerically (in general) from the dynamic programming equations (1)-(3) for any given set of system parameters. But to develop intuition and characterize the optimal policy, in this section, we analyse ${\pi^{*}}$ in the asymptotci regime of a large number of health states $H\gg 1$, i.e., a dynamic range of health states $H\to\infty$. This allows for analytic tractability of the optimal policy and closed-form conditions on our policies of interest. We make the following assumptions in this section.

Assumption 2.

Large $H$ Asymptotic Regime.

1. The number of health states is very large, i.e., the system operates in the asymptotic regime of $H\to\infty$.

2. Under ordinary monitoring the patient’s health drifts downwards, i.e., the improvement probability is $\lambda_{o}<0.5$ and the worsening one is $\mu_{o}=1-\lambda_{o}>0.5$

Assumption 2.a allows us to use tools from random walk analysis, as done in Lemma 1, making the analysis tractable. Assumption 2.b allows for the Markov chain to remain stable (positive recurrent) in the asymptotic regime.

The results obtained in this asymptotic regime can be thought of as an approximation for the RPM service with finite $H$, when $H$ grows large. In the next section, we numerically demonstrate that this asymptotic approximation tracks the optimal policy and the value function for our simplified RPM for a number of health states as low as $H=5$.

We define $\widetilde{\Pi}$ as the the set of policies under which the service chooses the same action irrespective of the monitoring state, that is,

$$\widetilde{\Pi}=\{\pi\mid\pi(i,h)=\pi(o,h),\;\forall h\geq 1\}.$$

Based on (1), for all $h\geq 1$, we then have

$${\pi^{*}}(o,h)={\pi^{*}}(i,h)$$

$$=\operatorname*{arg\,min}_{\{i,o\}}\{C_{i}+\gamma\lambda_{i}V^{*}(i,h+1)+\gamma(1-\lambda_{i})V^{*}(i,h-1),$$

$$\;\;\;\;\;\;C_{o}+\gamma\lambda_{o}V^{*}(o,h+1)+\gamma(1-\lambda_{o})V^{*}(o,h-1)\}$$

This implies that ${\pi^{*}}\in\widetilde{\Pi}$ and we can restrict our attention to the set $\widetilde{\Pi}$. For simplicity, we introduce notation $V^{*}(h)=V^{*}(o,h)=V^{*}(i,h)$ for $h\in\mathcal{H}$ and similarly the notation ${\pi^{*}}(h)$.

We next define an important policy $\pi_{o}$ where the patient stays under ordinary monitoring at all health states, i.e., $\pi_{o}(i,h)=\pi_{o}(o,h)=o$ for all $h\geq 1$. Note that $\pi_{o}\in\widetilde{\Pi}$ and we define the corresponding value function $V_{o}(h)$. Our first lemma gives the value function for this policy and presents an important property about the optimal policy.

Lemma 1.

For the simplified RPM (Definition 1) and under Assumption 2,

1. The value function $V_{o}$ for the policy $\pi_{o}$ is given by:

   $$V_{o}(h)=\phi^{h}C_{c},$$

   where

   $$\phi=\frac{1-\sqrt{1-4\lambda_{o}\mu_{o}\gamma^{2}}}{2\lambda_{o}\gamma}.$$

   Note that $\phi<1$ for $\gamma<1$.

2. For any choice of parameters, there exists $h^{\prime}$ such that, under the optimal policy, the patient prefers to stay in ordinary monitoring above health state $h^{\prime}$, i.e., ${\pi^{*}}(h)=o$, for all $h\geq h^{\prime}$.

Proof.

See Appendix A. ∎

For the simplified RPM, the cost of invasiveness under ordinary monitoring is zero. Then $V_{o}(h)=\mathbb{E}[\gamma^{T}C_{c}|h_{0}=h]$, where $T$ is the time taken to reach health state $0$, when started at health state $h$ and when the patient always stay under ordinary monitoring. $T$ here is precisely the hitting time of state $0$ for a random walk initiated at state $h$. The proof for this lemma then follows from the moment generating function of the hitting time for an $\infty$-state random walk.

(a) Optimal policy is $\pi_{o}$ (always in ordinary monitoring)

(b) Optimal policy is $\pi_{t,\bar{h}}$ (threshold policy with $\bar{h}=3$)

Figure 2: Optimal policies (numerically computed) for $H=6$ and two different parameter sets. (a) For $\lambda_{o}=0.2,\lambda_{i}=0.3,C_{c}=20,C_{i}=1,C_{o}=0,\gamma=0.9$ the optimal policy is $\pi_{o}$ (using ordinary monitoring at all health states). (b) For $\lambda_{o}=0.2,\lambda_{i}=0.3,C_{c}=60,C_{i}=1,C_{o}=0,\gamma=0.9$ the optimal policy is $\pi_{t,\bar{h}}$ with $\bar{h}=3$, that is, ordinary monitoring is used for $h>3$ and intensive for $h\leq 3$.

An important implication of the above lemma is that the policy where the patient chooses to stay in intensive monitoring for all health states is never optimal. Our next theorem shows that the policy $\pi_{o}$ is actually optimal for a large choice of parameters.

Theorem 1.

Under Assumption 2, the policy $\pi_{o}$ is optimal (${\pi^{*}}=\pi_{o}$) for the simplified RPM (Definition 1) when the parameters satisfy

$$\gamma(\lambda_{i}-\lambda_{o})(1-\phi^{2})\leq\frac{C_{i}}{C_{c}}.$$

Proof.

See Appendix A. ∎

We next define a threshold-policy $\pi_{t,\bar{h}}$ characterized by the health state ${\bar{h}}>0$. These are the policies under which there exists a threshold ${\bar{h}}$ such that the patient stays in intensive monitoring when the patient’s health is below or at the threshold ${\bar{h}}$ and in ordinary monitoring when their health is better than ${\bar{h}}$. Note that $\pi_{t,\bar{h}}\in\widetilde{\Pi}$. So $\pi_{t,\bar{h}}(h)=i$ for $1\leq h\leq{\bar{h}}$ and $\pi_{t,\bar{h}}(h)=o$ for $h>{\bar{h}}$. Our next theorem gives a set of conditions under which $\pi_{t,\bar{h}}$ is the optimal policy for some threshold ${\bar{h}}$.

Theorem 2.

Under Assumption 2, the policy $\pi_{t,\bar{h}}$ is optimal (${\pi^{*}}=\pi_{t,\bar{h}}$) for some threshold ${\bar{h}}$ for the simplified RPM (Definition 1) when the following two conditions are satisfied:

1. $\gamma(\lambda_{i}-\lambda_{o})(1-\phi^{2})>\frac{C_{i}}{C_{c}}$.

2. $\frac{\gamma\mu_{o}(1+\gamma\mu_{o})}{1-\gamma^{2}\lambda_{o}\mu_{o}}\leq 1$.

Proof.

See Appendix A. ∎

Condition a) here is precisely the complement of the condition in Theorem 1. Condition b) is an additional condition which our proof requires for the threshold policy to be optimal. In the asymptotic regime, we strongly believe that condition b) is not necessary, and condition a) alone is sufficient. Hence we believe that in the asymptotic regime, condition a) alone dictates what the optimal policy is and that the optimal policy can only be of two forms - $\pi_{o}$ and $\pi_{t,\bar{h}}$. This is reinforced by the numerical analysis, presented in the next section.

IV Performance

In this section, we glean insights on the optimal policy by numerically solving the dynamic programming equations given by (1)-(3) to find the optimal policy.

Figure 2 depicts the two policies discussed in the last section and a sample set of parameters under which they are optimal. Figure 2(a) shows the policy $\pi_{o}$, under which the patient stays in ordinary monitoring at all health states. Figure 2(b) shows the policy $\pi_{t,\bar{h}}$ with threshold $t=3$, where the patient stays in intensive monitoring for health states $h\leq 3$ and in ordinary monitoring for health states $h>3$. Let model 2(a) use the set of parameters $\lambda_{o}=0.2,\lambda_{i}=0.3,C_{c}=20,C_{i}=1,C_{o}=0,\gamma=0.9$ and model 2(b) use the set of parameters $\lambda_{o}=0.2,\lambda_{i}=0.3,C_{c}=60,C_{i}=1,C_{o}=0,\gamma=0.9$. Then $\pi_{o}$ and $\pi_{t,\bar{h}}$ with $t=3$ are optimal for model 2(a) and 2(b), respectively.

Note that the parameters for model 2(a) satisfy $\gamma(\lambda_{i}-\lambda_{o})(1-\phi^{2})\leq C_{i}/C_{c}$, which is sufficient for Theorem 1 to hold. Similarly the parameters for model 2(b) satisfy $\gamma(\lambda_{i}-\lambda_{o})(1-\phi^{2})>C_{i}/C_{c}$, which is condition (a) in Theorem 2. Note, however, that the parameters in model 2(b) do not satisfy condition (b) of Theorem 2, implying that the condition is not necessary.

When $H$ is finite, there also exist instances where the optimal policy is $\pi_{i}$, where the patient chooses to stay in intensive monitoring for all health states $h$. But we observed that this policy is optimal only in extreme cases where $H$ is very small or $\gamma$ is very close to $1$. Hence we do not further analyse this policy here.

Figure 3: The optimal (blue) value function $V^{*}(h)$ (numerically computed) compared to its asymptotic counterpart (red, $V_{o}(h)$ obtained from Lemma 1) for various health states $h$ in a system with $H=5,\lambda_{o}=0.2,\lambda_{i}=0.4,C_{c}=5,C_{i}=1,C_{o}=0,\gamma=0.9$. Note the close proximity of the two plots.

(a) Variation in optimal policy with $C_{c}/C_{i}$

(b) Variation in optimal policy with $\lambda_{i}$

(c) Variation in optimal policy with $\gamma$

Figure 4: Dependence of the (numerically computed) optimal monitoring policy on the (a) cost ratio $C_{c}/C_{i}$ (with fixed $\lambda_{o}=0.2,\lambda_{i}=0.4,C_{i}=1,C_{o}=0,\gamma=0.9$); on (b) $\lambda_{i}$ (with fixed $\lambda_{o}=0.2,C_{c}=50,C_{i}=1,C_{o}=0,\gamma=0.9$); and on (c) $\gamma$ (with fixed $\lambda_{o}=0.2,\lambda_{i}=0.4C_{c}=50,C_{i}=1,C_{o}=0$). We set $H=10$ in all cases. Below the vertical orange dashed line, the optimal policy is $\pi_{o}$ (ordinary monitoring is used for all health states). This line is positioned at the point where the condition of Theorem 1 achieves equality. Above this value the policy changes to $\pi_{t,\bar{h}}$ and each threshold $\bar{h}$ is marked in blue. Note that the threshold $\bar{h}$ is increasing as $C_{o}/C_{i}$ gets higher in (a), i.e., the patient stays “longer” under intensive monitoring. The same behavior is seen in (b) and (c).

Figure 3 shows how closely our asymptotic analysis in Section III relates to the actual solutions of the dynamic programming equations. The model parameters here are chosen such that the optimal policy is $\pi_{o}$. We calculate $V^{*}(h)$ for this model, and compare it with the value function $V_{o}(h)=\phi^{h}C_{c}$ obtained for the asymptotic case $H\to\infty$ (Lemma 1). As observed in the plot, the value function obtained for $H=5$ is almost identical to the asymptotic approximation $V_{o}(h)$, and they slightly differ only near the top health state $H$. The accuracy of the asymptotic approximation in predicting the optimal policy is further demonstrated by our next result.

We next study the impact of different parameters on the optimal policy ${\pi^{*}}$. Figure 4(a) shows how the optimal policy (numerically computed) varies with the cost ratio $C_{c}/C_{i}$. For $C_{c}/C_{i}<20.02$ the optimal policy is $\pi_{o}$, while for $C_{c}/C_{i}>20.02$ it is $\pi_{t,\bar{h}}$ with varied thresholds. Note that $C_{c}/C_{i}=20.02$ satisfies the condition in Theorem 1 with equality (the value of $C_{c}/C_{i}$ which satisfies $\gamma(\lambda_{i}-\lambda_{o})(1-\phi^{2})=C_{i}/C_{C}$). This shows that the condition obtained under the asymptotic assumption is a good indicator for our original problem with finite health states. As the ratio $C_{c}/C_{i}$ grows, the cost incurred on reaching the critical health state increases as compared to the cost of invasiveness, and it gets optimal for the patient to stay under intensive monitoring till their health significantly improves.

Figure 4(b) shows how the optimal policy varies as the probability $\lambda_{i}$ increases. The optimal policy is $\pi_{o}$ for $\lambda_{i}<0.28$ and $\pi_{t,\bar{h}}$ with varied thresholds for $\lambda_{i}>0.28$. Again, $\lambda_{i}=0.28$ solves the condition in Theorem 1 with equality. As the probability $\lambda_{i}$ increases, the probability of the patient’s health improving under intensive monitoring improves, incentivizing the patient to stay under intensive monitoring for longer. Finally, Figure 4(c) shows the impact of $\gamma$ on the optimal policy. As $\gamma$ increases, the patient incurs a higher discounted cost on reaching the critical state, and hence they stay under intensive monitoring for a broader range of states.

V Conclusions and Extensions

We have developed a two-tier service architecture for remote patient monitoring (RPM), where the service policy decides whether to place the patient under ordinary or intensive monitoring, given their health state. The optimal policy is first analyzed in asymptotic regimes and conditions are established for choosing ordinary vs intensive monitoring. The policy is then numerically computed and the dependence of its behavior on various key parameters is investigated.

An important extension would be to consider the general monitoring model, which includes non-zero transition costs. Based on numerical experiments performed in the general case, the optimal policy in this case would be a threshold policy with two thresholds instead of the one observed in this paper. A patient under ordinary monitoring would be switched to intensive when their health state deteriorates below a certain lower health threshold, and a patient under intensive monitoring would switch to ordinary when their health state improves above an upper health threshold.

There are many other direct extensions that we could not include in the limited space of this short paper. One can consider a service architecture with more than two tiers, where a patient can be under ordinary monitoring (tier m=0), or intensive monitoring tier m=1, or (more) intensive monitoring tier m=2 and so on, with corresponding health care features and attributes. The costs and probabilities of transitions could also be made dependent on the health state, allowing for a more realistic model.

Appendix A Proofs

A-A Proof for Lemma 1

Proof.

1. For the policy $\pi_{o}$, the value function is

   $$V_{o}(h)=\mathbb{E}\left[\sum_{t=0}^{T-1}(\gamma^{t}c(s_{t},\pi_{o}(s_{t})))+\gamma^{T}C_{c}|h_{0}=h\right].$$

   Here $T$ denotes the time at which the patient reaches health state $0$. Since $\pi_{o}(h)=o$, $c(s_{t},\pi_{o}(s_{t}))=0$ for all $t<T$. This implies that $V_{o}(h)=C_{c}\mathbb{E}[\gamma^{T}|h_{0}=h]$, where $T$ is the time at which the patient reaches health state $0$ and hence the hitting time of state $0$ for a random walk initialized at state $h$. Since $\mu_{o}>\lambda_{o}$, the probability that the patient reaches health state $0$ under this policy is $1$. Then [10, Chapter XIV, eqn. 4.8] gives us $V_{o}(h)=C_{c}\phi^{h},$ where

   $$\phi=\frac{1-\sqrt{1-4\lambda_{o}\mu_{o}\gamma^{2}}}{2\lambda_{o}\gamma}.$$

2. Let $h^{\prime}=\lceil\log(C_{i}/C_{c})/\log(\phi)\rceil+1$. Then for any $h\geq h^{\prime}$, $V_{o}(h)\leq V_{o}(h^{\prime})\leq\phi C_{i}$. Consider a policy $\pi^{\prime}$ such that $\pi^{\prime}(h)=i$. Then $V_{\pi^{\prime}}(h^{\prime})\geq C_{i}$. Now for any $h\geq h^{\prime}$, $V_{o}(h)<V_{\pi^{\prime}}(h)$ as $\phi<1$ for $\gamma<1$. This implies that the policy $\pi^{\prime}$ cannot be optimal, and hence under the optimal policy, the patient prefers to stay in ordinary monitoring for all health states $h\geq h^{\prime}$. This completes the proof for Lemma 1.

∎

A-B Proof for Theorem 1

Proof.

We know that a policy $\pi$ is optimal if and only if

$$c(s,\pi(s))+\gamma\sum_{s^{\prime}}p(s^{\prime}|s,\pi(s))V_{\pi}(s^{\prime})$$

$$=\min_{a}\left(c(s,a)+\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)V_{\pi}(s^{\prime})\right),$$

is true for all states $s$ [9, Proposition 2.2 and 2.3]. Hence policy $\pi_{o}$ is optimal if and only if

$$\gamma(\lambda_{o}V_{o}(h+1)+\mu_{o}V_{o}(h-1))$$

$$\leq C_{i}+\gamma(\lambda_{i}V_{o}(h+1)+\mu_{i}V_{o}(h-1)),$$ (4)

for all $1\geq h$. Using Lemma 1, we have that $\pi_{o}$ is optimal if and only if, for all $1\leq h$,

$$C_{c}\gamma(\lambda_{o}\phi^{h+1}+\mu_{o}\phi^{h-1})\leq C_{i}+C_{c}\gamma(\lambda_{i}\phi^{h+1}+\mu_{i}\phi^{h-1})$$

$$\iff C_{c}\gamma\phi^{h-1}((\lambda_{o}-\lambda_{i})\phi^{2}+(\mu_{o}-\mu_{i}))\leq C_{i}$$

$$\stackrel{{\scriptstyle(b)}}{{\iff}} C_{c}\gamma\phi^{h-1}(\lambda_{i}-\lambda_{o})(1-\phi^{2})\leq C_{i}.$$

Here (b) follows from the definition that $\mu_{o}=1-\lambda_{o}$ and $\mu_{i}=1-\lambda_{i}$. Note that if

$$\gamma(\lambda_{i}-\lambda_{o})(1-\phi^{2})\leq\frac{C_{i}}{C_{c}}$$

is true, then (A-B) is satisfied for all $h\geq 1$ which implies that the policy $\pi_{o}$ is optimal. This completes the proof for Theorem 1. ∎

A-C Proof for Theorem 2

Proof.

Since $\gamma(\lambda_{i}-\lambda_{o})(1-\phi^{2})>\frac{C_{i}}{C_{c}}$, eqn. (A-B) is not satisfied for $h=1$, and hence $\pi_{o}$ is not optimal, which implies that staying under intensive monitoring is optimal for some state. Define $Q^{*}(h,i)=C_{i}+\gamma(\lambda_{i}V^{*}(h{+}1)+\mu_{i}V^{*}(h{-}1))$ and $Q^{*}(h,o)=\gamma(\lambda_{o}V^{*}(h{+}1)+\mu_{o}V^{*}(h{-}1))$. This denotes the Q-functions, where action $i$ (or $o$, respectively) is taken when initialized at health state $h$ and then actions are taken using the optimal policy. Note that ${\pi^{*}}(h)=o$ if $Q^{*}(h,o)<Q^{*}(h,i)$, and ${\pi^{*}}(h)=i$ otherwise. If we show that $Q^{*}(h,o)-Q^{*}(h,i)$ is monotonically decreasing as $h$ increases, then if action $o$ is optimal at some $h$, then it will also be optimal at $h+1$ and so on. As condition (a) already enforces that policy $\pi_{o}$ is not optimal and we have shown in Lemma 1 part (b) that there exists $h^{\prime}$ such that the optimal policy for health states above $h^{\prime}$ is $o$, the optimal policy has to be $\pi_{t,\bar{h}}$ for some $\bar{h}$. We can show that

$$Q^{*}(h,o)-Q^{*}(h,i)=\gamma(\lambda_{i}-\lambda_{o})(V^{*}(h-1)-V^{*}(h+1))-C_{i}.$$

Now,

$$\big{(}Q^{*}(h+1,o)-Q^{*}(h+1,i)\big{)}-\big{(}Q^{*}(h,o)-Q^{*}(h,i)\big{)}$$

$$=\gamma(\lambda_{i}-\lambda_{o})\Big{(}V^{*}(h)-V^{*}(h+2)$$

$$\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;-V^{*}(h-1)+V^{*}(h+1)\Big{)}$$

Hence if $V^{*}(h)+V^{*}(h{+}1)\leq V^{*}(h{-}1)+V^{*}(h{+}2)$ is true for all $h\geq 1$ then $Q^{*}(h,o){-}Q^{*}(h,i)$ is monotonically decreasing with $h$. Now we know using [9, Proposition 2.2] that

$$V^{*}(h)\leq\gamma\lambda_{o}V^{*}(h+1)+\gamma\mu_{o}V^{*}(h-1),$$

and

$$V^{*}(h+1)\leq\gamma\lambda_{o}V^{*}(h+2)+\gamma\mu_{o}V^{*}(h).$$

With some further manipulation, we can show that

$$V^{*}(h)+V^{*}(h+1) \leq\frac{\gamma\lambda_{o}(1+\gamma\lambda_{o})}{1-\gamma^{2}\lambda_{o}\mu_{o}}V^{*}(h+2)$$

$$+\frac{\gamma\mu_{o}(1+\gamma\mu_{o})}{1-\gamma^{2}\lambda_{o}\mu_{o}}V^{*}(h-1).$$

We can show that $\frac{\gamma\lambda_{o}(1+\gamma\lambda_{o})}{1-\gamma^{2}\lambda_{o}\mu_{o}}$ is always less than $1$ for $\lambda_{o}\leq 0.5$. Hence if $\frac{\gamma\mu_{o}(1+\gamma\mu_{o})}{1-\gamma^{2}\lambda_{o}\mu_{o}}\leq 1$, then $V^{*}(h)+V^{*}(h{+}1)\leq V^{*}(h{-}1)+V^{*}(h{+}2)$ and hence $Q^{*}(h,o){-}Q^{*}(h,i)$ is monotonically decreasing with $h$. Under the additional condition that $\gamma(\lambda_{i}-\lambda_{o})(1-\phi^{2})>\frac{C_{i}}{C_{c}}$, this implies that $\pi_{t,\bar{h}}$ is the optimal policy for some threshold $\bar{h}$. ∎