-
Constrained Markov decision processes for response-adaptive procedures in clinical trials with binary outcomes
Authors:
Stef Baas,
Aleida Braaksma,
Richard J. Boucherie
Abstract:
A constrained Markov decision process (CMDP) approach is developed for response-adaptive procedures in clinical trials with binary outcomes. The resulting CMDP class of Bayesian response -- adaptive procedures can be used to target a certain objective, e.g., patient benefit or power while using constraints to keep other operating characteristics under control. In the CMDP approach, the constraints…
▽ More
A constrained Markov decision process (CMDP) approach is developed for response-adaptive procedures in clinical trials with binary outcomes. The resulting CMDP class of Bayesian response -- adaptive procedures can be used to target a certain objective, e.g., patient benefit or power while using constraints to keep other operating characteristics under control. In the CMDP approach, the constraints can be formulated under different priors, which can induce a certain behaviour of the policy under a given statistical hypothesis, or given that the parameters lie in a specific part of the parameter space. A solution method is developed to find the optimal policy, as well as a more efficient method, based on backward recursion, which often yields a near-optimal solution with an available optimality gap. Three applications are considered, involving type I error and power constraints, constraints on the mean squared error, and a constraint on prior robustness. While the CMDP approach slightly outperforms the constrained randomized dynamic programming (CRDP) procedure known from literature when focussing on type I and II error and mean squared error, showing the general quality of CRDP, CMDP significantly outperforms CRDP when the focus is on type I and II error only.
△ Less
Submitted 28 January, 2024;
originally announced January 2024.
-
A stochastic programming approach for dynamic allocation of bed capacity and assignment of patients to collaborating hospitals during pandemic outbreaks
Authors:
Stef Baas,
Sander Dijkstra,
Richard J. Boucherie,
Anne Zander
Abstract:
We consider a region containing several hospitals collaborating to treat patients during an infectious outbreak. Collaboration occurs through the dynamic allocation of hospital bed capacity to infectious patients and by assigning infectious patients to specific hospitals. Scaling up capacity for infectious patients means that beds are removed from regular care, which can only be done simultaneousl…
▽ More
We consider a region containing several hospitals collaborating to treat patients during an infectious outbreak. Collaboration occurs through the dynamic allocation of hospital bed capacity to infectious patients and by assigning infectious patients to specific hospitals. Scaling up capacity for infectious patients means that beds are removed from regular care, which can only be done simultaneously for all beds in a room. Moreover, as opening rooms for infectious patients takes some lead time, we have to decide on preparing to open rooms for infectious patients ahead of time. We apply a stochastic direct lookahead approach. Each day, we make decisions on room allocation and patient assignment based on the solutions of two stochastic programs with scenarios using short-time forecasts of the number of infectious hospitalizations in the region and the bed occupancy in each collaborating hospital. We aim to balance costs for bed shortages and unutilized beds for infectious patients and opening and closing rooms. We demonstrate that a stochastic lookahead is superior to a deterministic one. We furthermore compare our solution approach with two heuristic strategies in a simulation study based on historical COVID-19 data of a region with three hospitals in the Netherlands. In one strategy, hospitals decide on their capacity allocation individually. In another strategy, we assume a pandemic unit, where one hospital is designated to take all regional infectious patients until full. The numerical results show that our stochastic direct lookahead approach considerably outperforms these heuristics.
△ Less
Submitted 27 November, 2023;
originally announced November 2023.
-
A Sampling-Based Method for Gittins Index Approximation
Authors:
Stef Baas,
Richard J. Boucherie,
Aleida Braaksma
Abstract:
A sampling-based method is introduced to approximate the Gittins index for a general family of alternative bandit processes. The approximation consists of a truncation of the optimization horizon and support for the immediate rewards, an optimal stop** value approximation, and a stochastic approximation procedure. Finite-time error bounds are given for the three approximations, leading to a proc…
▽ More
A sampling-based method is introduced to approximate the Gittins index for a general family of alternative bandit processes. The approximation consists of a truncation of the optimization horizon and support for the immediate rewards, an optimal stop** value approximation, and a stochastic approximation procedure. Finite-time error bounds are given for the three approximations, leading to a procedure to construct a confidence interval for the Gittins index using a finite number of Monte Carlo samples, as well as an epsilon-optimal policy for the Bayesian multi-armed bandit. Proofs are given for almost sure convergence and convergence in distribution for the sampling based Gittins index approximation. In a numerical study, the approximation quality of the proposed method is verified for the Bernoulli bandit and Gaussian bandit with known variance, and the method is shown to significantly outperform Thompson sampling and the Bayesian Upper Confidence Bound algorithms for a novel random effects multi-armed bandit.
△ Less
Submitted 21 July, 2023;
originally announced July 2023.
-
On the cycle maximum of birth-death processes and networks of queues
Authors:
Richard J. Boucherie
Abstract:
This paper considers the cycle maximum in birth-death processes as a step** stone to characterisation of the cycle maximum in single queues and open Kelly-Whittle networks of queues. For positive recurrent birth-death processes we show that the sequence of sample maxima is stochastically compact. For transient birth-death processes we show that the sequence of sample maxima conditioned on the ma…
▽ More
This paper considers the cycle maximum in birth-death processes as a step** stone to characterisation of the cycle maximum in single queues and open Kelly-Whittle networks of queues. For positive recurrent birth-death processes we show that the sequence of sample maxima is stochastically compact. For transient birth-death processes we show that the sequence of sample maxima conditioned on the maximum being finite is stochastically compact.
We show that the Markov chain recording the total number of customers in a Kelly-Whittle network is a birth-death process with birth and death rates determined by the normalising constants in a suitably defined sequence of closed networks. Explicit or asymptotic expressions for these normalising constants allow asymptotic evaluation of the birth and death rates, which, in turn, allows characterisation of the cycle maximum in a single busy cycle, and convergence of the sequence of sample maxima for Kelly-Whittle networks of queues.
△ Less
Submitted 31 May, 2022;
originally announced May 2022.
-
Bayesian Covariance Structure Modeling of Multi-Way Nested Data
Authors:
Stef Baas,
Richard J. Boucherie,
Jean-Paul Fox
Abstract:
A Bayesian multivariate model with a structured covariance matrix for multi-way nested data is proposed. This flexible modeling framework allows for positive and for negative associations among clustered observations, and generalizes the well-known dependence structure implied by random effects. A conjugate shifted-inverse gamma prior is proposed for the covariance parameters which ensures that th…
▽ More
A Bayesian multivariate model with a structured covariance matrix for multi-way nested data is proposed. This flexible modeling framework allows for positive and for negative associations among clustered observations, and generalizes the well-known dependence structure implied by random effects. A conjugate shifted-inverse gamma prior is proposed for the covariance parameters which ensures that the covariance matrix remains positive definite under posterior analysis. A numerically efficient Gibbs sampling procedure is defined for balanced nested designs, and is validated using two simulation studies. For a top-layer unbalanced nested design, the procedure requires an additional data augmentation step. The proposed data augmentation procedure facilitates sampling latent variables from (truncated) univariate normal distributions, and avoids numerical computation of the inverse of the structured covariance matrix. The Bayesian multivariate (linear transformation) model is applied to two-way nested interval-censored event times to analyze differences in adverse events between three groups of patients, who were randomly allocated to treatment with different stents (BIO-RESORT). The parameters of the structured covariance matrix represent unobserved heterogeneity in treatment effects and are examined to detect differential treatment effects.
△ Less
Submitted 25 January, 2022;
originally announced January 2022.
-
Solving correlation clustering with QAOA and a Rydberg qudit system: a full-stack approach
Authors:
Jordi R. Weggemans,
Alexander Urech,
Alexander Rausch,
Robert Spreeuw,
Richard Boucherie,
Florian Schreck,
Kareljan Schoutens,
Jiří Minář,
Florian Speelman
Abstract:
We study the correlation clustering problem using the quantum approximate optimization algorithm (QAOA) and qudits, which constitute a natural platform for such non-binary problems. Specifically, we consider a neutral atom quantum computer and propose a full stack approach for correlation clustering, including Hamiltonian formulation of the algorithm, analysis of its performance, identification of…
▽ More
We study the correlation clustering problem using the quantum approximate optimization algorithm (QAOA) and qudits, which constitute a natural platform for such non-binary problems. Specifically, we consider a neutral atom quantum computer and propose a full stack approach for correlation clustering, including Hamiltonian formulation of the algorithm, analysis of its performance, identification of a suitable level structure for ${}^{87}{\rm Sr}$ and specific gate design. We show the qudit implementation is superior to the qubit encoding as quantified by the gate count. For single layer QAOA, we also prove (conjecture) a lower bound of $0.6367$ ($0.6699$) for the approximation ratio on 3-regular graphs. Our numerical studies evaluate the algorithm's performance by considering complete and Erdős-Rényi graphs of up to 7 vertices and clusters. We find that in all cases the QAOA surpasses the Swamy bound $0.7666$ for the approximation ratio for QAOA depths $p \geq 2$. Finally, by analysing the effect of errors when solving complete graphs we find that their inclusion severely limits the algorithm's performance.
△ Less
Submitted 25 March, 2022; v1 submitted 22 June, 2021;
originally announced June 2021.
-
Data retrieval time for energy harvesting wireless sensor networks
Authors:
Mihaela Mitici,
Jasper Goseling,
Maurits de Graaf,
Richard J. Boucherie
Abstract:
We consider the problem of retrieving a reliable estimate of an attribute monitored by a wireless sensor network, where the sensors harvest energy from the environment independently, at random. Each sensor stores the harvested energy in batteries of limited capacity. Moreover, provided they have sufficient energy, the sensors broadcast their measurements in a decentralized fashion. Clients arrive…
▽ More
We consider the problem of retrieving a reliable estimate of an attribute monitored by a wireless sensor network, where the sensors harvest energy from the environment independently, at random. Each sensor stores the harvested energy in batteries of limited capacity. Moreover, provided they have sufficient energy, the sensors broadcast their measurements in a decentralized fashion. Clients arrive at the sensor network according to a Poisson process and are interested in retrieving a fixed number of sensor measurements, based on which a reliable estimate is computed. We show that the time until an arbitrary sensor broadcasts has a phase-type distribution. Based on this result and the theory of order statistics of phase-type distributions, we determine the probability distribution of the time needed for a client to retrieve a reliable estimate of an attribute monitored by the sensor network. We also provide closed-form expression for the retrieval time of a reliable estimate when the capacity of the sensor battery or the rate at which energy is harvested is asymptotically large. In addition, we analyze numerically the retrieval time of a reliable estimate for various sizes of the sensor network, maximum capacity of the sensor batteries and rate at which energy is harvested. These results show that the energy harvesting rate and the broadcasting rate are the main parameters that influence the retrieval time of a reliable estimate, while deploying sensors with large batteries does not significantly reduce the retrieval time.
△ Less
Submitted 13 October, 2015;
originally announced October 2015.
-
Performance measures for the two-node queue with finite buffers
Authors:
Yanting Chen,
Xinwei Bai,
Richard J. Boucherie,
Jasper Goseling
Abstract:
We consider a two-node queue modeled as a two-dimensional random walk. In particular, we consider the case that one or both queues have finite buffers. We develop an approximation scheme based on the Markov reward approach to error bounds in order to bound performance measures of such random walks in terms of a perturbed random walk in which the transitions along the boundaries are different from…
▽ More
We consider a two-node queue modeled as a two-dimensional random walk. In particular, we consider the case that one or both queues have finite buffers. We develop an approximation scheme based on the Markov reward approach to error bounds in order to bound performance measures of such random walks in terms of a perturbed random walk in which the transitions along the boundaries are different from those in the original model and the invariant measure of the perturbed random walk is of product-form. We then apply this approximation scheme to a tandem queue and some variants of this model, for the case that both buffers are finite. We also apply our approximation scheme to a coupled-queue in which only one of the buffers has finite capacity.
△ Less
Submitted 2 November, 2017; v1 submitted 27 February, 2015;
originally announced February 2015.
-
Invariant measures and error bounds for random walks in the quarter-plane based on sums of geometric terms
Authors:
Yanting Chen,
Richard J. Boucherie,
Jasper Goseling
Abstract:
We consider homogeneous random walks in the quarter-plane. The necessary conditions which characterize random walks of which the invariant measure is a sum of geometric terms are provided in [2,3]. Based on these results, we first develop an algorithm to check whether the invariant measure of a given random walk is a sum of geometric terms. We also provide the explicit form of the invariant measur…
▽ More
We consider homogeneous random walks in the quarter-plane. The necessary conditions which characterize random walks of which the invariant measure is a sum of geometric terms are provided in [2,3]. Based on these results, we first develop an algorithm to check whether the invariant measure of a given random walk is a sum of geometric terms. We also provide the explicit form of the invariant measure if it is a sum of geometric terms. Secondly, for random walks of which the invariant measure is not a sum of geometric terms, we provide an approximation scheme to obtain error bounds for the performance measures. Finally, some numerical examples are provided.
△ Less
Submitted 25 February, 2015;
originally announced February 2015.
-
A Linear Programming Approach to Error Bounds for Random Walks in the Quarter-plane
Authors:
Jasper Goseling,
Richard J. Boucherie,
Jan-Kees van Ommeren
Abstract:
We consider the approximation of the performance of random walks in the quarter-plane. The approximation is in terms of a random walk with a product-form stationary distribution, which is obtained by perturbing the transition probabilities along the boundaries of the state space. A Markov reward approach is used to bound the approximation error. The main contribution of the work is the formulation…
▽ More
We consider the approximation of the performance of random walks in the quarter-plane. The approximation is in terms of a random walk with a product-form stationary distribution, which is obtained by perturbing the transition probabilities along the boundaries of the state space. A Markov reward approach is used to bound the approximation error. The main contribution of the work is the formulation of a linear program that provides the approximation error.
△ Less
Submitted 12 September, 2014;
originally announced September 2014.
-
Necessary conditions for the invariant measure of a random walk to be a sum of geometric terms
Authors:
Yanting Chen,
Richard J. Boucherie,
Jasper Goseling
Abstract:
We consider the invariant measure of homogeneous random walks in the quarter-plane. In particular, we consider measures that can be expressed as an infinite sum of geometric terms. We present necessary conditions for the invariant measure to be a sum of geometric terms. We demonstrate that, under a mild regularity condition, each geometric term must individually satisfy the balance equations in th…
▽ More
We consider the invariant measure of homogeneous random walks in the quarter-plane. In particular, we consider measures that can be expressed as an infinite sum of geometric terms. We present necessary conditions for the invariant measure to be a sum of geometric terms. We demonstrate that, under a mild regularity condition, each geometric term must individually satisfy the balance equations in the interior of the state space. We show that the geometric terms in an invariant measure must be the union of finitely many pairwise-coupled sets of infinite cardinality. We further show that for the invariant measure to be a sum of geometric terms, the random walk cannot have transitions to the north, northeast or east. Finally, we show that for an infinite weighted sum of geometric terms to be an invariant measure at least one of the weights must be negative.
△ Less
Submitted 2 July, 2014; v1 submitted 11 April, 2013;
originally announced April 2013.
-
An Optimal Query Assignment for Wireless Sensor Networks
Authors:
Mihaela Mitici,
Martijn Onderwater,
Maurits de Graaf,
Jan-Kees van Ommeren,
Nico van Dijk,
Jasper Goseling,
Richard J. Boucherie
Abstract:
A trade-off between two QoS requirements of wireless sensor networks: query waiting time and validity (age) of the data feeding the queries, is investigated. We propose a Continuous Time Markov Decision Process with a drift that trades-off between the two QoS requirements by assigning incoming queries to the wireless sensor network or to the database. To compute an optimal assignment policy, we ar…
▽ More
A trade-off between two QoS requirements of wireless sensor networks: query waiting time and validity (age) of the data feeding the queries, is investigated. We propose a Continuous Time Markov Decision Process with a drift that trades-off between the two QoS requirements by assigning incoming queries to the wireless sensor network or to the database. To compute an optimal assignment policy, we argue, by means of non-standard uniformization, a discrete time Markov decision process, stochastically equivalent to the initial continuous process. We determine an optimal query assignment policy for the discrete time process by means of dynamic programming. Next, we assess numerically the performance of the optimal policy and show that it outperforms in terms of average assignment costs three other heuristics, commonly used in practice. Lastly, the optimality of the our model is confirmed also in the case of real query traffic, where our proposed policy achieves significant cost savings compared to the heuristics.
△ Less
Submitted 11 March, 2014; v1 submitted 21 December, 2012;
originally announced December 2012.
-
The Invariant Measure of Random Walks in the Quarter-plane: Representation in Geometric Terms
Authors:
Yanting Chen,
Richard J. Boucherie,
Jasper Goseling
Abstract:
We consider the invariant measure of homogeneous random walks in the quarter-plane. In particular, we consider measures that can be expressed as a finite linear combination of geometric terms and present conditions on the structure of these linear combinations such that the resulting measure may yield an invariant measure of a random walk. We demonstrate that each geometric term must individually…
▽ More
We consider the invariant measure of homogeneous random walks in the quarter-plane. In particular, we consider measures that can be expressed as a finite linear combination of geometric terms and present conditions on the structure of these linear combinations such that the resulting measure may yield an invariant measure of a random walk. We demonstrate that each geometric term must individually satisfy the balance equations in the interior of the state space and further show that the geometric terms in an invariant measure must have a pairwise-coupled structure. Finally, we show that at least one of the coefficients in the linear combination must be negative.
△ Less
Submitted 1 July, 2014; v1 submitted 23 July, 2012;
originally announced July 2012.
-
The Invariant Measure of Homogeneous Markov Processes in The Quarter-Plane: Representation in Geometric Terms
Authors:
Yanting Chen,
Richard J. Boucherie,
Jasper Goseling
Abstract:
We consider the invariant measure of a homogeneous continuous- time Markov process in the quarter-plane. The basic solutions of the global balance equation are the geometric distributions. We first show that the invariant measure can not be a finite linear combination of basic geometric distributions, unless it consists of a single basic geo- metric distribution. Second, we show that a countable l…
▽ More
We consider the invariant measure of a homogeneous continuous- time Markov process in the quarter-plane. The basic solutions of the global balance equation are the geometric distributions. We first show that the invariant measure can not be a finite linear combination of basic geometric distributions, unless it consists of a single basic geo- metric distribution. Second, we show that a countable linear combina- tion of geometric terms can be an invariant measure only if it consists of pairwise-coupled terms. As a consequence, we obtain a complete characterization of all countable linear combinations of geometric dis- tributions that may yield an invariant measure for a homogeneous continuous-time Markov process in the quarter-plane.
△ Less
Submitted 24 February, 2014; v1 submitted 13 December, 2011;
originally announced December 2011.
-
Time-Limited and k-Limited Polling Systems: A Matrix Analytic Solution
Authors:
Ahmad Al Hanbali,
Roland de Haan,
Richard J. Boucherie,
Jan-Kees van Ommeren
Abstract:
In this paper, we will develop a tool to analyze polling systems with the autonomous-server, the time-limited, and the k-limited service discipline. It is known that these disciplines do not satisfy the well-known branching property in polling system, therefore, hardly any exact result exists in the literature for them. Our strategy is to apply an iterative scheme that is based on relating in cl…
▽ More
In this paper, we will develop a tool to analyze polling systems with the autonomous-server, the time-limited, and the k-limited service discipline. It is known that these disciplines do not satisfy the well-known branching property in polling system, therefore, hardly any exact result exists in the literature for them. Our strategy is to apply an iterative scheme that is based on relating in closed-form the joint queue-length at the beginning and the end of a server visit to a queue. These kernel relations are derived using the theory of absorbing Markov chains. Finally, we will show that our tool works also in the case of a tandem queueing network with a single server that can serve one queue at a time.
△ Less
Submitted 5 October, 2009;
originally announced October 2009.
