-
Discrete-Time Mean Field Control with Environment States
Authors:
Kai Cui,
Anam Tahir,
Mark Sinzger,
Heinz Koeppl
Abstract:
Multi-agent reinforcement learning methods have shown remarkable potential in solving complex multi-agent problems but mostly lack theoretical guarantees. Recently, mean field control and mean field games have been established as a tractable solution for large-scale multi-agent problems with many agents. In this work, driven by a motivating scheduling problem, we consider a discrete-time mean fiel…
▽ More
Multi-agent reinforcement learning methods have shown remarkable potential in solving complex multi-agent problems but mostly lack theoretical guarantees. Recently, mean field control and mean field games have been established as a tractable solution for large-scale multi-agent problems with many agents. In this work, driven by a motivating scheduling problem, we consider a discrete-time mean field control model with common environment states. We rigorously establish approximate optimality as the number of agents grows in the finite agent case and find that a dynamic programming principle holds, resulting in the existence of an optimal stationary policy. As exact solutions are difficult in general due to the resulting continuous action space of the limiting mean field Markov decision process, we apply established deep reinforcement learning methods to solve the associated mean field control problem. The performance of the learned mean field control policy is compared to typical multi-agent reinforcement learning approaches and is found to converge to the mean field performance for sufficiently many agents, verifying the obtained theoretical results and reaching competitive solutions.
△ Less
Submitted 17 December, 2021; v1 submitted 30 April, 2021;
originally announced April 2021.
-
Poisson channel with binary Markov input and average sojourn time constraint
Authors:
Mark Sinzger,
Maximilian Gehri,
Heinz Koeppl
Abstract:
A minimal model for gene expression, consisting of a switchable promoter together with the resulting messenger RNA, is equivalent to a Poisson channel with a binary Markovian input process. Determining its capacity is an optimization problem with respect to two parameters: the average sojourn times of the promoter's active (ON) and inactive (OFF) state. An expression for the mutual information is…
▽ More
A minimal model for gene expression, consisting of a switchable promoter together with the resulting messenger RNA, is equivalent to a Poisson channel with a binary Markovian input process. Determining its capacity is an optimization problem with respect to two parameters: the average sojourn times of the promoter's active (ON) and inactive (OFF) state. An expression for the mutual information is found by solving the associated filtering problem analytically on the level of distributions. For fixed peak power, three bandwidth-like constraints are imposed by lower-bounding (i) the average sojourn times (ii) the autocorrelation time and (iii) the average time until a transition. OFF-favoring optima are found for all three constraints, as commonly encountered for the Poisson channel. In addition, constraint (i) exhibits a region that favors the ON state, and (iii) shows ON-favoring local optima.
△ Less
Submitted 5 January, 2021;
originally announced January 2021.
-
Bounds on the spectral radius of real-valued non-negative Kernels on measurable spaces
Authors:
Wasiur R. KhudaBukhsh,
Mark Sinzger,
Heinz Koeppl
Abstract:
In this short technical note, we extend a recently published result [Liao2017] on the Perron root (or the spectral radius) of non-negative matrices to real-valued non-negative kernels on an arbitrary measurable space $(\mathrm{E}, \mathcal{E})$. To be precise, for any real-valued non-negative kernel $K : \mathrm{E}\times \mathcal{E} \rightarrow \mathbb{R}$, we prove that the spectral radius…
▽ More
In this short technical note, we extend a recently published result [Liao2017] on the Perron root (or the spectral radius) of non-negative matrices to real-valued non-negative kernels on an arbitrary measurable space $(\mathrm{E}, \mathcal{E})$. To be precise, for any real-valued non-negative kernel $K : \mathrm{E}\times \mathcal{E} \rightarrow \mathbb{R}$, we prove that the spectral radius $ρ(K)$ of $K$ satisfies $$
\inf_{x \in \mathrm{E} } \frac{ \mathcal{R} K \cdotp L (x) }{ \mathcal{R} L (x) } \le ρ(K) \le \sup_{x \in \mathrm{E} } \frac{ \mathcal{R} K\cdotp L (x) }{ \mathcal{R} L (x) }, $$ where $L$ is an arbitrary Kernel on $(\mathrm{E}, \mathcal{E})$, which is integrable with respect to the left eigenmeasure of $K$ and satisfies $ \mathcal{R} L (x) >0 $ for all $x \in \mathrm{E}$, and the operator $\mathcal{R}$ is defined by $\mathcal{R}L (x) :=\int_{\mathrm{E}} L(x, \mathrm{d}y) $.
△ Less
Submitted 31 August, 2018; v1 submitted 1 August, 2018;
originally announced August 2018.
