-
Reinforced Model Predictive Control via Trust-Region Quasi-Newton Policy Optimization
Authors:
Dean Brandner,
Sergio Lucia
Abstract:
Model predictive control can optimally deal with nonlinear systems under consideration of constraints. The control performance depends on the model accuracy and the prediction horizon. Recent advances propose to use reinforcement learning applied to a parameterized model predictive controller to recover the optimal control performance even if an imperfect model or short prediction horizons are use…
▽ More
Model predictive control can optimally deal with nonlinear systems under consideration of constraints. The control performance depends on the model accuracy and the prediction horizon. Recent advances propose to use reinforcement learning applied to a parameterized model predictive controller to recover the optimal control performance even if an imperfect model or short prediction horizons are used. However, common reinforcement learning algorithms rely on first order updates, which only have a linear convergence rate and hence need an excessive amount of dynamic data. Higher order updates are typically intractable if the policy is approximated with neural networks due to the large number of parameters.
In this work, we use a parameterized model predictive controller as policy, and leverage the small amount of necessary parameters to propose a trust-region constrained Quasi-Newton training algorithm for policy optimization with a superlinear convergence rate. We show that the required second order derivative information can be calculated by the solution of a linear system of equations. A simulation study illustrates that the proposed training algorithm outperforms other algorithms in terms of data efficiency and accuracy.
△ Less
Submitted 28 May, 2024;
originally announced May 2024.
-
Improved uncertainty quantification for neural networks with Bayesian last layer
Authors:
Felix Fiedler,
Sergio Lucia
Abstract:
Uncertainty quantification is an important task in machine learning - a task in which standardneural networks (NNs) have traditionally not excelled. This can be a limitation for safety-critical applications, where uncertainty-aware methods like Gaussian processes or Bayesian linear regression are often preferred. Bayesian neural networks are an approach to address this limitation. They assume prob…
▽ More
Uncertainty quantification is an important task in machine learning - a task in which standardneural networks (NNs) have traditionally not excelled. This can be a limitation for safety-critical applications, where uncertainty-aware methods like Gaussian processes or Bayesian linear regression are often preferred. Bayesian neural networks are an approach to address this limitation. They assume probability distributions for all parameters and yield distributed predictions. However, training and inference are typically intractable and approximations must be employed. A promising approximation is NNs with Bayesian last layer (BLL). They assume distributed weights only in the linear output layer and yield a normally distributed prediction. To approximate the intractable Bayesian neural network, point estimates of the distributed weights in all but the last layer should be obtained by maximizing the marginal likelihood. This has previously been challenging, as the marginal likelihood is expensive to evaluate in this setting. We present a reformulation of the log-marginal likelihood of a NN with BLL which allows for efficient training using backpropagation. Furthermore, we address the challenge of uncertainty quantification for extrapolation points. We provide a metric to quantify the degree of extrapolation and derive a method to improve the uncertainty quantification for these points. Our methods are derived for the multivariate case and demonstrated in a simulation study. In comparison to Bayesian linear regression with fixed features, and a Bayesian neural network trained with variational inference, our proposed method achieves the highest log-predictive density on test data.
△ Less
Submitted 3 January, 2024; v1 submitted 21 February, 2023;
originally announced February 2023.
-
Optimization-Based Predictive Congestion Control for the Tor Network: Opportunities and Challenges
Authors:
Christoph Döpmann,
Felix Fiedler,
Sergio Lucia,
Florian Tschorsch
Abstract:
Based on the principle of onion routing, the Tor network achieves anonymity for its users by relaying user data over a series of intermediate relays. This approach makes congestion control in the network a challenging task. As of today, this results in higher latencies due to considerable backlog as well as unfair data rate allocation. In this paper, we present a concept study of PredicTor, a nove…
▽ More
Based on the principle of onion routing, the Tor network achieves anonymity for its users by relaying user data over a series of intermediate relays. This approach makes congestion control in the network a challenging task. As of today, this results in higher latencies due to considerable backlog as well as unfair data rate allocation. In this paper, we present a concept study of PredicTor, a novel approach to congestion control that tackles clogged overlay networks. Unlike traditional approaches, it is built upon the idea of distributed model predictive control, a recent advancement from the area of control theory. PredicTor is tailored to minimizing latency in the network and achieving max-min fairness. We contribute a thorough evaluation of its behavior in both toy scenarios to assess the optimizer and complex networks to assess its potential. For this, we conduct large-scale simulation studies and compare PredicTor to existing congestion control mechanisms in Tor. We show that PredicTor is highly effective in reducing latency and realizing fair rate allocations. In addition, we strive to bring the ideas of modern control theory to the networking community, enabling the development of improved, future congestion control. We therefore demonstrate benefits and issues alike with this novel research direction.
△ Less
Submitted 8 December, 2022;
originally announced December 2022.
-
Towards Grassroots Peering at the Edge
Authors:
David Bermbach,
Sergio Lucia,
Vlado Handziski,
Adam Wolisz
Abstract:
Fog Computing allows applications to address their latency and privacy requirements while co** with bandwidth limitations of Internet service providers (ISPs). Existing research on fog systems has so far mostly taken a very high-level view on the actual fog infrastructure. In this position paper, we identify and discuss the problem of having multiple ISPs in edge-to-edge communication. As a poss…
▽ More
Fog Computing allows applications to address their latency and privacy requirements while co** with bandwidth limitations of Internet service providers (ISPs). Existing research on fog systems has so far mostly taken a very high-level view on the actual fog infrastructure. In this position paper, we identify and discuss the problem of having multiple ISPs in edge-to-edge communication. As a possible solution we propose that edge operators create direct edge-to-edge links in a grassroots fashion and discuss different implementation options. Based on this, we highlight some important open research challenges that result from this.
△ Less
Submitted 20 October, 2021;
originally announced January 2022.
-
Sensor Artificial Intelligence and its Application to Space Systems -- A White Paper
Authors:
Anko Börner,
Heinz-Wilhelm Hübers,
Odej Kao,
Florian Schmidt,
Sören Becker,
Joachim Denzler,
Daniel Matolin,
David Haber,
Sergio Lucia,
Wojciech Samek,
Rudolph Triebel,
Sascha Eichstädt,
Felix Biessmann,
Anna Kruspe,
Peter Jung,
Manon Kok,
Guillermo Gallego,
Ralf Berger
Abstract:
Information and communication technologies have accompanied our everyday life for years. A steadily increasing number of computers, cameras, mobile devices, etc. generate more and more data, but at the same time we realize that the data can only partially be analyzed with classical approaches. The research and development of methods based on artificial intelligence (AI) made enormous progress in t…
▽ More
Information and communication technologies have accompanied our everyday life for years. A steadily increasing number of computers, cameras, mobile devices, etc. generate more and more data, but at the same time we realize that the data can only partially be analyzed with classical approaches. The research and development of methods based on artificial intelligence (AI) made enormous progress in the area of interpretability of data in recent years. With growing experience, both, the potential and limitations of these new technologies are increasingly better understood. Typically, AI approaches start with the data from which information and directions for action are derived. However, the circumstances under which such data are collected and how they change over time are rarely considered. A closer look at the sensors and their physical properties within AI approaches will lead to more robust and widely applicable algorithms. This holistic approach which considers entire signal chains from the origin to a data product, "Sensor AI", is a highly relevant topic with great potential. It will play a decisive role in autonomous driving as well as in areas of automated production, predictive maintenance or space research. The goal of this white paper is to establish "Sensor AI" as a dedicated research topic. We want to exchange knowledge on the current state-of-the-art on Sensor AI, to identify synergies among research groups and thus boost the collaboration in this key technology for science and industry.
△ Less
Submitted 9 June, 2020;
originally announced June 2020.
-
PredicTor: Predictive Congestion Control for the Tor Network
Authors:
Felix Fiedler,
Christoph Döpmann,
Florian Tschorsch,
Sergio Lucia
Abstract:
In the Tor network, anonymity is achieved through a multi-layered architecture, which comes at the cost of a complex network. Scheduling data in this network is a challenging task and the current approach shows to be incapable of avoiding network congestion and allocating fair data rates. We propose PredicTor, a distributed model predictive control approach, to tackle these challenges. PredicTor i…
▽ More
In the Tor network, anonymity is achieved through a multi-layered architecture, which comes at the cost of a complex network. Scheduling data in this network is a challenging task and the current approach shows to be incapable of avoiding network congestion and allocating fair data rates. We propose PredicTor, a distributed model predictive control approach, to tackle these challenges. PredicTor is designed to schedule incoming and outgoing data rates on individual nodes of the Tor architecture, leading to a scalable approach. We successfully avoid congestion through exchanging information of predicted behavior with adjacent nodes. Furthermore, we formulate PredicTor with a focus on fair allocation of resources, for which we present and proof a novel optimization-based fairness approach. Our proposed controller is evaluated with the popular network simulator ns-3, where we compare it with the current Tor scheduler as well as with another recently proposed enhancement. PredicTor shows significant improvements over the previous approaches, especially with respect to latency.
△ Less
Submitted 21 April, 2020;
originally announced April 2020.
