-
Autonomous Port Navigation With Ranging Sensors Using Model-Based Reinforcement Learning
Authors:
Siemen Herremans,
Ali Anwar,
Arne Troch,
Ian Ravijts,
Maarten Vangeneugden,
Siegfried Mercelis,
Peter Hellinckx
Abstract:
Autonomous ship** has recently gained much interest in the research community. However, little research focuses on inland - and port navigation, even though this is identified by countries such as Belgium and the Netherlands as an essential step towards a sustainable future. These environments pose unique challenges, since they can contain dynamic obstacles that do not broadcast their location,…
▽ More
Autonomous ship** has recently gained much interest in the research community. However, little research focuses on inland - and port navigation, even though this is identified by countries such as Belgium and the Netherlands as an essential step towards a sustainable future. These environments pose unique challenges, since they can contain dynamic obstacles that do not broadcast their location, such as small vessels, kayaks or buoys. Therefore, this research proposes a navigational algorithm which can navigate an inland vessel in a wide variety of complex port scenarios using ranging sensors to observe the environment. The proposed methodology is based on a machine learning approach that has recently set benchmark results in various domains: model-based reinforcement learning. By randomizing the port environments during training, the trained model can navigate in scenarios that it never encountered during training. Furthermore, results show that our approach outperforms the commonly used dynamic window approach and a benchmark model-free reinforcement learning algorithm. This work is therefore a significant step towards vessels that can navigate autonomously in complex port scenarios.
△ Less
Submitted 17 November, 2023;
originally announced December 2023.
-
Safety Aware Autonomous Path Planning Using Model Predictive Reinforcement Learning for Inland Waterways
Authors:
Astrid Vanneste,
Simon Vanneste,
Olivier Vasseur,
Robin Janssens,
Mattias Billast,
Ali Anwar,
Kevin Mets,
Tom De Schepper,
Siegfried Mercelis,
Peter Hellinckx
Abstract:
In recent years, interest in autonomous ship** in urban waterways has increased significantly due to the trend of kee** cars and trucks out of city centers. Classical approaches such as Frenet frame based planning and potential field navigation often require tuning of many configuration parameters and sometimes even require a different configuration depending on the situation. In this paper, w…
▽ More
In recent years, interest in autonomous ship** in urban waterways has increased significantly due to the trend of kee** cars and trucks out of city centers. Classical approaches such as Frenet frame based planning and potential field navigation often require tuning of many configuration parameters and sometimes even require a different configuration depending on the situation. In this paper, we propose a novel path planning approach based on reinforcement learning called Model Predictive Reinforcement Learning (MPRL). MPRL calculates a series of waypoints for the vessel to follow. The environment is represented as an occupancy grid map, allowing us to deal with any shape of waterway and any number and shape of obstacles. We demonstrate our approach on two scenarios and compare the resulting path with path planning using a Frenet frame and path planning based on a proximal policy optimization (PPO) agent. Our results show that MPRL outperforms both baselines in both test scenarios. The PPO based approach was not able to reach the goal in either scenario while the Frenet frame approach failed in the scenario consisting of a corner with obstacles. MPRL was able to safely (collision free) navigate to the goal in both of the test scenarios.
△ Less
Submitted 16 November, 2023;
originally announced November 2023.
-
An In-Depth Analysis of Discretization Methods for Communication Learning using Backpropagation with Multi-Agent Reinforcement Learning
Authors:
Astrid Vanneste,
Simon Vanneste,
Kevin Mets,
Tom De Schepper,
Siegfried Mercelis,
Peter Hellinckx
Abstract:
Communication is crucial in multi-agent reinforcement learning when agents are not able to observe the full state of the environment. The most common approach to allow learned communication between agents is the use of a differentiable communication channel that allows gradients to flow between agents as a form of feedback. However, this is challenging when we want to use discrete messages to redu…
▽ More
Communication is crucial in multi-agent reinforcement learning when agents are not able to observe the full state of the environment. The most common approach to allow learned communication between agents is the use of a differentiable communication channel that allows gradients to flow between agents as a form of feedback. However, this is challenging when we want to use discrete messages to reduce the message size, since gradients cannot flow through a discrete communication channel. Previous work proposed methods to deal with this problem. However, these methods are tested in different communication learning architectures and environments, making it hard to compare them. In this paper, we compare several state-of-the-art discretization methods as well as a novel approach. We do this comparison in the context of communication learning using gradients from other agents and perform tests on several environments. In addition, we present COMA-DIAL, a communication learning approach based on DIAL and COMA extended with learning rate scaling and adapted exploration. Using COMA-DIAL allows us to perform experiments on more complex environments. Our results show that the novel ST-DRU method, proposed in this paper, achieves the best results out of all discretization methods across the different environments. It achieves the best or close to the best performance in each of the experiments and is the only method that does not fail on any of the tested environments.
△ Less
Submitted 9 August, 2023;
originally announced August 2023.
-
Scalability of Message Encoding Techniques for Continuous Communication Learned with Multi-Agent Reinforcement Learning
Authors:
Astrid Vanneste,
Thomas Somers,
Simon Vanneste,
Kevin Mets,
Tom De Schepper,
Siegfried Mercelis,
Peter Hellinckx
Abstract:
Many multi-agent systems require inter-agent communication to properly achieve their goal. By learning the communication protocol alongside the action protocol using multi-agent reinforcement learning techniques, the agents gain the flexibility to determine which information should be shared. However, when the number of agents increases we need to create an encoding of the information contained in…
▽ More
Many multi-agent systems require inter-agent communication to properly achieve their goal. By learning the communication protocol alongside the action protocol using multi-agent reinforcement learning techniques, the agents gain the flexibility to determine which information should be shared. However, when the number of agents increases we need to create an encoding of the information contained in these messages. In this paper, we investigate the effect of increasing the amount of information that should be contained in a message and increasing the number of agents. We evaluate these effects on two different message encoding methods, the mean message encoder and the attention message encoder. We perform our experiments on a matrix environment. Surprisingly, our results show that the mean message encoder consistently outperforms the attention message encoder. Therefore, we analyse the communication protocol used by the agents that use the mean message encoder and can conclude that the agents use a combination of an exponential and a logarithmic function in their communication policy to avoid the loss of important information after applying the mean message encoder.
△ Less
Submitted 9 August, 2023;
originally announced August 2023.
-
Deep set conditioned latent representations for action recognition
Authors:
Akash Singh,
Tom De Schepper,
Kevin Mets,
Peter Hellinckx,
Jose Oramas,
Steven Latre
Abstract:
In recent years multi-label, multi-class video action recognition has gained significant popularity. While reasoning over temporally connected atomic actions is mundane for intelligent species, standard artificial neural networks (ANN) still struggle to classify them. In the real world, atomic actions often temporally connect to form more complex composite actions. The challenge lies in recognisin…
▽ More
In recent years multi-label, multi-class video action recognition has gained significant popularity. While reasoning over temporally connected atomic actions is mundane for intelligent species, standard artificial neural networks (ANN) still struggle to classify them. In the real world, atomic actions often temporally connect to form more complex composite actions. The challenge lies in recognising composite action of varying durations while other distinct composite or atomic actions occur in the background. Drawing upon the success of relational networks, we propose methods that learn to reason over the semantic concept of objects and actions. We empirically show how ANNs benefit from pretraining, relational inductive biases and unordered set-based latent representations. In this paper we propose deep set conditioned I3D (SCI3D), a two stream relational network that employs latent representation of state and visual representation for reasoning over events and actions. They learn to reason about temporally connected actions in order to identify all of them in the video. The proposed method achieves an improvement of around 1.49% mAP in atomic action recognition and 17.57% mAP in composite action recognition, over a I3D-NL baseline, on the CATER dataset.
△ Less
Submitted 21 December, 2022;
originally announced December 2022.
-
An Analysis of Discretization Methods for Communication Learning with Multi-Agent Reinforcement Learning
Authors:
Astrid Vanneste,
Simon Vanneste,
Kevin Mets,
Tom De Schepper,
Siegfried Mercelis,
Steven Latré,
Peter Hellinckx
Abstract:
Communication is crucial in multi-agent reinforcement learning when agents are not able to observe the full state of the environment. The most common approach to allow learned communication between agents is the use of a differentiable communication channel that allows gradients to flow between agents as a form of feedback. However, this is challenging when we want to use discrete messages to redu…
▽ More
Communication is crucial in multi-agent reinforcement learning when agents are not able to observe the full state of the environment. The most common approach to allow learned communication between agents is the use of a differentiable communication channel that allows gradients to flow between agents as a form of feedback. However, this is challenging when we want to use discrete messages to reduce the message size since gradients cannot flow through a discrete communication channel. Previous work proposed methods to deal with this problem. However, these methods are tested in different communication learning architectures and environments, making it hard to compare them. In this paper, we compare several state-of-the-art discretization methods as well as two methods that have not been used for communication learning before. We do this comparison in the context of communication learning using gradients from other agents and perform tests on several environments. Our results show that none of the methods is best in all environments. The best choice in discretization method greatly depends on the environment. However, the discretize regularize unit (DRU), straight through DRU and the straight through gumbel softmax show the most consistent results across all the tested environments. Therefore, these methods prove to be the best choice for general use while the straight through estimator and the gumbel softmax may provide better results in specific environments but fail completely in others.
△ Less
Submitted 12 April, 2022;
originally announced April 2022.
-
Learning to Communicate with Reinforcement Learning for an Adaptive Traffic Control System
Authors:
Simon Vanneste,
Gauthier de Borrekens,
Stig Bosmans,
Astrid Vanneste,
Kevin Mets,
Siegfried Mercelis,
Steven Latré,
Peter Hellinckx
Abstract:
Recent work in multi-agent reinforcement learning has investigated inter agent communication which is learned simultaneously with the action policy in order to improve the team reward. In this paper, we investigate independent Q-learning (IQL) without communication and differentiable inter-agent learning (DIAL) with learned communication on an adaptive traffic control system (ATCS). In real world…
▽ More
Recent work in multi-agent reinforcement learning has investigated inter agent communication which is learned simultaneously with the action policy in order to improve the team reward. In this paper, we investigate independent Q-learning (IQL) without communication and differentiable inter-agent learning (DIAL) with learned communication on an adaptive traffic control system (ATCS). In real world ATCS, it is impossible to present the full state of the environment to every agent so in our simulation, the individual agents will only have a limited observation of the full state of the environment. The ATCS will be simulated using the Simulation of Urban MObility (SUMO) traffic simulator in which two connected intersections are simulated. Every intersection is controlled by an agent which has the ability to change the direction of the traffic flow. Our results show that a DIAL agent outperforms an independent Q-learner on both training time and on maximum achieved reward as it is able to share relevant information with the other agents.
△ Less
Submitted 29 October, 2021;
originally announced October 2021.
-
Mixed Cooperative-Competitive Communication Using Multi-Agent Reinforcement Learning
Authors:
Astrid Vanneste,
Wesley Van Wijnsberghe,
Simon Vanneste,
Kevin Mets,
Siegfried Mercelis,
Steven Latré,
Peter Hellinckx
Abstract:
By using communication between multiple agents in multi-agent environments, one can reduce the effects of partial observability by combining one agent's observation with that of others in the same dynamic environment. While a lot of successful research has been done towards communication learning in cooperative settings, communication learning in mixed cooperative-competitive settings is also impo…
▽ More
By using communication between multiple agents in multi-agent environments, one can reduce the effects of partial observability by combining one agent's observation with that of others in the same dynamic environment. While a lot of successful research has been done towards communication learning in cooperative settings, communication learning in mixed cooperative-competitive settings is also important and brings its own complexities such as the opposing team overhearing the communication. In this paper, we apply differentiable inter-agent learning (DIAL), designed for cooperative settings, to a mixed cooperative-competitive setting. We look at the difference in performance between communication that is private for a team and communication that can be overheard by the other team. Our research shows that communicating agents are able to achieve similar performance to fully observable agents after a given training period in our chosen environment. Overall, we find that sharing communication across teams results in decreased performance for the communicating team in comparison to results achieved with private communication.
△ Less
Submitted 29 October, 2021;
originally announced October 2021.
-
A Review of the Deep Sea Treasure problem as a Multi-Objective Reinforcement Learning Benchmark
Authors:
Amber Cassimon,
Reinout Eyckerman,
Siegfried Mercelis,
Steven Latré,
Peter Hellinckx
Abstract:
In this paper, the authors investigate the Deep Sea Treasure (DST) problem as proposed by Vamplew et al. Through a number of proofs, the authors show the original DST problem to be quite basic, and not always representative of practical Multi-Objective Optimization problems. In an attempt to bring theory closer to practice, the authors propose an alternative, improved version of the DST problem, a…
▽ More
In this paper, the authors investigate the Deep Sea Treasure (DST) problem as proposed by Vamplew et al. Through a number of proofs, the authors show the original DST problem to be quite basic, and not always representative of practical Multi-Objective Optimization problems. In an attempt to bring theory closer to practice, the authors propose an alternative, improved version of the DST problem, and prove that some of the properties that simplify the original DST problem no longer hold. The authors also provide a reference implementation and perform a comparison between their implementation, and other existing open-source implementations of the problem. Finally, the authors also provide a complete Pareto-front for their new DST problem.
△ Less
Submitted 21 May, 2024; v1 submitted 13 October, 2021;
originally announced October 2021.
-
Exploiting non-i.i.d. data towards more robust machine learning algorithms
Authors:
Wim Casteels,
Peter Hellinckx
Abstract:
In the field of machine learning there is a growing interest towards more robust and generalizable algorithms. This is for example important to bridge the gap between the environment in which the training data was collected and the environment where the algorithm is deployed. Machine learning algorithms have increasingly been shown to excel in finding patterns and correlations from data. Determini…
▽ More
In the field of machine learning there is a growing interest towards more robust and generalizable algorithms. This is for example important to bridge the gap between the environment in which the training data was collected and the environment where the algorithm is deployed. Machine learning algorithms have increasingly been shown to excel in finding patterns and correlations from data. Determining the consistency of these patterns and for example the distinction between causal correlations and nonsensical spurious relations has proven to be much more difficult. In this paper a regularization scheme is introduced that prefers universal causal correlations. This approach is based on 1) the robustness of causal correlations and 2) the data not being independently and identically distribute (i.i.d.). The scheme is demonstrated with a classification task by clustering the (non-i.i.d.) training set in subpopulations. A non-i.i.d. regularization term is then introduced that penalizes weights that are not invariant over these clusters. The resulting algorithm favours correlations that are universal over the subpopulations and indeed a better performance is obtained on an out-of-distribution test set with respect to a more conventional l_2-regularization.
△ Less
Submitted 7 October, 2020;
originally announced October 2020.
-
Learning to Communicate Using Counterfactual Reasoning
Authors:
Simon Vanneste,
Astrid Vanneste,
Kevin Mets,
Tom De Schepper,
Ali Anwar,
Siegfried Mercelis,
Steven Latré,
Peter Hellinckx
Abstract:
Learning to communicate in order to share state information is an active problem in the area of multi-agent reinforcement learning (MARL). The credit assignment problem, the non-stationarity of the communication environment and the creation of influenceable agents are major challenges within this research field which need to be overcome in order to learn a valid communication protocol. This paper…
▽ More
Learning to communicate in order to share state information is an active problem in the area of multi-agent reinforcement learning (MARL). The credit assignment problem, the non-stationarity of the communication environment and the creation of influenceable agents are major challenges within this research field which need to be overcome in order to learn a valid communication protocol. This paper introduces the novel multi-agent counterfactual communication learning (MACC) method which adapts counterfactual reasoning in order to overcome the credit assignment problem for communicating agents. Secondly, the non-stationarity of the communication environment while learning the communication Q-function is overcome by creating the communication Q-function using the action policy of the other agents and the Q-function of the action environment. Additionally, a social loss function is introduced in order to create influenceable agents which is required to learn a valid communication protocol. Our experiments show that MACC is able to outperform the state-of-the-art baselines in four different scenarios in the Particle environment.
△ Less
Submitted 26 April, 2022; v1 submitted 12 June, 2020;
originally announced June 2020.
-
Towards evaluating emergent behavior of the Internet of Things using large scale simulation techniques
Authors:
Stig Bosmans,
Siegfried Mercelis Peter Hellinckx,
Joachim Denil
Abstract:
With the increase in Internet of Things devices and more decentralized architectures we see a new type of application gain importance, a type where local interactions between individual entities lead to a global emergent behavior, Emergent-based IoT (EBI) Systems. In this position paper we explore techniques to evaluate this emergent behavior in IoT applications. Because of the required scale and…
▽ More
With the increase in Internet of Things devices and more decentralized architectures we see a new type of application gain importance, a type where local interactions between individual entities lead to a global emergent behavior, Emergent-based IoT (EBI) Systems. In this position paper we explore techniques to evaluate this emergent behavior in IoT applications. Because of the required scale and diversity this is not an easy task. Therefore, we mainly focus on a distributed simulation approach and provide an overview of possible techniques that could optimize the overall simulation performance. Our focus is both on modeling and simulation technology.
△ Less
Submitted 26 February, 2018;
originally announced February 2018.
