-
A Benchmark Environment for Offline Reinforcement Learning in Racing Games
Authors:
Girolamo Macaluso,
Alessandro Sestini,
Andrew D. Bagdanov
Abstract:
Offline Reinforcement Learning (ORL) is a promising approach to reduce the high sample complexity of traditional Reinforcement Learning (RL) by eliminating the need for continuous environmental interactions. ORL exploits a dataset of pre-collected transitions and thus expands the range of application of RL to tasks in which the excessive environment queries increase training time and decrease effi…
▽ More
Offline Reinforcement Learning (ORL) is a promising approach to reduce the high sample complexity of traditional Reinforcement Learning (RL) by eliminating the need for continuous environmental interactions. ORL exploits a dataset of pre-collected transitions and thus expands the range of application of RL to tasks in which the excessive environment queries increase training time and decrease efficiency, such as in modern AAA games. This paper introduces OfflineMania a novel environment for ORL research. It is inspired by the iconic TrackMania series and developed using the Unity 3D game engine. The environment simulates a single-agent racing game in which the objective is to complete the track through optimal navigation. We provide a variety of datasets to assess ORL performance. These datasets, created from policies of varying ability and in different sizes, aim to offer a challenging testbed for algorithm development and evaluation. We further establish a set of baselines for a range of Online RL, ORL, and hybrid Offline to Online RL approaches using our environment.
△ Less
Submitted 12 July, 2024;
originally announced July 2024.
-
Reinforcement Learning for High-Level Strategic Control in Tower Defense Games
Authors:
Joakim Bergdahl,
Alessandro Sestini,
Linus Gisslén
Abstract:
In strategy games, one of the most important aspects of game design is maintaining a sense of challenge for players. Many mobile titles feature quick gameplay loops that allow players to progress steadily, requiring an abundance of levels and puzzles to prevent them from reaching the end too quickly. As with any content creation, testing and validation are essential to ensure engaging gameplay mec…
▽ More
In strategy games, one of the most important aspects of game design is maintaining a sense of challenge for players. Many mobile titles feature quick gameplay loops that allow players to progress steadily, requiring an abundance of levels and puzzles to prevent them from reaching the end too quickly. As with any content creation, testing and validation are essential to ensure engaging gameplay mechanics, enjoyable game assets, and playable levels. In this paper, we propose an automated approach that can be leveraged for gameplay testing and validation that combines traditional scripted methods with reinforcement learning, rea** the benefits of both approaches while adapting to new situations similarly to how a human player would. We test our solution on a popular tower defense game, Plants vs. Zombies. The results show that combining a learned approach, such as reinforcement learning, with a scripted AI produces a higher-performing and more robust agent than using only heuristic AI, achieving a 57.12% success rate compared to 47.95% in a set of 40 levels. Moreover, the results demonstrate the difficulty of training a general agent for this type of puzzle-like game.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Leveraging Large Language Models for Efficient Failure Analysis in Game Development
Authors:
Leonardo Marini,
Linus Gisslén,
Alessandro Sestini
Abstract:
In games, and more generally in the field of software development, early detection of bugs is vital to maintain a high quality of the final product. Automated tests are a powerful tool that can catch a problem earlier in development by executing periodically. As an example, when new code is submitted to the code base, a new automated test verifies these changes. However, identifying the specific c…
▽ More
In games, and more generally in the field of software development, early detection of bugs is vital to maintain a high quality of the final product. Automated tests are a powerful tool that can catch a problem earlier in development by executing periodically. As an example, when new code is submitted to the code base, a new automated test verifies these changes. However, identifying the specific change responsible for a test failure becomes harder when dealing with batches of changes -- especially in the case of a large-scale project such as a AAA game, where thousands of people contribute to a single code base. This paper proposes a new approach to automatically identify which change in the code caused a test to fail. The method leverages Large Language Models (LLMs) to associate error messages with the corresponding code changes causing the failure. We investigate the effectiveness of our approach with quantitative and qualitative evaluations. Our approach reaches an accuracy of 71% in our newly created dataset, which comprises issues reported by developers at EA over a period of one year. We further evaluated our model through a user study to assess the utility and usability of the tool from a developer perspective, resulting in a significant reduction in time -- up to 60% -- spent investigating issues.
△ Less
Submitted 11 June, 2024;
originally announced June 2024.
-
Piecewise rational rotation-minimizing motions via data stream interpolation
Authors:
Carlotta Giannelli,
Lorenzo Sacco,
Alessandra Sestini,
Zbyněk Šír
Abstract:
When a moving frame defined along a space curve is required to keep an axis aligned with the tangent direction of motion, the use of rotation-minimizing frames (RMF) avoids unnecessary rotations in the normal plane. The construction of rigid body motions using a specific subset of quintic curves with rational RMFs (RRMFs) is here considered. In particular, a novel geometric characterization of suc…
▽ More
When a moving frame defined along a space curve is required to keep an axis aligned with the tangent direction of motion, the use of rotation-minimizing frames (RMF) avoids unnecessary rotations in the normal plane. The construction of rigid body motions using a specific subset of quintic curves with rational RMFs (RRMFs) is here considered. In particular, a novel geometric characterization of such subset enables the design of a local algorithm to interpolate an assigned stream of positions, together with an initial frame orientation. To achieve this, the translational part of the motion is described by a parametric $G^1$ spline curve whose segments are quintic RRMFs, with a globally continuous piecewise rational rotation-minimizing frame. A selection of numerical experiments illustrates the performances of the proposed method on synthetic and arbitrary data streams.
△ Less
Submitted 12 June, 2024; v1 submitted 23 May, 2024;
originally announced May 2024.
-
Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation
Authors:
Girolamo Macaluso,
Alessandro Sestini,
Andrew D. Bagdanov
Abstract:
Offline reinforcement learning leverages pre-collected datasets of transitions to train policies. It can serve as effective initialization for online algorithms, enhancing sample efficiency and speeding up convergence. However, when such datasets are limited in size and quality, offline pre-training can produce sub-optimal policies and lead to degraded online reinforcement learning performance. In…
▽ More
Offline reinforcement learning leverages pre-collected datasets of transitions to train policies. It can serve as effective initialization for online algorithms, enhancing sample efficiency and speeding up convergence. However, when such datasets are limited in size and quality, offline pre-training can produce sub-optimal policies and lead to degraded online reinforcement learning performance. In this paper we propose a model-based data augmentation strategy to maximize the benefits of offline reinforcement learning pre-training and reduce the scale of data needed to be effective. Our approach leverages a world model of the environment trained on the offline dataset to augment states during offline pre-training. We evaluate our approach on a variety of MuJoCo robotic tasks and our results show it can jump-start online fine-tuning and substantially reduce - in some cases by an order of magnitude - the required number of environment interactions.
△ Less
Submitted 19 December, 2023; v1 submitted 15 December, 2023;
originally announced December 2023.
-
Local spline refinement driven by fault jump estimates for scattered data approximation
Authors:
Cesare Bracco,
Carlotta Giannelli,
Francesco Patrizi,
Alessandra Sestini
Abstract:
We present new fault jump estimates to guide local refinement in surface approximation schemes with adaptive spline constructions. The proposed approach is based on the idea that, since discontinuities in the data should naturally correspond to sharp variations in the reconstructed surface, the location and size of jumps detected in the input point cloud should drive the mesh refinement algorithm.…
▽ More
We present new fault jump estimates to guide local refinement in surface approximation schemes with adaptive spline constructions. The proposed approach is based on the idea that, since discontinuities in the data should naturally correspond to sharp variations in the reconstructed surface, the location and size of jumps detected in the input point cloud should drive the mesh refinement algorithm. To exploit the possibility of inserting local meshlines in one or the other coordinate direction, as suggested by the jump estimates, we propose a quasi-interpolation (QI) scheme based on locally refined B-splines (LR B-splines). Particular attention is devoted to the construction of the local operator of the LR B-spline QI scheme, which properly adapts the spline approximation according to the nature and density of the scattered data configuration. A selection of numerical examples outlines the performance of the method on synthetic and real datasets characterized by different geographical features.
△ Less
Submitted 26 June, 2024; v1 submitted 15 November, 2023;
originally announced November 2023.
-
Improving Generalization in Game Agents with Data Augmentation in Imitation Learning
Authors:
Derek Yadgaroff,
Alessandro Sestini,
Konrad Tollmar,
Ayca Ozcelikkale,
Linus Gisslén
Abstract:
Imitation learning is an effective approach for training game-playing agents and, consequently, for efficient game production. However, generalization - the ability to perform well in related but unseen scenarios - is an essential requirement that remains an unsolved challenge for game AI. Generalization is difficult for imitation learning agents because it requires the algorithm to take meaningfu…
▽ More
Imitation learning is an effective approach for training game-playing agents and, consequently, for efficient game production. However, generalization - the ability to perform well in related but unseen scenarios - is an essential requirement that remains an unsolved challenge for game AI. Generalization is difficult for imitation learning agents because it requires the algorithm to take meaningful actions outside of the training distribution. In this paper we propose a solution to this challenge. Inspired by the success of data augmentation in supervised learning, we augment the training data so the distribution of states and actions in the dataset better represents the real state-action distribution. This study evaluates methods for combining and applying data augmentations to observations, to improve generalization of imitation learning agents. It also provides a performance benchmark of these augmentations across several 3D environments. These results demonstrate that data augmentation is a promising framework for improving generalization in imitation learning agents.
△ Less
Submitted 7 April, 2024; v1 submitted 22 September, 2023;
originally announced September 2023.
-
Generating Personas for Games with Multimodal Adversarial Imitation Learning
Authors:
William Ahlberg,
Alessandro Sestini,
Konrad Tollmar,
Linus Gisslén
Abstract:
Reinforcement learning has been widely successful in producing agents capable of playing games at a human level. However, this requires complex reward engineering, and the agent's resulting policy is often unpredictable. Going beyond reinforcement learning is necessary to model a wide range of human playstyles, which can be difficult to represent with a reward function. This paper presents a novel…
▽ More
Reinforcement learning has been widely successful in producing agents capable of playing games at a human level. However, this requires complex reward engineering, and the agent's resulting policy is often unpredictable. Going beyond reinforcement learning is necessary to model a wide range of human playstyles, which can be difficult to represent with a reward function. This paper presents a novel imitation learning approach to generate multiple persona policies for playtesting. Multimodal Generative Adversarial Imitation Learning (MultiGAIL) uses an auxiliary input parameter to learn distinct personas using a single-agent model. MultiGAIL is based on generative adversarial imitation learning and uses multiple discriminators as reward models, inferring the environment reward by comparing the agent and distinct expert policies. The reward from each discriminator is weighted according to the auxiliary input. Our experimental analysis demonstrates the effectiveness of our technique in two environments with continuous and discrete action spaces.
△ Less
Submitted 15 August, 2023;
originally announced August 2023.
-
Technical Challenges of Deploying Reinforcement Learning Agents for Game Testing in AAA Games
Authors:
Jonas Gillberg,
Joakim Bergdahl,
Alessandro Sestini,
Andrew Eakins,
Linus Gisslen
Abstract:
Going from research to production, especially for large and complex software systems, is fundamentally a hard problem. In large-scale game production, one of the main reasons is that the development environment can be very different from the final product. In this technical paper we describe an effort to add an experimental reinforcement learning system to an existing automated game testing soluti…
▽ More
Going from research to production, especially for large and complex software systems, is fundamentally a hard problem. In large-scale game production, one of the main reasons is that the development environment can be very different from the final product. In this technical paper we describe an effort to add an experimental reinforcement learning system to an existing automated game testing solution based on scripted bots in order to increase its capacity. We report on how this reinforcement learning system was integrated with the aim to increase test coverage similar to [1] in a set of AAA games including Battlefield 2042 and Dead Space (2023). The aim of this technical paper is to show a use-case of leveraging reinforcement learning in game production and cover some of the largest time sinks anyone who wants to make the same journey for their game may encounter. Furthermore, to help the game industry to adopt this technology faster, we propose a few research directions that we believe will be valuable and necessary for making machine learning, and especially reinforcement learning, an effective tool in game production.
△ Less
Submitted 19 July, 2023;
originally announced July 2023.
-
Efficient Ground Vehicle Path Following in Game AI
Authors:
Rodrigue de Schaetzen,
Alessandro Sestini
Abstract:
This short paper presents an efficient path following solution for ground vehicles tailored to game AI. Our focus is on adapting established techniques to design simple solutions with parameters that are easily tunable for an efficient benchmark path follower. Our solution pays particular attention to computing a target speed which uses quadratic Bezier curves to estimate the path curvature. The p…
▽ More
This short paper presents an efficient path following solution for ground vehicles tailored to game AI. Our focus is on adapting established techniques to design simple solutions with parameters that are easily tunable for an efficient benchmark path follower. Our solution pays particular attention to computing a target speed which uses quadratic Bezier curves to estimate the path curvature. The performance of the proposed path follower is evaluated through a variety of test scenarios in a first-person shooter game, demonstrating its effectiveness and robustness in handling different types of paths and vehicles. We achieved a 70% decrease in the total number of stuck events compared to an existing path following solution.
△ Less
Submitted 7 July, 2023;
originally announced July 2023.
-
Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learning
Authors:
Alessandro Sestini,
Joakim Bergdahl,
Konrad Tollmar,
Andrew D. Bagdanov,
Linus Gisslén
Abstract:
In games, as in and many other domains, design validation and testing is a huge challenge as systems are growing in size and manual testing is becoming infeasible. This paper proposes a new approach to automated game validation and testing. Our method leverages a data-driven imitation learning technique, which requires little effort and time and no knowledge of machine learning or programming, tha…
▽ More
In games, as in and many other domains, design validation and testing is a huge challenge as systems are growing in size and manual testing is becoming infeasible. This paper proposes a new approach to automated game validation and testing. Our method leverages a data-driven imitation learning technique, which requires little effort and time and no knowledge of machine learning or programming, that designers can use to efficiently train game testing agents. We investigate the validity of our approach through a user study with industry experts. The survey results show that our method is indeed a valid approach to game validation and that data-driven programming would be a useful aid to reducing effort and increasing quality of modern playtesting. The survey also highlights several open challenges. With the help of the most recent literature, we analyze the identified challenges and propose future research directions suitable for supporting and maximizing the utility of our approach.
△ Less
Submitted 19 August, 2022; v1 submitted 15 August, 2022;
originally announced August 2022.
-
Contextual Decision Trees
Authors:
Tommaso Aldinucci,
Enrico Civitelli,
Leonardo di Gangi,
Alessandro Sestini
Abstract:
Focusing on Random Forests, we propose a multi-armed contextual bandit recommendation framework for feature-based selection of a single shallow tree of the learned ensemble. The trained system, which works on top of the Random Forest, dynamically identifies a base predictor that is responsible for providing the final output. In this way, we obtain local interpretations by observing the rules of th…
▽ More
Focusing on Random Forests, we propose a multi-armed contextual bandit recommendation framework for feature-based selection of a single shallow tree of the learned ensemble. The trained system, which works on top of the Random Forest, dynamically identifies a base predictor that is responsible for providing the final output. In this way, we obtain local interpretations by observing the rules of the recommended tree. The carried out experiments reveal that our dynamic method is superior to an independent fitted CART decision tree and comparable to the whole black-box Random Forest in terms of predictive performances.
△ Less
Submitted 13 July, 2022;
originally announced July 2022.
-
IgA-BEM for 3D Helmholtz problems on multi-patch domains using B-spline tailored numerical integration
Authors:
Antonella Falini,
Tadej Kanduc,
Maria Lucia Sampoli,
Alessandra Sestini
Abstract:
An Isogeometric Boundary Element Method (IgA-BEM) is considered for the numerical solution of Helmholtz problems on 3D bounded or unbounded domains, admitting a smooth conformal multi-patch representation of their finite boundary surface. The discretization space is formed by $C^0$ inter-patch continuous basis functions whose restriction to a patch simplifies to the span of tensor product B-spline…
▽ More
An Isogeometric Boundary Element Method (IgA-BEM) is considered for the numerical solution of Helmholtz problems on 3D bounded or unbounded domains, admitting a smooth conformal multi-patch representation of their finite boundary surface. The discretization space is formed by $C^0$ inter-patch continuous basis functions whose restriction to a patch simplifies to the span of tensor product B-splines composed with the given patch parameterization. For both regular and singular integration, the proposed model utilizes a numerical procedure defined on the support of each trial B-spline function, which makes possible a function--by--function implementation of the matrix assembly phase. Spline quasi-interpolation is the common ingredient of all the considered quadrature rules; in the singular case it is combined with a B-spline recursion over the spline degree and with a singularity extraction technique, extended to the multi-patch setting for the first time. A threshold selection strategy is proposed to automatically distinguish between nearly singular and regular integrals. Numerical examples on relevant benchmarks show that the expected convergence orders are achieved with uniform discretization and a small number of uniformly spaced quadrature nodes.
△ Less
Submitted 23 December, 2022; v1 submitted 8 April, 2022;
originally announced April 2022.
-
A collocation IGA-BEM for 3D potential problems on unbounded domains
Authors:
Antonella Falini,
Carlotta Giannelli,
Tadej Kanduc,
Maria Lucia Sampoli,
Alessandra Sestini
Abstract:
In this paper the numerical solution of potential problems defined on 3D unbounded domains is addressed with Boundary Element Methods (BEMs), since in this way the problem is studied only on the boundary, and thus any finite approximation of the infinite domain can be avoided. The isogeometric analysis (IGA) setting is considered and in particular B-splines and NURBS functions are taken into accou…
▽ More
In this paper the numerical solution of potential problems defined on 3D unbounded domains is addressed with Boundary Element Methods (BEMs), since in this way the problem is studied only on the boundary, and thus any finite approximation of the infinite domain can be avoided. The isogeometric analysis (IGA) setting is considered and in particular B-splines and NURBS functions are taken into account. In order to exploit all the possible benefits from using spline spaces, an important point is the development of specific cubature formulas for weakly and nearly singular integrals. Our proposal for this aim is based on spline quasi-interpolation and on the use of a spline product formula. Besides that, a robust singularity extraction procedure is introduced as a preliminary step and an efficient function-by-function assembly phase is adopted. A selection of numerical examples confirms that the numerical solutions reach the expected convergence orders.
△ Less
Submitted 6 April, 2022;
originally announced April 2022.
-
CCPT: Automatic Gameplay Testing and Validation with Curiosity-Conditioned Proximal Trajectories
Authors:
Alessandro Sestini,
Linus Gisslén,
Joakim Bergdahl,
Konrad Tollmar,
Andrew D. Bagdanov
Abstract:
This paper proposes a novel deep reinforcement learning algorithm to perform automatic analysis and detection of gameplay issues in complex 3D navigation environments. The Curiosity-Conditioned Proximal Trajectories (CCPT) method combines curiosity and imitation learning to train agents to methodically explore in the proximity of known trajectories derived from expert demonstrations. We show how C…
▽ More
This paper proposes a novel deep reinforcement learning algorithm to perform automatic analysis and detection of gameplay issues in complex 3D navigation environments. The Curiosity-Conditioned Proximal Trajectories (CCPT) method combines curiosity and imitation learning to train agents to methodically explore in the proximity of known trajectories derived from expert demonstrations. We show how CCPT can explore complex environments, discover gameplay issues and design oversights in the process, and recognize and highlight them directly to game designers. We further demonstrate the effectiveness of the algorithm in a novel 3D navigation environment which reflects the complexity of modern AAA video games. Our results show a higher level of coverage and bug discovery than baselines methods, and it hence can provide a valuable tool for game designers to identify issues in game design automatically.
△ Less
Submitted 21 February, 2022;
originally announced February 2022.
-
A local $C^2$ Hermite interpolation scheme with PH quintic splines for 3D data streams
Authors:
Carlotta Giannelli,
Lorenzo Sacco,
Alessandra Sestini
Abstract:
The construction of smooth spatial paths with Pythagorean-hodograph (PH) quintic spline biarcs is proposed. To facilitate real-time computations of $C^2$ PH quintic splines, an efficient local data stream interpolation algorithm is introduced. Each spline segment interpolates second and first order Hermite data at the initial and final end-point, respectively. In the spline extension of the scheme…
▽ More
The construction of smooth spatial paths with Pythagorean-hodograph (PH) quintic spline biarcs is proposed. To facilitate real-time computations of $C^2$ PH quintic splines, an efficient local data stream interpolation algorithm is introduced. Each spline segment interpolates second and first order Hermite data at the initial and final end-point, respectively. In the spline extension of the scheme a $C^2$ smooth connection between successive spline segments is obtained by taking the locally required second-order derivative information from the previous segment. Consequently, the data stream spline interpolant is globally $C^2$ continuous and can be constructed for arbitrary $C^1$ Hermite data configurations. A simple and effective selection of the free parameters that arise in the interpolation problem is proposed. The developed theoretical analysis proves the fourth approximation order of the local scheme while a selection of numerical examples confirms the same accuracy of its spline extension. In addition, the performances of the algorithm are also validated by considering its application to point stream interpolation with automatically generated first-order derivative information.
△ Less
Submitted 29 August, 2021;
originally announced August 2021.
-
Policy Fusion for Adaptive and Customizable Reinforcement Learning Agents
Authors:
Alessandro Sestini,
Alexander Kuhnle,
Andrew D. Bagdanov
Abstract:
In this article we study the problem of training intelligent agents using Reinforcement Learning for the purpose of game development. Unlike systems built to replace human players and to achieve super-human performance, our agents aim to produce meaningful interactions with the player, and at the same time demonstrate behavioral traits as desired by game designers. We show how to combine distinct…
▽ More
In this article we study the problem of training intelligent agents using Reinforcement Learning for the purpose of game development. Unlike systems built to replace human players and to achieve super-human performance, our agents aim to produce meaningful interactions with the player, and at the same time demonstrate behavioral traits as desired by game designers. We show how to combine distinct behavioral policies to obtain a meaningful "fusion" policy which comprises all these behaviors. To this end, we propose four different policy fusion methods for combining pre-trained policies. We further demonstrate how these methods can be used in combination with Inverse Reinforcement Learning in order to create intelligent agents with specific behavioral styles as chosen by game designers, without having to define many and possibly poorly-designed reward functions. Experiments on two different environments indicate that entropy-weighted policy fusion significantly outperforms all others. We provide several practical examples and use-cases for how these methods are indeed useful for video game production and designers.
△ Less
Submitted 21 April, 2021;
originally announced April 2021.
-
Isogemetric Analysis and Symmetric Galerkin BEM: a 2D numerical study
Authors:
A. Aimi,
M. Diligenti,
M. L. Sampoli,
A. Sestini
Abstract:
Isogeometric approach applied to Boundary Element Methods is an emerging research area. In this context, the aim of the present contribution is that of investigating, from a numerical point of view, the Symmetric Galerkin Boundary Element Method (SGBEM) devoted to the solution of 2D boundary value problems for the Laplace equation, where the boundary and the unknowns on it are both represented by…
▽ More
Isogeometric approach applied to Boundary Element Methods is an emerging research area. In this context, the aim of the present contribution is that of investigating, from a numerical point of view, the Symmetric Galerkin Boundary Element Method (SGBEM) devoted to the solution of 2D boundary value problems for the Laplace equation, where the boundary and the unknowns on it are both represented by B-splines. We mainly compare this approach, which we call IGA-SGBEM, with a curvilinear SGBEM, which operates on any boundary given by explicit parametric representation and where the approximate solution is obtained using Lagrangian basis. Both techniques are further compared with a standard (conventional) SGBEM approach, where the boundary of the assigned problem is approximated by linear elements and the numerical solution is expressed in terms of Lagrangian basis. Several examples will be presented and discussed, underlying benefits and drawbacks of all the above-mentioned approaches.
△ Less
Submitted 16 March, 2021;
originally announced March 2021.
-
Deep Policy Networks for NPC Behaviors that Adapt to Changing Design Parameters in Roguelike Games
Authors:
Alessandro Sestini,
Alexander Kuhnle,
Andrew D. Bagdanov
Abstract:
Recent advances in Deep Reinforcement Learning (DRL) have largely focused on improving the performance of agents with the aim of replacing humans in known and well-defined environments. The use of these techniques as a game design tool for video game production, where the aim is instead to create Non-Player Character (NPC) behaviors, has received relatively little attention until recently. Turn-ba…
▽ More
Recent advances in Deep Reinforcement Learning (DRL) have largely focused on improving the performance of agents with the aim of replacing humans in known and well-defined environments. The use of these techniques as a game design tool for video game production, where the aim is instead to create Non-Player Character (NPC) behaviors, has received relatively little attention until recently. Turn-based strategy games like Roguelikes, for example, present unique challenges to DRL. In particular, the categorical nature of their complex game state, composed of many entities with different attributes, requires agents able to learn how to compare and prioritize these entities. Moreover, this complexity often leads to agents that overfit to states seen during training and that are unable to generalize in the face of design changes made during development. In this paper we propose two network architectures which, when combined with a \emph{procedural loot generation} system, are able to better handle complex categorical state spaces and to mitigate the need for retraining forced by design decisions. The first is based on a dense embedding of the categorical input space that abstracts the discrete observation model and renders trained agents more able to generalize. The second proposed architecture is more general and is based on a Transformer network able to reason relationally about input and input attributes. Our experimental evaluation demonstrates that new agents have better adaptation capacity with respect to a baseline architecture, making this framework more robust to dynamic gameplay changes during development. Based on the results shown in this paper, we believe that these solutions represent a step forward towards making DRL more accessible to the gaming industry.
△ Less
Submitted 7 December, 2020;
originally announced December 2020.
-
Demonstration-efficient Inverse Reinforcement Learning in Procedurally Generated Environments
Authors:
Alessandro Sestini,
Alexander Kuhnle,
Andrew D. Bagdanov
Abstract:
Deep Reinforcement Learning achieves very good results in domains where reward functions can be manually engineered. At the same time, there is growing interest within the community in using games based on Procedurally Content Generation (PCG) as benchmark environments since this type of environment is perfect for studying overfitting and generalization of agents under domain shift. Inverse Reinfo…
▽ More
Deep Reinforcement Learning achieves very good results in domains where reward functions can be manually engineered. At the same time, there is growing interest within the community in using games based on Procedurally Content Generation (PCG) as benchmark environments since this type of environment is perfect for studying overfitting and generalization of agents under domain shift. Inverse Reinforcement Learning (IRL) can instead extrapolate reward functions from expert demonstrations, with good results even on high-dimensional problems, however there are no examples of applying these techniques to procedurally-generated environments. This is mostly due to the number of demonstrations needed to find a good reward model. We propose a technique based on Adversarial Inverse Reinforcement Learning which can significantly decrease the need for expert demonstrations in PCG games. Through the use of an environment with a limited set of initial seed levels, plus some modifications to stabilize training, we show that our approach, DE-AIRL, is demonstration-efficient and still able to extrapolate reward functions which generalize to the fully procedural domain. We demonstrate the effectiveness of our technique on two procedural environments, MiniGrid and DeepCrawl, for a variety of tasks.
△ Less
Submitted 4 December, 2020;
originally announced December 2020.
-
DeepCrawl: Deep Reinforcement Learning for Turn-based Strategy Games
Authors:
Alessandro Sestini,
Alexander Kuhnle,
Andrew D. Bagdanov
Abstract:
In this paper we introduce DeepCrawl, a fully-playable Roguelike prototype for iOS and Android in which all agents are controlled by policy networks trained using Deep Reinforcement Learning (DRL). Our aim is to understand whether recent advances in DRL can be used to develop convincing behavioral models for non-player characters in videogames. We begin with an analysis of requirements that such a…
▽ More
In this paper we introduce DeepCrawl, a fully-playable Roguelike prototype for iOS and Android in which all agents are controlled by policy networks trained using Deep Reinforcement Learning (DRL). Our aim is to understand whether recent advances in DRL can be used to develop convincing behavioral models for non-player characters in videogames. We begin with an analysis of requirements that such an AI system should satisfy in order to be practically applicable in video game development, and identify the elements of the DRL model used in the DeepCrawl prototype. The successes and limitations of DeepCrawl are documented through a series of playability tests performed on the final game. We believe that the techniques we propose offer insight into innovative new avenues for the development of behaviors for non-player characters in video games, as they offer the potential to overcome critical issues with
△ Less
Submitted 3 December, 2020;
originally announced December 2020.
-
Cubature rules based on bivariate spline quasi-interpolation for weakly singular integrals
Authors:
A. Falini,
T. Kanduč,
M. L. Sampoli,
A. Sestini
Abstract:
In this paper we present a new class of cubature rules with the aim of accurately integrating weakly singular double integrals. In particular we focus on those integrals coming from the discretization of Boundary Integral Equations for 3D Laplace boundary value problems, using a collocation method within the Isogeometric Analysis paradigm. In such setting the regular part of the integrand can be d…
▽ More
In this paper we present a new class of cubature rules with the aim of accurately integrating weakly singular double integrals. In particular we focus on those integrals coming from the discretization of Boundary Integral Equations for 3D Laplace boundary value problems, using a collocation method within the Isogeometric Analysis paradigm. In such setting the regular part of the integrand can be defined as the product of a tensor product B-spline and a general function. The rules are derived by using first the spline quasi-interpolation approach to approximate such function and then the extension of a well known algorithm for spline product to the bivariate setting. In this way efficiency is ensured, since the locality of any spline quasi-interpolation scheme is combined with the capability of an ad--hoc treatment of the B-spline factor. The numerical integration is performed on the whole support of the B-spline factor by exploiting inter-element continuity of the integrands
△ Less
Submitted 15 August, 2020;
originally announced August 2020.
-
THB-spline approximations for turbine blade design with local B-spline approximations
Authors:
Cesare Bracco,
Carlotta Giannelli,
David Grossmann,
Sofia Imperatore,
Dominik Mokris,
Alessandra Sestini
Abstract:
We consider two-stage scattered data fitting with truncated hierarchical B-splines (THB-splines) for the adaptive reconstruction of industrial models. The first stage of the scheme is devoted to the computation of local least squares variational spline approximations, exploiting a simple fairness functional to handle data distributions with a locally varying density of points. Hierarchical spline…
▽ More
We consider two-stage scattered data fitting with truncated hierarchical B-splines (THB-splines) for the adaptive reconstruction of industrial models. The first stage of the scheme is devoted to the computation of local least squares variational spline approximations, exploiting a simple fairness functional to handle data distributions with a locally varying density of points. Hierarchical spline quasi-interpolation based on THB-splines is considered in the second stage of the method to construct the adaptive spline surface approximating the whole scattered data set and a suitable strategy to guide the adaptive refinement is introduced. A selection of examples on geometric models representing components of aircraft turbine blades highlights the performances of the scheme. The tests include a scattered data set with voids and the adaptive reconstruction of a cylinder-like surface.
△ Less
Submitted 8 July, 2020; v1 submitted 19 March, 2020;
originally announced March 2020.
-
An application of numerical differentiation formulas to discontinuity curve detection from irregularly sampled data
Authors:
Cesare Bracco,
Oleg Davydov,
Carlotta Giannelli,
Alessandra Sestini
Abstract:
We present a method to detect discontinuity curves, usually called faults, from a set of scattered data. The scheme first extracts from the data set a subset of points close to the faults. This selection is based on an indicator obtained by using numerical differentiation formulas with irregular centers for gradient approximation, since they can be directly applied to the scattered point cloud wit…
▽ More
We present a method to detect discontinuity curves, usually called faults, from a set of scattered data. The scheme first extracts from the data set a subset of points close to the faults. This selection is based on an indicator obtained by using numerical differentiation formulas with irregular centers for gradient approximation, since they can be directly applied to the scattered point cloud without intermediate approximations on a grid. The shape of the faults is reconstructed through local computations of regression lines and quadratic least squares approximations. In the final reconstruction stage, a suitable curve interpolation algorithm is applied to the selected set of ordered points previously associated with each fault.
△ Less
Submitted 15 May, 2019; v1 submitted 18 December, 2018;
originally announced December 2018.
-
An adaptive IGA-BEM with hierarchical B-splines based on quasi-interpolation quadrature schemes
Authors:
Antonella Falini,
Carlotta Giannelli,
Tadej Kanduc,
Maria Lucia Sampoli,
Alessandra Sestini
Abstract:
The isogeometric formulation of Boundary Element Method (BEM) is investigated within the adaptivity framework. Suitable weighted quadrature rules to evaluate integrals appearing in the Galerkin BEM formulation of 2D Laplace model problems are introduced. The new quadrature schemes are based on a spline quasi-interpolant (QI) operator and properly framed in the hierarchical setting. The local natur…
▽ More
The isogeometric formulation of Boundary Element Method (BEM) is investigated within the adaptivity framework. Suitable weighted quadrature rules to evaluate integrals appearing in the Galerkin BEM formulation of 2D Laplace model problems are introduced. The new quadrature schemes are based on a spline quasi-interpolant (QI) operator and properly framed in the hierarchical setting. The local nature of the QI perfectly fits with hierarchical spline constructions and leads to an efficient and accurate numerical scheme. An automatic adaptive refinement strategy is driven by a residual based error estimator. Numerical examples show that the optimal convergence rate of the BEM solution is recovered by the proposed adaptive method.
△ Less
Submitted 12 July, 2018; v1 submitted 10 July, 2018;
originally announced July 2018.
-
C2 continuous time dependent feedrate scheduling with configurable kinematic constraints
Authors:
Carlotta Giannelli,
Duccio Mugnaini,
Alessandra Sestini
Abstract:
We present a configurable trajectory planning strategy on planar paths for offline definition of time-dependent C2 piecewise quintic feedrates. The more conservative formulation ensures chord tolerance, as well as prescribed bounds on velocity, acceleration and jerk Cartesian components. Since the less restrictive formulations of our strategy can usually still ensure all the desired bounds while s…
▽ More
We present a configurable trajectory planning strategy on planar paths for offline definition of time-dependent C2 piecewise quintic feedrates. The more conservative formulation ensures chord tolerance, as well as prescribed bounds on velocity, acceleration and jerk Cartesian components. Since the less restrictive formulations of our strategy can usually still ensure all the desired bounds while simultaneously producing faster motions, the configurability feature is useful not only when reduced motion control is desired but also when full kinematic control has to be guaranteed. Our approach can be applied to any planar path with a piecewise sufficiently smooth parametric representation. When Pythagoreanhodograph spline curves are considered, the corresponding accurate and efficient CNC interpolator algorithms can be exploited.
△ Less
Submitted 21 November, 2017;
originally announced November 2017.
-
Adaptive scattered data fitting by extension of local approximations to hierarchical splines
Authors:
Cesare Bracco,
Carlotta Giannelli,
Alessandra Sestini
Abstract:
We introduce an adaptive scattered data fitting scheme as extension of local least squares approximations to hierarchical spline spaces. To efficiently deal with non-trivial data configurations, the local solutions are described in terms of (variable degree) polynomial approximations according not only to the number of data points locally available, but also to the smallest singular value of the l…
▽ More
We introduce an adaptive scattered data fitting scheme as extension of local least squares approximations to hierarchical spline spaces. To efficiently deal with non-trivial data configurations, the local solutions are described in terms of (variable degree) polynomial approximations according not only to the number of data points locally available, but also to the smallest singular value of the local collocation matrices. These local approximations are subsequently combined without the need of additional computations with the construction of hierarchical quasi-interpolants described in terms of truncated hierarchical B-splines. A selection of numerical experiments shows the effectivity of our approach for the approximation of real scattered data sets describing different terrain configurations.
△ Less
Submitted 27 April, 2017;
originally announced April 2017.
-
Efficient assembly based on B-spline tailored quadrature rules for the IgA-SGBEM
Authors:
A. Aimi,
F. Calabrò,
M. Diligenti,
M. L. Sampoli,
G. Sangalli,
A. Sestini
Abstract:
This paper deals with the discrete counterpart of 2D elliptic model problems rewritten in terms of Boundary Integral Equations. The study is done within the framework of Isogeometric Analysis based on B-splines. In such a context, the problem of constructing appropriate, accurate and efficient quadrature rules for the Symmetric Galerkin Boundary Element Method is here investigated. The new integra…
▽ More
This paper deals with the discrete counterpart of 2D elliptic model problems rewritten in terms of Boundary Integral Equations. The study is done within the framework of Isogeometric Analysis based on B-splines. In such a context, the problem of constructing appropriate, accurate and efficient quadrature rules for the Symmetric Galerkin Boundary Element Method is here investigated. The new integration schemes, together with row assembly and sum factorization, are used to build a more efficient strategy to derive the final linear system of equations. Key ingredients are weighted quadrature rules tailored for B--splines, that are constructed to be exact in the whole test space, also with respect to the singular kernel. Several simulations are presented and discussed, showing accurate evaluation of the involved integrals and outlining the superiority of the new approach in terms of computational cost and elapsed time with respect to the standard element-by-element assembly.
△ Less
Submitted 13 July, 2017; v1 submitted 29 March, 2017;
originally announced March 2017.
-
A comprehensive characterization of the set of polynomial curves with rational rotation-minimizing frames
Authors:
Rida T. Farouki,
Graziano Gentili,
Carlotta Giannelli,
Alessandra Sestini,
Caterina Stoppato
Abstract:
A rotation-minimizing frame $({\bf f}_1,{\bf f}_2,{\bf f}_3)$ on a space curve ${\bf r}(ξ)$ defines an orthonormal basis for $\mathbb{R}^3$ in which ${\bf f}_1={\bf r}'/|{\bf r}'|$ is the curve tangent, and the normal-plane vectors ${\bf f}_2$, ${\bf f}_3$ exhibit no instantaneous rotation about ${\bf f}_1$. Polynomial curves that admit rational rotation-minimizing frames (or RRMF curves) form a s…
▽ More
A rotation-minimizing frame $({\bf f}_1,{\bf f}_2,{\bf f}_3)$ on a space curve ${\bf r}(ξ)$ defines an orthonormal basis for $\mathbb{R}^3$ in which ${\bf f}_1={\bf r}'/|{\bf r}'|$ is the curve tangent, and the normal-plane vectors ${\bf f}_2$, ${\bf f}_3$ exhibit no instantaneous rotation about ${\bf f}_1$. Polynomial curves that admit rational rotation-minimizing frames (or RRMF curves) form a subset of the Pythagorean-hodograph (PH) curves, specified by integrating the form ${\bf r}'(ξ)={\cal A}(ξ)\,{\bf i}\,{\cal A}^*(ξ)$ for some quaternion polynomial ${\cal A}(ξ)$. By introducing the notion of rotation indicatrix and of core of the quaternion polynomial ${\cal A}(ξ)$, a comprehensive characterization of the complete space of RRMF curves is developed, that subsumes all previously known special cases. This novel characterization helps clarify the structure of the complete space of RRMF curves, distinguishes the spatial RRMF curves from trivial (planar) cases, and paves the way toward new construction algorithms.
△ Less
Submitted 24 April, 2016;
originally announced April 2016.
-
Bivariate hierarchical Hermite spline quasi--interpolation
Authors:
Cesare Bracco,
Carlotta Giannelli,
Francesca Mazzia,
Alessandra Sestini
Abstract:
Spline quasi-interpolation (QI) is a general and powerful approach for the construction of low cost and accurate approximations of a given function. In order to provide an efficient adaptive approximation scheme in the bivariate setting, we consider quasi-interpolation in hierarchical spline spaces. In particular, we study and experiment the features of the hierarchical extension of the tensor-pro…
▽ More
Spline quasi-interpolation (QI) is a general and powerful approach for the construction of low cost and accurate approximations of a given function. In order to provide an efficient adaptive approximation scheme in the bivariate setting, we consider quasi-interpolation in hierarchical spline spaces. In particular, we study and experiment the features of the hierarchical extension of the tensor-product formulation of the Hermite BS quasi-interpolation scheme. The convergence properties of this hierarchical operator, suitably defined in terms of truncated hierarchical B-spline bases, are analyzed. A selection of numerical examples is presented to compare the performances of the hierarchical and tensor-product versions of the scheme.
△ Less
Submitted 10 January, 2016;
originally announced January 2016.
-
Solution of a quadratic quaternion equation with mixed coefficients
Authors:
Rida T. Farouki,
Graziano Gentili,
Carlotta Giannelli,
Alessandra Sestini,
Caterina Stoppato
Abstract:
A comprehensive analysis of the morphology of the solution space for a special type of quadratic quaternion equation is presented. This equation, which arises in a surface construction problem, incorporates linear terms in a quaternion variable and its conjugate with right and left quaternion coefficients, while the quadratic term has a quaternion coefficient placed between the variable and its co…
▽ More
A comprehensive analysis of the morphology of the solution space for a special type of quadratic quaternion equation is presented. This equation, which arises in a surface construction problem, incorporates linear terms in a quaternion variable and its conjugate with right and left quaternion coefficients, while the quadratic term has a quaternion coefficient placed between the variable and its conjugate. It is proved that, for generic coefficients, the equation has two, one, or no solutions, but in certain special instances the solution set may comprise a circle or a 3-sphere in the quaternion space $\mathbb{H}$. The analysis yields solutions for each case, and intuitive interpretations of them in terms of the four-dimensional geometry of the quaternion space $\mathbb{H}$.
△ Less
Submitted 18 June, 2015;
originally announced June 2015.
-
Iterative solution of piecewise linear systems for the numerical solution of obstacle problems
Authors:
Luigi Brugnano,
Alessandra Sestini
Abstract:
We investigate the use of piecewise linear systems, whose coefficient matrix is a piecewise constant function of the solution itself. Such systems arise, for example, from the numerical solution of linear complementarity problems and in the numerical solution of free-surface problems. In particular, we here study their application to the numerical solution of both the (linear) parabolic obstacle…
▽ More
We investigate the use of piecewise linear systems, whose coefficient matrix is a piecewise constant function of the solution itself. Such systems arise, for example, from the numerical solution of linear complementarity problems and in the numerical solution of free-surface problems. In particular, we here study their application to the numerical solution of both the (linear) parabolic obstacle problem and the obstacle problem. We propose a class of effective semi-iterative Newton-type methods to find the exact solution of such piecewise linear systems. We prove that the semiiterative Newton-type methods have a global monotonic convergence property, i.e., the iterates converge monotonically to the exact solution in a finite number of steps. Numerical examples are presented to demonstrate the effectiveness of the proposed methods.
△ Less
Submitted 16 December, 2009;
originally announced December 2009.
