-
Solving the decision-making differential equations from eye fixation data in Unity software by using Hermite Long-Short-Term Memory neural network
Authors:
Kourosh Parand,
Saeed Setayeshi,
Mir Mohsen Pedram,
Ali Yoonesi,
Aida Pakniyat
Abstract:
Cognitive decision-making processes are crucial aspects of human behavior, influencing various personal and professional domains. This research delves into the application of differential equations in analyzing decision-making accuracy by leveraging eye-tracking data within a virtual industrial town setting. The study unveils a systematic approach to transforming raw data into a differential equat…
▽ More
Cognitive decision-making processes are crucial aspects of human behavior, influencing various personal and professional domains. This research delves into the application of differential equations in analyzing decision-making accuracy by leveraging eye-tracking data within a virtual industrial town setting. The study unveils a systematic approach to transforming raw data into a differential equation, essential for deciphering the relationship between eye movements during decision-making processes.
Mathematical relationship extraction and variable-parameter definition pave the way for deriving a differential equation that encapsulates the growth of fixations on characters. The key factors in this equation encompass the fixation rate $(λ)$ and separation rate $(μ)$, reflecting user interaction dynamics and their impact on decision-making complexities tied to user engagement with virtual characters.
For a comprehensive grasp of decision dynamics, solving this differential equation requires initial fixation counts, fixation rate, and separation rate. The formulation of differential equations incorporates various considerations such as engagement duration, character-player distance, relative speed, and character attributes, enabling the representation of fixation changes, speed dynamics, distance variations, and the effects of character attributes.
This comprehensive analysis not only enhances our comprehension of decision-making processes but also provides a foundational framework for predictive modeling and data-driven insights for future research and applications in cognitive science and virtual reality environments.
△ Less
Submitted 23 February, 2024; v1 submitted 20 February, 2024;
originally announced February 2024.
-
Hermite Neural Network Simulation for Solving the 2D Schrodinger Equation
Authors:
Kourosh Parand,
Aida Pakniyat
Abstract:
The Schrodinger equation is a mathematical equation describing the wave function's behavior in a quantum-mechanical system. It is a partial differential equation that provides valuable insights into the fundamental principles of quantum mechanics. In this paper, the aim was to solve the Schrodinger equation with sufficient accuracy by using a mixture of neural networks with the collocation method…
▽ More
The Schrodinger equation is a mathematical equation describing the wave function's behavior in a quantum-mechanical system. It is a partial differential equation that provides valuable insights into the fundamental principles of quantum mechanics. In this paper, the aim was to solve the Schrodinger equation with sufficient accuracy by using a mixture of neural networks with the collocation method base Hermite functions. Initially, the Hermite functions roots were employed as collocation points, enhancing the efficiency of the solution. The Schrodinger equation is defined in an infinite domain, the use of Hermite functions as activation functions resulted in excellent precision. Finally, the proposed method was simulated using MATLAB's Simulink tool. The results were then compared with those obtained using Physics-informed neural networks and the presented method.
△ Less
Submitted 16 February, 2024;
originally announced February 2024.
-
Solving Feynman-Kac Forward Backward SDEs Using McKean-Markov Branched Sampling
Authors:
Kelsey P. Hawkins,
Ali Pakniyat,
Evangelos Theodorou,
Panagiotis Tsiotras
Abstract:
We propose a new method for the numerical solution of the forward-backward stochastic differential equations (FBSDE) appearing in the Feynman-Kac representation of the value function in stochastic optimal control problems. Using Girsanov's change of probability measures, it is demonstrated how a McKean-Markov branched sampling method can be utilized for the forward integration pass, as long as the…
▽ More
We propose a new method for the numerical solution of the forward-backward stochastic differential equations (FBSDE) appearing in the Feynman-Kac representation of the value function in stochastic optimal control problems. Using Girsanov's change of probability measures, it is demonstrated how a McKean-Markov branched sampling method can be utilized for the forward integration pass, as long as the controlled drift term is appropriately compensated in the backward integration pass. Subsequently, a numerical approximation of the value function is proposed by solving a series of function approximation problems backwards in time along the edges of a space-filling tree consisting of trajectory samples. Moreover, a local entropy-weighted least squares Monte Carlo (LSMC) method is developed to concentrate function approximation accuracy in regions most likely to be visited by optimally controlled trajectories. The proposed methodology is numerically demonstrated on linear and nonlinear stochastic optimal control problems with non-quadratic running costs, which reveal significant convergence improvements over previous FBSDE-based numerical solution methods.
△ Less
Submitted 19 October, 2022;
originally announced October 2022.
-
Sha** Large Population Agent Behaviors Through Entropy-Regularized Mean-Field Games
Authors:
Yue Guan,
Mi Zhou,
Ali Pakniyat,
Panagiotis Tsiotras
Abstract:
Mean-field games (MFG) were introduced to efficiently analyze approximate Nash equilibria in large population settings. In this work, we consider entropy-regularized mean-field games with a finite state-action space in a discrete time setting. We show that entropy regularization provides the necessary regularity conditions, that are lacking in the standard finite mean field games. Such regularity…
▽ More
Mean-field games (MFG) were introduced to efficiently analyze approximate Nash equilibria in large population settings. In this work, we consider entropy-regularized mean-field games with a finite state-action space in a discrete time setting. We show that entropy regularization provides the necessary regularity conditions, that are lacking in the standard finite mean field games. Such regularity conditions enable us to design fixed-point iteration algorithms to find the unique mean-field equilibrium (MFE). Furthermore, the reference policy used in the regularization provides an extra parameter, through which one can control the behavior of the population. We first consider a stochastic game with a large population of $N$ homogeneous agents. We establish conditions for the existence of a Nash equilibrium in the limiting case as $N$ tends to infinity, and we demonstrate that the Nash equilibrium for the infinite population case is also an $ε$-Nash equilibrium for the $N$-agent system, where the sub-optimality $ε$ is of order $\mathcal{O}\big(1/\sqrt{N}\big)$. Finally, we verify the theoretical guarantees through a resource allocation example and demonstrate the efficacy of using a reference policy to control the behavior of a large population.
△ Less
Submitted 22 July, 2022; v1 submitted 14 October, 2021;
originally announced October 2021.
-
Value Function Estimators for Feynman-Kac Forward-Backward SDEs in Stochastic Optimal Control
Authors:
Kelsey P. Hawkins,
Ali Pakniyat,
Panagiotis Tsiotras
Abstract:
Two novel numerical estimators are proposed for solving forward-backward stochastic differential equations (FBSDEs) appearing in the Feynman-Kac representation of the value function in stochastic optimal control problems. In contrast to the current numerical approaches which are based on the discretization of the continuous-time FBSDE, we propose a converse approach, namely, we obtain a discrete-t…
▽ More
Two novel numerical estimators are proposed for solving forward-backward stochastic differential equations (FBSDEs) appearing in the Feynman-Kac representation of the value function in stochastic optimal control problems. In contrast to the current numerical approaches which are based on the discretization of the continuous-time FBSDE, we propose a converse approach, namely, we obtain a discrete-time approximation of the on-policy value function, and then we derive a discrete-time estimator that resembles the continuous-time counterpart. The proposed approach allows for the construction of higher accuracy estimators along with error analysis. The approach is applied to the policy improvement step in reinforcement learning. Numerical results and error analysis are demonstrated using (i) a scalar nonlinear stochastic optimal control problem and (ii) a four-dimensional linear quadratic regulator (LQR) problem. The proposed estimators show significant improvement in terms of accuracy in both cases over Euler-Maruyama-based estimators used in competing approaches. In the case of LQR problems, we demonstrate that our estimators result in near machine-precision level accuracy, in contrast to previously proposed methods that can potentially diverge on the same problems.
△ Less
Submitted 30 September, 2021; v1 submitted 25 March, 2021;
originally announced March 2021.
-
Forward-Backward Rapidly-Exploring Random Trees for Stochastic Optimal Control
Authors:
Kelsey P. Hawkins,
Ali Pakniyat,
Evangelos Theodorou,
Panagiotis Tsiotras
Abstract:
We propose a numerical method for the computation of the forward-backward stochastic differential equations (FBSDE) appearing in the Feynman-Kac representation of the value function in stochastic optimal control problems. By the use of the Girsanov change of probability measures, it is demonstrated how a rapidly-exploring random tree (RRT) method can be utilized for the forward integration pass, a…
▽ More
We propose a numerical method for the computation of the forward-backward stochastic differential equations (FBSDE) appearing in the Feynman-Kac representation of the value function in stochastic optimal control problems. By the use of the Girsanov change of probability measures, it is demonstrated how a rapidly-exploring random tree (RRT) method can be utilized for the forward integration pass, as long as the controlled drift terms are appropriately compensated in the backward integration pass. Subsequently, a numerical approximation of the value function is proposed by solving a series of function approximation problems backwards in time along the edges of the constructed RRT. Moreover, a local entropy-weighted least squares Monte Carlo (LSMC) method is developed to concentrate function approximation accuracy in regions most likely to be visited by optimally controlled trajectories. The results of the proposed methodology are demonstrated on linear and nonlinear stochastic optimal control problems with non-quadratic running costs, which reveal significant convergence improvements over previous FBSDE-based numerical solution methods.
△ Less
Submitted 25 March, 2021; v1 submitted 22 June, 2020;
originally announced June 2020.
-
A Class of Hybrid LQG Mean Field Games with State-Invariant Switching and Stop** Strategies
Authors:
Dena Firoozi,
Ali Pakniyat,
Peter E. Caines
Abstract:
A novel framework is presented that combines Mean Field Game (MFG) theory and Hybrid Optimal Control (HOC) theory to obtain a unique $ε$-Nash equilibrium for a non-cooperative game with switching and stop** times. We consider the case where there exists one major agent with a significant influence on the system together with a large number of minor agents constituting two subpopulations, each ag…
▽ More
A novel framework is presented that combines Mean Field Game (MFG) theory and Hybrid Optimal Control (HOC) theory to obtain a unique $ε$-Nash equilibrium for a non-cooperative game with switching and stop** times. We consider the case where there exists one major agent with a significant influence on the system together with a large number of minor agents constituting two subpopulations, each agent with individually asymptotically negligible effect on the whole system. Each agent has stochastic linear dynamics with quadratic costs, and the agents are coupled in their dynamics and costs by the average state of minor agents (i.e. the empirical mean field). It is shown that for a class of Hybrid LQG MFGs, the optimal switching and stop** times are state-invariant and only depend on the dynamical parameters of each agent. Accordingly, a hybrid systems formulation of the game is presented via the indexing by discrete events: (i) the switching of the major agent between alternative dynamics or (ii) the termination of the agents' trajectories in one or both of the subpopulations of minor agents. Optimal switchings and stop** time strategies together with best response control actions for, respectively, the major agent and all minor agents are established with respect to their individual cost criteria by an application of Hybrid LQG MFG theory.
△ Less
Submitted 9 January, 2022; v1 submitted 5 October, 2018;
originally announced October 2018.
-
On the Hybrid Minimum Principle
Authors:
Ali Pakniyat,
Peter E. Caines
Abstract:
The Hybrid Minimum Principle (HMP) is established for the optimal control of deterministic hybrid systems with both autonomous and controlled switchings and jumps where state jumps at the switching instants are permitted to be accompanied by changes in the dimension of the state space. First order variational analysis is performed via the needle variation methodology and the necessary optimality c…
▽ More
The Hybrid Minimum Principle (HMP) is established for the optimal control of deterministic hybrid systems with both autonomous and controlled switchings and jumps where state jumps at the switching instants are permitted to be accompanied by changes in the dimension of the state space. First order variational analysis is performed via the needle variation methodology and the necessary optimality conditions are established in the form of the HMP. A feature of special interest in this work is the explicit presentations of boundary conditions on the Hamiltonians and the adjoint processes before and after switchings and jumps. In addition to an analytic example, the HMP results are illustrated for the optimal control of an electric vehicle with transmission, where the modelling of the powertrain requires the consideration of both autonomous and controlled switchings accompanied by dimension changes.
△ Less
Submitted 17 May, 2018; v1 submitted 16 October, 2017;
originally announced October 2017.
-
On the Relation between the Minimum Principle and Dynamic Programming for Classical and Hybrid Control Systems
Authors:
Ali Pakniyat,
Peter E. Caines
Abstract:
Hybrid optimal control problems are studied for a general class of hybrid systems where autonomous and controlled state jumps are allowed at the switching instants and in addition to terminal and running costs switching between discrete states incurs costs. The statements of the Hybrid Minimum Principle and Hybrid Dynamic Programming are presented in this framework and it is shown that under certa…
▽ More
Hybrid optimal control problems are studied for a general class of hybrid systems where autonomous and controlled state jumps are allowed at the switching instants and in addition to terminal and running costs switching between discrete states incurs costs. The statements of the Hybrid Minimum Principle and Hybrid Dynamic Programming are presented in this framework and it is shown that under certain assumptions the adjoint process in the Hybrid Minimum Principle and the gradient of the value function in Hybrid Dynamic Programming are governed by the same set of differential equations and have the same boundary conditions and hence are almost everywhere identical to each other along optimal trajectories. Analytic examples are provided to illustrate the results and, in particular, a Riccati formalism for linear quadratic hybrid tracking problems is presented.
△ Less
Submitted 11 September, 2016;
originally announced September 2016.
