-
High-dimensional maximum-entropy phase space tomography using normalizing flows
Authors:
Austin Hoover,
Jonathan C. Wong
Abstract:
Particle accelerators generate charged particle beams with tailored distributions in six-dimensional (6D) position-momentum space (phase space). Knowledge of the phase space distribution enables model-based beam optimization and control. In the absence of direct measurements, the distribution must be tomographically reconstructed from its projections. In this paper, we highlight that such problems…
▽ More
Particle accelerators generate charged particle beams with tailored distributions in six-dimensional (6D) position-momentum space (phase space). Knowledge of the phase space distribution enables model-based beam optimization and control. In the absence of direct measurements, the distribution must be tomographically reconstructed from its projections. In this paper, we highlight that such problems can be severely underdetermined and that entropy maximization is the most conservative solution strategy. We leverage normalizing flows -- invertible generative models -- to extend maximum-entropy tomography to 6D phase space and perform numerical experiments to validate the model performance. Our numerical experiments demonstrate that flow-based entropy estimates are consistent with 2D maximum-entropy solutions and that normalizing flows can fit complex 6D phase space distributions to large measurement sets in reasonable time.
△ Less
Submitted 7 June, 2024; v1 submitted 31 May, 2024;
originally announced June 2024.
-
LSA-PINN: Linear Boundary Connectivity Loss for Solving PDEs on Complex Geometry
Authors:
Jian Cheng Wong,
Pao-Hsiung Chiu,
Chinchun Ooi,
My Ha Dao,
Yew-Soon Ong
Abstract:
We present a novel loss formulation for efficient learning of complex dynamics from governing physics, typically described by partial differential equations (PDEs), using physics-informed neural networks (PINNs). In our experiments, existing versions of PINNs are seen to learn poorly in many problems, especially for complex geometries, as it becomes increasingly difficult to establish appropriate…
▽ More
We present a novel loss formulation for efficient learning of complex dynamics from governing physics, typically described by partial differential equations (PDEs), using physics-informed neural networks (PINNs). In our experiments, existing versions of PINNs are seen to learn poorly in many problems, especially for complex geometries, as it becomes increasingly difficult to establish appropriate sampling strategy at the near boundary region. Overly dense sampling can adversely impede training convergence if the local gradient behaviors are too complex to be adequately modelled by PINNs. On the other hand, if the samples are too sparse, existing PINNs tend to overfit the near boundary region, leading to incorrect solution. To prevent such issues, we propose a new Boundary Connectivity (BCXN) loss function which provides linear local structure approximation (LSA) to the gradient behaviors at the boundary for PINN. Our BCXN-loss implicitly imposes local structure during training, thus facilitating fast physics-informed learning across entire problem domains with order of magnitude sparser training samples. This LSA-PINN method shows a few orders of magnitude smaller errors than existing methods in terms of the standard L2-norm metric, while using dramatically fewer training samples and iterations. Our proposed LSA-PINN does not pose any requirement on the differentiable property of the networks, and we demonstrate its benefits and ease of implementation on both multi-layer perceptron and convolutional neural network versions as commonly used in current PINN literature.
△ Less
Submitted 2 March, 2023; v1 submitted 2 February, 2023;
originally announced February 2023.
-
Neuroevolution of Physics-Informed Neural Nets: Benchmark Problems and Comparative Results
Authors:
Nicholas Sung Wei Yong,
Jian Cheng Wong,
Pao-Hsiung Chiu,
Abhishek Gupta,
Chinchun Ooi,
Yew-Soon Ong
Abstract:
The potential of learned models for fundamental scientific research and discovery is drawing increasing attention worldwide. Physics-informed neural networks (PINNs), where the loss function directly embeds governing equations of scientific phenomena, is one of the key techniques at the forefront of recent advances. PINNs are typically trained using stochastic gradient descent methods, akin to the…
▽ More
The potential of learned models for fundamental scientific research and discovery is drawing increasing attention worldwide. Physics-informed neural networks (PINNs), where the loss function directly embeds governing equations of scientific phenomena, is one of the key techniques at the forefront of recent advances. PINNs are typically trained using stochastic gradient descent methods, akin to their deep learning counterparts. However, analysis in this paper shows that PINNs' unique loss formulations lead to a high degree of complexity and ruggedness that may not be conducive for gradient descent. Unlike in standard deep learning, PINN training requires globally optimum parameter values that satisfy physical laws as closely as possible. Spurious local optimum, indicative of erroneous physics, must be avoided. Hence, neuroevolution algorithms, with their superior global search capacity, may be a better choice for PINNs relative to gradient descent methods. Here, we propose a set of five benchmark problems, with open-source codes, spanning diverse physical phenomena for novel neuroevolution algorithm development. Using this, we compare two neuroevolution algorithms against the commonly used stochastic gradient descent, and our baseline results support the claim that neuroevolution can surpass gradient descent, ensuring better physics compliance in the predicted outputs. %Furthermore, implementing neuroevolution with JAX leads to orders of magnitude speedup relative to standard implementations.
△ Less
Submitted 6 December, 2023; v1 submitted 15 December, 2022;
originally announced December 2022.
-
Design of Turing Systems with Physics-Informed Neural Networks
Authors:
Jordon Kho,
Winston Koh,
Jian Cheng Wong,
Pao-Hsiung Chiu,
Chin Chun Ooi
Abstract:
Reaction-diffusion (Turing) systems are fundamental to the formation of spatial patterns in nature and engineering. These systems are governed by a set of non-linear partial differential equations containing parameters that determine the rate of constituent diffusion and reaction. Critically, these parameters, such as diffusion coefficient, heavily influence the mode and type of the final pattern,…
▽ More
Reaction-diffusion (Turing) systems are fundamental to the formation of spatial patterns in nature and engineering. These systems are governed by a set of non-linear partial differential equations containing parameters that determine the rate of constituent diffusion and reaction. Critically, these parameters, such as diffusion coefficient, heavily influence the mode and type of the final pattern, and quantitative characterization and knowledge of these parameters can aid in bio-mimetic design or understanding of real-world systems. However, the use of numerical methods to infer these parameters can be difficult and computationally expensive. Typically, adjoint solvers may be used, but they are frequently unstable for very non-linear systems. Alternatively, massive amounts of iterative forward simulations are used to find the best match, but this is extremely effortful. Recently, physics-informed neural networks have been proposed as a means for data-driven discovery of partial differential equations, and have seen success in various applications. Thus, we investigate the use of physics-informed neural networks as a tool to infer key parameters in reaction-diffusion systems in the steady-state for scientific discovery or design. Our proof-of-concept results show that the method is able to infer parameters for different pattern modes and types with errors of less than 10\%. In addition, the stochastic nature of this method can be exploited to provide multiple parameter alternatives to the desired pattern, highlighting the versatility of this method for bio-mimetic design. This work thus demonstrates the utility of physics-informed neural networks for inverse parameter inference of reaction-diffusion systems to enhance scientific discovery and design.
△ Less
Submitted 24 November, 2022;
originally announced November 2022.
-
Robustness of Physics-Informed Neural Networks to Noise in Sensor Data
Authors:
Jian Cheng Wong,
Pao-Hsiung Chiu,
Chin Chun Ooi,
My Ha Da
Abstract:
Physics-Informed Neural Networks (PINNs) have been shown to be an effective way of incorporating physics-based domain knowledge into neural network models for many important real-world systems. They have been particularly effective as a means of inferring system information based on data, even in cases where data is scarce. Most of the current work however assumes the availability of high-quality…
▽ More
Physics-Informed Neural Networks (PINNs) have been shown to be an effective way of incorporating physics-based domain knowledge into neural network models for many important real-world systems. They have been particularly effective as a means of inferring system information based on data, even in cases where data is scarce. Most of the current work however assumes the availability of high-quality data. In this work, we further conduct a preliminary investigation of the robustness of physics-informed neural networks to the magnitude of noise in the data. Interestingly, our experiments reveal that the inclusion of physics in the neural network is sufficient to negate the impact of noise in data originating from hypothetical low quality sensors with high signal-to-noise ratios of up to 1. The resultant predictions for this test case are seen to still match the predictive value obtained for equivalent data obtained from high-quality sensors with potentially 10x less noise. This further implies the utility of physics-informed neural network modeling for making sense of data from sensor networks in the future, especially with the advent of Industry 4.0 and the increasing trend towards ubiquitous deployment of low-cost sensors which are typically noisier.
△ Less
Submitted 22 November, 2022;
originally announced November 2022.
-
FastFlow: AI for Fast Urban Wind Velocity Prediction
Authors:
Shi Jer Low,
Venugopalan,
S. G. Raghavan,
Harish Gopalan,
Jian Cheng Wong,
Justin Yeoh,
Chin Chun Ooi
Abstract:
Data-driven approaches, including deep learning, have shown great promise as surrogate models across many domains. These extend to various areas in sustainability. An interesting direction for which data-driven methods have not been applied much yet is in the quick quantitative evaluation of urban layouts for planning and design. In particular, urban designs typically involve complex trade-offs be…
▽ More
Data-driven approaches, including deep learning, have shown great promise as surrogate models across many domains. These extend to various areas in sustainability. An interesting direction for which data-driven methods have not been applied much yet is in the quick quantitative evaluation of urban layouts for planning and design. In particular, urban designs typically involve complex trade-offs between multiple objectives, including limits on urban build-up and/or consideration of urban heat island effect. Hence, it can be beneficial to urban planners to have a fast surrogate model to predict urban characteristics of a hypothetical layout, e.g. pedestrian-level wind velocity, without having to run computationally expensive and time-consuming high-fidelity numerical simulations. This fast surrogate can then be potentially integrated into other design optimization frameworks, including generative models or other gradient-based methods. Here we present the use of CNNs for urban layout characterization that is typically done via high-fidelity numerical simulation. We further apply this model towards a first demonstration of its utility for data-driven pedestrian-level wind velocity prediction. The data set in this work comprises results from high-fidelity numerical simulations of wind velocities for a diverse set of realistic urban layouts, based on randomized samples from a real-world, highly built-up urban city. We then provide prediction results obtained from the trained CNN, demonstrating test errors of under 0.1 m/s for previously unseen urban layouts. We further illustrate how this can be useful for purposes such as rapid evaluation of pedestrian wind velocity for a potential new layout. It is hoped that this data set will further accelerate research in data-driven urban AI, even as our baseline model facilitates quantitative comparison to future methods.
△ Less
Submitted 22 November, 2022;
originally announced November 2022.
-
CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method
Authors:
Pao-Hsiung Chiu,
Jian Cheng Wong,
Chinchun Ooi,
My Ha Dao,
Yew-Soon Ong
Abstract:
In this study, novel physics-informed neural network (PINN) methods for coupling neighboring support points and their derivative terms which are obtained by automatic differentiation (AD), are proposed to allow efficient training with improved accuracy. The computation of differential operators required for PINNs loss evaluation at collocation points are conventionally obtained via AD. Although AD…
▽ More
In this study, novel physics-informed neural network (PINN) methods for coupling neighboring support points and their derivative terms which are obtained by automatic differentiation (AD), are proposed to allow efficient training with improved accuracy. The computation of differential operators required for PINNs loss evaluation at collocation points are conventionally obtained via AD. Although AD has the advantage of being able to compute the exact gradients at any point, such PINNs can only achieve high accuracies with large numbers of collocation points, otherwise they are prone to optimizing towards unphysical solution. To make PINN training fast, the dual ideas of using numerical differentiation (ND)-inspired method and coupling it with AD are employed to define the loss function. The ND-based formulation for training loss can strongly link neighboring collocation points to enable efficient training in sparse sample regimes, but its accuracy is restricted by the interpolation scheme. The proposed coupled-automatic-numerical differentiation framework, labeled as can-PINN, unifies the advantages of AD and ND, providing more robust and efficient training than AD-based PINNs, while further improving accuracy by up to 1-2 orders of magnitude relative to ND-based PINNs. For a proof-of-concept demonstration of this can-scheme to fluid dynamic problems, two numerical-inspired instantiations of can-PINN schemes for the convection and pressure gradient terms were derived to solve the incompressible Navier-Stokes (N-S) equations. The superior performance of can-PINNs is demonstrated on several challenging problems, including the flow mixing phenomena, lid driven flow in a cavity, and channel flow over a backward facing step. The results reveal that for challenging problems like these, can-PINNs can consistently achieve very good accuracy whereas conventional AD-based PINNs fail.
△ Less
Submitted 27 March, 2022; v1 submitted 29 October, 2021;
originally announced October 2021.
-
Learning in Sinusoidal Spaces with Physics-Informed Neural Networks
Authors:
Jian Cheng Wong,
Chinchun Ooi,
Abhishek Gupta,
Yew-Soon Ong
Abstract:
A physics-informed neural network (PINN) uses physics-augmented loss functions, e.g., incorporating the residual term from governing partial differential equations (PDEs), to ensure its output is consistent with fundamental physics laws. However, it turns out to be difficult to train an accurate PINN model for many problems in practice. In this paper, we present a novel perspective of the merits o…
▽ More
A physics-informed neural network (PINN) uses physics-augmented loss functions, e.g., incorporating the residual term from governing partial differential equations (PDEs), to ensure its output is consistent with fundamental physics laws. However, it turns out to be difficult to train an accurate PINN model for many problems in practice. In this paper, we present a novel perspective of the merits of learning in sinusoidal spaces with PINNs. By analyzing behavior at model initialization, we first show that a PINN of increasing expressiveness induces an initial bias around flat output functions. Notably, this initial solution can be very close to satisfying many physics PDEs, i.e., falling into a local minimum of the PINN loss that only minimizes PDE residuals, while still being far from the true solution that jointly minimizes PDE residuals and the initial and/or boundary conditions. It is difficult for gradient descent optimization to escape from such a local minimum trap, often causing the training to stall. We then prove that the sinusoidal map** of inputs, in an architecture we label as sf-PINN, is effective to increase input gradient variability, thus avoiding being trapped in such deceptive local minimum. The level of variability can be effectively modulated to match high-frequency patterns in the problem at hand. A key facet of this paper is the comprehensive empirical study that demonstrates the efficacy of learning in sinusoidal spaces with PINNs for a wide range of forward and inverse modelling problems spanning multiple physics domains.
△ Less
Submitted 14 March, 2022; v1 submitted 20 September, 2021;
originally announced September 2021.
-
Improved Surrogate Modeling of Fluid Dynamics with Physics-Informed Neural Networks
Authors:
Jian Cheng Wong,
Chinchun Ooi,
Pao-Hsiung Chiu,
My Ha Dao
Abstract:
Physics-Informed Neural Networks (PINNs) have recently shown great promise as a way of incorporating physics-based domain knowledge, including fundamental governing equations, into neural network models for many complex engineering systems. They have been particularly effective in the area of inverse problems, where boundary conditions may be ill-defined, and data-absent scenarios, where typical s…
▽ More
Physics-Informed Neural Networks (PINNs) have recently shown great promise as a way of incorporating physics-based domain knowledge, including fundamental governing equations, into neural network models for many complex engineering systems. They have been particularly effective in the area of inverse problems, where boundary conditions may be ill-defined, and data-absent scenarios, where typical supervised learning approaches will fail. Here, we further explore the use of this modeling methodology to surrogate modeling of a fluid dynamical system, and demonstrate additional undiscussed and interesting advantages of such a modeling methodology over conventional data-driven approaches: 1) improving the model's predictive performance even with incomplete description of the underlying physics; 2) improving the robustness of the model to noise in the dataset; 3) reduced effort to convergence during optimization for a new, previously unseen scenario by transfer optimization of a pre-existing model. Hence, we noticed the inclusion of a physics-based regularization term can substantially improve the equivalent data-driven surrogate model in many substantive ways, including an order of magnitude improvement in test error when the dataset is very noisy, and a 2-3x improvement when only partial physics is included. In addition, we propose a novel transfer optimization scheme for use in such surrogate modeling scenarios and demonstrate an approximately 3x improvement in speed to convergence and an order of magnitude improvement in predictive performance over conventional Xavier initialization for training of new scenarios.
△ Less
Submitted 4 May, 2021;
originally announced May 2021.
-
Can Transfer Neuroevolution Tractably Solve Your Differential Equations?
Authors:
Jian Cheng Wong,
Abhishek Gupta,
Yew-Soon Ong
Abstract:
This paper introduces neuroevolution for solving differential equations. The solution is obtained through optimizing a deep neural network whose loss function is defined by the residual terms from the differential equations. Recent studies have focused on learning such physics-informed neural networks through stochastic gradient descent (SGD) variants, yet they face the difficulty of obtaining an…
▽ More
This paper introduces neuroevolution for solving differential equations. The solution is obtained through optimizing a deep neural network whose loss function is defined by the residual terms from the differential equations. Recent studies have focused on learning such physics-informed neural networks through stochastic gradient descent (SGD) variants, yet they face the difficulty of obtaining an accurate solution due to optimization challenges. In the context of solving differential equations, we are faced with the problem of finding globally optimum parameters of the network, instead of being concerned with out-of-sample generalization. SGD, which searches along a single gradient direction, is prone to become trapped in local optima, so it may not be the best approach here. In contrast, neuroevolution carries out a parallel exploration of diverse solutions with the goal of circumventing local optima. It could potentially find more accurate solutions with better optimized neural networks. However, neuroevolution can be slow, raising tractability issues in practice. With that in mind, a novel and computationally efficient transfer neuroevolution algorithm is proposed in this paper. Our method is capable of exploiting relevant experiential priors when solving a new problem, with adaptation to protect against the risk of negative transfer. The algorithm is applied on a variety of differential equations to empirically demonstrate that transfer neuroevolution can indeed achieve better accuracy and faster convergence than SGD. The experimental outcomes thus establish transfer neuroevolution as a noteworthy approach for solving differential equations, one that has never been studied in the past. Our work expands the resource of available algorithms for optimizing physics-informed neural networks.
△ Less
Submitted 4 May, 2021; v1 submitted 6 January, 2021;
originally announced January 2021.
-
Breaking the fundamental energy dissipation limit in ferroelectric-dielectric capacitors
Authors:
Justin C. Wong,
Sayeef Salahuddin
Abstract:
Half of the energy is always lost when charging a capacitor. Even in the limit of vanishing resistance, half of the charging energy is still lost--to radiation instead of heat. While this fraction can technically be reduced by charging adiabatically, it otherwise places a fundamental limit on the charging efficiency of a capacitor. Here we show that this 1/2 limit can be broken by coupling a ferro…
▽ More
Half of the energy is always lost when charging a capacitor. Even in the limit of vanishing resistance, half of the charging energy is still lost--to radiation instead of heat. While this fraction can technically be reduced by charging adiabatically, it otherwise places a fundamental limit on the charging efficiency of a capacitor. Here we show that this 1/2 limit can be broken by coupling a ferroelectric to the capacitor dielectric. Maxwell's equations are solved for the coupled system to analyze energy flow from the perspective of Poynting's theorem and show that (1) total energy dissipation is reduced below the fundamental limit during charging and discharging; (2) energy is saved by "recycling" the energy already stored in the ferroelectric phase transition; and (3) this phase transition energy is directly transferred between the ferroelectric and dielectric during charging and discharging. These results demystify recent works on low energy negative capacitance devices as well as lay the foundation for improving fundamental energy efficiency in all devices that rely on energy storage in electric fields.
△ Less
Submitted 11 May, 2018;
originally announced May 2018.
-
Electrically Induced, Non-Volatile, Metal Insulator Transition in a Ferroelectric Gated MoS$_2$ Transistor
Authors:
Zhongyuan Lu,
Claudy Serrao,
Asif I. Khan,
James D. Clarkson,
Justin C. Wong,
Ramamoorthy Ramesh,
Sayeef Salahuddin
Abstract:
We demonstrate an electrically induced, non-volatile, metal-insulator phase transition in a MoS$_2$ transistor. A single crystalline, epitaxially grown, PbZr$_{0.2}$Ti$_{0.8}$O$_3$ (PZT) was placed in the gate of a field effect transistor made of thin film MoS$_2$. When a gate voltage is applied to this ferroelectric gated transistor, a clear transition from insulator to metal and vice versa is ob…
▽ More
We demonstrate an electrically induced, non-volatile, metal-insulator phase transition in a MoS$_2$ transistor. A single crystalline, epitaxially grown, PbZr$_{0.2}$Ti$_{0.8}$O$_3$ (PZT) was placed in the gate of a field effect transistor made of thin film MoS$_2$. When a gate voltage is applied to this ferroelectric gated transistor, a clear transition from insulator to metal and vice versa is observed. Importantly, when the gate voltage is turned off, the remnant polarization in the ferroelectric can keep the MoS$_2$ in its original phase, thereby providing a non-volatile state. Thus a metallic or insulating phase can be written, erased or retained simply by applying a gate voltage to the transistor.
△ Less
Submitted 24 July, 2017; v1 submitted 17 May, 2017;
originally announced May 2017.
-
Non Volatile MoS$_{2}$ Field Effect Transistors Directly Gated By Single Crystalline Epitaxial Ferroelectric
Authors:
Zhongyuan Lu,
Claudy Serrao,
Asif Islam Khan,
Long You,
Justin C. Wong,
Yu Ye,
Hanyu Zhu,
Xiang Zhang,
Sayeef Salahuddin
Abstract:
We demonstrate non-volatile, n-type, back-gated, MoS$_{2}$ transistors, placed directly on an epitaxial grown, single crystalline, PbZr$_{0.2}$Ti$_{0.8}$O$_{3}$ (PZT) ferroelectric. The transistors show decent ON current (19 $μA/μ$m), high on-off ratio (10$^{7}$), and a subthreshold swing of (SS ~ 92 mV/dec) with a 100 nm thick PZT layer as the back gate oxide. Importantly, the ferroelectric polar…
▽ More
We demonstrate non-volatile, n-type, back-gated, MoS$_{2}$ transistors, placed directly on an epitaxial grown, single crystalline, PbZr$_{0.2}$Ti$_{0.8}$O$_{3}$ (PZT) ferroelectric. The transistors show decent ON current (19 $μA/μ$m), high on-off ratio (10$^{7}$), and a subthreshold swing of (SS ~ 92 mV/dec) with a 100 nm thick PZT layer as the back gate oxide. Importantly, the ferroelectric polarization can directly control the channel charge, showing a clear anti-clockwise hysteresis. We have selfconsistently confirmed the switching of the ferroelectric and corresponding change in channel current from a direct time-dependent measurement. Our results demonstrate that it is possible to obtain transistor operation directly on polar surfaces and therefore it should be possible to integrate 2D electronics with single crystalline functional oxides.
△ Less
Submitted 24 July, 2017; v1 submitted 1 May, 2017;
originally announced May 2017.
