-
Denoising Low-dose Images Using Deep Learning of Time Series Images
Authors:
Yang Shao,
Toshie Yaguchi,
Toshiaki Tanigaki
Abstract:
Digital image devices have been widely applied in many fields, including scientific imaging, recognition of individuals, and remote sensing. As the application of these imaging technologies to autonomous driving and measurement, image noise generated when observation cannot be performed with a sufficient dose has become a major problem. Machine learning denoise technology is expected to be the sol…
▽ More
Digital image devices have been widely applied in many fields, including scientific imaging, recognition of individuals, and remote sensing. As the application of these imaging technologies to autonomous driving and measurement, image noise generated when observation cannot be performed with a sufficient dose has become a major problem. Machine learning denoise technology is expected to be the solver of this problem, but there are the following problems. Here we report, artifacts generated by machine learning denoise in ultra-low dose observation using an in-situ observation video of an electron microscope as an example. And as a method to solve this problem, we propose a method to decompose a time series image into a 2D image of the spatial axis and time to perform machine learning denoise. Our method opens new avenues accurate and stable reconstruction of continuous high-resolution images from low-dose imaging in science, industry, and life.
△ Less
Submitted 30 March, 2024;
originally announced April 2024.
-
Neural Operators Meet Energy-based Theory: Operator Learning for Hamiltonian and Dissipative PDEs
Authors:
Yusuke Tanaka,
Takaharu Yaguchi,
Tomoharu Iwata,
Naonori Ueda
Abstract:
The operator learning has received significant attention in recent years, with the aim of learning a map** between function spaces. Prior works have proposed deep neural networks (DNNs) for learning such a map**, enabling the learning of solution operators of partial differential equations (PDEs). However, these works still struggle to learn dynamics that obeys the laws of physics. This paper…
▽ More
The operator learning has received significant attention in recent years, with the aim of learning a map** between function spaces. Prior works have proposed deep neural networks (DNNs) for learning such a map**, enabling the learning of solution operators of partial differential equations (PDEs). However, these works still struggle to learn dynamics that obeys the laws of physics. This paper proposes Energy-consistent Neural Operators (ENOs), a general framework for learning solution operators of PDEs that follows the energy conservation or dissipation law from observed solution trajectories. We introduce a novel penalty function inspired by the energy-based theory of physics for training, in which the energy functional is modeled by another DNN, allowing one to bias the outputs of the DNN-based solution operators to ensure energetic consistency without explicit PDEs. Experiments on multiple physical systems show that ENO outperforms existing DNN models in predicting solutions from data, especially in super-resolution settings.
△ Less
Submitted 14 February, 2024;
originally announced February 2024.
-
Good Lattice Training: Physics-Informed Neural Networks Accelerated by Number Theory
Authors:
Takashi Matsubara,
Takaharu Yaguchi
Abstract:
Physics-informed neural networks (PINNs) offer a novel and efficient approach to solving partial differential equations (PDEs). Their success lies in the physics-informed loss, which trains a neural network to satisfy a given PDE at specific points and to approximate the solution. However, the solutions to PDEs are inherently infinite-dimensional, and the distance between the output and the soluti…
▽ More
Physics-informed neural networks (PINNs) offer a novel and efficient approach to solving partial differential equations (PDEs). Their success lies in the physics-informed loss, which trains a neural network to satisfy a given PDE at specific points and to approximate the solution. However, the solutions to PDEs are inherently infinite-dimensional, and the distance between the output and the solution is defined by an integral over the domain. Therefore, the physics-informed loss only provides a finite approximation, and selecting appropriate collocation points becomes crucial to suppress the discretization errors, although this aspect has often been overlooked. In this paper, we propose a new technique called good lattice training (GLT) for PINNs, inspired by number theoretic methods for numerical analysis. GLT offers a set of collocation points that are effective even with a small number of points and for multi-dimensional spaces. Our experiments demonstrate that GLT requires 2--20 times fewer collocation points (resulting in lower computational cost) than uniformly random sampling or Latin hypercube sampling, while achieving competitive performance.
△ Less
Submitted 25 July, 2023;
originally announced July 2023.
-
FINDE: Neural Differential Equations for Finding and Preserving Invariant Quantities
Authors:
Takashi Matsubara,
Takaharu Yaguchi
Abstract:
Many real-world dynamical systems are associated with first integrals (a.k.a. invariant quantities), which are quantities that remain unchanged over time. The discovery and understanding of first integrals are fundamental and important topics both in the natural sciences and in industrial applications. First integrals arise from the conservation laws of system energy, momentum, and mass, and from…
▽ More
Many real-world dynamical systems are associated with first integrals (a.k.a. invariant quantities), which are quantities that remain unchanged over time. The discovery and understanding of first integrals are fundamental and important topics both in the natural sciences and in industrial applications. First integrals arise from the conservation laws of system energy, momentum, and mass, and from constraints on states; these are typically related to specific geometric structures of the governing equations. Existing neural networks designed to ensure such first integrals have shown excellent accuracy in modeling from data. However, these models incorporate the underlying structures, and in most situations where neural networks learn unknown systems, these structures are also unknown. This limitation needs to be overcome for scientific discovery and modeling of unknown systems. To this end, we propose first integral-preserving neural differential equation (FINDE). By leveraging the projection method and the discrete gradient method, FINDE finds and preserves first integrals from data, even in the absence of prior knowledge about underlying structures. Experimental results demonstrate that FINDE can predict future states of target systems much longer and find various quantities consistent with well-known first integrals in a unified manner.
△ Less
Submitted 27 March, 2023; v1 submitted 1 October, 2022;
originally announced October 2022.
-
An Error Analysis Framework for Neural Network Modeling of Dynamical Systems
Authors:
Shunpei Terakawa,
Takashi Matsubara,
Takaharu Yaguchi
Abstract:
We propose a theoretical framework for investigating a modeling error caused by numerical integration in the learning process of dynamics. Recently, learning equations of motion to describe dynamics from data using neural networks has been attracting attention. During such training, numerical integration is used to compare the data with the solution of the neural network model; however, discretiza…
▽ More
We propose a theoretical framework for investigating a modeling error caused by numerical integration in the learning process of dynamics. Recently, learning equations of motion to describe dynamics from data using neural networks has been attracting attention. During such training, numerical integration is used to compare the data with the solution of the neural network model; however, discretization errors due to numerical integration prevent the model from being trained correctly. In this study, we formulate the modeling error using the Dahlquist test equation that is commonly used in the analysis of numerical methods and apply it to some of the Runge--Kutta methods.
△ Less
Submitted 28 December, 2021;
originally announced December 2021.
-
Symplecticity of coupled Hamiltonian systems
Authors:
Shunpei Terakawa,
Takaharu Yaguchi
Abstract:
We derived a condition under which a coupled system consisting of two finite-dimensional Hamiltonian systems becomes a Hamiltonian system. In many cases, an industrial system can be modeled as a coupled system of some subsystems. Although it is known that symplectic integrators are suitable for discretizing Hamiltonian systems,the composition of Hamiltonian systems may not be Hamiltonian. In this…
▽ More
We derived a condition under which a coupled system consisting of two finite-dimensional Hamiltonian systems becomes a Hamiltonian system. In many cases, an industrial system can be modeled as a coupled system of some subsystems. Although it is known that symplectic integrators are suitable for discretizing Hamiltonian systems,the composition of Hamiltonian systems may not be Hamiltonian. In this paper, focusing on a property of Hamiltonian systems, that is, the conservation of the symplectic form, we provide a condition under which two Hamiltonian systems coupled with interactions compose a Hamiltonian system.
△ Less
Submitted 27 December, 2021;
originally announced December 2021.
-
Comparison of Numerical Solvers for Differential Equations for Holonomic Gradient Method in Statistics
Authors:
Nobuki Takayama,
Takaharu Yaguchi,
Yi Zhang
Abstract:
Definite integrals with parameters of holonomic functions satisfy holonomic systems of linear partial differential equations. When we restrict parameters to a one dimensional curve, the system becomes a linear ordinary differential equation (ODE) with respect to a curve in the parameter space. We can evaluate the integral by solving the linear ODE numerically. This approach to evaluate numerically…
▽ More
Definite integrals with parameters of holonomic functions satisfy holonomic systems of linear partial differential equations. When we restrict parameters to a one dimensional curve, the system becomes a linear ordinary differential equation (ODE) with respect to a curve in the parameter space. We can evaluate the integral by solving the linear ODE numerically. This approach to evaluate numerically definite integrals is called the holonomic gradient method (HGM) and it is useful to evaluate several normalizing constants in statistics. We will discuss and compare methods to solve linear ODE's to evaluate normalizing constants.
△ Less
Submitted 14 August, 2023; v1 submitted 21 November, 2021;
originally announced November 2021.
-
KAM Theory Meets Statistical Learning Theory: Hamiltonian Neural Networks with Non-Zero Training Loss
Authors:
Yuhan Chen,
Takashi Matsubara,
Takaharu Yaguchi
Abstract:
Many physical phenomena are described by Hamiltonian mechanics using an energy function (the Hamiltonian). Recently, the Hamiltonian neural network, which approximates the Hamiltonian as a neural network, and its extensions have attracted much attention. This is a very powerful method, but its use in theoretical studies remains limited. In this study, by combining the statistical learning theory a…
▽ More
Many physical phenomena are described by Hamiltonian mechanics using an energy function (the Hamiltonian). Recently, the Hamiltonian neural network, which approximates the Hamiltonian as a neural network, and its extensions have attracted much attention. This is a very powerful method, but its use in theoretical studies remains limited. In this study, by combining the statistical learning theory and Kolmogorov-Arnold-Moser (KAM) theory, we provide a theoretical analysis of the behavior of Hamiltonian neural networks when the learning error is not completely zero. A Hamiltonian neural network with non-zero errors can be considered as a perturbation from the true dynamics, and the perturbation theory of the Hamilton equation is widely known as the KAM theory. To apply the KAM theory, we provide a generalization error bound for Hamiltonian neural networks by deriving an estimate of the covering number of the gradient of the multi-layer perceptron, which is the key ingredient of the model. This error bound gives an $L^\infty$ bound on the Hamiltonian that is required in the application of the KAM theory.
△ Less
Submitted 22 March, 2022; v1 submitted 22 February, 2021;
originally announced February 2021.
-
Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memory
Authors:
Takashi Matsubara,
Yuto Miyatake,
Takaharu Yaguchi
Abstract:
A neural network model of a differential equation, namely neural ODE, has enabled the learning of continuous-time dynamical systems and probabilistic distributions with high accuracy. The neural ODE uses the same network repeatedly during a numerical integration. The memory consumption of the backpropagation algorithm is proportional to the number of uses times the network size. This is true even…
▽ More
A neural network model of a differential equation, namely neural ODE, has enabled the learning of continuous-time dynamical systems and probabilistic distributions with high accuracy. The neural ODE uses the same network repeatedly during a numerical integration. The memory consumption of the backpropagation algorithm is proportional to the number of uses times the network size. This is true even if a checkpointing scheme divides the computation graph into sub-graphs. Otherwise, the adjoint method obtains a gradient by a numerical integration backward in time. Although this method consumes memory only for a single network use, it requires high computational cost to suppress numerical errors. This study proposes the symplectic adjoint method, which is an adjoint method solved by a symplectic integrator. The symplectic adjoint method obtains the exact gradient (up to rounding error) with memory proportional to the number of uses plus the network size. The experimental results demonstrate that the symplectic adjoint method consumes much less memory than the naive backpropagation algorithm and checkpointing schemes, performs faster than the adjoint method, and is more robust to rounding errors.
△ Less
Submitted 19 October, 2021; v1 submitted 19 February, 2021;
originally announced February 2021.
-
Method for estimating hidden structures determined by unidentifiable state-space models and time-series data based on the Groebner basis
Authors:
Mizuka Komatsu,
Takaharu Yaguchi
Abstract:
In this study, we propose a method for extracting the hidden algebraic structures of model parameters that are uniquely determined by observed time-series data and unidentifiable state-space models, explicitly and exhaustively. State-space models are often constructed based on the domain, for example, physical or biological. Such models include parameters that are assigned specific meanings in rel…
▽ More
In this study, we propose a method for extracting the hidden algebraic structures of model parameters that are uniquely determined by observed time-series data and unidentifiable state-space models, explicitly and exhaustively. State-space models are often constructed based on the domain, for example, physical or biological. Such models include parameters that are assigned specific meanings in relation to the system under consideration, which is examined by estimating the parameters using the corresponding data. As the parameters of unidentifiable models cannot be uniquely determined from the given data, it is difficult to examine the systems described by such models. To overcome this difficulty, multiple possible sets of parameters are estimated and analysed in the exiting approaches; however, in general, all the possible parameters cannot be explored; therefore, considerations on the system using the estimated parameters become insufficient. In this study, focusing on certain structures determined by the observed data and models uniquely, even if they are unidentifiable, we introduce the concept of parameter variety. This is newly defined and proven to form algebraic varieties, in general. A computational algebraic method that relies on the Groebner basis for deriving the explicit representation of the varieties is presented along with the supporting theory. Furthermore, its application in the analysis of a model that describes virus dynamics is presented. With this, new insight on the dynamics overlooked by the conventional approach are discovered, confirming the applicability of our idea and the proposed method.
△ Less
Submitted 22 December, 2020;
originally announced December 2020.
-
Observing and modeling the sequential pairwise reactions that drive solid-state ceramic synthesis
Authors:
Akira Miura,
Christopher J. Bartel,
Yusuke Goto,
Yoshikazu Mizuguchi,
Chikako Moriyoshi,
Yoshihiro Kuroiwa,
Yongming Wang,
Toshie Yaguchi,
Manabu Shirai,
Masanori Nagao,
Nataly Carolina Rosero-Navarro,
Kiyoharu Tadanaga,
Gerbrand Ceder,
Wenhao Sun
Abstract:
Solid-state synthesis from powder precursors is the primary processing route to advanced multicomponent ceramic materials. Designing ceramic synthesis routes is usually a laborious, trial-and-error process, as heterogeneous mixtures of powder precursors often evolve through a complicated series of reaction intermediates. Here, we show that phase evolution from multiple precursors can be modeled as…
▽ More
Solid-state synthesis from powder precursors is the primary processing route to advanced multicomponent ceramic materials. Designing ceramic synthesis routes is usually a laborious, trial-and-error process, as heterogeneous mixtures of powder precursors often evolve through a complicated series of reaction intermediates. Here, we show that phase evolution from multiple precursors can be modeled as a sequence of pairwise interfacial reactions, with thermodynamic driving forces that can be efficiently calculated using ab initio methods. Using the synthesis of the classic high-temperature superconductor YBa$_2$Cu$_3$O$_{6+x}$ (YBCO) as a representative system, we rationalize how replacing the common BaCO$_3$ precursor with BaO$_2$ redirects phase evolution through a kinetically-facile pathway. Our model is validated from in situ X-ray diffraction and in situ microscopy observations, which show rapid YBCO formation from BaO$_2$ in only 30 minutes. By combining thermodynamic modeling with in situ characterization, we introduce a new computable framework to interpret and ultimately design synthesis pathways to complex ceramic materials.
△ Less
Submitted 13 January, 2021; v1 submitted 22 September, 2020;
originally announced September 2020.
-
Deep Energy-Based Modeling of Discrete-Time Physics
Authors:
Takashi Matsubara,
Ai Ishikawa,
Takaharu Yaguchi
Abstract:
Physical phenomena in the real world are often described by energy-based modeling theories, such as Hamiltonian mechanics or the Landau theory, which yield various physical laws. Recent developments in neural networks have enabled the mimicking of the energy conservation law by learning the underlying continuous-time differential equations. However, this may not be possible in discrete time, which…
▽ More
Physical phenomena in the real world are often described by energy-based modeling theories, such as Hamiltonian mechanics or the Landau theory, which yield various physical laws. Recent developments in neural networks have enabled the mimicking of the energy conservation law by learning the underlying continuous-time differential equations. However, this may not be possible in discrete time, which is often the case in practical learning and computation. Moreover, other physical laws have been overlooked in the previous neural network models. In this study, we propose a deep energy-based physical model that admits a specific differential geometric structure. From this structure, the conservation or dissipation law of energy and the mass conservation law follow naturally. To ensure the energetic behavior in discrete time, we also propose an automatic discrete differential algorithm that enables neural networks to employ the discrete gradient method.
△ Less
Submitted 31 October, 2020; v1 submitted 21 May, 2019;
originally announced May 2019.
-
Preserving multiple first integrals by discrete gradients
Authors:
Morten Dahlby,
Brynjulf Owren,
Takaharu Yaguchi
Abstract:
We consider systems of ordinary differential equations with known first integrals. The notion of a discrete tangent space is introduced as the orthogonal complement of an arbitrary set of discrete gradients. Integrators which exactly conserve all the first integrals simultaneously are then defined. In both cases we start from an arbitrary method of a prescribed order (say, a Runge-Kutta scheme) an…
▽ More
We consider systems of ordinary differential equations with known first integrals. The notion of a discrete tangent space is introduced as the orthogonal complement of an arbitrary set of discrete gradients. Integrators which exactly conserve all the first integrals simultaneously are then defined. In both cases we start from an arbitrary method of a prescribed order (say, a Runge-Kutta scheme) and modify it using two approaches: one based on projection and one based one local coordinates. The methods are tested on the Kepler problem.
△ Less
Submitted 8 June, 2011; v1 submitted 1 November, 2010;
originally announced November 2010.
