-
Constructing Extreme Learning Machines with zero Spectral Bias
Authors:
Kaumudi Joshi,
Vukka Snigdha,
Arya Kumar Bhattacharya
Abstract:
The phenomena of Spectral Bias, where the higher frequency components of a function being learnt in a feedforward Artificial Neural Network (ANN) are seen to converge more slowly than the lower frequencies, is observed ubiquitously across ANNs. This has created technology challenges in fields where resolution of higher frequencies is crucial, like in Physics Informed Neural Networks (PINNs). Extre…
▽ More
The phenomena of Spectral Bias, where the higher frequency components of a function being learnt in a feedforward Artificial Neural Network (ANN) are seen to converge more slowly than the lower frequencies, is observed ubiquitously across ANNs. This has created technology challenges in fields where resolution of higher frequencies is crucial, like in Physics Informed Neural Networks (PINNs). Extreme Learning Machines (ELMs) that obviate an iterative solution process which provides the theoretical basis of Spectral Bias (SB), should in principle be free of the same. This work verifies the reliability of this assumption, and shows that it is incorrect. However, the structure of ELMs makes them naturally amenable to implementation of variants of Fourier Feature Embeddings, which have been shown to mitigate SB in ANNs. This approach is implemented and verified to completely eliminate SB, thus bringing into feasibility the application of ELMs for practical problems like PINNs where resolution of higher frequencies is essential.
△ Less
Submitted 19 July, 2023;
originally announced July 2023.
-
Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders
Authors:
Mayank Deshpande,
Siddharth Agarwal,
Vukka Snigdha,
Arya Kumar Bhattacharya
Abstract:
An important inference from Neural Tangent Kernel (NTK) theory is the existence of spectral bias (SB), that is, low frequency components of the target function of a fully connected Artificial Neural Network (ANN) being learnt significantly faster than the higher frequencies during training. This is established for Mean Square Error (MSE) loss functions with very low learning rate parameters. Physi…
▽ More
An important inference from Neural Tangent Kernel (NTK) theory is the existence of spectral bias (SB), that is, low frequency components of the target function of a fully connected Artificial Neural Network (ANN) being learnt significantly faster than the higher frequencies during training. This is established for Mean Square Error (MSE) loss functions with very low learning rate parameters. Physics Informed Neural Networks (PINNs) are designed to learn the solutions of differential equations (DE) of arbitrary orders; in PINNs the loss functions are obtained as the residues of the conservative form of the DEs and represent the degree of dissatisfaction of the equations. So there has been an open question whether (a) PINNs also exhibit SB and (b) if so, how does this bias vary across the orders of the DEs. In this work, a series of numerical experiments are conducted on simple sinusoidal functions of varying frequencies, compositions and equation orders to investigate these issues. It is firmly established that under normalized conditions, PINNs do exhibit strong spectral bias, and this increases with the order of the differential equation.
△ Less
Submitted 7 January, 2023;
originally announced January 2023.
-
Function reconstruction as a classical moment problem: A maximum entropy approach
Authors:
Parthapratim Biswas,
Arun K. Bhattacharya
Abstract:
We present a systematic study of the reconstruction of a non-negative function via maximum entropy approach utilizing the information contained in a finite number of moments of the function. For testing the efficacy of the approach, we reconstruct a set of functions using an iterative entropy optimization scheme, and study the convergence profile as the number of moments is increased. We consider…
▽ More
We present a systematic study of the reconstruction of a non-negative function via maximum entropy approach utilizing the information contained in a finite number of moments of the function. For testing the efficacy of the approach, we reconstruct a set of functions using an iterative entropy optimization scheme, and study the convergence profile as the number of moments is increased. We consider a wide variety of functions that include a distribution with a sharp discontinuity, a rapidly oscillatory function, a distribution with singularities, and finally a distribution with several spikes and fine structure. The last example is important in the context of the determination of the natural density of the logistic map. The convergence of the method is studied by comparing the moments of the approximated functions with the exact ones. Furthermore, by varying the number of moments and iterations, we examine to what extent the features of the functions, such as the divergence behavior at singular points within the interval, is reproduced. The proximity of the reconstructed maximum entropy solution to the exact solution is examined via Kullback-Leibler divergence and variation measures for different number of moments.
△ Less
Submitted 27 April, 2010;
originally announced April 2010.
-
Maximum entropy and the problem of moments: A stable algorithm
Authors:
K. Bandyopadhyay,
A. K. Bhattacharya,
Parthapratim Biswas,
D. A. Drabold
Abstract:
We present a technique for entropy optimization to calculate a distribution from its moments. The technique is based upon maximizing a discretized form of the Shannon entropy functional by map** the problem onto a dual space where an optimal solution can be constructed iteratively. We demonstrate the performance and stability of our algorithm with several tests on numerically difficult functio…
▽ More
We present a technique for entropy optimization to calculate a distribution from its moments. The technique is based upon maximizing a discretized form of the Shannon entropy functional by map** the problem onto a dual space where an optimal solution can be constructed iteratively. We demonstrate the performance and stability of our algorithm with several tests on numerically difficult functions. We then consider an electronic structure application, the electronic density of states of amorphous silica and study the convergence of Fermi level with increasing number of moments.
△ Less
Submitted 27 December, 2004;
originally announced December 2004.
-
Structure and stability of copper clusters : A tight-binding molecular dynamics study
Authors:
Mukul Kabir,
Abhijit Mookerjee,
A. K. Bhattacharya
Abstract:
In this paper we propose a tight-binding molecular dynamics with parameters fitted to first-principles calculations on the smaller clusters and with an environment correction, to be a powerful technique for studying large transition/noble metal clusters. In particular, the structure and stability of $Cu_n$ clusters for $n=3-55$ are studied by using this technique. The results for small $Cu_n$ cl…
▽ More
In this paper we propose a tight-binding molecular dynamics with parameters fitted to first-principles calculations on the smaller clusters and with an environment correction, to be a powerful technique for studying large transition/noble metal clusters. In particular, the structure and stability of $Cu_n$ clusters for $n=3-55$ are studied by using this technique. The results for small $Cu_n$ clusters ($n=3-9$) show good agreement with {\it ab initio} calculations and available experimental results. In the size range $10\le n \le 55$ most of the clusters adopt icosahedral structure which can be derived from the 13-atom icosahedron, the polyicosahedral 19-, 23-, and 26-atom clusters and the 55-atom icosahedron, by adding or removing atoms. However, a local geometrical change from icosahedral to decahedral structure is observed for $n = 40-44$ and return to the icosahedral growth pattern is found at $n=45$ which continues. Electronic "magic numbers" ($n=2$, 8, 20, 34, 40) in this regime are correctly reproduced. Due to electron pairing in HOMOs, even-odd alternation is found. A sudden loss of even-odd alternation in second difference of cluster binding energy, HOMO-LUMO gap energy and ionization potential is observed in the region $n\sim40$ due to structural change there. Interplay between electronic and geometrical structure is found.
△ Less
Submitted 29 October, 2003;
originally announced October 2003.
-
Electronic and Optical Properties of ZnIn$_2$Te$_4$
Authors:
Biplab Ganguli,
Kamal Krishna Saha,
Tanusri Saha-Dasgupta,
Abhijit Mookerjee,
A. K. Bhattacharya
Abstract:
Band structure and optical properties of defect- Chalcopyrite type semiconductor ZnIn$_2$Te$_4$ have been studied by TB-LMTO first principle technique. The optical absorption calculation suggest that ZnIn$_2$Te$_4$ is a direct-gap semiconductor having a band gap of 1.40 eV., which confirms the experimentally measured value. The calculated complex dielectric-function $ε(E) = ε_1(E) + iε_2(E)$ rev…
▽ More
Band structure and optical properties of defect- Chalcopyrite type semiconductor ZnIn$_2$Te$_4$ have been studied by TB-LMTO first principle technique. The optical absorption calculation suggest that ZnIn$_2$Te$_4$ is a direct-gap semiconductor having a band gap of 1.40 eV., which confirms the experimentally measured value. The calculated complex dielectric-function $ε(E) = ε_1(E) + iε_2(E)$ reveal distinct structures at energies of the critical points in the Brillouin zone.
△ Less
Submitted 5 December, 2002;
originally announced December 2002.
-
An augmented space recursion study of the electronic structure of rough epitaxial overlayers
Authors:
Biplab Sanyal,
Parthapratim Biswas,
Abhijit Mookerjee,
Hemant G. Salunke,
G. P. Das,
A. K. Bhattacharya
Abstract:
In this communication we propose the use of the Augmented Space Recursion as an ideal methodology for the study of electronic and magnetic structures of rough surfaces, interfaces and overlayers. The method can take into account roughness, short-ranged clustering effects, surface dilatation and interdiffusion. We illustrate our method by an application of Fe overlayer on Ag (100) surface.
In this communication we propose the use of the Augmented Space Recursion as an ideal methodology for the study of electronic and magnetic structures of rough surfaces, interfaces and overlayers. The method can take into account roughness, short-ranged clustering effects, surface dilatation and interdiffusion. We illustrate our method by an application of Fe overlayer on Ag (100) surface.
△ Less
Submitted 19 March, 1999;
originally announced March 1999.
