-
Altering Backward Pass Gradients improves Convergence
Authors:
Bishshoy Das,
Milton Mondal,
Brejesh Lall,
Shiv Dutt Joshi,
Sumantra Dutta Roy
Abstract:
In standard neural network training, the gradients in the backward pass are determined by the forward pass. As a result, the two stages are coupled. This is how most neural networks are trained currently. However, gradient modification in the backward pass has seldom been studied in the literature. In this paper we explore decoupled training, where we alter the gradients in the backward pass. We p…
▽ More
In standard neural network training, the gradients in the backward pass are determined by the forward pass. As a result, the two stages are coupled. This is how most neural networks are trained currently. However, gradient modification in the backward pass has seldom been studied in the literature. In this paper we explore decoupled training, where we alter the gradients in the backward pass. We propose a simple yet powerful method called PowerGrad Transform, that alters the gradients before the weight update in the backward pass and significantly enhances the predictive performance of the neural network. PowerGrad Transform trains the network to arrive at a better optima at convergence. It is computationally extremely efficient, virtually adding no additional cost to either memory or compute, but results in improved final accuracies on both the training and test sets. PowerGrad Transform is easy to integrate into existing training routines, requiring just a few lines of code. PowerGrad Transform accelerates training and makes it possible for the network to better fit the training data. With decoupled training, PowerGrad Transform improves baseline accuracies for ResNet-50 by 0.73%, for SE-ResNet-50 by 0.66% and by more than 1.0% for the non-normalized ResNet-18 network on the ImageNet classification task.
△ Less
Submitted 20 September, 2022; v1 submitted 24 November, 2021;
originally announced November 2021.
-
The Generalized Fourier Transform: A Unified Framework for the Fourier, Laplace, Mellin and $Z$ Transforms
Authors:
Pushpendra Singh,
Anubha Gupta,
Shiv Dutt Joshi
Abstract:
This paper introduces Generalized Fourier transform (GFT) that is an extension or the generalization of the Fourier transform (FT). The Unilateral Laplace transform (LT) is observed to be the special case of GFT. GFT, as proposed in this work, contributes significantly to the scholarly literature. There are many salient contribution of this work. Firstly, GFT is applicable to a much larger class o…
▽ More
This paper introduces Generalized Fourier transform (GFT) that is an extension or the generalization of the Fourier transform (FT). The Unilateral Laplace transform (LT) is observed to be the special case of GFT. GFT, as proposed in this work, contributes significantly to the scholarly literature. There are many salient contribution of this work. Firstly, GFT is applicable to a much larger class of signals, some of which cannot be analyzed with FT and LT. For example, we have shown the applicability of GFT on the polynomially decaying functions and super exponentials. Secondly, we demonstrate the efficacy of GFT in solving the initial value problems (IVPs). Thirdly, the generalization presented for FT is extended for other integral transforms with examples shown for wavelet transform and cosine transform. Likewise, generalized Gamma function is also presented. One interesting application of GFT is the computation of generalized moments, for the otherwise non-finite moments, of any random variable such as the Cauchy random variable. Fourthly, we introduce Fourier scale transform (FST) that utilizes GFT with the topological isomorphism of an exponential map. Lastly, we propose Generalized Discrete-Time Fourier transform (GDTFT). The DTFT and unilateral $z$-transform are shown to be the special cases of the proposed GDTFT. The properties of GFT and GDTFT have also been discussed.
△ Less
Submitted 12 February, 2021;
originally announced March 2021.
-
Image fusion using symmetric skip autoencodervia an Adversarial Regulariser
Authors:
Snigdha Bhagat,
S. D. Joshi,
Brejesh Lall
Abstract:
It is a challenging task to extract the best of both worlds by combining the spatial characteristics of a visible image and the spectral content of an infrared image. In this work, we propose a spatially constrained adversarial autoencoder that extracts deep features from the infrared and visible images to obtain a more exhaustive and global representation. In this paper, we propose a residual aut…
▽ More
It is a challenging task to extract the best of both worlds by combining the spatial characteristics of a visible image and the spectral content of an infrared image. In this work, we propose a spatially constrained adversarial autoencoder that extracts deep features from the infrared and visible images to obtain a more exhaustive and global representation. In this paper, we propose a residual autoencoder architecture, regularised by a residual adversarial network, to generate a more realistic fused image. The residual module serves as primary building for the encoder, decoder and adversarial network, as an add on the symmetric skip connections perform the functionality of embedding the spatial characteristics directly from the initial layers of encoder structure to the decoder part of the network. The spectral information in the infrared image is incorporated by adding the feature maps over several layers in the encoder part of the fusion structure, which makes inference on both the visual and infrared images separately. In order to efficiently optimize the parameters of the network, we propose an adversarial regulariser network which would perform supervised learning on the fused image and the original visual image.
△ Less
Submitted 4 June, 2020; v1 submitted 1 May, 2020;
originally announced May 2020.
-
Unified Functorial Signal Representation III: Foundations, Redundancy, $L^0$ and $L^2$ functors
Authors:
Salil Samant,
Shiv Dutt Joshi
Abstract:
In this paper we propose and lay the foundations of a functorial framework for representing signals. By incorporating additional category-theoretic relative and generative perspective alongside the classic set-theoretic measure theory the fundamental concepts of redundancy, compression are formulated in a novel authentic arrow-theoretic way. The existing classic framework representing a signal as…
▽ More
In this paper we propose and lay the foundations of a functorial framework for representing signals. By incorporating additional category-theoretic relative and generative perspective alongside the classic set-theoretic measure theory the fundamental concepts of redundancy, compression are formulated in a novel authentic arrow-theoretic way. The existing classic framework representing a signal as a vector of appropriate linear space is shown as a special case of the proposed framework.
Next in the context of signal-spaces as a categories we study the various covariant and contravariant forms of $L^0$ and $L^2$ functors using categories of measurable or measure spaces and their opposites involving Boolean and measure algebras along with partial extension. Finally we contribute a novel definition of intra-signal redundancy using general concept of isomorphism arrow in a category covering the translation case and others as special cases. Through category-theory we provide a simple yet precise explanation for the well-known heuristic of lossless differential encoding standards yielding better compressions in image types such as line drawings, iconic image, text etc; as compared to classic representation techniques such as JPEG which choose bases or frames in a global Hilbert space.
△ Less
Submitted 27 October, 2017;
originally announced October 2017.
-
Orthogonal Ramanujan Sums, its properties and Applications in Multiresolution Analysis
Authors:
Devendra Kumar Yadav,
Gajraj Kuldeep,
S. D. Joshi
Abstract:
Signal processing community has recently shown interest in Ramanujan sums which was defined by S.Ramanujan in 1918. In this paper we have proposed Orthog- onal Ramanujan Sums (ORS) based on Ramanujan sums. In this paper we present two novel application of ORS. Firstly a new representation of a finite length signal is given using ORS which is defined as Orthogonal Ramanujan Periodic Transform.Secon…
▽ More
Signal processing community has recently shown interest in Ramanujan sums which was defined by S.Ramanujan in 1918. In this paper we have proposed Orthog- onal Ramanujan Sums (ORS) based on Ramanujan sums. In this paper we present two novel application of ORS. Firstly a new representation of a finite length signal is given using ORS which is defined as Orthogonal Ramanujan Periodic Transform.Secondly ORS has been applied to multiresolution analysis and it is shown that Haar transform is a spe- cial case.
△ Less
Submitted 24 May, 2017;
originally announced July 2017.
-
Some studies on multidimensional Fourier theory for Hilbert transform, analytic signal and space-time series analysis
Authors:
Pushpendra Singh,
Shiv Dutt Joshi
Abstract:
In this paper, we propose the Fourier frequency vector (FFV), inherently, associated with multidimensional Fourier transform. With the help of FFV, we are able to provide physical meaning of so called negative frequencies in multidimensional Fourier transform (MDFT), which in turn provide multidimensional spatial and space-time series analysis. The complex exponential representation of sinusoidal…
▽ More
In this paper, we propose the Fourier frequency vector (FFV), inherently, associated with multidimensional Fourier transform. With the help of FFV, we are able to provide physical meaning of so called negative frequencies in multidimensional Fourier transform (MDFT), which in turn provide multidimensional spatial and space-time series analysis. The complex exponential representation of sinusoidal function always yields two frequencies, negative frequency corresponding to positive frequency and vice versa, in the multidimensional Fourier spectrum. Thus, using the MDFT, we propose multidimensional Hilbert transform (MDHT) and associated multidimensional analytic signal (MDAS) with following properties: (a) the extra and redundant positive, negative, or both frequencies, introduced due to complex exponential representation of multidimensional Fourier spectrum, are suppressed, (b) real part of MDAS is original signal, (c) real and imaginary part of MDAS are orthogonal, and (d) the magnitude envelope of a original signal is obtained as the magnitude of its associated MDAS, which is the instantaneous amplitude of the MDAS. The proposed MDHT and associated DMAS are generalization of the 1D HT and AS, respectively. We also provide the decomposition of an image into the AM-FM image model by the Fourier method and obtain explicit expression for the analytic image computation by 2DDFT.
△ Less
Submitted 29 July, 2015;
originally announced July 2015.
-
Computationally efficient MIMO system identification using Signal Matched Synthesis Filter Bank
Authors:
Binish Fatimah,
Shiv Dutt Joshi
Abstract:
We propose a multi input multi output(MIMO) system identification framework by interpreting the MIMO system in terms of a multirate synthesis filter bank. The proposed methodology is discussed in two steps: in the first step the MIMO system is interpreted as a synthesis filter bank and the second step is to convert the MIMO system into a SISO system "without any loss of information", which re-stru…
▽ More
We propose a multi input multi output(MIMO) system identification framework by interpreting the MIMO system in terms of a multirate synthesis filter bank. The proposed methodology is discussed in two steps: in the first step the MIMO system is interpreted as a synthesis filter bank and the second step is to convert the MIMO system into a SISO system "without any loss of information", which re-structures the system identification problem into a SISO form. The system identification problem, in its new form, is identical to the problem of obtaining the signal matched synthesis filter bank (SMSFB) as proposed in Part II. Since we have developed fast algorithms to obtain the filter bank coefficients in Part II, for "the given data case" as well as "the given statistics case", we can use these algorithm for the MIMO system identification as well. This framework can have an adaptive as well as block processing implementation. The algorithms, used here, involve only scalar computations, unlike the conventional MIMO system identification algorithms where one requires matrix computations. These order recursive algorithm can also be used to obtain approximate smaller order model for large order systems without using any model order reduction algorithm. The proposed identification framework can also be used for SISO LPTV system identification and also for a SIMO or MISO system. The efficacy of the proposed scheme is validated and its performance in the presence of measurement noise is illustrated using simulation results.
△ Less
Submitted 26 May, 2015;
originally announced May 2015.
-
The Hilbert spectrum and the Energy Preserving Empirical Mode Decomposition
Authors:
Pushpendra Singh,
Shiv Dutt Joshi,
Rakesh Kumar Patney,
Kaushik Saha
Abstract:
In this paper, we propose algorithms which preserve energy in empirical mode decomposition (EMD), generating finite $n$ number of band limited Intrinsic Mode Functions (IMFs). In the first energy preserving EMD (EPEMD) algorithm, a signal is decomposed into linearly independent (LI), non orthogonal yet energy preserving (LINOEP) IMFs and residue (EPIMFs). It is shown that a vector in an inner prod…
▽ More
In this paper, we propose algorithms which preserve energy in empirical mode decomposition (EMD), generating finite $n$ number of band limited Intrinsic Mode Functions (IMFs). In the first energy preserving EMD (EPEMD) algorithm, a signal is decomposed into linearly independent (LI), non orthogonal yet energy preserving (LINOEP) IMFs and residue (EPIMFs). It is shown that a vector in an inner product space can be represented as a sum of LI and non orthogonal vectors in such a way that Parseval's type property is satisfied. From the set of $n$ IMFs, through Gram-Schmidt orthogonalization method (GSOM), $n!$ set of orthogonal functions can be obtained. In the second algorithm, we show that if the orthogonalization process proceeds from lowest frequency IMF to highest frequency IMF, then the GSOM yields functions which preserve the properties of IMFs and the energy of a signal. With the Hilbert transform, these IMFs yield instantaneous frequencies and amplitudes as functions of time that reveal the imbedded structures of a signal. The instantaneous frequencies and square of amplitudes as functions of time produce a time-frequency-energy distribution, referred as the Hilbert spectrum, of a signal. Simulations have been carried out for the analysis of various time series and real life signals to show comparison among IMFs produced by EMD, EPEMD, ensemble EMD and multivariate EMD algorithms. Simulation results demonstrate the power of this proposed method.
△ Less
Submitted 16 April, 2015;
originally announced April 2015.
-
The Fourier Decomposition Method for nonlinear and nonstationary time series analysis
Authors:
Pushpendra Singh,
Shiv Dutt Joshi,
Rakesh Kumar Patney,
Kaushik Saha
Abstract:
Since many decades, there is a general perception in literature that the Fourier methods are not suitable for the analysis of nonlinear and nonstationary data. In this paper, we propose a Fourier Decomposition Method (FDM) and demonstrate its efficacy for the analysis of nonlinear (i.e. data generated by nonlinear systems) and nonstationary time series. The proposed FDM decomposes any data into a…
▽ More
Since many decades, there is a general perception in literature that the Fourier methods are not suitable for the analysis of nonlinear and nonstationary data. In this paper, we propose a Fourier Decomposition Method (FDM) and demonstrate its efficacy for the analysis of nonlinear (i.e. data generated by nonlinear systems) and nonstationary time series. The proposed FDM decomposes any data into a small number of `Fourier intrinsic band functions' (FIBFs). The FDM presents a generalized Fourier expansion with variable amplitudes and frequencies of a time series by the Fourier method itself. We propose an idea of zero-phase filter bank based multivariate FDM (MFDM) algorithm, for the analysis of multivariate nonlinear and nonstationary time series, from the FDM. We also present an algorithm to obtain cutoff frequencies for MFDM. The MFDM algorithm is generating finite number of band limited multivariate FIBFs (MFIBFs). The MFDM preserves some intrinsic physical properties of the multivariate data, such as scale alignment, trend and instantaneous frequency. The proposed methods produce the results in a time-frequency-energy distribution that reveal the intrinsic structures of a data. Simulations have been carried out and comparison is made with the Empirical Mode Decomposition (EMD) methods in the analysis of various simulated as well as real life time series, and results show that the proposed methods are powerful tools for analyzing and obtaining the time-frequency-energy representation of any data.
△ Less
Submitted 31 August, 2015; v1 submitted 26 February, 2015;
originally announced March 2015.
-
Exact Least Squares Algorithm for Signal Matched Synthesis Filter Bank: Part II
Authors:
Binish Fatimah,
S. D. Joshi
Abstract:
In the companion paper, we proposed a concept of signal matched whitening filter bank and developed a time and order recursive, fast least squares algorithm for the same. Objective of part II of the paper is two fold: first is to define a concept of signal matched synthesis filter bank, hence combining definitions of part I and part II we obtain a filter bank matched to a given signal. We also dev…
▽ More
In the companion paper, we proposed a concept of signal matched whitening filter bank and developed a time and order recursive, fast least squares algorithm for the same. Objective of part II of the paper is two fold: first is to define a concept of signal matched synthesis filter bank, hence combining definitions of part I and part II we obtain a filter bank matched to a given signal. We also develop a fast time and order recursive, least squares algorithm for obtaining the same. The synthesis filters, obtained here, reconstruct the given signal only and not every signal from the finite energy signal space (i.e. belonging to L^2(R)), as is usually done. The recursions, so obtained, result in a lattice-like structure. Since the filter parameters are not directly available, we also present an order recursive algorithm for the computation of signal matched synthesis filter bank coefficients from the lattice parameters. The second objective is to explore the possibility of using synthesis side for modeling of a given stochastic process. Simulation results have also been presented to validate the theory.
△ Less
Submitted 16 September, 2014;
originally announced September 2014.
-
Exact Least Squares Algorithm for Signal Matched Multirate Whitening Filter Bank: Part I
Authors:
Binish Fatimah,
S. D. Joshi
Abstract:
In this paper, we define a concept of signal matched multirate whitening filter bank which provides an optimum coding gain. This is achieved by whitening the outputs, of the analysis filter bank, within as well as across the channels, by solving a constrained projection problem. We also present a fast time and order recursive least squares algorithm to obtain the vector output of the proposed anal…
▽ More
In this paper, we define a concept of signal matched multirate whitening filter bank which provides an optimum coding gain. This is achieved by whitening the outputs, of the analysis filter bank, within as well as across the channels, by solving a constrained projection problem. We also present a fast time and order recursive least squares algorithm to obtain the vector output of the proposed analysis filter bank. The recursive algorithm, developed here, gives rise to a lattice-like structure. Since the proposed signal matched analysis filter bank coefficients are not available directly, an order recursive algorithm is also presented for estimating these from the lattice parameters. Simulation results are presented to validate the theory. It is also observed that the proposed algorithm can be used to whiten Gaussian/non-Gaussian processes with minimum as well as non-minimum phase.
△ Less
Submitted 17 September, 2014;
originally announced September 2014.
