-
Learning Robust Kernel Ensembles with Kernel Average Pooling
Authors:
Pouya Bashivan,
Adam Ibrahim,
Amirozhan Dehghani,
Yifei Ren
Abstract:
Model ensembles have long been used in machine learning to reduce the variance in individual model predictions, making them more robust to input perturbations. Pseudo-ensemble methods like dropout have also been commonly used in deep learning models to improve generalization. However, the application of these techniques to improve neural networks' robustness against input perturbations remains und…
▽ More
Model ensembles have long been used in machine learning to reduce the variance in individual model predictions, making them more robust to input perturbations. Pseudo-ensemble methods like dropout have also been commonly used in deep learning models to improve generalization. However, the application of these techniques to improve neural networks' robustness against input perturbations remains underexplored. We introduce Kernel Average Pooling (KAP), a neural network building block that applies the mean filter along the kernel dimension of the layer activation tensor. We show that ensembles of kernels with similar functionality naturally emerge in convolutional neural networks equipped with KAP and trained with backpropagation. Moreover, we show that when trained on inputs perturbed with additive Gaussian noise, KAP models are remarkably robust against various forms of adversarial attacks. Empirical evaluations on CIFAR10, CIFAR100, TinyImagenet, and Imagenet datasets show substantial improvements in robustness against strong adversarial attacks such as AutoAttack without training on any adversarial examples.
△ Less
Submitted 30 May, 2023; v1 submitted 30 September, 2022;
originally announced October 2022.
-
The pure spectrum of a residuated lattice
Authors:
Saeed Rasouli,
Amin Dehghani
Abstract:
This paper studies a fascinating type of filter in residuated lattices, the so-called pure filters. A combination of algebraic and topological methods on the pure filters of a residuated lattice is applied to obtain some new structural results. The notion of purely-prime filters of a residuated lattice has been investigated, and a Cohen-type theorem has been obtained. It is shown that the pure spe…
▽ More
This paper studies a fascinating type of filter in residuated lattices, the so-called pure filters. A combination of algebraic and topological methods on the pure filters of a residuated lattice is applied to obtain some new structural results. The notion of purely-prime filters of a residuated lattice has been investigated, and a Cohen-type theorem has been obtained. It is shown that the pure spectrum of a residuated lattice is a compact sober space, and a Grothendieck-type theorem has been demonstrated. It is proved that the pure spectrum of a Gelfand residuated lattice is a Hausdorff space, and deduced that the pure spectrum of a Gelfand residuated lattice is homeomorphic to its usual maximal spectrum. Finally, the pure spectrum of an mp-residuated lattice is investigated and verified that a given residuated lattice is mp iff its minimal prime spectrum is equipped with the induced dual hull-kernel topology, and its pure spectrum is the same.
△ Less
Submitted 16 July, 2022;
originally announced July 2022.
-
Human Recognition based on Retinal Bifurcations and Modified Correlation Function
Authors:
Amin Dehghani
Abstract:
Nowadays high security is an important issue for most of the secure places and recent advances increase the needs of high-security systems. Therefore, needs to high security for controlling and permitting the allowable people to enter the high secure places, increases and extends the use of conventional recognition methods. Therefore, a novel identification method using retinal images is proposed…
▽ More
Nowadays high security is an important issue for most of the secure places and recent advances increase the needs of high-security systems. Therefore, needs to high security for controlling and permitting the allowable people to enter the high secure places, increases and extends the use of conventional recognition methods. Therefore, a novel identification method using retinal images is proposed in this paper. For this purpose, new mathematical functions are applied on corners and bifurcations. To evaluate the proposed method we use 40 retinal images from the DRIVE database, 20 normal retinal image from STARE database and 140 normal retinal images from local collected database and the accuracy rate is 99.34 percent.
△ Less
Submitted 18 September, 2021;
originally announced September 2021.
-
Time-Frequency Localization Using Deep Convolutional Maxout Neural Network in Persian Speech Recognition
Authors:
Arash Dehghani,
Seyyed Ali Seyyedsalehi
Abstract:
In this paper, a CNN-based structure for the time-frequency localization of information is proposed for Persian speech recognition. Research has shown that the receptive fields' spectrotemporal plasticity of some neurons in mammals' primary auditory cortex and midbrain makes localization facilities improve recognition performance. Over the past few years, much work has been done to localize time-f…
▽ More
In this paper, a CNN-based structure for the time-frequency localization of information is proposed for Persian speech recognition. Research has shown that the receptive fields' spectrotemporal plasticity of some neurons in mammals' primary auditory cortex and midbrain makes localization facilities improve recognition performance. Over the past few years, much work has been done to localize time-frequency information in ASR systems, using the spatial or temporal immutability properties of methods such as HMMs, TDNNs, CNNs, and LSTM-RNNs. However, most of these models have large parameter volumes and are challenging to train. For this purpose, we have presented a structure called Time-Frequency Convolutional Maxout Neural Network (TFCMNN) in which parallel time-domain and frequency-domain 1D-CMNNs are applied simultaneously and independently to the spectrogram, and then their outputs are concatenated and applied jointly to a fully connected Maxout network for classification. To improve the performance of this structure, we have used newly developed methods and models such as Dropout, maxout, and weight normalization. Two sets of experiments were designed and implemented on the FARSDAT dataset to evaluate the performance of this model compared to conventional 1D-CMNN models. According to the experimental results, the average recognition score of TFCMNN models is about 1.6% higher than the average of conventional 1D-CMNN models. In addition, the average training time of the TFCMNN models is about 17 hours lower than the average training time of traditional models. Therefore, as proven in other sources, time-frequency localization in ASR systems increases system accuracy and speeds up the training process.
△ Less
Submitted 30 August, 2022; v1 submitted 9 August, 2021;
originally announced August 2021.
-
Performance Evaluation of Deep Convolutional Maxout Neural Network in Speech Recognition
Authors:
Arash Dehghani,
Seyyed Ali Seyyedsalehi
Abstract:
In this paper, various structures and methods of Deep Artificial Neural Networks (DNN) will be evaluated and compared for the purpose of continuous Persian speech recognition. One of the first models of neural networks used in speech recognition applications were fully connected Neural Networks (FCNNs) and, consequently, Deep Neural Networks (DNNs). Although these models have better performance co…
▽ More
In this paper, various structures and methods of Deep Artificial Neural Networks (DNN) will be evaluated and compared for the purpose of continuous Persian speech recognition. One of the first models of neural networks used in speech recognition applications were fully connected Neural Networks (FCNNs) and, consequently, Deep Neural Networks (DNNs). Although these models have better performance compared to GMM / HMM models, they do not have the proper structure to model local speech information. Convolutional Neural Network (CNN) is a good option for modeling the local structure of biological signals, including speech signals. Another issue that Deep Artificial Neural Networks face, is the convergence of networks on training data. The main inhibitor of convergence is the presence of local minima in the process of training. Deep Neural Network Pre-training methods, despite a large amount of computing, are powerful tools for crossing the local minima. But the use of appropriate neuronal models in the network structure seems to be a better solution to this problem. The Rectified Linear Unit neuronal model and the Maxout model are the most suitable neuronal models presented to this date. Several experiments were carried out to evaluate the performance of the methods and structures mentioned. After verifying the proper functioning of these methods, a combination of all models was implemented on FARSDAT speech database for continuous speech recognition. The results obtained from the experiments show that the combined model (CMDNN) improves the performance of ANNs in speech recognition versus the pre-trained fully connected NNs with sigmoid neurons by about 3%.
△ Less
Submitted 4 May, 2021;
originally announced May 2021.
-
Subject Cross Validation in Human Activity Recognition
Authors:
Akbar Dehghani,
Tristan Glatard,
Emad Shihab
Abstract:
K-fold Cross Validation is commonly used to evaluate classifiers and tune their hyperparameters. However, it assumes that data points are Independent and Identically Distributed (i.i.d.) so that samples used in the training and test sets can be selected randomly and uniformly. In Human Activity Recognition datasets, we note that the samples produced by the same subjects are likely to be correlated…
▽ More
K-fold Cross Validation is commonly used to evaluate classifiers and tune their hyperparameters. However, it assumes that data points are Independent and Identically Distributed (i.i.d.) so that samples used in the training and test sets can be selected randomly and uniformly. In Human Activity Recognition datasets, we note that the samples produced by the same subjects are likely to be correlated due to diverse factors. Hence, k-fold cross validation may overestimate the performance of activity recognizers, in particular when overlap** sliding windows are used. In this paper, we investigate the effect of Subject Cross Validation on the performance of Human Activity Recognition, both with non-overlap** and with overlap** sliding windows. Results show that k-fold cross validation artificially increases the performance of recognizers by about 10%, and even by 16% when overlap** windows are used. In addition, we do not observe any performance gain from the use of overlap** windows. We conclude that Human Activity Recognition systems should be evaluated by Subject Cross Validation, and that overlap** windows are not worth their extra computational cost.
△ Less
Submitted 9 April, 2019; v1 submitted 4 April, 2019;
originally announced April 2019.
