-
Training Artificial Neural Networks by Coordinate Search Algorithm
Authors:
Ehsan Rokhsatyazdi,
Shahryar Rahnamayan,
Sevil Zanjani Miyandoab,
Azam Asilian Bidgoli,
H. R. Tizhoosh
Abstract:
Training Artificial Neural Networks poses a challenging and critical problem in machine learning. Despite the effectiveness of gradient-based learning methods, such as Stochastic Gradient Descent (SGD), in training neural networks, they do have several limitations. For instance, they require differentiable activation functions, and cannot optimize a model based on several independent non-different…
▽ More
Training Artificial Neural Networks poses a challenging and critical problem in machine learning. Despite the effectiveness of gradient-based learning methods, such as Stochastic Gradient Descent (SGD), in training neural networks, they do have several limitations. For instance, they require differentiable activation functions, and cannot optimize a model based on several independent non-differentiable loss functions simultaneously; for example, the F1-score, which is used during testing, can be used during training when a gradient-free optimization algorithm is utilized. Furthermore, the training in any DNN can be possible with a small size of the training dataset. To address these concerns, we propose an efficient version of the gradient-free Coordinate Search (CS) algorithm, an instance of General Pattern Search methods, for training neural networks. The proposed algorithm can be used with non-differentiable activation functions and tailored to multi-objective/multi-loss problems. Finding the optimal values for weights of ANNs is a large-scale optimization problem. Therefore instead of finding the optimal value for each variable, which is the common technique in classical CS, we accelerate optimization and convergence by bundling the weights. In fact, this strategy is a form of dimension reduction for optimization problems. Based on the experimental results, the proposed method, in some cases, outperforms the gradient-based approach, particularly, in situations with insufficient labeled training data. The performance plots demonstrate a high convergence rate, highlighting the capability of our suggested method to find a reasonable solution with fewer function calls. As of now, the only practical and efficient way of training ANNs with hundreds of thousands of weights is gradient-based algorithms such as SGD or Adam. In this paper we introduce an alternative method for training ANN.
△ Less
Submitted 19 February, 2024;
originally announced February 2024.
-
Compact NSGA-II for Multi-objective Feature Selection
Authors:
Sevil Zanjani Miyandoab,
Shahryar Rahnamayan,
Azam Asilian Bidgoli
Abstract:
Feature selection is an expensive challenging task in machine learning and data mining aimed at removing irrelevant and redundant features. This contributes to an improvement in classification accuracy, as well as the budget and memory requirements for classification, or any other post-processing task conducted after feature selection. In this regard, we define feature selection as a multi-objecti…
▽ More
Feature selection is an expensive challenging task in machine learning and data mining aimed at removing irrelevant and redundant features. This contributes to an improvement in classification accuracy, as well as the budget and memory requirements for classification, or any other post-processing task conducted after feature selection. In this regard, we define feature selection as a multi-objective binary optimization task with the objectives of maximizing classification accuracy and minimizing the number of selected features. In order to select optimal features, we have proposed a binary Compact NSGA-II (CNSGA-II) algorithm. Compactness represents the population as a probability distribution to enhance evolutionary algorithms not only to be more memory-efficient but also to reduce the number of fitness evaluations. Instead of holding two populations during the optimization process, our proposed method uses several Probability Vectors (PVs) to generate new individuals. Each PV efficiently explores a region of the search space to find non-dominated solutions instead of generating candidate solutions from a small population as is the common approach in most evolutionary algorithms. To the best of our knowledge, this is the first compact multi-objective algorithm proposed for feature selection. The reported results for expensive optimization cases with a limited budget on five datasets show that the CNSGA-II performs more efficiently than the well-known NSGA-II method in terms of the hypervolume (HV) performance metric requiring less memory. The proposed method and experimental results are explained and analyzed in detail.
△ Less
Submitted 19 February, 2024;
originally announced February 2024.
-
Multi-objective Binary Coordinate Search for Feature Selection
Authors:
Sevil Zanjani Miyandoab,
Shahryar Rahnamayan,
Azam Asilian Bidgoli
Abstract:
A supervised feature selection method selects an appropriate but concise set of features to differentiate classes, which is highly expensive for large-scale datasets. Therefore, feature selection should aim at both minimizing the number of selected features and maximizing the accuracy of classification, or any other task. However, this crucial task is computationally highly demanding on many real-…
▽ More
A supervised feature selection method selects an appropriate but concise set of features to differentiate classes, which is highly expensive for large-scale datasets. Therefore, feature selection should aim at both minimizing the number of selected features and maximizing the accuracy of classification, or any other task. However, this crucial task is computationally highly demanding on many real-world datasets and requires a very efficient algorithm to reach a set of optimal features with a limited number of fitness evaluations. For this purpose, we have proposed the binary multi-objective coordinate search (MOCS) algorithm to solve large-scale feature selection problems. To the best of our knowledge, the proposed algorithm in this paper is the first multi-objective coordinate search algorithm. In this method, we generate new individuals by flip** a variable of the candidate solutions on the Pareto front. This enables us to investigate the effectiveness of each feature in the corresponding subset. In fact, this strategy can play the role of crossover and mutation operators to generate distinct subsets of features. The reported results indicate the significant superiority of our method over NSGA-II, on five real-world large-scale datasets, particularly when the computing budget is limited. Moreover, this simple hyper-parameter-free algorithm can solve feature selection much faster and more efficiently than NSGA-II.
△ Less
Submitted 19 February, 2024;
originally announced February 2024.
-
Ranking Loss and Sequestering Learning for Reducing Image Search Bias in Histopathology
Authors:
Pooria Mazaheri,
Azam Asilian Bidgoli,
Shahryar Rahnamayan,
H. R. Tizhoosh
Abstract:
Recently, deep learning has started to play an essential role in healthcare applications, including image search in digital pathology. Despite the recent progress in computer vision, significant issues remain for image searching in histopathology archives. A well-known problem is AI bias and lack of generalization. A more particular shortcoming of deep models is the ignorance toward search functio…
▽ More
Recently, deep learning has started to play an essential role in healthcare applications, including image search in digital pathology. Despite the recent progress in computer vision, significant issues remain for image searching in histopathology archives. A well-known problem is AI bias and lack of generalization. A more particular shortcoming of deep models is the ignorance toward search functionality. The former affects every model, the latter only search and matching. Due to the lack of ranking-based learning, researchers must train models based on the classification error and then use the resultant embedding for image search purposes. Moreover, deep models appear to be prone to internal bias even if using a large image repository of various hospitals. This paper proposes two novel ideas to improve image search performance. First, we use a ranking loss function to guide feature extraction toward the matching-oriented nature of the search. By forcing the model to learn the ranking of matched outputs, the representation learning is customized toward image search instead of learning a class label. Second, we introduce the concept of sequestering learning to enhance the generalization of feature extraction. By excluding the images of the input hospital from the matched outputs, i.e., sequestering the input domain, the institutional bias is reduced. The proposed ideas are implemented and validated through the largest public dataset of whole slide images. The experiments demonstrate superior results compare to the-state-of-art.
△ Less
Submitted 14 April, 2023;
originally announced April 2023.
-
Evolutionary Computation in Action: Feature Selection for Deep Embedding Spaces of Gigapixel Pathology Images
Authors:
Azam Asilian Bidgoli,
Shahryar Rahnamayan,
Taher Dehkharghanian,
Abtin Riasatian,
H. R. Tizhoosh
Abstract:
One of the main obstacles of adopting digital pathology is the challenge of efficient processing of hyperdimensional digitized biopsy samples, called whole slide images (WSIs). Exploiting deep learning and introducing compact WSI representations are urgently needed to accelerate image analysis and facilitate the visualization and interpretability of pathology results in a postpandemic world. In th…
▽ More
One of the main obstacles of adopting digital pathology is the challenge of efficient processing of hyperdimensional digitized biopsy samples, called whole slide images (WSIs). Exploiting deep learning and introducing compact WSI representations are urgently needed to accelerate image analysis and facilitate the visualization and interpretability of pathology results in a postpandemic world. In this paper, we introduce a new evolutionary approach for WSI representation based on large-scale multi-objective optimization (LSMOP) of deep embeddings. We start with patch-based sampling to feed KimiaNet , a histopathology-specialized deep network, and to extract a multitude of feature vectors. Coarse multi-objective feature selection uses the reduced search space strategy guided by the classification accuracy and the number of features. In the second stage, the frequent features histogram (FFH), a novel WSI representation, is constructed by multiple runs of coarse LSMOP. Fine evolutionary feature selection is then applied to find a compact (short-length) feature vector based on the FFH and contributes to a more robust deep-learning approach to digital pathology supported by the stochastic power of evolutionary algorithms. We validate the proposed schemes using The Cancer Genome Atlas (TCGA) images in terms of WSI representation, classification accuracy, and feature quality. Furthermore, a novel decision space for multicriteria decision making in the LSMOP field is introduced. Finally, a patch-level visualization approach is proposed to increase the interpretability of deep features. The proposed evolutionary algorithm finds a very compact feature vector to represent a WSI (almost 14,000 times smaller than the original feature vectors) with 8% higher accuracy compared to the codes provided by the state-of-the-art methods.
△ Less
Submitted 18 April, 2023; v1 submitted 1 March, 2023;
originally announced March 2023.
-
Image-Based Benchmarking and Visualization for Large-Scale Global Optimization
Authors:
Kyle Robert Harrison,
Azam Asilian Bidgoli,
Shahryar Rahnamayan,
Kalyanmoy Deb
Abstract:
In the context of optimization, visualization techniques can be useful for understanding the behaviour of optimization algorithms and can even provide a means to facilitate human interaction with an optimizer. Towards this goal, an image-based visualization framework, without dimension reduction, that visualizes the solutions to large-scale global optimization problems as images is proposed. In th…
▽ More
In the context of optimization, visualization techniques can be useful for understanding the behaviour of optimization algorithms and can even provide a means to facilitate human interaction with an optimizer. Towards this goal, an image-based visualization framework, without dimension reduction, that visualizes the solutions to large-scale global optimization problems as images is proposed. In the proposed framework, the pixels visualize decision variables while the entire image represents the overall solution quality. This framework affords a number of benefits over existing visualization techniques including enhanced scalability (in terms of the number of decision variables), facilitation of standard image processing techniques, providing nearly infinite benchmark cases, and explicit alignment with human perception. Furthermore, image-based visualization can be used to visualize the optimization process in real-time, thereby allowing the user to ascertain characteristics of the search process as it is progressing. To the best of the authors' knowledge, this is the first realization of a dimension-preserving, scalable visualization framework that embeds the inherent relationship between decision space and objective space. The proposed framework is utilized with 10 different map** schemes on an image-reconstruction problem that encompass continuous, discrete, binary, combinatorial, constrained, dynamic, and multi-objective optimization. The proposed framework is then demonstrated on arbitrary benchmark problems with known optima. Experimental results elucidate the flexibility and demonstrate how valuable information about the search process can be gathered via the proposed visualization framework.
△ Less
Submitted 23 July, 2020;
originally announced July 2020.
-
Artistic Style in Robotic Painting; a Machine Learning Approach to Learning Brushstroke from Human Artists
Authors:
Ardavan Bidgoli,
Manuel Ladron De Guevara,
Cinnie Hsiung,
Jean Oh,
Eunsu Kang
Abstract:
Robotic painting has been a subject of interest among both artists and roboticists since the 1970s. Researchers and interdisciplinary artists have employed various painting techniques and human-robot collaboration models to create visual mediums on canvas. One of the challenges of robotic painting is to apply a desired artistic style to the painting. Style transfer techniques with machine learning…
▽ More
Robotic painting has been a subject of interest among both artists and roboticists since the 1970s. Researchers and interdisciplinary artists have employed various painting techniques and human-robot collaboration models to create visual mediums on canvas. One of the challenges of robotic painting is to apply a desired artistic style to the painting. Style transfer techniques with machine learning models have helped us address this challenge with the visual style of a specific painting. However, other manual elements of style, i.e., painting techniques and brushstrokes of an artist, have not been fully addressed. We propose a method to integrate an artistic style to the brushstrokes and the painting process through collaboration with a human artist. In this paper, we describe our approach to 1) collect brushstrokes and hand-brush motion samples from an artist, and 2) train a generative model to generate brushstrokes that pertains to the artist's style, and 3) fine tune a stroke-based rendering model to work with our robotic painting setup. We will report on the integration of these three steps in a separate publication. In a preliminary study, 71% of human evaluators find our reconstructed brushstrokes are pertaining to the characteristics of the artist's style. Moreover, 58% of participants could not distinguish a painting made by our method from a visually similar painting created by a human artist.
△ Less
Submitted 28 July, 2020; v1 submitted 7 July, 2020;
originally announced July 2020.
-
V-Dream: Immersive Exploration of Generative Design Solution Space
Authors:
Mohammad Keshavarzi,
Ardavan Bidgoli,
Hans Kellner
Abstract:
Generative Design workflows have introduced alternative paradigms in the domain of computational design, allowing designers to generate large pools of valid solutions by defining a set of goals and constraints. However, analyzing and narrowing down the generated solution space, which usually consists of various high-dimensional properties, has been a major challenge in current generative workflows…
▽ More
Generative Design workflows have introduced alternative paradigms in the domain of computational design, allowing designers to generate large pools of valid solutions by defining a set of goals and constraints. However, analyzing and narrowing down the generated solution space, which usually consists of various high-dimensional properties, has been a major challenge in current generative workflows. By taking advantage of the interactive unbounded spatial exploration, and the visual immersion offered in virtual reality platforms, we propose V-Dream, a virtual reality generative analysis framework for exploring large-scale solution spaces. V-Dream proposes a hybrid search workflow in which a spatial stochastic search approach is combined with a recommender system allowing users to pick desired candidates and eliminate the undesired ones iteratively. In each cycle, V-Dream reorganizes the remaining options in clusters based on the defined features. Moreover, our framework allows users to inspect design solutions and evaluate their performance metrics in various hierarchical levels, assisting them in narrowing down the solution space through iterative cycles of search/select/re-clustering of the solutions in an immersive fashion. Finally, we present a prototype of our proposed framework, illustrating how users can navigate and narrow down desired solutions from a pool of over 16000 monitor stands generated by Autodesk's Dreamcatcher software.
△ Less
Submitted 19 June, 2020;
originally announced June 2020.
-
Machinic Surrogates: Human-Machine Relationships in Computational Creativity
Authors:
Ardavan Bidgoli,
Eunsu Kang,
Daniel Cardoso Llach
Abstract:
Recent advancements in artificial intelligence (AI) and its sub-branch machine learning (ML) promise machines that go beyond the boundaries of automation and behave autonomously. Applications of these machines in creative practices such as art and design entail relationships between users and machines that have been described as a form of collaboration or co-creation between computational and huma…
▽ More
Recent advancements in artificial intelligence (AI) and its sub-branch machine learning (ML) promise machines that go beyond the boundaries of automation and behave autonomously. Applications of these machines in creative practices such as art and design entail relationships between users and machines that have been described as a form of collaboration or co-creation between computational and human agents. This paper uses examples from art and design to argue that this frame is incomplete as it fails to acknowledge the socio-technical nature of AI systems, and the different human agencies involved in their design, implementation, and operation. Situating applications of AI-enabled tools in creative practices in a spectrum between automation and autonomy, this paper distinguishes different kinds of human engagement elicited by systems deemed automated or autonomous. Reviewing models of artistic collaboration during the late 20th century, it suggests that collaboration is at the core of these artistic practices. We build upon the growing literature of machine learning and art to look for the human agencies inscribed in works of computational creativity, and expand the co-creation frame to incorporate emerging forms of human-human collaboration mediated through technical artifacts such as algorithms and data.
△ Less
Submitted 3 August, 2019;
originally announced August 2019.
-
DeepCloud. The Application of a Data-driven, Generative Model in Design
Authors:
Ardavan Bidgoli,
Pedro Veloso
Abstract:
Generative systems have a significant potential to synthesize innovative design alternatives. Still, most of the common systems that have been adopted in design require the designer to explicitly define the specifications of the procedures and in some cases the design space. In contrast, a generative system could potentially learn both aspects through processing a database of existing solutions wi…
▽ More
Generative systems have a significant potential to synthesize innovative design alternatives. Still, most of the common systems that have been adopted in design require the designer to explicitly define the specifications of the procedures and in some cases the design space. In contrast, a generative system could potentially learn both aspects through processing a database of existing solutions without the supervision of the designer. To explore this possibility, we review recent advancements of generative models in machine learning and current applications of learning techniques in design. Then, we describe the development of a data-driven generative system titled DeepCloud. It combines an autoencoder architecture for point clouds with a web-based interface and analog input devices to provide an intuitive experience for data-driven generation of design alternatives. We delineate the implementation of two prototypes of DeepCloud, their contributions, and potentials for generative design.
△ Less
Submitted 1 April, 2019;
originally announced April 2019.
-
A Hybrid Feature Selection Method to Improve Performance of a Group of Classification Algorithms
Authors:
Mehdi Naseriparsa,
Amir-Masoud Bidgoli,
Touraj Varaee
Abstract:
In this paper a hybrid feature selection method is proposed which takes advantages of wrapper subset evaluation with a lower cost and improves the performance of a group of classifiers. The method uses combination of sample domain filtering and resampling to refine the sample domain and two feature subset evaluation methods to select reliable features. This method utilizes both feature space and s…
▽ More
In this paper a hybrid feature selection method is proposed which takes advantages of wrapper subset evaluation with a lower cost and improves the performance of a group of classifiers. The method uses combination of sample domain filtering and resampling to refine the sample domain and two feature subset evaluation methods to select reliable features. This method utilizes both feature space and sample domain in two phases. The first phase filters and resamples the sample domain and the second phase adopts a hybrid procedure by information gain, wrapper subset evaluation and genetic search to find the optimal feature space. Experiments carried out on different types of datasets from UCI Repository of Machine Learning databases and the results show a rise in the average performance of five classifiers (Naive Bayes, Logistic, Multilayer Perceptron, Best First Decision Tree and JRIP) simultaneously and the classification error for these classifiers decreases considerably. The experiments also show that this method outperforms other feature selection methods with a lower cost.
△ Less
Submitted 8 March, 2014;
originally announced March 2014.
-
Improving Performance of a Group of Classification Algorithms Using Resampling and Feature Selection
Authors:
Mehdi Naseriparsa,
Amir-masoud Bidgoli,
Touraj Varaee
Abstract:
In recent years the importance of finding a meaningful pattern from huge datasets has become more challenging. Data miners try to adopt innovative methods to face this problem by applying feature selection methods. In this paper we propose a new hybrid method in which we use a combination of resampling, filtering the sample domain and wrapper subset evaluation method with genetic search to reduce…
▽ More
In recent years the importance of finding a meaningful pattern from huge datasets has become more challenging. Data miners try to adopt innovative methods to face this problem by applying feature selection methods. In this paper we propose a new hybrid method in which we use a combination of resampling, filtering the sample domain and wrapper subset evaluation method with genetic search to reduce dimensions of Lung-Cancer dataset that we received from UCI Repository of Machine Learning databases. Finally, we apply some well- known classification algorithms (Naïve Bayes, Logistic, Multilayer Perceptron, Best First Decision Tree and JRIP) to the resulting dataset and compare the results and prediction rates before and after the application of our feature selection method on that dataset. The results show a substantial progress in the average performance of five classification algorithms simultaneously and the classification error for these classifiers decreases considerably. The experiments also show that this method outperforms other feature selection methods with a lower cost.
△ Less
Submitted 8 March, 2014;
originally announced March 2014.
