-
Neuroevolution-Enhanced Multi-Objective Optimization for Mixed-Precision Quantization
Authors:
Santiago Miret,
Vui Seng Chua,
Mattias Marder,
Mariano Phielipp,
Nilesh Jain,
Somdeb Majumdar
Abstract:
Mixed-precision quantization is a powerful tool to enable memory and compute savings of neural network workloads by deploying different sets of bit-width precisions on separate compute operations. In this work, we present a flexible and scalable framework for automated mixed-precision quantization that concurrently optimizes task performance, memory compression, and compute savings through multi-o…
▽ More
Mixed-precision quantization is a powerful tool to enable memory and compute savings of neural network workloads by deploying different sets of bit-width precisions on separate compute operations. In this work, we present a flexible and scalable framework for automated mixed-precision quantization that concurrently optimizes task performance, memory compression, and compute savings through multi-objective evolutionary computing. Our framework centers on Neuroevolution-Enhanced Multi-Objective Optimization (NEMO), a novel search method, which combines established search methods with the representational power of neural networks. Within NEMO, the population is divided into structurally distinct sub-populations, or species, which jointly create the Pareto frontier of solutions for the multi-objective problem. At each generation, species perform separate mutation and crossover operations, and are re-sized in proportion to the goodness of their contribution to the Pareto frontier. In our experiments, we define a graph-based representation to describe the underlying workload, enabling us to deploy graph neural networks trained by NEMO via neuroevolution, to find Pareto optimal configurations for MobileNet-V2, ResNet50 and ResNeXt-101-32x8d. Compared to the state-of-the-art, we achieve competitive results on memory compression and superior results for compute compression. Further analysis reveals that the graph representation and the species-based approach employed by NEMO are critical to finding optimal solutions.
△ Less
Submitted 1 April, 2022; v1 submitted 14 June, 2021;
originally announced June 2021.
-
Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning
Authors:
Shauharda Khadka,
Estelle Aflalo,
Mattias Marder,
Avrech Ben-David,
Santiago Miret,
Shie Mannor,
Tamir Hazan,
Hanlin Tang,
Somdeb Majumdar
Abstract:
For deep neural network accelerators, memory movement is both energetically expensive and can bound computation. Therefore, optimal map** of tensors to memory hierarchies is critical to performance. The growing complexity of neural networks calls for automated memory map** instead of manual heuristic approaches; yet the search space of neural network computational graphs have previously been p…
▽ More
For deep neural network accelerators, memory movement is both energetically expensive and can bound computation. Therefore, optimal map** of tensors to memory hierarchies is critical to performance. The growing complexity of neural networks calls for automated memory map** instead of manual heuristic approaches; yet the search space of neural network computational graphs have previously been prohibitively large. We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces, that combines graph neural networks, reinforcement learning, and evolutionary search. A set of fast, stateless policies guide the evolutionary search to improve its sample-efficiency. We train and validate our approach directly on the Intel NNP-I chip for inference. EGRL outperforms policy-gradient, evolutionary search and dynamic programming baselines on BERT, ResNet-101 and ResNet-50. We additionally achieve 28-78\% speed-up compared to the native NNP-I compiler on all three workloads.
△ Less
Submitted 15 October, 2020; v1 submitted 14 July, 2020;
originally announced July 2020.
-
Delta-encoder: an effective sample synthesis method for few-shot object recognition
Authors:
Eli Schwartz,
Leonid Karlinsky,
Joseph Shtok,
Sivan Harary,
Mattias Marder,
Rogerio Feris,
Abhishek Kumar,
Raja Giryes,
Alex M. Bronstein
Abstract:
Learning to classify new categories based on just one or a few examples is a long-standing challenge in modern computer vision. In this work, we proposes a simple yet effective method for few-shot (and one-shot) object recognition. Our approach is based on a modified auto-encoder, denoted Delta-encoder, that learns to synthesize new samples for an unseen category just by seeing few examples from i…
▽ More
Learning to classify new categories based on just one or a few examples is a long-standing challenge in modern computer vision. In this work, we proposes a simple yet effective method for few-shot (and one-shot) object recognition. Our approach is based on a modified auto-encoder, denoted Delta-encoder, that learns to synthesize new samples for an unseen category just by seeing few examples from it. The synthesized samples are then used to train a classifier. The proposed approach learns to both extract transferable intra-class deformations, or "deltas", between same-class pairs of training examples, and to apply those deltas to the few provided examples of a novel class (unseen during training) in order to efficiently synthesize samples from that new class. The proposed method improves over the state-of-the-art in one-shot object-recognition and compares favorably in the few-shot case. Upon acceptance code will be made available.
△ Less
Submitted 29 November, 2018; v1 submitted 12 June, 2018;
originally announced June 2018.
-
Patterns in Illinois Educational School Data
Authors:
Cacey S. Stevens,
Michael P. Marder,
Sidney R. Nagel
Abstract:
We examine Illinois educational data from standardized exams and analyze primary factors affecting the achievement of public school students. We focus on the simplest possible models: representation of data through visualizations and regressions on single variables. Exam scores are shown to depend on school type, location, and poverty concentration. For most schools in Illinois, student test score…
▽ More
We examine Illinois educational data from standardized exams and analyze primary factors affecting the achievement of public school students. We focus on the simplest possible models: representation of data through visualizations and regressions on single variables. Exam scores are shown to depend on school type, location, and poverty concentration. For most schools in Illinois, student test scores decline linearly with poverty concentration. However Chicago must be treated separately. Selective schools in Chicago, as well as some traditional and charter schools, deviate from this pattern based on poverty. For any poverty level, Chicago schools perform better than those in the rest of Illinois. Selective programs for gifted students show high performance at each grade level, most notably at the high school level, when compared to other Illinois school types. The case of Chicago charter schools is more complex. In the last six years, their students' scores overtook those of students in traditional Chicago high schools.
△ Less
Submitted 29 January, 2015;
originally announced February 2015.
