-
PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks
Authors:
Marina Neseem,
Conor McCullough,
Randy Hsin,
Chas Leichner,
Shan Li,
In Suk Chong,
Andrew G. Howard,
Lukasz Lew,
Sherief Reda,
Ville-Mikko Rautio,
Daniele Moro
Abstract:
Low-precision quantization is recognized for its efficacy in neural network optimization. Our analysis reveals that non-quantized elementwise operations which are prevalent in layers such as parameterized activation functions, batch normalization, and quantization scaling dominate the inference cost of low-precision models. These non-quantized elementwise operations are commonly overlooked in SOTA…
▽ More
Low-precision quantization is recognized for its efficacy in neural network optimization. Our analysis reveals that non-quantized elementwise operations which are prevalent in layers such as parameterized activation functions, batch normalization, and quantization scaling dominate the inference cost of low-precision models. These non-quantized elementwise operations are commonly overlooked in SOTA efficiency metrics such as Arithmetic Computation Effort (ACE). In this paper, we propose ACEv2 - an extended version of ACE which offers a better alignment with the inference cost of quantized models and their energy consumption on ML hardware. Moreover, we introduce PikeLPN, a model that addresses these efficiency issues by applying quantization to both elementwise operations and multiply-accumulate operations. In particular, we present a novel quantization technique for batch normalization layers named QuantNorm which allows for quantizing the batch normalization parameters without compromising the model performance. Additionally, we propose applying Double Quantization where the quantization scaling parameters are quantized. Furthermore, we recognize and resolve the issue of distribution mismatch in Separable Convolution layers by introducing Distribution-Heterogeneous Quantization which enables quantizing them to low-precision. PikeLPN achieves Pareto-optimality in efficiency-accuracy trade-off with up to 3X efficiency improvement compared to SOTA low-precision models.
△ Less
Submitted 29 March, 2024;
originally announced April 2024.
-
Towards an Open and Scalable Music Metadata Layer
Authors:
Thomas Hardjono,
George Howard,
Eric Scace,
Mizan Chowdury,
Lucas Novak,
Meghan Gaudet,
Justin Anderson,
Nicole d'avis,
Christopher Kulis,
Edward Sweeney,
Chandler Vaughan
Abstract:
One of the significant issues in the music supply chain today is the lack of consistent, complete and authoritative information or metadata regarding the creation of a given musical work. In many cases multiple entities in the music supply chain have each created their own version of the metadata for a musical work, often by manually re-entering the same information or through scra** data from o…
▽ More
One of the significant issues in the music supply chain today is the lack of consistent, complete and authoritative information or metadata regarding the creation of a given musical work. In many cases multiple entities in the music supply chain have each created their own version of the metadata for a musical work, often by manually re-entering the same information or through scra** data from other sites. In such cases, the effort to synchronize or to correct the information becomes manually laborious and error-prone. Furthermore, confidential information regarding the legal ownership of the musical work is often commingled in the same metadata, making the entire database proprietary and thus closed. In this paper we explore an alternative model for creation metadata following the open access paradigm found in other industries, such as in book publishing, library systems and in the automotive parts supply chain. The vision is to create a new music metadata layer for creation metadata that is open, scalable and provides an authoritative source of information that is available to all entities in the music supply chain globally.
△ Less
Submitted 7 November, 2019;
originally announced November 2019.
-
A Staged Approach to Evolving Real-world UAV Controllers
Authors:
Gerard David Howard,
Alberto Elfes
Abstract:
A testbed has recently been introduced that evolves controllers for arbitrary hover-capable UAVs, with evaluations occurring directly on the robot. To prepare the testbed for real-world deployment, we investigate the effects of state-space limitations brought about by physical tethering (which prevents damage to the UAV during stochastic tuning), on the generality of the evolved controllers. We id…
▽ More
A testbed has recently been introduced that evolves controllers for arbitrary hover-capable UAVs, with evaluations occurring directly on the robot. To prepare the testbed for real-world deployment, we investigate the effects of state-space limitations brought about by physical tethering (which prevents damage to the UAV during stochastic tuning), on the generality of the evolved controllers. We identify generalisation issues in some controllers, and propose an improved method that comprises two stages: in the first stage, controllers are evolved as normal using standard tethers, but experiments are terminated when the population displays basic flight competency. Optimisation then continues on a much less restrictive tether, effectively free-flying, and is allowed to explore a larger state-space envelope. We compare the two methods on a hover task using a real UAV, and show that more general solutions are generated in fewer generations using the two-stage approach. A secondary experiment undertakes a sensitivity analysis of the evolved controllers.
△ Less
Submitted 26 May, 2019;
originally announced May 2019.
-
Deep learning cardiac motion analysis for human survival prediction
Authors:
Ghalib A. Bello,
Timothy J. W. Dawes,
**ming Duan,
Carlo Biffi,
Antonio de Marvao,
Luke S. G. E. Howard,
J. Simon R. Gibbs,
Martin R. Wilkins,
Stuart A. Cook,
Daniel Rueckert,
Declan P. O'Regan
Abstract:
Motion analysis is used in computer vision to understand the behaviour of moving objects in sequences of images. Optimising the interpretation of dynamic biological systems requires accurate and precise motion tracking as well as efficient representations of high-dimensional motion trajectories so that these can be used for prediction tasks. Here we use image sequences of the heart, acquired using…
▽ More
Motion analysis is used in computer vision to understand the behaviour of moving objects in sequences of images. Optimising the interpretation of dynamic biological systems requires accurate and precise motion tracking as well as efficient representations of high-dimensional motion trajectories so that these can be used for prediction tasks. Here we use image sequences of the heart, acquired using cardiac magnetic resonance imaging, to create time-resolved three-dimensional segmentations using a fully convolutional network trained on anatomical shape priors. This dense motion model formed the input to a supervised denoising autoencoder (4Dsurvival), which is a hybrid network consisting of an autoencoder that learns a task-specific latent code representation trained on observed outcome data, yielding a latent representation optimised for survival prediction. To handle right-censored survival outcomes, our network used a Cox partial likelihood loss function. In a study of 302 patients the predictive accuracy (quantified by Harrell's C-index) was significantly higher (p < .0001) for our model C=0.73 (95$\%$ CI: 0.68 - 0.78) than the human benchmark of C=0.59 (95$\%$ CI: 0.53 - 0.65). This work demonstrates how a complex computer vision task using high-dimensional medical image data can efficiently predict human survival.
△ Less
Submitted 8 October, 2018;
originally announced October 2018.
-
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Authors:
Andrew G. Howard,
Menglong Zhu,
Bo Chen,
Dmitry Kalenichenko,
Weijun Wang,
Tobias Weyand,
Marco Andreetto,
Hartwig Adam
Abstract:
We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choo…
▽ More
We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.
△ Less
Submitted 16 April, 2017;
originally announced April 2017.
-
On Self-Adaptive Mutation Restarts for Evolutionary Robotics with Real Rotorcraft
Authors:
Gerard David Howard
Abstract:
Self-adaptive parameters are increasingly used in the field of Evolutionary Robotics, as they allow key evolutionary rates to vary autonomously in a context-sensitive manner throughout the optimisation process. A significant limitation to self-adaptive mutation is that rates can be set unfavourably, which hinders convergence. Rate restarts are typically employed to remedy this, but thus far have o…
▽ More
Self-adaptive parameters are increasingly used in the field of Evolutionary Robotics, as they allow key evolutionary rates to vary autonomously in a context-sensitive manner throughout the optimisation process. A significant limitation to self-adaptive mutation is that rates can be set unfavourably, which hinders convergence. Rate restarts are typically employed to remedy this, but thus far have only been applied in Evolutionary Robotics for mutation-only algorithms. This paper focuses on the level at which evolutionary rate restarts are applied in population-based algorithms with more than 1 evolutionary operator. After testing on a real hexacopter hovering task, we conclude that individual-level restarting results in higher fitness solutions without fitness stagnation, and population restarts provide a more stable rate evolution. Without restarts, experiments can become stuck in suboptimal controller/rate combinations which can be difficult to escape from.
△ Less
Submitted 2 April, 2017; v1 submitted 31 March, 2017;
originally announced March 2017.
-
Evolving Spiking Networks with Variable Resistive Memories
Authors:
Gerard David Howard,
Larry Bull,
Ben de Lacy Costello,
Andrew Adamatzky,
Ella Gale
Abstract:
Neuromorphic computing is a brainlike information processing paradigm that requires adaptive learning mechanisms. A spiking neuro-evolutionary system is used for this purpose; plastic resistive memories are implemented as synapses in spiking neural networks. The evolutionary design process exploits parameter self-adaptation and allows the topology and synaptic weights to be evolved for each networ…
▽ More
Neuromorphic computing is a brainlike information processing paradigm that requires adaptive learning mechanisms. A spiking neuro-evolutionary system is used for this purpose; plastic resistive memories are implemented as synapses in spiking neural networks. The evolutionary design process exploits parameter self-adaptation and allows the topology and synaptic weights to be evolved for each network in an autonomous manner. Variable resistive memories are the focus of this research; each synapse has its own conductance profile which modifies the plastic behaviour of the device and may be altered during evolution. These variable resistive networks are evaluated on a noisy robotic dynamic-reward scenario against two static resistive memories and a system containing standard connections only. Results indicate that the extra behavioural degrees of freedom available to the networks incorporating variable resistive memories enable them to outperform the comparative synapse types.
△ Less
Submitted 17 May, 2015;
originally announced May 2015.
-
Some Improvements on Deep Convolutional Neural Network Based Image Classification
Authors:
Andrew G. Howard
Abstract:
We investigate multiple techniques to improve upon the current state of the art deep convolutional neural network based image classification pipeline. The techiques include adding more image transformations to training data, adding more transformations to generate additional predictions at test time and using complementary models applied to higher resolution images. This paper summarizes our entry…
▽ More
We investigate multiple techniques to improve upon the current state of the art deep convolutional neural network based image classification pipeline. The techiques include adding more image transformations to training data, adding more transformations to generate additional predictions at test time and using complementary models applied to higher resolution images. This paper summarizes our entry in the Imagenet Large Scale Visual Recognition Challenge 2013. Our system achieved a top 5 classification error rate of 13.55% using no external data which is over a 20% relative improvement on the previous year's winner.
△ Less
Submitted 18 December, 2013;
originally announced December 2013.
-
Evolution of Plastic Learning in Spiking Networks via Memristive Connections
Authors:
Gerard Howard,
Ella Gale,
Larry Bull,
Ben de Lacy Costello,
Andy Adamatzky
Abstract:
This article presents a spiking neuroevolutionary system which implements memristors as plastic connections, i.e. whose weights can vary during a trial. The evolutionary design process exploits parameter self-adaptation and variable topologies, allowing the number of neurons, connection weights, and inter-neural connectivity pattern to emerge. By comparing two phenomenological real-world memristor…
▽ More
This article presents a spiking neuroevolutionary system which implements memristors as plastic connections, i.e. whose weights can vary during a trial. The evolutionary design process exploits parameter self-adaptation and variable topologies, allowing the number of neurons, connection weights, and inter-neural connectivity pattern to emerge. By comparing two phenomenological real-world memristor implementations with networks comprised of (i) linear resistors (ii) constant-valued connections, we demonstrate that this approach allows the evolution of networks of appropriate complexity to emerge whilst exploiting the memristive properties of the connections to reduce learning time. We extend this approach to allow for heterogeneous mixtures of memristors within the networks; our approach provides an in-depth analysis of network structure. Our networks are evaluated on simulated robotic navigation tasks; results demonstrate that memristive plasticity enables higher performance than constant-weighted connections in both static and dynamic reward scenarios, and that mixtures of memristive elements provide performance advantages when compared to homogeneous memristive networks.
△ Less
Submitted 14 December, 2012;
originally announced December 2012.
-
Organic Memristor Devices for Logic Elements with Memory
Authors:
Victor Erokhin,
Gerard David Howard,
Andrew Adamatzky
Abstract:
Memristors are promising next-generation memory candidates that are nonvolatile, possess low power requirements and are capable of nanoscale fabrication. In this article we physically realise and describe the use of organic memristors in designing statefull boolean logic gates for the AND OR and NOT operations. The output of these gates is analog and dependent on the length of time that suitable c…
▽ More
Memristors are promising next-generation memory candidates that are nonvolatile, possess low power requirements and are capable of nanoscale fabrication. In this article we physically realise and describe the use of organic memristors in designing statefull boolean logic gates for the AND OR and NOT operations. The output of these gates is analog and dependent on the length of time that suitable charge is applied to the inputs, displaying a learning property. Results may be also interpreted in a traditional binary manner through use of a suitable thresholding function at the output. The memristive property of the gate allows the for the production of analog outputs that vary based on the charge-dependent nonvolatile state of the memristor. We provide experimental results of physical fabrication of three types of logic gate. A simulation of a one-bit full adder comprised of memristive logic gates is also included, displaying varying response to two distinct input patterns.
△ Less
Submitted 14 December, 2012;
originally announced December 2012.
-
A Spiking Neural Learning Classifier System
Authors:
Gerard Howard,
Larry Bull,
Pier-Luca Lanzi
Abstract:
Learning Classifier Systems (LCS) are population-based reinforcement learners used in a wide variety of applications. This paper presents a LCS where each traditional rule is represented by a spiking neural network, a type of network with dynamic internal state. We employ a constructivist model of growth of both neurons and dendrites that realise flexible learning by evolving structures of suffici…
▽ More
Learning Classifier Systems (LCS) are population-based reinforcement learners used in a wide variety of applications. This paper presents a LCS where each traditional rule is represented by a spiking neural network, a type of network with dynamic internal state. We employ a constructivist model of growth of both neurons and dendrites that realise flexible learning by evolving structures of sufficient complexity to solve a well-known problem involving continuous, real-valued inputs. Additionally, we extend the system to enable temporal state decomposition. By allowing our LCS to chain together sequences of heterogeneous actions into macro-actions, it is shown to perform optimally in a problem where traditional methods can fail to find a solution in a reasonable amount of time. Our final system is tested on a simulated robotics platform.
△ Less
Submitted 16 January, 2012;
originally announced January 2012.
