-
A flexible smoother adapted to censored data with outliers and its application to SARS-CoV-2 monitoring in wastewater
Authors:
Marie Courbariaux,
Nicolas Cluzel,
Siyun Wang,
Vincent Maréchal,
Laurent Moulin,
Sébastien Wurtzer,
Obépine consortium,
Jean-Marie Mouchel,
Yvon Maday,
Grégory Nuel
Abstract:
A sentinel network, Obépine, has been designed to monitor SARS-CoV-2 viral load in wastewaters arriving at wastewater treatment plants (WWTPs) in France as an indirect macro-epidemiological parameter. The sources of uncertainty in such monitoring system are numerous and the concentration measurements it provides are left-censored and contain outliers, which biases the results of usual smoothing me…
▽ More
A sentinel network, Obépine, has been designed to monitor SARS-CoV-2 viral load in wastewaters arriving at wastewater treatment plants (WWTPs) in France as an indirect macro-epidemiological parameter. The sources of uncertainty in such monitoring system are numerous and the concentration measurements it provides are left-censored and contain outliers, which biases the results of usual smoothing methods. Hence the need for an adapted pre-processing in order to evaluate the real daily amount of virus arriving to each WWTP. We propose a method based on an auto-regressive model adapted to censored data with outliers. Inference and prediction are produced via a discretised smoother which makes it a very flexible tool. This method is both validated on simulations and on real data from Obépine. The resulting smoothed signal shows a good correlation with other epidemiological indicators and is currently used by Obépine to provide an estimate of virus circulation over the watersheds corresponding to about 200 WWTPs.
△ Less
Submitted 4 January, 2022; v1 submitted 4 August, 2021;
originally announced August 2021.
-
BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
Authors:
Miloš Nikolić,
Ghouthi Boukli Hacene,
Ciaran Bannon,
Alberto Delmas Lascorz,
Matthieu Courbariaux,
Yoshua Bengio,
Vincent Gripon,
Andreas Moshovos
Abstract:
Neural networks have demonstrably achieved state-of-the art accuracy using low-bitlength integer quantization, yielding both execution time and energy benefits on existing hardware designs that support short bitlengths. However, the question of finding the minimum bitlength for a desired accuracy remains open. We introduce a training method for minimizing inference bitlength at any granularity whi…
▽ More
Neural networks have demonstrably achieved state-of-the art accuracy using low-bitlength integer quantization, yielding both execution time and energy benefits on existing hardware designs that support short bitlengths. However, the question of finding the minimum bitlength for a desired accuracy remains open. We introduce a training method for minimizing inference bitlength at any granularity while maintaining accuracy. Namely, we propose a regularizer that penalizes large bitlength representations throughout the architecture and show how it can be modified to minimize other quantifiable criteria, such as number of operations or memory footprint. We demonstrate that our method learns thrifty representations while maintaining accuracy. With ImageNet, the method produces an average per layer bitlength of 4.13, 3.76 and 4.36 bits on AlexNet, ResNet18 and MobileNet V2 respectively, remaining within 2.0%, 0.5% and 0.5% of the base TOP-1 accuracy.
△ Less
Submitted 11 August, 2020; v1 submitted 7 February, 2020;
originally announced February 2020.
-
Attention Based Pruning for Shift Networks
Authors:
Ghouthi Boukli Hacene,
Carlos Lassance,
Vincent Gripon,
Matthieu Courbariaux,
Yoshua Bengio
Abstract:
In many application domains such as computer vision, Convolutional Layers (CLs) are key to the accuracy of deep learning methods. However, it is often required to assemble a large number of CLs, each containing thousands of parameters, in order to reach state-of-the-art accuracy, thus resulting in complex and demanding systems that are poorly fitted to resource-limited devices. Recently, methods h…
▽ More
In many application domains such as computer vision, Convolutional Layers (CLs) are key to the accuracy of deep learning methods. However, it is often required to assemble a large number of CLs, each containing thousands of parameters, in order to reach state-of-the-art accuracy, thus resulting in complex and demanding systems that are poorly fitted to resource-limited devices. Recently, methods have been proposed to replace the generic convolution operator by the combination of a shift operation and a simpler 1x1 convolution. The resulting block, called Shift Layer (SL), is an efficient alternative to CLs in the sense it allows to reach similar accuracies on various tasks with faster computations and fewer parameters. In this contribution, we introduce Shift Attention Layers (SALs), which extend SLs by using an attention mechanism that learns which shifts are the best at the same time the network function is trained. We demonstrate SALs are able to outperform vanilla SLs (and CLs) on various object recognition benchmarks while significantly reducing the number of float operations and parameters for the inference.
△ Less
Submitted 29 May, 2019;
originally announced May 2019.
-
Regularized Binary Network Training
Authors:
Sajad Darabi,
Mouloud Belbahri,
Matthieu Courbariaux,
Vahid Partovi Nia
Abstract:
There is a significant performance gap between Binary Neural Networks (BNNs) and floating point Deep Neural Networks (DNNs). We propose to improve the binary training method, by introducing a new regularization function that encourages training weights around binary values. In addition, we add trainable scaling factors to our regularization functions. Additionally, an improved approximation of the…
▽ More
There is a significant performance gap between Binary Neural Networks (BNNs) and floating point Deep Neural Networks (DNNs). We propose to improve the binary training method, by introducing a new regularization function that encourages training weights around binary values. In addition, we add trainable scaling factors to our regularization functions. Additionally, an improved approximation of the derivative of the sign activation function in the backward computation. These modifications are based on linear operations that are easily implementable into the binary training framework. Experimental results on ImageNet shows our method outperforms the traditional BNN method and XNOR-net.
△ Less
Submitted 21 April, 2020; v1 submitted 31 December, 2018;
originally announced December 2018.
-
Post-processing multi-ensemble temperature and precipitation forecasts through an Exchangeable Gamma Normal model and its Tobit extension
Authors:
Marie Courbariaux,
Pierre Barbillon,
Luc Perreault,
Éric Parent
Abstract:
Meteorological ensembles are a collection of scenarios for future weather delivered by a meteorological center. Such ensembles form the main source of valuable information for probabilistic forecasting which aims at producing a predictive probability distribution of the quantity of interest instead of a single best guess estimate. Unfortunately, ensembles cannot generally be considered as a sample…
▽ More
Meteorological ensembles are a collection of scenarios for future weather delivered by a meteorological center. Such ensembles form the main source of valuable information for probabilistic forecasting which aims at producing a predictive probability distribution of the quantity of interest instead of a single best guess estimate. Unfortunately, ensembles cannot generally be considered as a sample from such a predictive probability distribution without a preliminary post-processing treatment to calibrate the ensemble. Two main families of post-processing methods, either competing such as BMA or collaborative such as EMOS, can be found in the literature. This paper proposes a mixed effect model belonging to the collaborative family. The structure of the model is based on the hypothesis of invariance under the relabelling of the ensemble members. Its interesting specificities are as follows: 1) exchangeability, which contributes to parsimony, with a latent pivot variable synthesizing the essential meteorological features of the ensembles, 2) a multi-ensemble implementation, allowing to take advantage of various information so as to increase the sharpness of the forecasting procedure. Focus is cast onto Normal statistical structures, first with a direct application for temperatures, then with its Tobit extension for precipitation. Inference is performed by EM algorithms with recourse made to stochastic conditional simulations in the precipitation case. After checking its good behavior on artificial data, the proposed post-processing technique is applied to temperature and precipitation ensemble forecasts produced over five river basins managed by Hydro-Qu$é$bec. These ensemble forecasts were extracted from the THORPEX Interactive Grand Global Ensemble (TIGGE) database. The results indicate that post-processed ensemble are calibrated and generally sharper than the raw ensembles.
△ Less
Submitted 5 March, 2019; v1 submitted 24 April, 2018;
originally announced April 2018.
-
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
Authors:
Itay Hubara,
Matthieu Courbariaux,
Daniel Soudry,
Ran El-Yaniv,
Yoshua Bengio
Abstract:
We introduce a method to train Quantized Neural Networks (QNNs) --- neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At train-time the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operati…
▽ More
We introduce a method to train Quantized Neural Networks (QNNs) --- neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At train-time the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. As a result, power consumption is expected to be drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to their 32-bit counterparts. For example, our quantized version of AlexNet with 1-bit weights and 2-bit activations achieves $51\%$ top-1 accuracy. Moreover, we quantize the parameter gradients to 6-bits as well which enables gradients computation using only bit-wise operation. Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits. Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The QNN code is available online.
△ Less
Submitted 22 September, 2016;
originally announced September 2016.
-
Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
Authors:
Matthieu Courbariaux,
Itay Hubara,
Daniel Soudry,
Ran El-Yaniv,
Yoshua Bengio
Abstract:
We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At training-time the binary weights and activations are used for computing the parameters gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substan…
▽ More
We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At training-time the binary weights and activations are used for computing the parameters gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs we conduct two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.
△ Less
Submitted 17 March, 2016; v1 submitted 8 February, 2016;
originally announced February 2016.
-
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Authors:
Matthieu Courbariaux,
Yoshua Bengio,
Jean-Pierre David
Abstract:
Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on…
▽ More
Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and power-hungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.
△ Less
Submitted 18 April, 2016; v1 submitted 1 November, 2015;
originally announced November 2015.
-
Neural Networks with Few Multiplications
Authors:
Zhouhan Lin,
Matthieu Courbariaux,
Roland Memisevic,
Yoshua Bengio
Abstract:
For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden…
▽ More
For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden states to sign changes. Second, while back-propagating error derivatives, in addition to binarizing the weights, we quantize the representations at each layer to convert the remaining multiplications into binary shifts. Experimental results across 3 popular datasets (MNIST, CIFAR10, SVHN) show that this approach not only does not hurt classification performance but can result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardware-friendly training of neural networks.
△ Less
Submitted 26 February, 2016; v1 submitted 11 October, 2015;
originally announced October 2015.
-
Training deep neural networks with low precision multiplications
Authors:
Matthieu Courbariaux,
Yoshua Bengio,
Jean-Pierre David
Abstract:
Multipliers are the most space and power-hungry arithmetic operators of the digital implementation of deep neural networks. We train a set of state-of-the-art neural networks (Maxout networks) on three benchmark datasets: MNIST, CIFAR-10 and SVHN. They are trained with three distinct formats: floating point, fixed point and dynamic fixed point. For each of those datasets and for each of those form…
▽ More
Multipliers are the most space and power-hungry arithmetic operators of the digital implementation of deep neural networks. We train a set of state-of-the-art neural networks (Maxout networks) on three benchmark datasets: MNIST, CIFAR-10 and SVHN. They are trained with three distinct formats: floating point, fixed point and dynamic fixed point. For each of those datasets and for each of those formats, we assess the impact of the precision of the multiplications on the final error after training. We find that very low precision is sufficient not just for running trained networks but also for training them. For example, it is possible to train Maxout networks with 10 bits multiplications.
△ Less
Submitted 22 September, 2015; v1 submitted 22 December, 2014;
originally announced December 2014.