-
Autoencoders
Authors:
Dor Bank,
Noam Koenigstein,
Raja Giryes
Abstract:
An autoencoder is a specific type of a neural network, which is mainly designed to encode the input into a compressed and meaningful representation, and then decode it back such that the reconstructed input is similar as possible to the original one. This chapter surveys the different types of autoencoders that are mainly used today. It also describes various applications and use-cases of autoenco…
▽ More
An autoencoder is a specific type of a neural network, which is mainly designed to encode the input into a compressed and meaningful representation, and then decode it back such that the reconstructed input is similar as possible to the original one. This chapter surveys the different types of autoencoders that are mainly used today. It also describes various applications and use-cases of autoencoders.
△ Less
Submitted 3 April, 2021; v1 submitted 12 March, 2020;
originally announced March 2020.
-
An ETF view of Dropout regularization
Authors:
Dor Bank,
Raja Giryes
Abstract:
Dropout is a popular regularization technique in deep learning. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to recent developments in analog channel coding, we suggest that for a certain family of autoencoders with a linear encoder, optimizing the encoder with dropout reg…
▽ More
Dropout is a popular regularization technique in deep learning. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to recent developments in analog channel coding, we suggest that for a certain family of autoencoders with a linear encoder, optimizing the encoder with dropout regularization leads to an equiangular tight frame (ETF). Since this optimization is non-convex, we add another regularization that promotes such structures by minimizing the cross-correlation between filters in the network. We demonstrate its applicability in convolutional and fully connected layers in both feed-forward and recurrent networks. All these results suggest that there is indeed a relationship between dropout and ETF structure of the regularized linear operations.
△ Less
Submitted 19 August, 2020; v1 submitted 14 October, 2018;
originally announced October 2018.
-
Reaching Distributed Equilibrium with Limited ID Space
Authors:
Dor Bank,
Moshe Sulamy,
Eyal Waserman
Abstract:
We examine the relation between the size of the id space and the number of rational agents in a network under which equilibrium in distributed algorithms is possible. When the number of agents in the network is not a-priori known, a single agent may duplicate to gain an advantage, pretending to be more than one agent. However, when the id space is limited, each duplication involves a risk of being…
▽ More
We examine the relation between the size of the id space and the number of rational agents in a network under which equilibrium in distributed algorithms is possible. When the number of agents in the network is not a-priori known, a single agent may duplicate to gain an advantage, pretending to be more than one agent. However, when the id space is limited, each duplication involves a risk of being caught. By comparing the risk against the advantage, given an id space of size $L$, we provide a method of calculating the minimal threshold $t$, the required number of agents in the network, such that the algorithm is in equilibrium. That is, it is the minimal value of $t$ such that if agents a-priori know that $n \geq t$ then the algorithm is in equilibrium. We demonstrate this method by applying it to two problems, Leader Election and Knowledge Sharing, as well as providing a constant-time approximation $t \approx \frac{L}{5}$ of the minimal threshold for Leader Election.
△ Less
Submitted 18 April, 2018; v1 submitted 17 April, 2018;
originally announced April 2018.
-
Improved Training for Self-Training by Confidence Assessments
Authors:
Gal Hyams,
Daniel Greenfeld,
Dor Bank
Abstract:
It is well known that for some tasks, labeled data sets may be hard to gather. Therefore, we wished to tackle here the problem of having insufficient training data. We examined learning methods from unlabeled data after an initial training on a limited labeled data set. The suggested approach can be used as an online learning method on the unlabeled test set. In the general classification task, wh…
▽ More
It is well known that for some tasks, labeled data sets may be hard to gather. Therefore, we wished to tackle here the problem of having insufficient training data. We examined learning methods from unlabeled data after an initial training on a limited labeled data set. The suggested approach can be used as an online learning method on the unlabeled test set. In the general classification task, whenever we predict a label with high enough confidence, we treat it as a true label and train the data accordingly. For the semantic segmentation task, a classic example for an expensive data labeling process, we do so pixel-wise. Our suggested approaches were applied on the MNIST data-set as a proof of concept for a vision classification task and on the ADE20K data-set in order to tackle the semi-supervised semantic segmentation problem.
△ Less
Submitted 5 April, 2018; v1 submitted 30 September, 2017;
originally announced October 2017.
