Showing 1–1 of 1 results for author: Glandorf, P
-
HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization
Authors:
Patrick Glandorf,
Timo Kaiser,
Bodo Rosenhahn
Abstract:
Sparse neural networks are a key factor in develo** resource-efficient machine learning applications. We propose the novel and powerful sparse learning method Adaptive Regularized Training (ART) to compress dense into sparse networks. Instead of the commonly used binary mask during training to reduce the number of model weights, we inherently shrink weights close to zero in an iterative manner w…
▽ More
Sparse neural networks are a key factor in develo** resource-efficient machine learning applications. We propose the novel and powerful sparse learning method Adaptive Regularized Training (ART) to compress dense into sparse networks. Instead of the commonly used binary mask during training to reduce the number of model weights, we inherently shrink weights close to zero in an iterative manner with increasing weight regularization. Our method compresses the pre-trained model knowledge into the weights of highest magnitude. Therefore, we introduce a novel regularization loss named HyperSparse that exploits the highest weights while conserving the ability of weight exploration. Extensive experiments on CIFAR and TinyImageNet show that our method leads to notable performance gains compared to other sparsification methods, especially in extremely high sparsity regimes up to 99.8 percent model sparsity. Additional investigations provide new insights into the patterns that are encoded in weights with high magnitudes.
△ Less
Submitted 16 August, 2023; v1 submitted 14 August, 2023;
originally announced August 2023.
