-
On conceptualisation and an overview of learning path recommender systems in e-learning
Authors:
A. Fuster-López,
J. M. Cruz,
P. Guerrero-García,
E. M. T. Hendrix,
A. Košir,
I. Nowak,
L. Oneto,
S. Sirmakessis,
M. F. Pacheco,
F. P. Fernandes,
A. I. Pereira
Abstract:
The use of e-learning systems has a long tradition, where students can study online helped by a system. In this context, the use of recommender systems is relatively new. In our research project, we investigated various ways to create a recommender system. They all aim at facilitating the learning and understanding of a student. We present a common concept of the learning path and its learning ind…
▽ More
The use of e-learning systems has a long tradition, where students can study online helped by a system. In this context, the use of recommender systems is relatively new. In our research project, we investigated various ways to create a recommender system. They all aim at facilitating the learning and understanding of a student. We present a common concept of the learning path and its learning indicators and embed 5 different recommenders in this context.
△ Less
Submitted 7 June, 2024;
originally announced June 2024.
-
Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization
Authors:
Aleksandra Irena Nowak,
Łukasz Gniecki,
Filip Szatkowski,
Jacek Tabor
Abstract:
Static sparse training aims to train sparse models from scratch, achieving remarkable results in recent years. A key design choice is given by the sparse initialization, which determines the trainable sub-network through a binary mask. Existing methods mainly select such mask based on a predefined dense initialization. Such an approach may not efficiently leverage the mask's potential impact on th…
▽ More
Static sparse training aims to train sparse models from scratch, achieving remarkable results in recent years. A key design choice is given by the sparse initialization, which determines the trainable sub-network through a binary mask. Existing methods mainly select such mask based on a predefined dense initialization. Such an approach may not efficiently leverage the mask's potential impact on the optimization. An alternative direction, inspired by research into dynamical isometry, is to introduce orthogonality in the sparse subnetwork, which helps in stabilizing the gradient signal. In this work, we propose Exact Orthogonal Initialization (EOI), a novel sparse orthogonal initialization scheme based on composing random Givens rotations. Contrary to other existing approaches, our method provides exact (not approximated) orthogonality and enables the creation of layers with arbitrary densities. We demonstrate the superior effectiveness and efficiency of EOI through experiments, consistently outperforming common sparse initialization techniques. Our method enables training highly sparse 1000-layer MLP and CNN networks without residual connections or normalization techniques, emphasizing the crucial role of weight initialization in static sparse training alongside sparse mask selection. The code is available at https://github.com/woocash2/sparser-better-deeper-stronger
△ Less
Submitted 3 June, 2024;
originally announced June 2024.
-
Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training
Authors:
Aleksandra I. Nowak,
Bram Grooten,
Decebal Constantin Mocanu,
Jacek Tabor
Abstract:
Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training. It has been shown that under specific conditions, DST is able to outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training proces…
▽ More
Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training. It has been shown that under specific conditions, DST is able to outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process to adjust the network's sparse connectivity. While the growing criterion's impact on DST performance is relatively well studied, the influence of the pruning criterion remains overlooked. To address this issue, we design and perform an extensive empirical analysis of various pruning criteria to better understand their impact on the dynamics of DST solutions. Surprisingly, we find that most of the studied methods yield similar results. The differences become more significant in the low-density regime, where the best performance is predominantly given by the simplest technique: magnitude-based pruning. The code is provided at https://github.com/alooow/fantastic_weights_paper
△ Less
Submitted 29 November, 2023; v1 submitted 21 June, 2023;
originally announced June 2023.
