Showing 1–2 of 2 results for author: Gideoni, Y

Does Double Descent Occur in Self-Supervised Learning?

Authors: Alisia Lupidi, Yonatan Gideoni, Dulhan Jayalath

Abstract: Most investigations into double descent have focused on supervised models while the few works studying self-supervised settings find a surprising lack of the phenomenon. These results imply that double descent may not exist in self-supervised models. We show this empirically using a standard and linear autoencoder, two previously unstudied settings. The test loss is found to have either a classica… ▽ More Most investigations into double descent have focused on supervised models while the few works studying self-supervised settings find a surprising lack of the phenomenon. These results imply that double descent may not exist in self-supervised models. We show this empirically using a standard and linear autoencoder, two previously unstudied settings. The test loss is found to have either a classical U-shape or to monotonically decrease instead of exhibiting a double-descent curve. We hope that further work on this will help elucidate the theoretical underpinnings of this phenomenon. △ Less

Submitted 15 July, 2023; originally announced July 2023.

Comments: 7 pages, 2 tables, 3 figures. Accepted for the workshop on High-Dimensional Learning Dynamics at ICML 2023

Group Invariant Global Pooling

Authors: Kamil Bujel, Yonatan Gideoni, Chaitanya K. Joshi, Pietro Liò

Abstract: Much work has been devoted to devising architectures that build group-equivariant representations, while invariance is often induced using simple global pooling mechanisms. Little work has been done on creating expressive layers that are invariant to given symmetries, despite the success of permutation invariant pooling in various molecular tasks. In this work, we present Group Invariant Global Po… ▽ More Much work has been devoted to devising architectures that build group-equivariant representations, while invariance is often induced using simple global pooling mechanisms. Little work has been done on creating expressive layers that are invariant to given symmetries, despite the success of permutation invariant pooling in various molecular tasks. In this work, we present Group Invariant Global Pooling (GIGP), an invariant pooling layer that is provably sufficiently expressive to represent a large class of invariant functions. We validate GIGP on rotated MNIST and QM9, showing improvements for the latter while attaining identical results for the former. By making the pooling process group orbit-aware, this invariant aggregation method leads to improved performance, while performing well-principled group aggregation. △ Less

Submitted 30 May, 2023; originally announced May 2023.