-
Attending to Topological Spaces: The Cellular Transformer
Authors:
Rubén Ballester,
Pablo Hernández-García,
Mathilde Papillon,
Claudio Battiloro,
Nina Miolane,
Tolga Birdal,
Carles Casacuberta,
Sergio Escalera,
Mustafa Hajij
Abstract:
Topological Deep Learning seeks to enhance the predictive performance of neural network models by harnessing topological structures in input data. Topological neural networks operate on spaces such as cell complexes and hypergraphs, that can be seen as generalizations of graphs. In this work, we introduce the Cellular Transformer (CT), a novel architecture that generalizes graph-based transformers…
▽ More
Topological Deep Learning seeks to enhance the predictive performance of neural network models by harnessing topological structures in input data. Topological neural networks operate on spaces such as cell complexes and hypergraphs, that can be seen as generalizations of graphs. In this work, we introduce the Cellular Transformer (CT), a novel architecture that generalizes graph-based transformers to cell complexes. First, we propose a new formulation of the usual self- and cross-attention mechanisms, tailored to leverage incidence relations in cell complexes, e.g., edge-face and node-edge relations. Additionally, we propose a set of topological positional encodings specifically designed for cell complexes. By transforming three graph datasets into cell complex datasets, our experiments reveal that CT not only achieves state-of-the-art performance, but it does so without the need for more complex enhancements such as virtual nodes, in-domain structural encodings, or graph rewiring.
△ Less
Submitted 26 May, 2024; v1 submitted 22 May, 2024;
originally announced May 2024.
-
TopoX: A Suite of Python Packages for Machine Learning on Topological Domains
Authors:
Mustafa Hajij,
Mathilde Papillon,
Florian Frantzen,
Jens Agerberg,
Ibrahem AlJabea,
Ruben Ballester,
Claudio Battiloro,
Guillermo Bernárdez,
Tolga Birdal,
Aiden Brent,
Peter Chin,
Sergio Escalera,
Simone Fiorellino,
Odin Hoff Gardaa,
Gurusankar Gopalakrishnan,
Devendra Govil,
Josef Hoppe,
Maneel Reddy Karri,
Jude Khouja,
Manuel Lecha,
Neal Livesay,
Jan Meißner,
Soham Mukherjee,
Alexander Nikitin,
Theodore Papamarkou
, et al. (18 additional authors not shown)
Abstract:
We introduce TopoX, a Python software suite that provides reliable and user-friendly building blocks for computing and machine learning on topological domains that extend graphs: hypergraphs, simplicial, cellular, path and combinatorial complexes. TopoX consists of three packages: TopoNetX facilitates constructing and computing on these domains, including working with nodes, edges and higher-order…
▽ More
We introduce TopoX, a Python software suite that provides reliable and user-friendly building blocks for computing and machine learning on topological domains that extend graphs: hypergraphs, simplicial, cellular, path and combinatorial complexes. TopoX consists of three packages: TopoNetX facilitates constructing and computing on these domains, including working with nodes, edges and higher-order cells; TopoEmbedX provides methods to embed topological domains into vector spaces, akin to popular graph-based embedding algorithms such as node2vec; TopoModelx is built on top of PyTorch and offers a comprehensive toolbox of higher-order message passing functions for neural networks on topological domains. The extensively documented and unit-tested source code of TopoX is available under MIT license at https://pyt-team.github.io/.
△ Less
Submitted 17 February, 2024; v1 submitted 4 February, 2024;
originally announced February 2024.
-
Topological Data Analysis for Neural Network Analysis: A Comprehensive Survey
Authors:
Rubén Ballester,
Carles Casacuberta,
Sergio Escalera
Abstract:
This survey provides a comprehensive exploration of applications of Topological Data Analysis (TDA) within neural network analysis. Using TDA tools such as persistent homology and Mapper, we delve into the intricate structures and behaviors of neural networks and their datasets. We discuss different strategies to obtain topological information from data and neural networks by means of TDA. Additio…
▽ More
This survey provides a comprehensive exploration of applications of Topological Data Analysis (TDA) within neural network analysis. Using TDA tools such as persistent homology and Mapper, we delve into the intricate structures and behaviors of neural networks and their datasets. We discuss different strategies to obtain topological information from data and neural networks by means of TDA. Additionally, we review how topological information can be leveraged to analyze properties of neural networks, such as their generalization capacity or expressivity. We explore practical implications of deep learning, specifically focusing on areas like adversarial detection and model selection. Our survey organizes the examined works into four broad domains: 1. Characterization of neural network architectures; 2. Analysis of decision regions and boundaries; 3. Study of internal representations, activations, and parameters; 4. Exploration of training dynamics and loss functions. Within each category, we discuss several articles, offering background information to aid in understanding the various methodologies. We conclude with a synthesis of key insights gained from our study, accompanied by a discussion of challenges and potential advancements in the field.
△ Less
Submitted 3 January, 2024; v1 submitted 10 December, 2023;
originally announced December 2023.
-
ICML 2023 Topological Deep Learning Challenge : Design and Results
Authors:
Mathilde Papillon,
Mustafa Hajij,
Helen Jenne,
Johan Mathe,
Audun Myers,
Theodore Papamarkou,
Tolga Birdal,
Tamal Dey,
Tim Doster,
Tegan Emerson,
Gurusankar Gopalakrishnan,
Devendra Govil,
Aldo Guzmán-Sáenz,
Henry Kvinge,
Neal Livesay,
Soham Mukherjee,
Shreyas N. Samaga,
Karthikeyan Natesan Ramamurthy,
Maneel Reddy Karri,
Paul Rosen,
Sophia Sanborn,
Robin Walters,
Jens Agerberg,
Sadrodin Barikbin,
Claudio Battiloro
, et al. (31 additional authors not shown)
Abstract:
This paper presents the computational challenge on topological deep learning that was hosted within the ICML 2023 Workshop on Topology and Geometry in Machine Learning. The competition asked participants to provide open-source implementations of topological neural networks from the literature by contributing to the python packages TopoNetX (data processing) and TopoModelX (deep learning). The chal…
▽ More
This paper presents the computational challenge on topological deep learning that was hosted within the ICML 2023 Workshop on Topology and Geometry in Machine Learning. The competition asked participants to provide open-source implementations of topological neural networks from the literature by contributing to the python packages TopoNetX (data processing) and TopoModelX (deep learning). The challenge attracted twenty-eight qualifying submissions in its two-month duration. This paper describes the design of the challenge and summarizes its main findings.
△ Less
Submitted 18 January, 2024; v1 submitted 26 September, 2023;
originally announced September 2023.
-
Decorrelating neurons using persistence
Authors:
Rubén Ballester,
Carles Casacuberta,
Sergio Escalera
Abstract:
We propose a novel way to improve the generalisation capacity of deep learning models by reducing high correlations between neurons. For this, we present two regularisation terms computed from the weights of a minimum spanning tree of the clique whose vertices are the neurons of a given network (or a sample of those), where weights on edges are correlation dissimilarities. We provide an extensive…
▽ More
We propose a novel way to improve the generalisation capacity of deep learning models by reducing high correlations between neurons. For this, we present two regularisation terms computed from the weights of a minimum spanning tree of the clique whose vertices are the neurons of a given network (or a sample of those), where weights on edges are correlation dissimilarities. We provide an extensive set of experiments to validate the effectiveness of our terms, showing that they outperform popular ones. Also, we demonstrate that naive minimisation of all correlations between neurons obtains lower accuracies than our regularisation terms, suggesting that redundancies play a significant role in artificial neural networks, as evidenced by some studies in neuroscience for real networks. We include a proof of differentiability of our regularisers, thus develo** the first effective topological persistence-based regularisation terms that consider the whole set of neurons and that can be applied to a feedforward architecture in any deep learning task such as classification, data generation, or regression.
△ Less
Submitted 9 August, 2023;
originally announced August 2023.
-
On the Expressivity of Persistent Homology in Graph Learning
Authors:
Rubén Ballester,
Bastian Rieck
Abstract:
Persistent homology, a technique from computational topology, has recently shown strong empirical performance in the context of graph classification. Being able to capture long range graph properties via higher-order topological features, such as cycles of arbitrary length, in combination with multi-scale topological descriptors, has improved predictive performance for data sets with prominent top…
▽ More
Persistent homology, a technique from computational topology, has recently shown strong empirical performance in the context of graph classification. Being able to capture long range graph properties via higher-order topological features, such as cycles of arbitrary length, in combination with multi-scale topological descriptors, has improved predictive performance for data sets with prominent topological structures, such as molecules. At the same time, the theoretical properties of persistent homology have not been formally assessed in this context. This paper intends to bridge the gap between computational topology and graph machine learning by providing a brief introduction to persistent homology in the context of graphs, as well as a theoretical discussion and empirical analysis of its expressivity for graph learning tasks.
△ Less
Submitted 3 June, 2024; v1 submitted 20 February, 2023;
originally announced February 2023.
-
Predicting the generalization gap in neural networks using topological data analysis
Authors:
Rubén Ballester,
Xavier Arnal Clemente,
Carles Casacuberta,
Meysam Madadi,
Ciprian A. Corneanu,
Sergio Escalera
Abstract:
Understanding how neural networks generalize on unseen data is crucial for designing more robust and reliable models. In this paper, we study the generalization gap of neural networks using methods from topological data analysis. For this purpose, we compute homological persistence diagrams of weighted graphs constructed from neuron activation correlations after a training phase, aiming to capture…
▽ More
Understanding how neural networks generalize on unseen data is crucial for designing more robust and reliable models. In this paper, we study the generalization gap of neural networks using methods from topological data analysis. For this purpose, we compute homological persistence diagrams of weighted graphs constructed from neuron activation correlations after a training phase, aiming to capture patterns that are linked to the generalization capacity of the network. We compare the usefulness of different numerical summaries from persistence diagrams and show that a combination of some of them can accurately predict and partially explain the generalization gap without the need of a test set. Evaluation on two computer vision recognition tasks (CIFAR10 and SVHN) shows competitive generalization gap prediction when compared against state-of-the-art methods.
△ Less
Submitted 12 August, 2023; v1 submitted 23 March, 2022;
originally announced March 2022.
