-
Attending to Topological Spaces: The Cellular Transformer
Authors:
Rubén Ballester,
Pablo Hernández-García,
Mathilde Papillon,
Claudio Battiloro,
Nina Miolane,
Tolga Birdal,
Carles Casacuberta,
Sergio Escalera,
Mustafa Hajij
Abstract:
Topological Deep Learning seeks to enhance the predictive performance of neural network models by harnessing topological structures in input data. Topological neural networks operate on spaces such as cell complexes and hypergraphs, that can be seen as generalizations of graphs. In this work, we introduce the Cellular Transformer (CT), a novel architecture that generalizes graph-based transformers…
▽ More
Topological Deep Learning seeks to enhance the predictive performance of neural network models by harnessing topological structures in input data. Topological neural networks operate on spaces such as cell complexes and hypergraphs, that can be seen as generalizations of graphs. In this work, we introduce the Cellular Transformer (CT), a novel architecture that generalizes graph-based transformers to cell complexes. First, we propose a new formulation of the usual self- and cross-attention mechanisms, tailored to leverage incidence relations in cell complexes, e.g., edge-face and node-edge relations. Additionally, we propose a set of topological positional encodings specifically designed for cell complexes. By transforming three graph datasets into cell complex datasets, our experiments reveal that CT not only achieves state-of-the-art performance, but it does so without the need for more complex enhancements such as virtual nodes, in-domain structural encodings, or graph rewiring.
△ Less
Submitted 26 May, 2024; v1 submitted 22 May, 2024;
originally announced May 2024.
-
Topological Data Analysis for Neural Network Analysis: A Comprehensive Survey
Authors:
Rubén Ballester,
Carles Casacuberta,
Sergio Escalera
Abstract:
This survey provides a comprehensive exploration of applications of Topological Data Analysis (TDA) within neural network analysis. Using TDA tools such as persistent homology and Mapper, we delve into the intricate structures and behaviors of neural networks and their datasets. We discuss different strategies to obtain topological information from data and neural networks by means of TDA. Additio…
▽ More
This survey provides a comprehensive exploration of applications of Topological Data Analysis (TDA) within neural network analysis. Using TDA tools such as persistent homology and Mapper, we delve into the intricate structures and behaviors of neural networks and their datasets. We discuss different strategies to obtain topological information from data and neural networks by means of TDA. Additionally, we review how topological information can be leveraged to analyze properties of neural networks, such as their generalization capacity or expressivity. We explore practical implications of deep learning, specifically focusing on areas like adversarial detection and model selection. Our survey organizes the examined works into four broad domains: 1. Characterization of neural network architectures; 2. Analysis of decision regions and boundaries; 3. Study of internal representations, activations, and parameters; 4. Exploration of training dynamics and loss functions. Within each category, we discuss several articles, offering background information to aid in understanding the various methodologies. We conclude with a synthesis of key insights gained from our study, accompanied by a discussion of challenges and potential advancements in the field.
△ Less
Submitted 3 January, 2024; v1 submitted 10 December, 2023;
originally announced December 2023.
-
Decorrelating neurons using persistence
Authors:
Rubén Ballester,
Carles Casacuberta,
Sergio Escalera
Abstract:
We propose a novel way to improve the generalisation capacity of deep learning models by reducing high correlations between neurons. For this, we present two regularisation terms computed from the weights of a minimum spanning tree of the clique whose vertices are the neurons of a given network (or a sample of those), where weights on edges are correlation dissimilarities. We provide an extensive…
▽ More
We propose a novel way to improve the generalisation capacity of deep learning models by reducing high correlations between neurons. For this, we present two regularisation terms computed from the weights of a minimum spanning tree of the clique whose vertices are the neurons of a given network (or a sample of those), where weights on edges are correlation dissimilarities. We provide an extensive set of experiments to validate the effectiveness of our terms, showing that they outperform popular ones. Also, we demonstrate that naive minimisation of all correlations between neurons obtains lower accuracies than our regularisation terms, suggesting that redundancies play a significant role in artificial neural networks, as evidenced by some studies in neuroscience for real networks. We include a proof of differentiability of our regularisers, thus develo** the first effective topological persistence-based regularisation terms that consider the whole set of neurons and that can be applied to a feedforward architecture in any deep learning task such as classification, data generation, or regression.
△ Less
Submitted 9 August, 2023;
originally announced August 2023.
-
A topological classifier to characterize brain states: When shape matters more than variance
Authors:
Aina Ferrà,
Gloria Cecchini,
Fritz-Pere Nobbe Fisas,
Carles Casacuberta,
Ignasi Cos
Abstract:
Despite the remarkable accuracies attained by machine learning classifiers to separate complex datasets in a supervised fashion, most of their operation falls short to provide an informed intuition about the structure of data, and, what is more important, about the phenomena being characterized by the given datasets. By contrast, topological data analysis (TDA) is devoted to study the shape of dat…
▽ More
Despite the remarkable accuracies attained by machine learning classifiers to separate complex datasets in a supervised fashion, most of their operation falls short to provide an informed intuition about the structure of data, and, what is more important, about the phenomena being characterized by the given datasets. By contrast, topological data analysis (TDA) is devoted to study the shape of data clouds by means of persistence descriptors and provides a quantitative characterization of specific topological features of the dataset under scrutiny.
In this article we introduce a novel TDA-based classifier that works on the principle of assessing quantifiable changes on topological metrics caused by the addition of new input to a subset of data. We used this classifier with a high-dimensional electro-encephalographic (EEG) dataset recorded from eleven participants during a decision-making experiment in which three motivational states were induced through a manipulation of social pressure. After processing a band-pass filtered version of EEG signals, we calculated silhouettes from persistence diagrams associated with each motivated state, and classified unlabeled signals according to their impact on each reference silhouette. Our results show that in addition to providing accuracies within the range of those of a nearest neighbour classifier, the TDA classifier provides formal intuition of the structure of the dataset as well as an estimate of its intrinsic dimension. Towards this end, we incorporated dimensionality reduction methods to our procedure and found that the accuracy of our TDA classifier is generally not sensitive to explained variance but rather to shape, contrary to what happens with most machine learning classifiers.
△ Less
Submitted 7 March, 2023;
originally announced March 2023.
-
Importance attribution in neural networks by means of persistence landscapes of time series
Authors:
Aina Ferrà,
Carles Casacuberta,
Oriol Pujol
Abstract:
We propose and implement a method to analyze time series with a neural network using a matrix of area-normalized persistence landscapes obtained through topological data analysis. We include a gating layer in the network's architecture that is able to identify the most relevant landscape levels for the classification task, thus working as an importance attribution system. Next, we perform a matchi…
▽ More
We propose and implement a method to analyze time series with a neural network using a matrix of area-normalized persistence landscapes obtained through topological data analysis. We include a gating layer in the network's architecture that is able to identify the most relevant landscape levels for the classification task, thus working as an importance attribution system. Next, we perform a matching between the selected landscape functions and the corresponding critical points of the original time series. From this matching we are able to reconstruct an approximate shape of the time series that gives insight into the classification decision. We test this technique with input data from a dataset of electrocardiographic signals.
△ Less
Submitted 6 February, 2023;
originally announced February 2023.
-
Predicting the generalization gap in neural networks using topological data analysis
Authors:
Rubén Ballester,
Xavier Arnal Clemente,
Carles Casacuberta,
Meysam Madadi,
Ciprian A. Corneanu,
Sergio Escalera
Abstract:
Understanding how neural networks generalize on unseen data is crucial for designing more robust and reliable models. In this paper, we study the generalization gap of neural networks using methods from topological data analysis. For this purpose, we compute homological persistence diagrams of weighted graphs constructed from neuron activation correlations after a training phase, aiming to capture…
▽ More
Understanding how neural networks generalize on unseen data is crucial for designing more robust and reliable models. In this paper, we study the generalization gap of neural networks using methods from topological data analysis. For this purpose, we compute homological persistence diagrams of weighted graphs constructed from neuron activation correlations after a training phase, aiming to capture patterns that are linked to the generalization capacity of the network. We compare the usefulness of different numerical summaries from persistence diagrams and show that a combination of some of them can accurately predict and partially explain the generalization gap without the need of a test set. Evaluation on two computer vision recognition tasks (CIFAR10 and SVHN) shows competitive generalization gap prediction when compared against state-of-the-art methods.
△ Less
Submitted 12 August, 2023; v1 submitted 23 March, 2022;
originally announced March 2022.
-
Reconstruction of univariate functions from directional persistence diagrams
Authors:
Aina Ferrà,
Carles Casacuberta,
Oriol Pujol
Abstract:
We describe a method for approximating a single-variable function $f$ using persistence diagrams of sublevel sets of $f$ from height functions in different directions. We provide algorithms for the piecewise linear case and for the smooth case. Three directions suffice to locate all local maxima and minima of a piecewise linear continuous function from its collection of directional persistence dia…
▽ More
We describe a method for approximating a single-variable function $f$ using persistence diagrams of sublevel sets of $f$ from height functions in different directions. We provide algorithms for the piecewise linear case and for the smooth case. Three directions suffice to locate all local maxima and minima of a piecewise linear continuous function from its collection of directional persistence diagrams, while five directions are needed in the case of smooth functions with non-degenerate critical points.
Our approximation of functions by means of persistence diagrams is motivated by a study of importance attribution in machine learning, where one seeks to reduce the number of critical points of signal functions without a significant loss of information for a neural network classifier.
△ Less
Submitted 8 February, 2023; v1 submitted 3 March, 2022;
originally announced March 2022.
-
Combinatorial homotopy categories
Authors:
Carles Casacuberta,
Jiri Rosicky
Abstract:
A model category is called combinatorial if it is cofibrantly generated and its underlying category is locally presentable. As shown in recent years, homotopy categories of combinatorial model categories share useful properties, such as being well generated and satisfying a very general form of Ohkawa's theorem.
A model category is called combinatorial if it is cofibrantly generated and its underlying category is locally presentable. As shown in recent years, homotopy categories of combinatorial model categories share useful properties, such as being well generated and satisfying a very general form of Ohkawa's theorem.
△ Less
Submitted 9 April, 2017; v1 submitted 1 February, 2017;
originally announced February 2017.
-
Comparing localizations across adjunctions
Authors:
Carles Casacuberta,
Oriol Raventós,
Andrew Tonks
Abstract:
We show that several apparently unrelated formulas involving left or right Bousfield localizations in homotopy theory are induced by comparison maps associated with pairs of adjoint functors. Such comparison maps are used in the article to discuss the existence of functorial liftings of homotopical localizations and cellularizations to categories of algebras over monads acting on model categories,…
▽ More
We show that several apparently unrelated formulas involving left or right Bousfield localizations in homotopy theory are induced by comparison maps associated with pairs of adjoint functors. Such comparison maps are used in the article to discuss the existence of functorial liftings of homotopical localizations and cellularizations to categories of algebras over monads acting on model categories, with emphasis on the cases of module spectra and algebras over simplicial operads. Some of our results hold for algebras up to homotopy as well; for example, if $T$ is the reduced monad associated with a simplicial operad and $f$ is any map of pointed simplicial sets, then $f$-localization coincides with $Tf$-localization on spaces underlying homotopy $T$-algebras, and similarly for cellularizations.
△ Less
Submitted 16 September, 2019; v1 submitted 29 April, 2014;
originally announced April 2014.
-
A generalization of Ohkawa's theorem
Authors:
Carles Casacuberta,
Javier J. Gutiérrez,
Jirí Rosický
Abstract:
A theorem due to Ohkawa states that the collection of Bousfield equivalence classes of spectra is a set. We extend this result to arbitrary combinatorial model categories.
A theorem due to Ohkawa states that the collection of Bousfield equivalence classes of spectra is a set. We extend this result to arbitrary combinatorial model categories.
△ Less
Submitted 4 October, 2013; v1 submitted 28 March, 2012;
originally announced March 2012.
-
Are all localizing subcategories of stable homotopy categories coreflective?
Authors:
Carles Casacuberta,
Javier J. Gutiérrez,
Jiří Rosický
Abstract:
We prove that, in a triangulated category with combinatorial models, every localizing subcategory is coreflective and every colocalizing subcategory is reflective if a certain large-cardinal axiom (Vopenka's principle) is assumed true. It follows that, under the same assumptions, orthogonality sets up a bijective correspondence between localizing subcategories and colocalizing subcategories. The e…
▽ More
We prove that, in a triangulated category with combinatorial models, every localizing subcategory is coreflective and every colocalizing subcategory is reflective if a certain large-cardinal axiom (Vopenka's principle) is assumed true. It follows that, under the same assumptions, orthogonality sets up a bijective correspondence between localizing subcategories and colocalizing subcategories. The existence of such a bijection was left as an open problem by Hovey, Palmieri and Strickland in their axiomatic study of stable homotopy categories and also by Neeman in the context of well-generated triangulated categories.
△ Less
Submitted 13 April, 2012; v1 submitted 11 June, 2011;
originally announced June 2011.
-
Definable orthogonality classes in accessible categories are small
Authors:
Joan Bagaria,
Carles Casacuberta,
A. R. D. Mathias,
Jiri Rosicky
Abstract:
We lower substantially the strength of the assumptions needed for the validity of certain results in category theory and homotopy theory which were known to follow from Vopenka's principle. We prove that the necessary large-cardinal hypotheses depend on the complexity of the formulas defining the given classes, in the sense of the Levy hierarchy. For example, the statement that, for a class S of m…
▽ More
We lower substantially the strength of the assumptions needed for the validity of certain results in category theory and homotopy theory which were known to follow from Vopenka's principle. We prove that the necessary large-cardinal hypotheses depend on the complexity of the formulas defining the given classes, in the sense of the Levy hierarchy. For example, the statement that, for a class S of morphisms in a locally presentable category C of structures, the orthogonal class of objects is a small-orthogonality class (hence reflective) is provable in ZFC if S is Σ_1, while it follows from the existence of a proper class of supercompact cardinals if S is Σ_2, and from the existence of a proper class of what we call C(n)-extendible cardinals if S is Σ_{n+2} for n bigger than or equal to 1. These cardinals form a new hierarchy, and we show that Vopenka's principle is equivalent to the existence of C(n)-extendible cardinals for all n. As a consequence, we prove that the existence of cohomological localizations of simplicial sets, a long-standing open problem in algebraic topology, is implied by the existence of arbitrarily large supercompact cardinals. This result follows from the fact that cohomology equivalences are Σ_2. In contrast with this fact, homology equivalences are Σ_1, from which it follows (as is well known) that the existence of homological localizations is provable in ZFC.
△ Less
Submitted 2 December, 2012; v1 submitted 14 January, 2011;
originally announced January 2011.
-
Brown representability does not come for free
Authors:
Carles Casacuberta,
Amnon Neeman
Abstract:
We exhibit a triangulated category T having both products and coproducts, and a triangulated subcategory S of T which is both localizing and colocalizing, for which neither a Bousfield localization nor a colocalization exists. It follows that neither the category S nor its dual satisfy Brown representability. Our example involves an abelian category whose derived category does not have small Hom…
▽ More
We exhibit a triangulated category T having both products and coproducts, and a triangulated subcategory S of T which is both localizing and colocalizing, for which neither a Bousfield localization nor a colocalization exists. It follows that neither the category S nor its dual satisfy Brown representability. Our example involves an abelian category whose derived category does not have small Hom-sets.
△ Less
Submitted 11 July, 2008;
originally announced July 2008.
-
Localization of algebras over coloured operads
Authors:
Carles Casacuberta,
Javier J. Gutierrez,
Ieke Moerdijk,
Rainer M. Vogt
Abstract:
We give sufficient conditions for homotopical localization functors to preserve algebras over coloured operads in monoidal model categories. Our approach encompasses a number of previous results about preservation of structures under localizations, such as loop spaces or infinite loop spaces, and provides new results of the same kind. For instance, under suitable assumptions, homotopical localiz…
▽ More
We give sufficient conditions for homotopical localization functors to preserve algebras over coloured operads in monoidal model categories. Our approach encompasses a number of previous results about preservation of structures under localizations, such as loop spaces or infinite loop spaces, and provides new results of the same kind. For instance, under suitable assumptions, homotopical localizations preserve ring spectra (in the strict sense, not only up to homotopy), modules over ring spectra, and algebras over commutative ring spectra, as well as ring maps, module maps, and algebra maps. It is principally the treatment of module spectra and their maps that led us to the use of coloured operads (also called enriched multicategories) in this context.
△ Less
Submitted 24 June, 2008;
originally announced June 2008.
-
Epireflections and supercompact cardinals
Authors:
Joan Bagaria,
Carles Casacuberta,
Adrian R. D. Mathias
Abstract:
We prove that, under suitable assumptions on a category C, the existence of supercompact cardinals implies that every absolute epireflective class of objects of C is a small-orthogonality class. More precisely, if L is a localization functor on an accessible category C such that the unit morphism X \to LX is an extremal epimorphism for all X, and the class of L-local objects is defined by an abs…
▽ More
We prove that, under suitable assumptions on a category C, the existence of supercompact cardinals implies that every absolute epireflective class of objects of C is a small-orthogonality class. More precisely, if L is a localization functor on an accessible category C such that the unit morphism X \to LX is an extremal epimorphism for all X, and the class of L-local objects is defined by an absolute formula with parameters, then the existence of a supercompact cardinal above the cardinalities of the parameters implies that L is a localization with respect to some set of morphisms.
△ Less
Submitted 5 March, 2007;
originally announced March 2007.
-
The orthogonal subcategory problem in homotopy theory
Authors:
Carles Casacuberta,
Boris Chorny
Abstract:
It is known that the existence of localization with respect to an arbitrary (possibly proper) class of maps in the category of simplicial sets is implied by a large-cardinal axiom called Vopenka's principle.In this article we extend the validity of this result to any left proper, combinatorial, simplicial model category $\cat M$ and show that, under additional assumptions on $\cat M$, every homo…
▽ More
It is known that the existence of localization with respect to an arbitrary (possibly proper) class of maps in the category of simplicial sets is implied by a large-cardinal axiom called Vopenka's principle.In this article we extend the validity of this result to any left proper, combinatorial, simplicial model category $\cat M$ and show that, under additional assumptions on $\cat M$, every homotopy idempotent functor is in fact a localization with respect to some set of maps. These results are valid for the homotopy category of spectra, among other applications.
△ Less
Submitted 15 February, 2005;
originally announced February 2005.
