-
Higher-order mutual information reveals synergistic sub-networks for multi-neuron importance
Authors:
Kenzo Clauw,
Sebastiano Stramaglia,
Daniele Marinazzo
Abstract:
Quantifying which neurons are important with respect to the classification decision of a trained neural network is essential for understanding their inner workings. Previous work primarily attributed importance to individual neurons. In this work, we study which groups of neurons contain synergistic or redundant information using a multivariate mutual information method called the O-information. W…
▽ More
Quantifying which neurons are important with respect to the classification decision of a trained neural network is essential for understanding their inner workings. Previous work primarily attributed importance to individual neurons. In this work, we study which groups of neurons contain synergistic or redundant information using a multivariate mutual information method called the O-information. We observe the first layer is dominated by redundancy suggesting general shared features (i.e. detecting edges) while the last layer is dominated by synergy indicating local class-specific features (i.e. concepts). Finally, we show the O-information can be used for multi-neuron importance. This can be demonstrated by re-training a synergistic sub-network, which results in a minimal change in performance. These results suggest our method can be used for pruning and unsupervised representation learning.
△ Less
Submitted 8 November, 2022; v1 submitted 1 November, 2022;
originally announced November 2022.
-
Gradients of O-information: low-order descriptors of high-order dependencies
Authors:
Tomas Scagliarini,
Davide Nuzzi,
Yuri Antonacci,
Luca Faes,
Fernando E. Rosas,
Daniele Marinazzo,
Sebastiano Stramaglia
Abstract:
O-information is an information-theoretic metric that captures the overall balance between redundant and synergistic information shared by groups of three or more variables. To complement the global assessment provided by this metric, here we propose the gradients of the O-information as low-order descriptors that can characterise how high-order effects are localised across a system of interest. W…
▽ More
O-information is an information-theoretic metric that captures the overall balance between redundant and synergistic information shared by groups of three or more variables. To complement the global assessment provided by this metric, here we propose the gradients of the O-information as low-order descriptors that can characterise how high-order effects are localised across a system of interest. We illustrate the capabilities of the proposed framework by revealing the role of specific spins in Ising models with frustration, and on practical data analysis on US macroeconomic data. Our theoretical and empirical analyses demonstrate the potential of these gradients to highlight the contribution of variables in forming high-order informational circuits
△ Less
Submitted 1 July, 2022;
originally announced July 2022.
-
Disentangling high-order mechanisms and high-order behaviours in complex systems
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Andrea I. Luppi,
Thomas F. Varley,
Joseph T. Lizier,
Sebastiano Stramaglia,
Henrik J. Jensen,
Daniele Marinazzo
Abstract:
Battiston et al. (arXiv:2110.06023) provide a comprehensive overview of how investigations of complex systems should take into account interactions between more than two elements, which can be modelled by hypergraphs and studied via topological data analysis. Following a separate line of enquiry, a broad literature has developed information-theoretic tools to characterize high-order interdependenc…
▽ More
Battiston et al. (arXiv:2110.06023) provide a comprehensive overview of how investigations of complex systems should take into account interactions between more than two elements, which can be modelled by hypergraphs and studied via topological data analysis. Following a separate line of enquiry, a broad literature has developed information-theoretic tools to characterize high-order interdependencies from observed data. While these could seem to be competing approaches aiming to address the same question, in this correspondence we clarify that this is not the case, and that a complete account of higher-order phenomena needs to embrace both.
△ Less
Submitted 21 March, 2022;
originally announced March 2022.
-
Beyond pairwise network similarity: exploring Mediation and Suppression between networks
Authors:
Lucas Lacasa,
Sebastiano Stramaglia,
Daniele Marinazzo
Abstract:
Network similarity measures quantify how and when two networks are symmetrically related, including measures of statistical association such as pairwise distance or other correlation measures between networks or between the layers of a multiplex network, but neither can directly unveil whether there are hidden confounding network factors nor can they estimate when such correlation is underpinned b…
▽ More
Network similarity measures quantify how and when two networks are symmetrically related, including measures of statistical association such as pairwise distance or other correlation measures between networks or between the layers of a multiplex network, but neither can directly unveil whether there are hidden confounding network factors nor can they estimate when such correlation is underpinned by a causal relation. In this work we extend this pairwise conceptual framework to triplets of networks and quantify how and when a network is related to a second network directly or via the indirect mediation or interaction with a third network. Accordingly, we develop a simple and intuitive set-theoretic approach to quantify mediation and suppression between networks. We validate our theory with synthetic models and further apply it to triplets of real-world networks, unveiling mediation and suppression effects which emerge when considering different modes of interaction in online social networks and different routes of information processing in the brain.
△ Less
Submitted 15 December, 2020;
originally announced December 2020.
-
Multiscale Analysis of Information Dynamics for Linear Multivariate Processes
Authors:
Luca Faes,
Alessandro Montalto,
Sebastiano Stramaglia,
Giandomenico Nollo,
Daniele Marinazzo
Abstract:
In the study of complex physical and physiological systems represented by multivariate time series, an issue of great interest is the description of the system dynamics over a range of different temporal scales. While information-theoretic approaches to the multiscale analysis of complex dynamics are being increasingly used, the theoretical properties of the applied measures are poorly understood.…
▽ More
In the study of complex physical and physiological systems represented by multivariate time series, an issue of great interest is the description of the system dynamics over a range of different temporal scales. While information-theoretic approaches to the multiscale analysis of complex dynamics are being increasingly used, the theoretical properties of the applied measures are poorly understood. This study introduces for the first time a framework for the analytical computation of information dynamics for linear multivariate stochastic processes explored at different time scales. After showing that the multiscale processing of a vector autoregressive (VAR) process introduces a moving average (MA) component, we describe how to represent the resulting VARMA process using state-space (SS) models and how to exploit the SS model parameters to compute analytical measures of information storage and information transfer for the original and rescaled processes. The framework is then used to quantify multiscale information dynamics for simulated unidirectionally and bidirectionally coupled VAR processes, showing that rescaling may lead to insightful patterns of information storage and transfer but also to potentially misleading behaviors.
△ Less
Submitted 24 February, 2016; v1 submitted 19 February, 2016;
originally announced February 2016.
-
Synergetic and redundant information flow detected by unnormalized Granger causality: application to resting state fMRI
Authors:
Sebastiano Stramaglia,
Leonardo Angelini,
Guorong Wu,
Jesus M. Cortés,
Luca Faes,
Daniele Marinazzo
Abstract:
Objectives: We develop a framework for the analysis of synergy and redundancy in the pattern of information flow between subsystems of a complex network. Methods: The presence of redundancy and/or synergy in multivariate time series data renders difficult to estimate the neat flow of information from each driver variable to a given target. We show that adopting an unnormalized definition of Grange…
▽ More
Objectives: We develop a framework for the analysis of synergy and redundancy in the pattern of information flow between subsystems of a complex network. Methods: The presence of redundancy and/or synergy in multivariate time series data renders difficult to estimate the neat flow of information from each driver variable to a given target. We show that adopting an unnormalized definition of Granger causality one may put in evidence redundant multiplets of variables influencing the target by maximizing the total Granger causality to a given target, over all the possible partitions of the set of driving variables. Consequently we introduce a pairwise index of synergy which is zero when two independent sources additively influence the future state of the system, differently from previous definitions of synergy.
Results: We report the application of the proposed approach to resting state fMRI data from the Human Connectome Project, showing that redundant pairs of regions arise mainly due to space contiguity and interhemispheric symmetry, whilst synergy occurs mainly between non-homologous pairs of regions in opposite hemispheres. Conclusions: Redundancy and synergy, in healthy resting brains, display characteristic patterns, revealed by the proposed approach.
Significance: The pairwise synergy index, here introduced, maps the informational character of the system at hand into a weighted complex network: the same approach can be applied to other complex systems whose normal state corresponds to a balance between redundant and synergetic circuits.
△ Less
Submitted 16 May, 2016; v1 submitted 14 April, 2015;
originally announced April 2015.
-
Synergy and redundancy in the Granger causal analysis of dynamical networks
Authors:
Sebastiano Stramaglia,
Jesus M. Cortes,
Daniele Marinazzo
Abstract:
We analyze by means of Granger causality the effect of synergy and redundancy in the inference (from time series data) of the information flow between subsystems of a complex network. Whilst we show that fully conditioned Granger causality is not affected by synergy, the pairwise analysis fails to put in evidence synergetic effects.
In cases when the number of samples is low, thus making the ful…
▽ More
We analyze by means of Granger causality the effect of synergy and redundancy in the inference (from time series data) of the information flow between subsystems of a complex network. Whilst we show that fully conditioned Granger causality is not affected by synergy, the pairwise analysis fails to put in evidence synergetic effects.
In cases when the number of samples is low, thus making the fully conditioned approach unfeasible, we show that partially conditioned Granger causality is an effective approach if the set of conditioning variables is properly chosen. We consider here two different strategies (based either on informational content for the candidate driver or on selecting the variables with highest pairwise influences) for partially conditioned Granger causality and show that depending on the data structure either one or the other might be valid. On the other hand, we observe that fully conditioned approaches do not work well in presence of redundancy, thus suggesting the strategy of separating the pairwise links in two subsets: those corresponding to indirect connections of the fully conditioned Granger causality (which should thus be excluded) and links that can be ascribed to redundancy effects and, together with the results from the fully connected approach, provide a better description of the causality pattern in presence of redundancy. We finally apply these methods to two different real datasets. First, analyzing electrophysiological data from an epileptic brain, we show that synergetic effects are dominant just before seizure occurrences. Second, our analysis applied to gene expression time series from HeLa culture shows that the underlying regulatory networks are characterized by both redundancy and synergy.
△ Less
Submitted 31 July, 2014; v1 submitted 20 March, 2014;
originally announced March 2014.
-
Expanding the Transfer Entropy to Identify Information Subgraphs in Complex Systems
Authors:
S. Stramaglia,
Guo-Rong Wu,
M. Pellicoro,
D. Marinazzo
Abstract:
We propose a formal expansion of the transfer entropy to put in evidence irreducible sets of variables which provide information for the future state of each assigned target. Multiplets characterized by a large contribution to the expansion are associated to informational circuits present in the system, with an informational character which can be associated to the sign of the contribution. For th…
▽ More
We propose a formal expansion of the transfer entropy to put in evidence irreducible sets of variables which provide information for the future state of each assigned target. Multiplets characterized by a large contribution to the expansion are associated to informational circuits present in the system, with an informational character which can be associated to the sign of the contribution. For the sake of computational complexity, we adopt the assumption of Gaussianity and use the corresponding exact formula for the conditional mutual information. We report the application of the proposed methodology on two EEG data sets.
△ Less
Submitted 24 September, 2012; v1 submitted 14 March, 2012;
originally announced March 2012.
-
Information flow in a network model and the law of diminishing marginal returns
Authors:
Daniele Marinazzo,
Mario Pellicoro,
Guorong Wu,
Leonardo Angelini,
Sebastiano Stramaglia
Abstract:
We analyze a simple dynamical network model which describes the limited capacity of nodes to process the input information. For a suitable choice of the parameters, the information flow pattern is characterized by exponential distribution of the incoming information and a fat-tailed distribution of the outgoing information, as a signature of the law of diminishing marginal returns. The analysis of…
▽ More
We analyze a simple dynamical network model which describes the limited capacity of nodes to process the input information. For a suitable choice of the parameters, the information flow pattern is characterized by exponential distribution of the incoming information and a fat-tailed distribution of the outgoing information, as a signature of the law of diminishing marginal returns. The analysis of a real EEG data-set shows that similar phenomena may be relevant for brain signals.
△ Less
Submitted 18 April, 2012; v1 submitted 22 February, 2012;
originally announced February 2012.
-
Nonlinear parametric model for Granger causality of time series
Authors:
Daniele Marinazzo,
Mario Pellicoro,
Sebastiano Stramaglia
Abstract:
We generalize a previously proposed approach for nonlinear Granger causality of time series, based on radial basis function. The proposed model is not constrained to be additive in variables from the two time series and can approximate any function of these variables, still being suitable to evaluate causality. Usefulness of this measure of causality is shown in a physiological example and in th…
▽ More
We generalize a previously proposed approach for nonlinear Granger causality of time series, based on radial basis function. The proposed model is not constrained to be additive in variables from the two time series and can approximate any function of these variables, still being suitable to evaluate causality. Usefulness of this measure of causality is shown in a physiological example and in the study of the feed-back loop in a model of excitatory and inhibitory neurons.
△ Less
Submitted 7 February, 2006;
originally announced February 2006.
