-
A scalable, synergy-first backbone decomposition of higher-order structures in complex systems
Authors:
Thomas F. Varley
Abstract:
Since its introduction in 2011, the partial information decomposition (PID) has triggered an explosion of interest in the field of multivariate information theory and the study of emergent, higher-order ("synergistic") interactions in complex systems. Despite its power, however, the PID has a number of limitations that restrict its general applicability: it scales poorly with system size and the s…
▽ More
Since its introduction in 2011, the partial information decomposition (PID) has triggered an explosion of interest in the field of multivariate information theory and the study of emergent, higher-order ("synergistic") interactions in complex systems. Despite its power, however, the PID has a number of limitations that restrict its general applicability: it scales poorly with system size and the standard approach to decomposition hinges on a definition of "redundancy", leaving synergy only vaguely defined as "that information not redundant." Other heuristic measures, such as the O-information, have been introduced, although these measures typically only provided a summary statistic of redundancy/synergy dominance, rather than direct insight into the synergy itself. To address this issue, we present an alternative decomposition that is synergy-first, scales much more gracefully than the PID, and has a straightforward interpretation. Our approach defines synergy as that information in a set that would be lost following the minimally invasive perturbation on any single element. By generalizing this idea to sets of elements, we construct a totally ordered "backbone" of partial synergy atoms that sweeps systems scales. Our approach starts with entropy, but can be generalized to the Kullback-Leibler divergence, and by extension, to the total correlation and the single-target mutual information. Finally, we show that this approach can be used to decompose higher-order interactions beyond just information theory: we demonstrate this by showing how synergistic combinations of pairwise edges in a complex network supports signal communicability and global integration. We conclude by discussing how this perspective on synergistic structure (information-based or otherwise) can deepen our understanding of part-whole relationships in complex systems.
△ Less
Submitted 12 February, 2024;
originally announced February 2024.
-
Evolving higher-order synergies reveals a trade-off between stability and information integration capacity in complex systems
Authors:
Thomas F. Varley,
Joshua Bongard
Abstract:
There has recently been an explosion of interest in how "higher-order" structures emerge in complex systems. This "emergent" organization has been found in a variety of natural and artificial systems, although at present the field lacks a unified understanding of what the consequences of higher-order synergies and redundancies are for systems. Typical research treat the presence (or absence) of sy…
▽ More
There has recently been an explosion of interest in how "higher-order" structures emerge in complex systems. This "emergent" organization has been found in a variety of natural and artificial systems, although at present the field lacks a unified understanding of what the consequences of higher-order synergies and redundancies are for systems. Typical research treat the presence (or absence) of synergistic information as a dependent variable and report changes in the level of synergy in response to some change in the system. Here, we attempt to flip the script: rather than treating higher-order information as a dependent variable, we use evolutionary optimization to evolve boolean networks with significant higher-order redundancies, synergies, or statistical complexity. We then analyse these evolved populations of networks using established tools for characterizing discrete dynamics: the number of attractors, average transient length, and Derrida coefficient. We also assess the capacity of the systems to integrate information. We find that high-synergy systems are unstable and chaotic, but with a high capacity to integrate information. In contrast, evolved redundant systems are extremely stable, but have negligible capacity to integrate information. Finally, the complex systems that balance integration and segregation (known as Tononi-Sporns-Edelman complexity) show features of both chaosticity and stability, with a greater capacity to integrate information than the redundant systems while being more stable than the random and synergistic systems. We conclude that there may be a fundamental trade-off between the robustness of a systems dynamics and its capacity to integrate information (which inherently requires flexibility and sensitivity), and that certain kinds of complexity naturally balance this trade-off.
△ Less
Submitted 26 January, 2024; v1 submitted 25 January, 2024;
originally announced January 2024.
-
The serotonergic psychedelic N,N-dipropyltryptamine alters information-processing dynamics in cortical neural circuits
Authors:
Thomas F. Varley,
Daniel Havert,
Leandro Fosque,
Abolfazl Alipour,
Naruepon Weerawongphrom,
Hiroki Naganobori,
Lily O'Shea,
Maria Pope,
John Beggs
Abstract:
Most of the recent work in psychedelic neuroscience has been done using non-invasive neuroimaging, with data recorded from the brains of adult volunteers under the influence of a variety of drugs. While this data provides holistic insights into the effects of psychedelics on whole-brain dynamics, the effects of psychedelics on the meso-scale dynamics of cortical circuits remains much less explored…
▽ More
Most of the recent work in psychedelic neuroscience has been done using non-invasive neuroimaging, with data recorded from the brains of adult volunteers under the influence of a variety of drugs. While this data provides holistic insights into the effects of psychedelics on whole-brain dynamics, the effects of psychedelics on the meso-scale dynamics of cortical circuits remains much less explored. Here, we report the effects of the serotonergic psychedelic N,N-diproptyltryptamine (DPT) on information-processing dynamics in a sample of in vitro organotypic cultures made from rat cortical tissue. Three hours of spontaneous activity were recorded: an hour of pre-drug control, and hour of exposure to 10$μ$M DPT solution, and a final hour of washout, once again under control conditions. We found that DPT reversibly alters information dynamics in multiple ways: first, the DPT condition was associated with higher entropy of spontaneous firing activity and reduced the amount of time information was stored in individual neurons. Second, DPT also reduced the reversibility of neural activity, increasing the entropy produced and suggesting a drive away from equilibrium. Third, DPT altered the structure of neuronal circuits, decreasing the overall information flow coming into each neuron, but increasing the number of weak connections, creating a dynamic that combines elements of integration and disintegration. Finally, DPT decreased the higher-order statistical synergy present in sets of three neurons. Collectively, these results paint a complex picture of how psychedelics regulate information processing in meso-scale cortical tissue. Implications for existing hypotheses of psychedelic action, such as the Entropic Brain Hypothesis, are discussed.
△ Less
Submitted 31 October, 2023;
originally announced October 2023.
-
Generalized Decomposition of Multivariate Information
Authors:
Thomas F. Varley
Abstract:
Since its introduction, the partial information decomposition (PID) has emerged as a powerful, information-theoretic technique useful for studying the structure of (potentially higher-order) interactions in complex systems. Despite its utility, the applicability of the PID is restricted by the need to assign elements as either inputs or targets, as well as the specific structure of the mutual info…
▽ More
Since its introduction, the partial information decomposition (PID) has emerged as a powerful, information-theoretic technique useful for studying the structure of (potentially higher-order) interactions in complex systems. Despite its utility, the applicability of the PID is restricted by the need to assign elements as either inputs or targets, as well as the specific structure of the mutual information itself. Here, we introduce a generalized information decomposition that relaxes the source/target distinction while still satisfying the basic intuitions about information. This approach is based on the decomposition of the Kullback-Leibler divergence, and consequently allows for the analysis of any information gained when updating from an arbitrary prior to an arbitrary posterior. Consequently, any information-theoretic measure that can be written in as a Kullback-Leibler divergence admits a decomposition in the style of Williams and Beer, including the total correlation, the negentropy, and the mutual information as special cases. In this paper, we explore how the generalized information decomposition can reveal novel insights into existing measures, as well as the nature of higher-order synergies. We show that synergistic information is intimately related to the well-known Tononi-Sporns-Edelman (TSE) complexity, and that synergistic information requires a similar integration/segregation balance as a high TSE complexity. Finally, we end with a discussion of how this approach fits into other attempts to generalize the PID and the possibilities for empirical applications.
△ Less
Submitted 8 December, 2023; v1 submitted 14 September, 2023;
originally announced September 2023.
-
Information Theory for Complex Systems Scientists
Authors:
Thomas F. Varley
Abstract:
In the 21st century, many of the crucial scientific and technical issues facing humanity can be understood as problems associated with understanding, modelling, and ultimately controlling complex systems: systems comprised of a large number of non-trivially interacting components whose collective behaviour can be difficult to predict. Information theory, a branch of mathematics historically associ…
▽ More
In the 21st century, many of the crucial scientific and technical issues facing humanity can be understood as problems associated with understanding, modelling, and ultimately controlling complex systems: systems comprised of a large number of non-trivially interacting components whose collective behaviour can be difficult to predict. Information theory, a branch of mathematics historically associated with questions about encoding and decoding messages, has emerged as something of a lingua franca for those studying complex systems, far exceeding its original narrow domain of communication systems engineering. In the context of complexity science, information theory provides a set of tools which allow researchers to uncover the statistical and effective dependencies between interacting components; relationships between systems and their environment; mereological whole-part relationships; and is sensitive to non-linearities missed by commonly parametric statistical models.
In this review, we aim to provide an accessible introduction to the core of modern information theory, aimed specifically at aspiring (and established) complex systems scientists. This includes standard measures, such as Shannon entropy, relative entropy, and mutual information, before building to more advanced topics, including: information dynamics, measures of statistical complexity, information decomposition, and effective network inference. In addition to detailing the formal definitions, in this review we make an effort to discuss how information theory can be interpreted and develop the intuition behind abstract concepts like "entropy," in the hope that this will enable interested readers to understand what information is, and how it is used, at a more fundamental level.
△ Less
Submitted 26 April, 2023; v1 submitted 24 April, 2023;
originally announced April 2023.
-
Partial entropy decomposition reveals higher-order structures in human brain activity
Authors:
Thomas F Varley,
Maria Pope,
Maria Grazia Puxeddu,
Joshua Faskowitz,
Olaf Sporns
Abstract:
The standard approach to modeling the human brain as a complex system is with a network, where the basic unit of interaction is a pairwise link between two brain regions. While powerful, this approach is limited by the inability to assess higher-order interactions involving three or more elements directly. In this work, we present a method for capturing higher-order dependencies in discrete data b…
▽ More
The standard approach to modeling the human brain as a complex system is with a network, where the basic unit of interaction is a pairwise link between two brain regions. While powerful, this approach is limited by the inability to assess higher-order interactions involving three or more elements directly. In this work, we present a method for capturing higher-order dependencies in discrete data based on partial entropy decomposition (PED). Our approach decomposes the joint entropy of the whole system into a set of strictly non-negative partial entropy atoms that describe the redundant, unique, and synergistic interactions that compose the system's structure. We begin by showing how the PED can provide insights into the mathematical structure of both the FC network itself, as well as established measures of higher-order dependency such as the O-information. When applied to resting state fMRI data, we find robust evidence of higher-order synergies that are largely invisible to standard functional connectivity analyses. This synergistic structure distinct from structural features based on redundancy that have previously dominated FC analyses. Our approach can also be localized in time, allowing a frame-by-frame analysis of how the distributions of redundancies and synergies change over the course of a recording. We find that different ensembles of regions can transiently change from being redundancy-dominated to synergy-dominated, and that the temporal pattern is structured in time. These results provide strong evidence that there exists a large space of unexplored structures in human brain data that have been largely missed by a focus on bivariate network connectivity models. This synergistic "shadow structures" is dynamic in time and, likely will illuminate new and interesting links between brain and behavior.
△ Less
Submitted 12 January, 2023;
originally announced January 2023.
-
Flickering emergences: The question of locality in information-theoretic approaches to emergence
Authors:
Thomas F. Varley
Abstract:
"Emergence", the phenomenon where a complex system displays properties, behaviours, or dynamics not trivially reducible to its constituent elements, is one of the defining properties of complex systems. Recently, there has been a concerted effort to formally define emergence using the mathematical framework of information theory, which proposes that emergence can be understood in terms of how the…
▽ More
"Emergence", the phenomenon where a complex system displays properties, behaviours, or dynamics not trivially reducible to its constituent elements, is one of the defining properties of complex systems. Recently, there has been a concerted effort to formally define emergence using the mathematical framework of information theory, which proposes that emergence can be understood in terms of how the states of wholes and parts collectively disclose information about the system's collective future. In this paper, we show how a common, foundational component of information-theoretic approaches to emergence implies an inherent instability to emergent properties, which we call flickering emergence. A system may on average display a meaningful emergent property (be it an informative coarse-graining, or higher-order synergy), but for particular configurations, that emergent property falls apart and becomes misinformative. We show existence proofs that flickering emergence occurs in two different frameworks (one based on coarse-graining and another based on multivariate information decomposition) and argue that any approach based on temporal mutual information will display it. Finally, we argue that flickering emergence should not be a disqualifying property of any model of emergence, but that it should be accounted for when attempting to theorize about how emergence relates to practical models of the natural world.
△ Less
Submitted 30 August, 2022;
originally announced August 2022.
-
Multivariate Information Theory Uncovers Synergistic Subsystems of the Human Cerebral Cortex
Authors:
Thomas F. Varley,
Maria Pope,
Joshua Faskowitz,
Olaf Sporns
Abstract:
One of the most well-established tools for modeling the brain as a complex system is the functional connectivity network, which examines the correlations between pairs of interacting brain regions. While powerful, the network model is limited by the restriction that only pairwise dependencies are visible and potentially higher-order structures are missed. In this work, we explore how multivariate…
▽ More
One of the most well-established tools for modeling the brain as a complex system is the functional connectivity network, which examines the correlations between pairs of interacting brain regions. While powerful, the network model is limited by the restriction that only pairwise dependencies are visible and potentially higher-order structures are missed. In this work, we explore how multivariate information theory can reveal higher-order, synergistic dependencies in the human brain. Using the O-information, a measure of whether the structure of a system is redundancy- or synergy-dominated, we show that synergistic subsystems are widespread in the human brain. We provide a mathematical analysis of the O-information to locate it within a larger taxonomy of multivariate complexity measures. We also show the O-information is related to a previously established measure, the Tononi-Sporns-Edelman complexity, and can be understood as an expected difference in integration between system scales. Highly synergistic subsystems typically sit between canonical functional networks, and may serve to integrate those networks. We then use simulated annealing to find maximally synergistic subsystems, finding that such systems typically comprise $\approx$10 brain regions, also recruited from multiple canonical brain systems. Though ubiquitous, highly synergistic subsystems are invisible when considering pairwise functional connectivity, suggesting that higher-order dependencies form a kind of ``shadow structure" that has been unrecognized by established network-based analyses. We assert that higher-order interactions in the brain represent a vast and under-explored space that, made accessible with tools of multivariate information theory, may offer novel scientific insights.
△ Less
Submitted 13 June, 2022;
originally announced June 2022.
-
Disentangling high-order mechanisms and high-order behaviours in complex systems
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Andrea I. Luppi,
Thomas F. Varley,
Joseph T. Lizier,
Sebastiano Stramaglia,
Henrik J. Jensen,
Daniele Marinazzo
Abstract:
Battiston et al. (arXiv:2110.06023) provide a comprehensive overview of how investigations of complex systems should take into account interactions between more than two elements, which can be modelled by hypergraphs and studied via topological data analysis. Following a separate line of enquiry, a broad literature has developed information-theoretic tools to characterize high-order interdependenc…
▽ More
Battiston et al. (arXiv:2110.06023) provide a comprehensive overview of how investigations of complex systems should take into account interactions between more than two elements, which can be modelled by hypergraphs and studied via topological data analysis. Following a separate line of enquiry, a broad literature has developed information-theoretic tools to characterize high-order interdependencies from observed data. While these could seem to be competing approaches aiming to address the same question, in this correspondence we clarify that this is not the case, and that a complete account of higher-order phenomena needs to embrace both.
△ Less
Submitted 21 March, 2022;
originally announced March 2022.
-
Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions
Authors:
Thomas F. Varley
Abstract:
A core feature of complex systems is that the interactions between elements in the present causally constrain each-other as the system evolves through time. To fully model all of these interactions (between elements, as well as ensembles of elements), we can decompose the total information flowing from past to future into a set of non-overlap** temporal interactions that describe all the differe…
▽ More
A core feature of complex systems is that the interactions between elements in the present causally constrain each-other as the system evolves through time. To fully model all of these interactions (between elements, as well as ensembles of elements), we can decompose the total information flowing from past to future into a set of non-overlap** temporal interactions that describe all the different modes by which information can flow. To achieve this, we propose a novel information-theoretic measure of temporal dependency ($I_{τsx}$) based on informative and misinformative local probability mass exclusions. To demonstrate the utility of this framework, we apply the decomposition to spontaneous spiking activity recorded from dissociated neural cultures of rat cerebral cortex to show how different modes of information processing are distributed over the system. Furthermore, being a localizable analysis, we show that $I_{τsx}$ can provide insight into the computational structure of single moments. We explore the time-resolved computational structure of neuronal avalanches and find that different types of information atoms have distinct profiles over the course of an avalanche, with the majority of non-trivial information dynamics happening before the first half of the cascade is completed. These analyses allow us to move beyond the historical focus on single measures of dependency such as information transfer or information integration, and explore a panoply of different relationships between elements (and groups of elements) in complex systems.
△ Less
Submitted 1 March, 2022; v1 submitted 25 February, 2022;
originally announced February 2022.
-
Intersectional synergies: untangling irreducible effects of intersecting identities via information decomposition
Authors:
Thomas F. Varley,
Patrick Kaminski
Abstract:
The idea of intersectionality has become a frequent topic of discussion both in academic sociology, as well as among popular movements for social justice such as Black Lives Matter, intersectional feminism, and LGBT rights. Intersectionality proposes that an individual's experience of society has aspects that are irreducible to the sum of one's various identities considered individually, but are "…
▽ More
The idea of intersectionality has become a frequent topic of discussion both in academic sociology, as well as among popular movements for social justice such as Black Lives Matter, intersectional feminism, and LGBT rights. Intersectionality proposes that an individual's experience of society has aspects that are irreducible to the sum of one's various identities considered individually, but are "greater than the sum of their parts." In this work, we show that the effects of intersectional identities can be statistically observed in empirical data using information theory. We show that, when considering the predictive relationship between various identities categories such as race, sex, and income (as a proxy for class) on outcomes such as health and wellness, robust statistical synergies appear. These synergies show that there are joint-effects of identities on outcomes that are irreducible to any identity considered individually and only appear when specific categories are considered together (for example, there is a large, synergistic effect of race and sex considered jointly on income irreducible to either race or sex). We then show using synthetic data that the current gold-standard method of assessing intersectionalities in data (linear regression with multiplicative interaction coefficients) fails to disambiguate between truly synergistic, greater-than-the-sum-of-their-parts interactions, and redundant interactions. We explore the significance of these two distinct types of interactions in the context of making inferences about intersectional relationships in data and the importance of being able to reliably differentiate the two. Finally, we conclude that information theory, as a model-free framework sensitive to nonlinearities and synergies in data, is a natural method by which to explore the space of higher-order social dynamics.
△ Less
Submitted 24 October, 2021; v1 submitted 18 June, 2021;
originally announced June 2021.
-
Causal Emergence in Discrete and Continuous Dynamical Systems
Authors:
Thomas F. Varley
Abstract:
Emergence, the phenomena where a system's micro-scale dynamics facilitate the development of non-trivial, informative higher scales, has become a foundational concept in modern sciences, tying together fields as diverse as physics, biology, economics, and ecology. Despite it's apparent universality and the considerable interest, historically researchers have struggled to provide a rigorous, formal…
▽ More
Emergence, the phenomena where a system's micro-scale dynamics facilitate the development of non-trivial, informative higher scales, has become a foundational concept in modern sciences, tying together fields as diverse as physics, biology, economics, and ecology. Despite it's apparent universality and the considerable interest, historically researchers have struggled to provide a rigorous, formal definition of emergence that is applicable across fields. Recent theoretical work using information theory and network science to formalize emergence in state-transition networks (causal emergence) has provided a promising way forward, however the relationship between this new framework and other well-studied system dynamics is unknown. In this study, we apply causal emergence analysis to two well-described dynamical systems: the 88 unique elementary cellular automata and the continuous Rossler system in periodic, critical, and chaotic regimes. We find that emergence, as well as its component elements (determinism, degeneracy, and effectiveness) vary dramatically in different dynamical regimes in sometimes unexpected ways. We conclude that the causal emergence framework provides a rich new area of research to explore both to theoreticians and natural scientists in many fields.
△ Less
Submitted 29 March, 2020;
originally announced March 2020.
