-
Characterising directed and undirected metrics of high-order interdependence
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Michael Gastpar
Abstract:
Systems of interest for theoretical or experimental work often exhibit high-order interactions, corresponding to statistical interdependencies in groups of variables that cannot be reduced to dependencies in subsets of them. While still under active development, the framework of partial information decomposition (PID) has emerged as the dominant approach to conceptualise and calculate high-order i…
▽ More
Systems of interest for theoretical or experimental work often exhibit high-order interactions, corresponding to statistical interdependencies in groups of variables that cannot be reduced to dependencies in subsets of them. While still under active development, the framework of partial information decomposition (PID) has emerged as the dominant approach to conceptualise and calculate high-order interdependencies. PID approaches can be grouped in two types: directed approaches that divide variables into sources and targets, and undirected approaches that treat all variables equally. Directed and undirected approaches are usually employed to investigate different scenarios, and hence little is known about how these two types of approaches may relate to each other, or if their corresponding quantities are linked in some way. In this paper we investigate the relationship between the redundancy-synergy index (RSI) and the O-information, which are practical metrics of directed and undirected high-order interdependencies, respectively. Our results reveal tight links between these two quantities, and provide interpretations of them in terms of likelihood ratios in a hypothesis testing setting, as well as in terms of projections in information geometry.
△ Less
Submitted 10 April, 2024;
originally announced April 2024.
-
Software in the natural world: A computational approach to hierarchical emergence
Authors:
Fernando E. Rosas,
Bernhard C. Geiger,
Andrea I Luppi,
Anil K. Seth,
Daniel Polani,
Michael Gastpar,
Pedro A. M. Mediano
Abstract:
Understanding the functional architecture of complex systems is crucial to illuminate their inner workings and enable effective methods for their prediction and control. Recent advances have introduced tools to characterise emergent macroscopic levels; however, while these approaches are successful in identifying when emergence takes place, they are limited in the extent they can determine how it…
▽ More
Understanding the functional architecture of complex systems is crucial to illuminate their inner workings and enable effective methods for their prediction and control. Recent advances have introduced tools to characterise emergent macroscopic levels; however, while these approaches are successful in identifying when emergence takes place, they are limited in the extent they can determine how it does. Here we address this limitation by develo** a computational approach to emergence, which characterises macroscopic processes in terms of their computational capabilities. Concretely, we articulate a view on emergence based on how software works, which is rooted on a mathematical formalism that articulates how macroscopic processes can express self-contained informational, interventional, and computational properties. This framework establishes a hierarchy of nested self-contained processes that determines what computations take place at what level, which in turn delineates the functional architecture of a complex system. This approach is illustrated on paradigmatic models from the statistical physics and computational neuroscience literature, which are shown to exhibit macroscopic processes that are akin to software in human-engineered systems. Overall, this framework enables a deeper understanding of the multi-level structure of complex systems, revealing specific ways in which they can be efficiently simulated, predicted, and controlled.
△ Less
Submitted 5 June, 2024; v1 submitted 14 February, 2024;
originally announced February 2024.
-
Quantifying Hierarchical Selection
Authors:
Hardik Rajpal,
Clem von Stengel,
Pedro A. M. Mediano,
Fernando E. Rosas,
Eduardo Viegas,
Pablo A. Marquet,
Henrik J. Jensen
Abstract:
At what level does selective pressure effectively act? When considering the reproductive dynamics of interacting and mutating agents, it has long been debated whether selection is better understood by focusing on the individual or if hierarchical selection emerges as a consequence of joint adaptation. Despite longstanding efforts in theoretical ecology there is still no consensus on this fundament…
▽ More
At what level does selective pressure effectively act? When considering the reproductive dynamics of interacting and mutating agents, it has long been debated whether selection is better understood by focusing on the individual or if hierarchical selection emerges as a consequence of joint adaptation. Despite longstanding efforts in theoretical ecology there is still no consensus on this fundamental issue, most likely due to the difficulty in obtaining adequate data spanning sufficient number of generations and the lack of adequate tools to quantify the effect of hierarchical selection. Here we capitalise on recent advances in information-theoretic data analysis to advance this state of affairs by investigating the emergence of high-order structures -- such as groups of species -- in the collective dynamics of the Tangled Nature model of evolutionary ecology. Our results show that evolutionary dynamics can lead to clusters of species that act as a selective group, that acquire information-theoretic agency. Overall, our findings provide quantitative evidence supporting the relevance of high-order structures in evolutionary ecology, which can emerge even from relatively simple processes of adaptation and selection.
△ Less
Submitted 1 November, 2023; v1 submitted 31 October, 2023;
originally announced October 2023.
-
Quantifying High-Order Interdependencies in Entangled Quantum States
Authors:
Marco Alberto Javarone,
Fernando E. Rosas,
Paolo Facchi,
Saverio Pascazio,
Sebastiano Stramaglia
Abstract:
Here, we leverage recent advances in information theory to develop a novel method to characterise the dominant character of the high-order dependencies of quantum systems. To this end, we introduce the Q-information: an information-theoretic measure capable of distinguishing quantum states dominated by synergy or redundancy. We illustrate the measure by investigating the properties of paradigmatic…
▽ More
Here, we leverage recent advances in information theory to develop a novel method to characterise the dominant character of the high-order dependencies of quantum systems. To this end, we introduce the Q-information: an information-theoretic measure capable of distinguishing quantum states dominated by synergy or redundancy. We illustrate the measure by investigating the properties of paradigmatic entangled Qubit states and find that -- in contrast to classical systems -- quantum systems need at least four variables to exhibit high-order properties. Furthermore, our results reveal that unitary evolution can radically affect the internal information organisation in a way that strongly depends on the corresponding Hamiltonian. Overall, the Q-information sheds light on novel aspects of the internal organisation of quantum systems and their time evolution, opening new avenues for studying several quantum phenomena and related technologies.
△ Less
Submitted 5 October, 2023;
originally announced October 2023.
-
Information decomposition reveals hidden high-order contributions to temporal irreversibility
Authors:
Andrea I Luppi,
Fernando E. Rosas,
Gustavo Deco,
Morten L. Kringelbach,
Pedro A. M. Mediano
Abstract:
Temporal irreversibility, often referred to as the arrow of time, is a fundamental concept in statistical mechanics. Markers of irreversibility also provide a powerful characterisation of information processing in biological systems. However, current approaches tend to describe temporal irreversibility in terms of a single scalar quantity, without disentangling the underlying dynamics that contrib…
▽ More
Temporal irreversibility, often referred to as the arrow of time, is a fundamental concept in statistical mechanics. Markers of irreversibility also provide a powerful characterisation of information processing in biological systems. However, current approaches tend to describe temporal irreversibility in terms of a single scalar quantity, without disentangling the underlying dynamics that contribute to irreversibility. Here we propose a broadly applicable information-theoretic framework to characterise the arrow of time in multivariate time series, which yields qualitatively different types of irreversible information dynamics. This multidimensional characterisation reveals previously unreported high-order modes of irreversibility, and establishes a formal connection between recent heuristic markers of temporal irreversibility and metrics of information processing. We demonstrate the prevalence of high-order irreversibility in the hyperactive regime of a biophysical model of brain dynamics, showing that our framework is both theoretically principled and empirically useful. This work challenges the view of the arrow of time as a monolithic entity, enhancing both our theoretical understanding of irreversibility and our ability to detect it in practical applications.
△ Less
Submitted 10 August, 2023;
originally announced August 2023.
-
Quantifying synergy and redundancy in multiplex networks
Authors:
Andrea I. Luppi,
Eckehard Olbrich,
Conor Finn,
Laura E. Suárez,
Fernando E. Rosas,
Pedro A. M. Mediano,
Jürgen Jost
Abstract:
Understanding how different networks relate to each other is key for obtaining a greater insight into complex systems. Here, we introduce an intuitive yet powerful framework to characterise the relationship between two networks comprising the same nodes. We showcase our framework by decomposing the shortest paths between nodes as being contributed uniquely by one or the other source network, or re…
▽ More
Understanding how different networks relate to each other is key for obtaining a greater insight into complex systems. Here, we introduce an intuitive yet powerful framework to characterise the relationship between two networks comprising the same nodes. We showcase our framework by decomposing the shortest paths between nodes as being contributed uniquely by one or the other source network, or redundantly by either, or synergistically by the two together. Our approach takes into account the networks' full topology, and it also provides insights at multiple levels of resolution: from global statistics, to individual paths of different length. We show that this approach is widely applicable, from brains to the London public transport system. In humans and across 123 other mammalian species, we demonstrate that reliance on unique contributions by long-range white matter fibers is a conserved feature of mammalian structural brain networks. Across species, we also find that efficient communication relies on significantly greater synergy between long-range and short-range fibers than expected by chance, and significantly less redundancy. Our framework may find applications to help decide how to trade-off different desiderata when designing network systems, or to evaluate their relative presence in existing systems, whether biological or artificial.
△ Less
Submitted 8 August, 2023; v1 submitted 2 June, 2023;
originally announced June 2023.
-
Dynamical noise can enhance high-order statistical structure in complex systems
Authors:
Patricio Orio,
Pedro A. M. Mediano,
Fernando E. Rosas
Abstract:
Recent research has provided a wealth of evidence highlighting the pivotal role of high-order interdependencies in supporting the information-processing capabilities of distributed complex systems. These findings may suggest that high-order interdependencies constitute a powerful resource that is, however, challenging to harness and can be readily disrupted. In this paper we contest this perspecti…
▽ More
Recent research has provided a wealth of evidence highlighting the pivotal role of high-order interdependencies in supporting the information-processing capabilities of distributed complex systems. These findings may suggest that high-order interdependencies constitute a powerful resource that is, however, challenging to harness and can be readily disrupted. In this paper we contest this perspective by demonstrating that high-order interdependencies can not only exhibit robustness to stochastic perturbations, but can in fact be enhanced by them. Using elementary cellular automata as a general testbed, our results unveil the capacity of dynamical noise to enhance the statistical regularities between agents and, intriguingly, even alter the prevailing character of their interdependencies. Furthermore, our results show that these effects are related to the high-order structure of the local rules, which affect the system's susceptibility to noise and characteristic times-scales. These results deepen our understanding of how high-order interdependencies may spontaneously emerge within distributed systems interacting with stochastic environments, thus providing an initial step towards elucidating their origin and function in complex systems like the human brain.
△ Less
Submitted 22 May, 2023;
originally announced May 2023.
-
Bayesian at heart: Towards autonomic outflow estimation via generative state-space modelling of heart rate dynamics
Authors:
Fernando E. Rosas,
Diego Candia-Rivera,
Andrea I Luppi,
Yike Guo,
Pedro A. M. Mediano
Abstract:
Recent research is revealing how cognitive processes are supported by a complex interplay between the brain and the rest of the body, which can be investigated by the analysis of physiological features such as breathing rhythms, heart rate, and skin conductance. Heart rate dynamics are of particular interest as they provide a way to track the sympathetic and parasympathetic outflow from the autono…
▽ More
Recent research is revealing how cognitive processes are supported by a complex interplay between the brain and the rest of the body, which can be investigated by the analysis of physiological features such as breathing rhythms, heart rate, and skin conductance. Heart rate dynamics are of particular interest as they provide a way to track the sympathetic and parasympathetic outflow from the autonomic nervous system, which is known to play a key role in modulating attention, memory, decision-making, and emotional processing. However, extracting useful information from heartbeats about the autonomic outflow is still challenging due to the noisy estimates that result from standard signal-processing methods. To advance this state of affairs, we propose a paradigm shift in how we conceptualise and model heart rate: instead of being a mere summary of the observed inter-beat intervals, we introduce a modelling framework that views heart rate as a hidden stochastic process that drives the observed heartbeats. Moreover, by leveraging the rich literature of state-space modelling and Bayesian inference, our proposed framework delivers a description of heart rate dynamics that is not a point estimate but a posterior distribution of a generative model. We illustrate the capabilities of our method by showing that it recapitulates linear properties of conventional heart rate estimators, while exhibiting a better discriminative power for metrics of dynamical complexity compared across different physiological states.
△ Less
Submitted 8 March, 2023;
originally announced March 2023.
-
Thermodynamics of exponential Kolmogorov-Nagumo averages
Authors:
Pablo A. Morales,
Jan Korbel,
Fernando E. Rosas
Abstract:
This paper investigates generalized thermodynamic relationships in physical systems where relevant macroscopic variables are determined by the exponential Kolmogorov-Nagumo average. We show that while the thermodynamic entropy of such systems is naturally described by Rényi's entropy with parameter $γ$, an ordinary Boltzmann distribution still describes their statistics under equilibrium thermodyn…
▽ More
This paper investigates generalized thermodynamic relationships in physical systems where relevant macroscopic variables are determined by the exponential Kolmogorov-Nagumo average. We show that while the thermodynamic entropy of such systems is naturally described by Rényi's entropy with parameter $γ$, an ordinary Boltzmann distribution still describes their statistics under equilibrium thermodynamics. Our results show that systems described by exponential Kolmogorov-Nagumo averages can be interpreted as systems originally in thermal equilibrium with a heat reservoir with inverse temperature $β$ that are suddenly quenched to another heat reservoir with inverse temperature $β' = (1-γ)β$. Furthermore, we show the connection with multifractal thermodynamics. For the non-equilibrium case, we show that the dynamics of systems described by exponential Kolmogorov-Nagumo averages still observe a second law of thermodynamics and the H-theorem. We further discuss the applications of stochastic thermodynamics in those systems -- namely, the validity of fluctuation theorems -- and the connection with thermodynamic length. namic length.
△ Less
Submitted 4 August, 2023; v1 submitted 14 February, 2023;
originally announced February 2023.
-
Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks
Authors:
Alexandra M. Proca,
Fernando E. Rosas,
Andrea I. Luppi,
Daniel Bor,
Matthew Crosby,
Pedro A. M. Mediano
Abstract:
Striking progress has recently been made in understanding human cognition by analyzing how its neuronal underpinnings are engaged in different modes of information processing. Specifically, neural information can be decomposed into synergistic, redundant, and unique features, with synergistic components being particularly aligned with complex cognition. However, two fundamental questions remain un…
▽ More
Striking progress has recently been made in understanding human cognition by analyzing how its neuronal underpinnings are engaged in different modes of information processing. Specifically, neural information can be decomposed into synergistic, redundant, and unique features, with synergistic components being particularly aligned with complex cognition. However, two fundamental questions remain unanswered: (a) precisely how and why a cognitive system can become highly synergistic; and (b) how these informational states map onto artificial neural networks in various learning modes. To address these questions, here we employ an information-decomposition framework to investigate the information processing strategies adopted by simple artificial neural networks performing a variety of cognitive tasks in both supervised and reinforcement learning settings. Our results show that synergy increases as neural networks learn multiple diverse tasks. Furthermore, performance in tasks requiring integration of multiple information sources critically relies on synergistic neurons. Finally, randomly turning off neurons during training through dropout increases network redundancy, corresponding to an increase in robustness. Overall, our results suggest that while redundant information is required for robustness to perturbations in the learning process, synergistic information is used to combine information from multiple modalities -- and more generally for flexible and efficient learning. These findings open the door to new ways of investigating how and why learning systems employ specific information-processing strategies, and support the principle that the capacity for general-purpose learning critically relies in the system's information dynamics.
△ Less
Submitted 6 October, 2022;
originally announced October 2022.
-
Gradients of O-information: low-order descriptors of high-order dependencies
Authors:
Tomas Scagliarini,
Davide Nuzzi,
Yuri Antonacci,
Luca Faes,
Fernando E. Rosas,
Daniele Marinazzo,
Sebastiano Stramaglia
Abstract:
O-information is an information-theoretic metric that captures the overall balance between redundant and synergistic information shared by groups of three or more variables. To complement the global assessment provided by this metric, here we propose the gradients of the O-information as low-order descriptors that can characterise how high-order effects are localised across a system of interest. W…
▽ More
O-information is an information-theoretic metric that captures the overall balance between redundant and synergistic information shared by groups of three or more variables. To complement the global assessment provided by this metric, here we propose the gradients of the O-information as low-order descriptors that can characterise how high-order effects are localised across a system of interest. We illustrate the capabilities of the proposed framework by revealing the role of specific spins in Ising models with frustration, and on practical data analysis on US macroeconomic data. Our theoretical and empirical analyses demonstrate the potential of these gradients to highlight the contribution of variables in forming high-order informational circuits
△ Less
Submitted 1 July, 2022;
originally announced July 2022.
-
An information-theoretic approach to hypergraph psychometrics
Authors:
Daniele Marinazzo,
Jan Van Roozendaal,
Fernando E. Rosas,
Massimo Stella,
Renzo Comolatti,
Nigel Colenbier,
Sebastiano Stramaglia,
Yves Rosseel
Abstract:
Psychological network approaches propose to see symptoms or questionnaire items as interconnected nodes, with links between them reflecting pairwise statistical dependencies evaluated cross-sectional, time-series, or panel data. These networks constitute an established methodology to assess the interactions and relative importance of nodes/indicators, providing an important complement to other app…
▽ More
Psychological network approaches propose to see symptoms or questionnaire items as interconnected nodes, with links between them reflecting pairwise statistical dependencies evaluated cross-sectional, time-series, or panel data. These networks constitute an established methodology to assess the interactions and relative importance of nodes/indicators, providing an important complement to other approaches such as factor analysis. However, focusing the modelling solely on pairwise relationships can neglect potentially critical information shared by groups of three or more variables in the form of higher-order interdependencies. To overcome this important limitation, here we propose an information-theoretic framework based on hypergraphs as psychometric models. As edges in hypergraphs are capable of encompassing several nodes together, this extension can thus provide a richer representation of the interactions that may exist among sets of psychological variables. Our results show how psychometric hypergraphs can highlight meaningful redundant and synergistic interactions on either simulated or state-of-art, re-analyzed psychometric datasets. Overall, our framework extends current network approaches while leading to new ways of assessing the data that differ at their core from other methods, extending the psychometric toolbox and opening promising avenues for future investigation.
△ Less
Submitted 13 May, 2022; v1 submitted 2 May, 2022;
originally announced May 2022.
-
Geometric Structures Induced by Deformations of the Legendre Transform
Authors:
Pablo A. Morales,
Jan Korbel,
Fernando E. Rosas
Abstract:
The recent link discovered between generalized Legendre transforms and non-dually flat statistical manifolds suggests a fundamental reason behind the ubiquity of Rényi's divergence and entropy in a wide range of physical phenomena. However, these early findings still provide little intuition on the nature of this relationship and its implications for physical systems. Here we shed new light on the…
▽ More
The recent link discovered between generalized Legendre transforms and non-dually flat statistical manifolds suggests a fundamental reason behind the ubiquity of Rényi's divergence and entropy in a wide range of physical phenomena. However, these early findings still provide little intuition on the nature of this relationship and its implications for physical systems. Here we shed new light on the Legendre transform by revealing the consequences of its deformation via symplectic geometry and complexification. These findings reveal a novel common framework that leads to a principled and unified understanding of physical systems that are not well-described by classic information-theoretic quantities.
△ Less
Submitted 18 April, 2023; v1 submitted 25 March, 2022;
originally announced March 2022.
-
Disentangling high-order mechanisms and high-order behaviours in complex systems
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Andrea I. Luppi,
Thomas F. Varley,
Joseph T. Lizier,
Sebastiano Stramaglia,
Henrik J. Jensen,
Daniele Marinazzo
Abstract:
Battiston et al. (arXiv:2110.06023) provide a comprehensive overview of how investigations of complex systems should take into account interactions between more than two elements, which can be modelled by hypergraphs and studied via topological data analysis. Following a separate line of enquiry, a broad literature has developed information-theoretic tools to characterize high-order interdependenc…
▽ More
Battiston et al. (arXiv:2110.06023) provide a comprehensive overview of how investigations of complex systems should take into account interactions between more than two elements, which can be modelled by hypergraphs and studied via topological data analysis. Following a separate line of enquiry, a broad literature has developed information-theoretic tools to characterize high-order interdependencies from observed data. While these could seem to be competing approaches aiming to address the same question, in this correspondence we clarify that this is not the case, and that a complete account of higher-order phenomena needs to embrace both.
△ Less
Submitted 21 March, 2022;
originally announced March 2022.
-
Greater than the parts: A review of the information decomposition approach to causal emergence
Authors:
Pedro A. M. Mediano,
Fernando E. Rosas,
Andrea I. Luppi,
Henrik J. Jensen,
Anil K. Seth,
Adam B. Barrett,
Robin L. Carhart-Harris,
Daniel Bor
Abstract:
Emergence is a profound subject that straddles many scientific disciplines, including the formation of galaxies and how consciousness arises from the collective activity of neurons. Despite the broad interest that exists on this concept, the study of emergence has suffered from a lack of formalisms that could be used to guide discussions and advance theories. Here we summarise, elaborate on, and e…
▽ More
Emergence is a profound subject that straddles many scientific disciplines, including the formation of galaxies and how consciousness arises from the collective activity of neurons. Despite the broad interest that exists on this concept, the study of emergence has suffered from a lack of formalisms that could be used to guide discussions and advance theories. Here we summarise, elaborate on, and extend a recent formal theory of causal emergence based on information decomposition, which is quantifiable and amenable to empirical testing. This theory relates emergence with information about a system's temporal evolution that cannot be obtained from the parts of the system separately. This article provides an accessible but rigorous introduction to the framework, discussing the merits of the approach in various scenarios of interest. We also discuss several interpretation issues and potential misunderstandings, while highlighting the distinctive benefits of this formalism.
△ Less
Submitted 11 November, 2021;
originally announced November 2021.
-
Towards an extended taxonomy of information dynamics via Integrated Information Decomposition
Authors:
Pedro A. M. Mediano,
Fernando E. Rosas,
Andrea I Luppi,
Robin L. Carhart-Harris,
Daniel Bor,
Anil K. Seth,
Adam B. Barrett
Abstract:
Complex systems, from the human brain to the global economy, are made of multiple elements that interact in such ways that the behaviour of the `whole' often seems to be more than what is readily explainable in terms of the `sum of the parts.' Our ability to understand and control these systems remains limited, one reason being that we still don't know how best to describe -- and quantify -- the h…
▽ More
Complex systems, from the human brain to the global economy, are made of multiple elements that interact in such ways that the behaviour of the `whole' often seems to be more than what is readily explainable in terms of the `sum of the parts.' Our ability to understand and control these systems remains limited, one reason being that we still don't know how best to describe -- and quantify -- the higher-order dynamical interactions that characterise their complexity. To address this limitation, we combine principles from the theories of Information Decomposition and Integrated Information into what we call Integrated Information Decomposition, or $Φ$ID. $Φ$ID provides a comprehensive framework to reason about, evaluate, and understand the information dynamics of complex multivariate systems. $Φ$ID reveals the existence of previously unreported modes of collective information flow, providing tools to express well-known measures of information transfer and dynamical complexity as aggregates of these modes. Via computational and empirical examples, we demonstrate that $Φ$ID extends our explanatory power beyond traditional causal discovery methods -- with profound implications for the study of complex systems across disciplines.
△ Less
Submitted 27 September, 2021;
originally announced September 2021.
-
Quantifying high-order interdependencies on individual patterns via the local O-information: theory and applications to music analysis
Authors:
Tomas Scagliarini,
Daniele Marinazzo,
Yike Guo,
Sebastiano Stramaglia,
Fernando E. Rosas
Abstract:
High-order, beyond-pairwise interdependencies are at the core of biological, economic, and social complex systems, and their adequate analysis is paramount to understand, engineer, and control such systems. This paper presents a framework to measure high-order interdependence that disentangles their effect on each individual pattern exhibited by a multivariate system. The approach is centred on th…
▽ More
High-order, beyond-pairwise interdependencies are at the core of biological, economic, and social complex systems, and their adequate analysis is paramount to understand, engineer, and control such systems. This paper presents a framework to measure high-order interdependence that disentangles their effect on each individual pattern exhibited by a multivariate system. The approach is centred on the 'local O-information', a new measure that assesses the balance between synergistic and redundant interdependencies at each pattern. To illustrate the potential of this framework, we present a detailed analysis of music scores from J.S. Bach, which reveals how high-order interdependence is deeply connected with highly non-trivial aspects of the musical discourse. Our results place the local O-information as a promising tool of wide applicability, which opens new perspectives for analysing high-order relationships in the patterns exhibited by complex systems.
△ Less
Submitted 26 August, 2021;
originally announced August 2021.
-
Integrated information as a common signature of dynamical and information-processing complexity
Authors:
Pedro A. M. Mediano,
Fernando E. Rosas,
Juan Carlos Farah,
Murray Shanahan,
Daniel Bor,
Adam B. Barrett
Abstract:
The apparent dichotomy between information-processing and dynamical approaches to complexity science forces researchers to choose between two diverging sets of tools and explanations, creating conflict and often hindering scientific progress. Nonetheless, given the shared theoretical goals between both approaches, it is reasonable to conjecture the existence of underlying common signatures that ca…
▽ More
The apparent dichotomy between information-processing and dynamical approaches to complexity science forces researchers to choose between two diverging sets of tools and explanations, creating conflict and often hindering scientific progress. Nonetheless, given the shared theoretical goals between both approaches, it is reasonable to conjecture the existence of underlying common signatures that capture interesting behaviour in both dynamical and information-processing systems. Here we argue that a pragmatic use of Integrated Information Theory (IIT), originally conceived in theoretical neuroscience, can provide a potential unifying framework to study complexity in general multivariate systems. Furthermore, by leveraging metrics put forward by the integrated information decomposition ($Φ$ID) framework, our results reveal that integrated information can effectively capture surprisingly heterogeneous signatures of complexity -- including metastability and criticality in networks of coupled oscillators as well as distributed computation and emergent stable particles in cellular automata -- without relying on idiosyncratic, ad-hoc criteria. These results show how an agnostic use of IIT can provide important steps towards bridging the gap between informational and dynamical approaches to complex systems.
△ Less
Submitted 18 June, 2021;
originally announced June 2021.
-
A generalization of the maximum entropy principle for curved statistical manifolds
Authors:
Pablo A. Morales,
Fernando E. Rosas
Abstract:
The maximum entropy principle (MEP) is one of the most prominent methods to investigate and model complex systems. Despite its popularity, the standard form of the MEP can only generate Boltzmann-Gibbs distributions, which are ill-suited for many scenarios of interest. As a principled approach to extend the reach of the MEP, this paper revisits its foundations in information geometry and shows how…
▽ More
The maximum entropy principle (MEP) is one of the most prominent methods to investigate and model complex systems. Despite its popularity, the standard form of the MEP can only generate Boltzmann-Gibbs distributions, which are ill-suited for many scenarios of interest. As a principled approach to extend the reach of the MEP, this paper revisits its foundations in information geometry and shows how the geometry of curved statistical manifolds naturally leads to a generalization of the MEP based on the Rényi entropy. By establishing a bridge between non-Euclidean geometry and the MEP, our proposal sets a solid foundation for the numerous applications of the Rényi entropy, and enables a range of novel methods for complex systems analysis.
△ Less
Submitted 29 March, 2022; v1 submitted 13 May, 2021;
originally announced May 2021.
-
Learning, compression, and leakage: Minimising classification error via meta-universal compression principles
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Michael Gastpar
Abstract:
Learning and compression are driven by the common aim of identifying and exploiting statistical regularities in data, which opens the door for fertile collaboration between these areas. A promising group of compression techniques for learning scenarios is normalised maximum likelihood (NML) coding, which provides strong guarantees for compression of small datasets - in contrast with more popular e…
▽ More
Learning and compression are driven by the common aim of identifying and exploiting statistical regularities in data, which opens the door for fertile collaboration between these areas. A promising group of compression techniques for learning scenarios is normalised maximum likelihood (NML) coding, which provides strong guarantees for compression of small datasets - in contrast with more popular estimators whose guarantees hold only in the asymptotic limit. Here we consider a NML-based decision strategy for supervised classification problems, and show that it attains heuristic PAC learning when applied to a wide variety of models. Furthermore, we show that the misclassification rate of our method is upper bounded by the maximal leakage, a recently proposed metric to quantify the potential of data leakage in privacy-sensitive scenarios.
△ Less
Submitted 31 January, 2021; v1 submitted 14 October, 2020;
originally announced October 2020.
-
Hyperharmonic analysis for the study of high-order information-theoretic signals
Authors:
Anibal M. Medina-Mardones,
Fernando E. Rosas,
Sebastián E. Rodríguez,
Rodrigo Cofré
Abstract:
Network representations often cannot fully account for the structural richness of complex systems spanning multiple levels of organisation. Recently proposed high-order information-theoretic signals are well-suited to capture synergistic phenomena that transcend pairwise interactions; however, the exponential-growth of their cardinality severely hinders their applicability. In this work, we combin…
▽ More
Network representations often cannot fully account for the structural richness of complex systems spanning multiple levels of organisation. Recently proposed high-order information-theoretic signals are well-suited to capture synergistic phenomena that transcend pairwise interactions; however, the exponential-growth of their cardinality severely hinders their applicability. In this work, we combine methods from harmonic analysis and combinatorial topology to construct efficient representations of high-order information-theoretic signals. The core of our method is the diagonalisation of a discrete version of the Laplace-de Rham operator, that geometrically encodes structural properties of the system. We capitalise on these ideas by develo** a complete workflow for the construction of hyperharmonic representations of high-order signals, which is applicable to a wide range of scenarios.
△ Less
Submitted 22 February, 2021; v1 submitted 2 October, 2020;
originally announced October 2020.
-
Decomposing spectral and phasic differences in non-linear features between datasets
Authors:
Pedro A. M. Mediano,
Fernando E. Rosas,
Adam B. Barrett,
Daniel Bor
Abstract:
When employing non-linear methods to characterise complex systems, it is important to determine to what extent they are capturing genuine non-linear phenomena that could not be assessed by simpler spectral methods. Specifically, we are concerned with the problem of quantifying spectral and phasic effects on an observed difference in a non-linear feature between two systems (or two states of the sa…
▽ More
When employing non-linear methods to characterise complex systems, it is important to determine to what extent they are capturing genuine non-linear phenomena that could not be assessed by simpler spectral methods. Specifically, we are concerned with the problem of quantifying spectral and phasic effects on an observed difference in a non-linear feature between two systems (or two states of the same system). Here we derive, from a sequence of null models, a decomposition of the difference in an observable into spectral, phasic, and spectrum-phase interaction components. Our approach makes no assumptions about the structure of the data and adds nuance to a wide range of time series analyses.
△ Less
Submitted 21 September, 2020;
originally announced September 2020.
-
Causal blankets: Theory and algorithmic framework
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Martin Biehl,
Shamil Chandaria,
Daniel Polani
Abstract:
We introduce a novel framework to identify perception-action loops (PALOs) directly from data based on the principles of computational mechanics. Our approach is based on the notion of causal blanket, which captures sensory and active variables as dynamical sufficient statistics -- i.e. as the "differences that make a difference." Moreover, our theory provides a broadly applicable procedure to con…
▽ More
We introduce a novel framework to identify perception-action loops (PALOs) directly from data based on the principles of computational mechanics. Our approach is based on the notion of causal blanket, which captures sensory and active variables as dynamical sufficient statistics -- i.e. as the "differences that make a difference." Moreover, our theory provides a broadly applicable procedure to construct PALOs that requires neither a steady-state nor Markovian dynamics. Using our theory, we show that every bipartite stochastic process has a causal blanket, but the extent to which this leads to an effective PALO formulation varies depending on the integrated information of the bipartition.
△ Less
Submitted 29 September, 2020; v1 submitted 28 August, 2020;
originally announced August 2020.
-
Whole-brain models to explore altered states of consciousness from the bottom up
Authors:
Rodrigo Cofré,
Rubén Herzog,
Pedro A. M. Mediano,
Juan Piccinini,
Fernando E. Rosas,
Yonatan Sanz Perl,
Enzo Tagliazucchi
Abstract:
The scope of human consciousness includes states departing from what most of us experience as ordinary wakefulness. These altered states of consciousness constitute a prime opportunity to study how global changes in brain activity relate to different varieties of subjective experience. We consider the problem of explaining how global signatures of altered consciousness arise from the interplay bet…
▽ More
The scope of human consciousness includes states departing from what most of us experience as ordinary wakefulness. These altered states of consciousness constitute a prime opportunity to study how global changes in brain activity relate to different varieties of subjective experience. We consider the problem of explaining how global signatures of altered consciousness arise from the interplay between large-scale connectivity and local dynamical rules that can be traced to known properties of neural tissue. For this purpose, we advocate a research program aimed at bridging the gap between bottom-up generative models of whole-brain activity and the top-down signatures proposed by theories of consciousness. Throughout this paper, we define altered states of consciousness, discuss relevant signatures of consciousness observed in brain activity, and introduce whole-brain models to explore the mechanisms of altered consciousness from the bottom-up. We discuss the potential of our proposal in view of the current state of the art, give specific examples of how this research agenda might play out, and emphasise how a systematic investigation of altered states of consciousness via bottom-up modelling may help us better understand the biophysical, informational, and dynamical underpinnings of consciousness.
△ Less
Submitted 6 August, 2020;
originally announced August 2020.
-
Understanding complexity via network theory: a gentle introduction
Authors:
Vaiva Vasiliauskaite,
Fernando E. Rosas
Abstract:
Network theory provides tools which are particularly appropriate for assessing the complex interdependencies that characterise our modern connected world. This article presents an introduction to network theory, in a way that doesn't require a strong mathematical background. We explore how network theory unveils commonalities in the interdependency profiles of various systems, ranging from biologi…
▽ More
Network theory provides tools which are particularly appropriate for assessing the complex interdependencies that characterise our modern connected world. This article presents an introduction to network theory, in a way that doesn't require a strong mathematical background. We explore how network theory unveils commonalities in the interdependency profiles of various systems, ranging from biological, to social, and artistic domains. Our aim is to enable an intuitive understanding while conveying the fundamental principles and aims of complexity science. Additionally, various network-theoretic tools are discussed, and numerous references for more advanced materials are provided.
△ Less
Submitted 30 April, 2020;
originally announced April 2020.
-
Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Henrik J. Jensen,
Anil K. Seth,
Adam B. Barrett,
Robin L. Carhart-Harris,
Daniel Bor
Abstract:
The broad concept of emergence is instrumental in various of the most challenging open scientific questions -- yet, few quantitative theories of what constitutes emergent phenomena have been proposed. This article introduces a formal theory of causal emergence in multivariate systems, which studies the relationship between the dynamics of parts of a system and macroscopic features of interest. Our…
▽ More
The broad concept of emergence is instrumental in various of the most challenging open scientific questions -- yet, few quantitative theories of what constitutes emergent phenomena have been proposed. This article introduces a formal theory of causal emergence in multivariate systems, which studies the relationship between the dynamics of parts of a system and macroscopic features of interest. Our theory provides a quantitative definition of downward causation, and introduces a complementary modality of emergent behaviour -- which we refer to as causal decoupling. Moreover, the theory allows practical criteria that can be efficiently calculated in large systems, making our framework applicable in a range of scenarios of practical interest. We illustrate our findings in a number of case studies, including Conway's Game of Life, Reynolds' flocking model, and neural activity as measured by electrocorticography.
△ Less
Submitted 17 April, 2020;
originally announced April 2020.
-
Data Disclosure under Perfect Sample Privacy
Authors:
Borzoo Rassouli,
Fernando E. Rosas,
Deniz Gunduz
Abstract:
Perfect data privacy seems to be in fundamental opposition to the economical and scientific opportunities associated with extensive data exchange. Defying this intuition, this paper develops a framework that allows the disclosure of collective properties of datasets without compromising the privacy of individual data samples. We present an algorithm to build an optimal disclosure strategy/map**,…
▽ More
Perfect data privacy seems to be in fundamental opposition to the economical and scientific opportunities associated with extensive data exchange. Defying this intuition, this paper develops a framework that allows the disclosure of collective properties of datasets without compromising the privacy of individual data samples. We present an algorithm to build an optimal disclosure strategy/map**, and discuss it fundamental limits on finite and asymptotically large datasets. Furthermore, we present explicit expressions to the asymptotic performance of this scheme in some scenarios, and study cases where our approach attains maximal efficiency. We finally discuss suboptimal schemes to provide sample privacy guarantees to large datasets with a reduced computational cost.
△ Less
Submitted 2 April, 2019;
originally announced April 2019.
