Information Flows? A Critique of Transfer Entropies
Authors:
Ryan G. James,
Nix Barnett,
James P. Crutchfield
Abstract:
A central task in analyzing complex dynamics is to determine the loci of information storage and the communication topology of information flows within a system. Over the last decade and a half, diagnostics for the latter have come to be dominated by the transfer entropy. Via straightforward examples, we show that it and a derivative quantity, the causation entropy, do not, in fact, quantify the f…
▽ More
A central task in analyzing complex dynamics is to determine the loci of information storage and the communication topology of information flows within a system. Over the last decade and a half, diagnostics for the latter have come to be dominated by the transfer entropy. Via straightforward examples, we show that it and a derivative quantity, the causation entropy, do not, in fact, quantify the flow of information. At one and the same time they can overestimate flow or underestimate influence. We isolate why this is the case and propose several avenues to alternate measures for information flow. We also address an auxiliary consequence: The proliferation of networks as a now-common theoretical model for large-scale systems, in concert with the use of transfer-like entropies, has shoehorned dyadic relationships into our structural interpretation of the organization and behavior of complex systems. This interpretation thus fails to include the effects of polyadic dependencies. The net result is that much of the sophisticated organization of complex systems may go undetected.
△ Less
Submitted 17 June, 2016; v1 submitted 20 December, 2015;
originally announced December 2015.
Computational Mechanics of Input-Output Processes: Structured transformations and the $ε$-transducer
Authors:
Nix Barnett,
James P. Crutchfield
Abstract:
Computational mechanics quantifies structure in a stochastic process via its causal states, leading to the process's minimal, optimal predictor---the $ε$-machine. We extend computational mechanics to communication channels between two processes, obtaining an analogous optimal model---the $ε$-transducer---of the stochastic map** between them. Here, we lay the foundation of a structural analysis o…
▽ More
Computational mechanics quantifies structure in a stochastic process via its causal states, leading to the process's minimal, optimal predictor---the $ε$-machine. We extend computational mechanics to communication channels between two processes, obtaining an analogous optimal model---the $ε$-transducer---of the stochastic map** between them. Here, we lay the foundation of a structural analysis of communication channels, treating joint processes and processes with input. The result is a principled structural analysis of mechanisms that support information flow between processes. It is the first in a series on the structural information theory of memoryful channels, channel composition, and allied conditional information measures.
△ Less
Submitted 26 January, 2016; v1 submitted 8 December, 2014;
originally announced December 2014.
