-
Memory-Augmented Generative Adversarial Transformers
Authors:
Stephan Raaijmakers,
Roos Bakker,
Anita Cremers,
Roy de Kleijn,
Tom Kouwenhoven,
Tessa Verhoef
Abstract:
Conversational AI systems that rely on Large Language Models, like Transformers, have difficulty interweaving external data (like facts) with the language they generate. Vanilla Transformer architectures are not designed for answering factual questions with high accuracy. This paper investigates a possible route for addressing this problem. We propose to extend the standard Transformer architectur…
▽ More
Conversational AI systems that rely on Large Language Models, like Transformers, have difficulty interweaving external data (like facts) with the language they generate. Vanilla Transformer architectures are not designed for answering factual questions with high accuracy. This paper investigates a possible route for addressing this problem. We propose to extend the standard Transformer architecture with an additional memory bank holding extra information (such as facts drawn from a knowledge base), and an extra attention layer for addressing this memory. We add this augmented memory to a Generative Adversarial Network-inspired Transformer architecture. This setup allows for implementing arbitrary felicity conditions on the generated language of the Transformer. We first demonstrate how this machinery can be deployed for handling factual questions in goal-oriented dialogues. Secondly, we demonstrate that our approach can be useful for applications like {\it style adaptation} as well: the adaptation of utterances according to certain stylistic (external) constraints, like social properties of human interlocutors in dialogues.
△ Less
Submitted 29 February, 2024;
originally announced February 2024.
-
Hierarchical network structure as the source of hierarchical dynamics (power law frequency spectra) in living and non-living systems: how state-trait continua (body plans, personalities) emerge from first principles in biophysics
Authors:
Rutger Goekoop,
Roy de Kleijn
Abstract:
Living systems are hierarchical control systems that display a small world network structure, in which many smaller clusters are nested within fewer larger ones, producing a fractal-like structure with a power-law cluster size distribution (a mereology). Apart from their structure, the dynamics of living systems also shows fractal-like qualities: the timeseries of inner message passing and overt b…
▽ More
Living systems are hierarchical control systems that display a small world network structure, in which many smaller clusters are nested within fewer larger ones, producing a fractal-like structure with a power-law cluster size distribution (a mereology). Apart from their structure, the dynamics of living systems also shows fractal-like qualities: the timeseries of inner message passing and overt behavior contain high frequencies or states (treble) that are nested within lower frequencies or traits (bass), producing a power-law frequency spectrum that is known as a state-trait continuum in the behavioral sciences. Here, we argue that the power-law dynamics of living systems results from their power-law network structure: organisms vertically encode the deep spatiotemporal structure of their (anticipated) environments, to the effect that many small clusters near the base of the hierarchy produce high frequency signal changes and fewer larger clusters at its top produce ultra-low frequencies. Such ultra-low frequencies produce physical as well as behavioral traits (i.e. body plans and personalities). Nested-modular structure then causes higher frequencies to be embedded within lower frequencies, producing a power law state-trait continuum. At the heart of such dynamics lies the need for efficient energy dissipation through networks of coupled oscillators, which also governs the dynamics of non-living systems (e.g. earthquakes, stock market fluctuations). Since hierarchical structure produces hierarchical dynamics, the development and collapse of hierarchical structure (e.g. during maturation and disease) should leave specific traces in the dynamics of nested modular systems that may serve as early warning signs to system failure. The applications of this idea range from (bio)physics and phylogenesis to ontogenesis and clinical medicine.
△ Less
Submitted 21 June, 2023; v1 submitted 14 April, 2023;
originally announced April 2023.
-
How higher goals are constructed and collapse under stress: a hierarchical Bayesian control systems perspective
Authors:
Rutger Goekoop,
Roy de Kleijn
Abstract:
In this paper, we show that organisms can be modeled as hierarchical Bayesian control systems with small world and information bottleneck (bow-tie) network structure. Such systems combine hierarchical perception with hierarchical goal setting and hierarchical action control. We argue that hierarchical Bayesian control systems produce deep hierarchies of goal states, from which it follows that orga…
▽ More
In this paper, we show that organisms can be modeled as hierarchical Bayesian control systems with small world and information bottleneck (bow-tie) network structure. Such systems combine hierarchical perception with hierarchical goal setting and hierarchical action control. We argue that hierarchical Bayesian control systems produce deep hierarchies of goal states, from which it follows that organisms must have some form of 'highest goals'. For all organisms, these involve internal (self) models, external (social) models and overarching (normative) models. We show that goal hierarchies tend to decompose in a top-down manner under severe and prolonged levels of stress. This produces behavior that favors short-term and self-referential goals over long term, social and/or normative goals. The collapse of goal hierarchies is universally accompanied by an increase in entropy (disorder) in control systems that can serve as an early warning sign for tip** points (disease or death of the organism). In humans, learning goal hierarchies corresponds to personality development (maturation). The failure of goal hierarchies to mature properly corresponds to personality deficits. A top-down collapse of such hierarchies under stress is identified as a common factor in all forms of episodic mental disorders (psychopathology). The paper concludes by discussing ways of testing these hypotheses empirically.
△ Less
Submitted 2 February, 2021; v1 submitted 20 April, 2020;
originally announced April 2020.
