Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformers
Authors:
Romain Menegaux,
Emmanuel Jehanno,
Margot Selosse,
Julien Mairal
Abstract:
We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention), which extends the notion of attention scores to attention _filters_, independently modulating the feature channels. We showcase CSA in a fully-attentional graph Transformer CGT (Chromatic Graph Transformer) which integrates both graph structural information and edge features, completely bypassing the need…
▽ More
We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention), which extends the notion of attention scores to attention _filters_, independently modulating the feature channels. We showcase CSA in a fully-attentional graph Transformer CGT (Chromatic Graph Transformer) which integrates both graph structural information and edge features, completely bypassing the need for local message-passing components. Our method flexibly encodes graph structure through node-node interactions, by enriching the original edge features with a relative positional encoding scheme. We propose a new scheme based on random walks that encodes both structural and positional information, and show how to incorporate higher-order topological information, such as rings in molecular graphs. Our approach achieves state-of-the-art results on the ZINC benchmark dataset, while providing a flexible framework for encoding graph structure and incorporating higher-order topology.
△ Less
Submitted 21 April, 2023;
originally announced April 2023.
GraphiT: Encoding Graph Structure in Transformers
Authors:
Grégoire Mialon,
Dexiong Chen,
Margot Selosse,
Julien Mairal
Abstract:
We show that viewing graphs as sets of node features and incorporating structural and positional information into a transformer architecture is able to outperform representations learned with classical graph neural networks (GNNs). Our model, GraphiT, encodes such information by (i) leveraging relative positional encoding strategies in self-attention scores based on positive definite kernels on gr…
▽ More
We show that viewing graphs as sets of node features and incorporating structural and positional information into a transformer architecture is able to outperform representations learned with classical graph neural networks (GNNs). Our model, GraphiT, encodes such information by (i) leveraging relative positional encoding strategies in self-attention scores based on positive definite kernels on graphs, and (ii) enumerating and encoding local sub-structures such as paths of short length. We thoroughly evaluate these two ideas on many classification and regression tasks, demonstrating the effectiveness of each of them independently, as well as their combination. In addition to performing well on standard benchmarks, our model also admits natural visualization mechanisms for interpreting graph motifs explaining the predictions, making it a potentially strong candidate for scientific applications where interpretation is important. Code available at https://github.com/inria-thoth/GraphiT.
△ Less
Submitted 10 June, 2021;
originally announced June 2021.
A community perspective on the concept of marine holobionts: state-of-the-art, challenges, and future directions
Authors:
Simon Dittami,
Enrique Arboleda,
Jean-Christophe Auguet,
Arite Bigalke,
Enora Briand,
Paco Cárdenas,
Ulisse Cardini,
Johan Decelle,
Ashwin Engelen,
Damien Eveillard,
Claire Gachon,
Sarah Griffiths,
Tilmann Harder,
Ehsan Kayal,
Elena Kazamia,
François Lallier,
Mónica Medina,
Ezequiel Marzinelli,
Teresa Morganti,
Laura Pons,
Soizic Prado,
José Pintado Valverde,
Mahasweta Saha,
Marc-Andre Selosse,
Derek Skillings
, et al. (6 additional authors not shown)
Abstract:
Host-microbe interactions play crucial roles in marine ecosystems, but we still have very little understanding of the mechanisms that govern these relationships, the evolutionary processes that shape them, and their ecological consequences. The holobiont concept is a renewed paradigm in biology that can help describe and understand these complex systems. It posits that a host and its associated mi…
▽ More
Host-microbe interactions play crucial roles in marine ecosystems, but we still have very little understanding of the mechanisms that govern these relationships, the evolutionary processes that shape them, and their ecological consequences. The holobiont concept is a renewed paradigm in biology that can help describe and understand these complex systems. It posits that a host and its associated microbiota, living together in a long-lasting relationship, form the holobiont, and have to be studied together, as a coherent biological and functional unit, in order to understand the biology, ecology and evolution of the organisms. Here we discuss critical concepts and opportunities in marine holobiont research and identify key challenges in the field. We highlight the potential economic, sociological, and environmental impacts of the holobiont concept in marine biological, evolutionary, and environmental sciences with comparisons to terrestrial science whenever appropriate. A deeper understanding of such complex systems, however, will require further technological and conceptual advances. The most significant challenge will be to bridge functional research on simple and tractable model systems and global approaches. This will require scientists to work together as an (inter)active community in order to address, for instance, ecological and evolutionary questions and the roles of holobionts in biogeochemical cycles.
△ Less
Submitted 11 July, 2019;
originally announced July 2019.
