Transformers are efficient hierarchical chemical graph learners
Authors:
Zihan Pengmei,
Zimu Li,
Chih-chan Tien,
Risi Kondor,
Aaron R. Dinner
Abstract:
Transformers, adapted from natural language processing, are emerging as a leading approach for graph representation learning. Contemporary graph transformers often treat nodes or edges as separate tokens. This approach leads to computational challenges for even moderately-sized graphs due to the quadratic scaling of self-attention complexity with token count. In this paper, we introduce SubFormer,…
▽ More
Transformers, adapted from natural language processing, are emerging as a leading approach for graph representation learning. Contemporary graph transformers often treat nodes or edges as separate tokens. This approach leads to computational challenges for even moderately-sized graphs due to the quadratic scaling of self-attention complexity with token count. In this paper, we introduce SubFormer, a graph transformer that operates on subgraphs that aggregate information by a message-passing mechanism. This approach reduces the number of tokens and enhances learning long-range interactions. We demonstrate SubFormer on benchmarks for predicting molecular properties from chemical structures and show that it is competitive with state-of-the-art graph transformers at a fraction of the computational cost, with training times on the order of minutes on a consumer-grade graphics card. We interpret the attention weights in terms of chemical structures. We show that SubFormer exhibits limited over-smoothing and avoids over-squashing, which is prevalent in traditional graph neural networks.
△ Less
Submitted 2 October, 2023;
originally announced October 2023.
Matching Targets Across Domains with RADON, the Re-Identification Across Domain Network
Authors:
Cassandra Burgess,
Cordelia Neisinger,
Rafael Dinner
Abstract:
We present a novel convolutional neural network that learns to match images of an object taken from different viewpoints or by different optical sensors. Our Re-Identification Across Domain Network (RADON) scores pairs of input images from different domains on similarity. Our approach extends previous work on Siamese networks and modifies them to more challenging use cases, including low- and no-s…
▽ More
We present a novel convolutional neural network that learns to match images of an object taken from different viewpoints or by different optical sensors. Our Re-Identification Across Domain Network (RADON) scores pairs of input images from different domains on similarity. Our approach extends previous work on Siamese networks and modifies them to more challenging use cases, including low- and no-shot learning, in which few images of a specific target are available for training. RADON shows strong performance on cross-view vehicle matching and cross-domain person identification in a no-shot learning environment.
△ Less
Submitted 25 May, 2021;
originally announced May 2021.
