Self-Distillation Improves DNA Sequence Inference
Authors:
Tong Yu,
Lei Cheng,
Ruslan Khalitov,
Erland Brandser Olsson,
Zhirong Yang
Abstract:
Self-supervised pretraining (SSP) has been recognized as a method to enhance prediction accuracy in various downstream tasks. However, its efficacy for DNA sequences remains somewhat constrained. This limitation stems primarily from the fact that most existing SSP approaches in genomics focus on masked language modeling of individual sequences, neglecting the crucial aspect of encoding statistics…
▽ More
Self-supervised pretraining (SSP) has been recognized as a method to enhance prediction accuracy in various downstream tasks. However, its efficacy for DNA sequences remains somewhat constrained. This limitation stems primarily from the fact that most existing SSP approaches in genomics focus on masked language modeling of individual sequences, neglecting the crucial aspect of encoding statistics across multiple sequences. To overcome this challenge, we introduce an innovative deep neural network model, which incorporates collaborative learning between a `student' and a `teacher' subnetwork. In this model, the student subnetwork employs masked learning on nucleotides and progressively adapts its parameters to the teacher subnetwork through an exponential moving average approach. Concurrently, both subnetworks engage in contrastive learning, deriving insights from two augmented representations of the input sequences. This self-distillation process enables our model to effectively assimilate both contextual information from individual sequences and distributional data across the sequence population. We validated our approach with preliminary pretraining using the human reference genome, followed by applying it to 20 downstream inference tasks. The empirical results from these experiments demonstrate that our novel method significantly boosts inference performance across the majority of these tasks. Our code is available at https://github.com/wiedersehne/FinDNA.
△ Less
Submitted 14 May, 2024;
originally announced May 2024.
A matrix math facility for Power ISA(TM) processors
Authors:
José E. Moreira,
Kit Barton,
Steven Battle,
Peter Bergner,
Ramon Bertran,
Puneeth Bhat,
Pedro Caldeira,
David Edelsohn,
Gordon Fossum,
Brad Frey,
Nemanja Ivanovic,
Chip Kerchner,
Vincent Lim,
Shakti Kapoor,
Tulio Machado Filho,
Silvia Melitta Mueller,
Brett Olsson,
Satish Sadasivam,
Baptiste Saleil,
Bill Schmidt,
Rajalakshmi Srinivasaraghavan,
Shricharan Srivatsan,
Brian Thompto,
Andreas Wagner,
Nelson Wu
Abstract:
Power ISA(TM) Version 3.1 has introduced a new family of matrix math instructions, collectively known as the Matrix-Multiply Assist (MMA) facility. The instructions in this facility implement numerical linear algebra operations on small matrices and are meant to accelerate computation-intensive kernels, such as matrix multiplication, convolution and discrete Fourier transform. These instructions h…
▽ More
Power ISA(TM) Version 3.1 has introduced a new family of matrix math instructions, collectively known as the Matrix-Multiply Assist (MMA) facility. The instructions in this facility implement numerical linear algebra operations on small matrices and are meant to accelerate computation-intensive kernels, such as matrix multiplication, convolution and discrete Fourier transform. These instructions have led to a power- and area-efficient implementation of a high throughput math engine in the future POWER10 processor. Performance per core is 4 times better, at constant frequency, than the previous generation POWER9 processor. We also advocate the use of compiler built-ins as the preferred way of leveraging these instructions, which we illustrate through case studies covering matrix multiplication and convolution.
△ Less
Submitted 7 April, 2021;
originally announced April 2021.
