General-purpose, long-context autoregressive modeling with Perceiver AR
Authors:
Curtis Hawthorne,
Andrew Jaegle,
Cătălina Cangea,
Sebastian Borgeaud,
Charlie Nash,
Mateusz Malinowski,
Sander Dieleman,
Oriol Vinyals,
Matthew Botvinick,
Ian Simon,
Hannah Sheahan,
Neil Zeghidour,
Jean-Baptiste Alayrac,
João Carreira,
Jesse Engel
Abstract:
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure. We develop Perceiver AR, an autoregressive, modality-agnostic…
▽ More
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure. We develop Perceiver AR, an autoregressive, modality-agnostic architecture which uses cross-attention to map long-range inputs to a small number of latents while also maintaining end-to-end causal masking. Perceiver AR can directly attend to over a hundred thousand tokens, enabling practical long-context density estimation without the need for hand-crafted sparsity patterns or memory mechanisms. When trained on images or music, Perceiver AR generates outputs with clear long-term coherence and structure. Our architecture also obtains state-of-the-art likelihood on long-sequence benchmarks, including 64 x 64 ImageNet images and PG-19 books.
△ Less
Submitted 14 June, 2022; v1 submitted 15 February, 2022;
originally announced February 2022.
Gradient Forward-Propagation for Large-Scale Temporal Video Modelling
Authors:
Mateusz Malinowski,
Dimitrios Vytiniotis,
Grzegorz Swirszcz,
Viorica Patraucean,
Joao Carreira
Abstract:
How can neural networks be trained on large-volume temporal data efficiently? To compute the gradients required to update parameters, backpropagation blocks computations until the forward and backward passes are completed. For temporal signals, this introduces high latency and hinders real-time learning. It also creates a coupling between consecutive layers, which limits model parallelism and incr…
▽ More
How can neural networks be trained on large-volume temporal data efficiently? To compute the gradients required to update parameters, backpropagation blocks computations until the forward and backward passes are completed. For temporal signals, this introduces high latency and hinders real-time learning. It also creates a coupling between consecutive layers, which limits model parallelism and increases memory consumption. In this paper, we build upon Sideways, which avoids blocking by propagating approximate gradients forward in time, and we propose mechanisms for temporal integration of information based on different variants of skip connections. We also show how to decouple computation and delegate individual neural modules to different devices, allowing distributed and parallel training. The proposed Skip-Sideways achieves low latency training, model parallelism, and, importantly, is capable of extracting temporal features, leading to more stable training and improved performance on real-world action recognition video datasets such as HMDB51, UCF101, and the large-scale Kinetics-600. Finally, we also show that models trained with Skip-Sideways generate better future frames than Sideways models, and hence they can better utilize motion cues.
△ Less
Submitted 12 July, 2021; v1 submitted 15 June, 2021;
originally announced June 2021.
