-
Diffusion Models for Video Prediction and Infilling
Authors:
Tobias Höppe,
Arash Mehrjou,
Stefan Bauer,
Didrik Nielsen,
Andrea Dittadi
Abstract:
Predicting and anticipating future outcomes or reasoning about missing information in a sequence are critical skills for agents to be able to make intelligent decisions. This requires strong, temporally coherent generative capabilities. Diffusion models have shown remarkable success in several generative tasks, but have not been extensively explored in the video domain. We present Random-Mask Vide…
▽ More
Predicting and anticipating future outcomes or reasoning about missing information in a sequence are critical skills for agents to be able to make intelligent decisions. This requires strong, temporally coherent generative capabilities. Diffusion models have shown remarkable success in several generative tasks, but have not been extensively explored in the video domain. We present Random-Mask Video Diffusion (RaMViD), which extends image diffusion models to videos using 3D convolutions, and introduces a new conditioning technique during training. By varying the mask we condition on, the model is able to perform video prediction, infilling, and upsampling. Due to our simple conditioning scheme, we can utilize the same architecture as used for unconditional training, which allows us to train the model in a conditional and unconditional fashion at the same time. We evaluate RaMViD on two benchmark datasets for video prediction, on which we achieve state-of-the-art results, and one for video generation. High-resolution videos are provided at https://sites.google.com/view/video-diffusion-prediction.
△ Less
Submitted 14 November, 2022; v1 submitted 15 June, 2022;
originally announced June 2022.
-
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Authors:
Leo Gao,
Stella Biderman,
Sid Black,
Laurence Golding,
Travis Hoppe,
Charles Foster,
Jason Phang,
Horace He,
Anish Thite,
Noa Nabeshima,
Shawn Presser,
Connor Leahy
Abstract:
Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present \textit{the Pile}: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and new…
▽ More
Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present \textit{the Pile}: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations. Through an in-depth exploratory analysis, we document potentially concerning aspects of the data for prospective users. We make publicly available the code used in its construction.
△ Less
Submitted 31 December, 2020;
originally announced January 2021.
-
QURATOR: Innovative Technologies for Content and Data Curation
Authors:
Georg Rehm,
Peter Bourgonje,
Stefanie Hegele,
Florian Kintzel,
Julián Moreno Schneider,
Malte Ostendorff,
Karolina Zaczynska,
Armin Berger,
Stefan Grill,
Sören Räuchle,
Jens Rauenbusch,
Lisa Rutenburg,
André Schmidt,
Mikka Wild,
Henry Hoffmann,
Julian Fink,
Sarah Schulz,
Jurica Seva,
Joachim Quantz,
Joachim Böttger,
Josefine Matthey,
Rolf Fricke,
Jan Thomsen,
Adrian Paschke,
Jamal Al Qundus
, et al. (15 additional authors not shown)
Abstract:
In all domains and sectors, the demand for intelligent systems to support the processing and generation of digital content is rapidly increasing. The availability of vast amounts of content and the pressure to publish new content quickly and in rapid succession requires faster, more efficient and smarter processing and generation methods. With a consortium of ten partners from research and industr…
▽ More
In all domains and sectors, the demand for intelligent systems to support the processing and generation of digital content is rapidly increasing. The availability of vast amounts of content and the pressure to publish new content quickly and in rapid succession requires faster, more efficient and smarter processing and generation methods. With a consortium of ten partners from research and industry and a broad range of expertise in AI, Machine Learning and Language Technologies, the QURATOR project, funded by the German Federal Ministry of Education and Research, develops a sustainable and innovative technology platform that provides services to support knowledge workers in various industries to address the challenges they face when curating digital content. The project's vision and ambition is to establish an ecosystem for content curation technologies that significantly pushes the current state of the art and transforms its region, the metropolitan area Berlin-Brandenburg, into a global centre of excellence for curation technologies.
△ Less
Submitted 25 April, 2020;
originally announced April 2020.
