-
DataComp-LM: In search of the next generation of training sets for language models
Authors:
Jeffrey Li,
Alex Fang,
Georgios Smyrnis,
Maor Ivgi,
Matt Jordan,
Samir Gadre,
Hritik Bansal,
Etash Guha,
Sedrick Keh,
Kushal Arora,
Saurabh Garg,
Rui Xin,
Niklas Muennighoff,
Reinhard Heckel,
Jean Mercat,
Mayee Chen,
Suchin Gururangan,
Mitchell Wortsman,
Alon Albalak,
Yonatan Bitton,
Marianna Nezhurina,
Amro Abbas,
Cheng-Yu Hsieh,
Dhruba Ghosh,
Josh Gardner
, et al. (34 additional authors not shown)
Abstract:
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with dat…
▽ More
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters. As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set. The resulting dataset, DCLM-Baseline enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-Baseline represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6x less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation.
△ Less
Submitted 20 June, 2024; v1 submitted 17 June, 2024;
originally announced June 2024.
-
Computational Tradeoffs in Image Synthesis: Diffusion, Masked-Token, and Next-Token Prediction
Authors:
Maciej Kilian,
Varun Jampani,
Luke Zettlemoyer
Abstract:
Nearly every recent image synthesis approach, including diffusion, masked-token prediction, and next-token prediction, uses a Transformer network architecture. Despite this common backbone, there has been no direct, compute controlled comparison of how these approaches affect performance and efficiency. We analyze the scalability of each approach through the lens of compute budget measured in FLOP…
▽ More
Nearly every recent image synthesis approach, including diffusion, masked-token prediction, and next-token prediction, uses a Transformer network architecture. Despite this common backbone, there has been no direct, compute controlled comparison of how these approaches affect performance and efficiency. We analyze the scalability of each approach through the lens of compute budget measured in FLOPs. We find that token prediction methods, led by next-token prediction, significantly outperform diffusion on prompt following. On image quality, while next-token prediction initially performs better, scaling trends suggest it is eventually matched by diffusion. We compare the inference compute efficiency of each approach and find that next token prediction is by far the most efficient. Based on our findings we recommend diffusion for applications targeting image quality and low latency; and next-token prediction when prompt following or throughput is more important.
△ Less
Submitted 24 May, 2024; v1 submitted 21 May, 2024;
originally announced May 2024.
-
Shape Reconstruction of Trapezoidal Surfaces
Authors:
Arvin Rasoulzadeh,
Martin Kilian,
Georg Nawratil
Abstract:
A smooth T-surface can be thought of as a generalization of a surface of revolution in such a way that the axis of rotation is not fixed at one point but rather traces a smooth path on the base plane. Furthermore, the action, by which the aforementioned surface is obtained does not need to be merely rotation but any ``suitable" planar equiform transformation applied to the points of a certain smoo…
▽ More
A smooth T-surface can be thought of as a generalization of a surface of revolution in such a way that the axis of rotation is not fixed at one point but rather traces a smooth path on the base plane. Furthermore, the action, by which the aforementioned surface is obtained does not need to be merely rotation but any ``suitable" planar equiform transformation applied to the points of a certain smooth profile curve. In analogy to the smooth setting, if the axis footpoints sweep a polyline on the base plane and if the profile curve is discretely chosen then a T-hedra (discrete T-surface) with trapezoidal faces is obtained. The goal of this article is to reconstruct a T-hedron from an already given point cloud of a T-surface. In doing so, a kinematic approach is taken into account, where the algorithm at first tries to find the aforementioned axis direction associated with the point cloud. Then the algorithm finds a polygonal path through which the axis footpoint moves. Finally, by properly cutting the point cloud with the planes passing through the axis and its footpoints, it reconstructs the surface. The presented method is demonstrated on base of examples. From an applied point of view, the straightforwardness of the generation of these surfaces predestines them for building and design processes. In fact, one can find many built objects belonging to the sub-classes of T-surfaces such as \emph{surfaces of revolution} and \emph{moulding surfaces}. Furthermore, the planarity of the faces of the discrete version paves the way for steel/glass construction in industry. Finally, these surfaces are also suitable for transformable designs as they allow an isometric deformation.
△ Less
Submitted 14 December, 2023;
originally announced December 2023.
-
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
Authors:
Andreas Blattmann,
Tim Dockhorn,
Sumith Kulal,
Daniel Mendelevitch,
Maciej Kilian,
Dominik Lorenz,
Yam Levi,
Zion English,
Vikram Voleti,
Adam Letts,
Varun Jampani,
Robin Rombach
Abstract:
We present Stable Video Diffusion - a latent video diffusion model for high-resolution, state-of-the-art text-to-video and image-to-video generation. Recently, latent diffusion models trained for 2D image synthesis have been turned into generative video models by inserting temporal layers and finetuning them on small, high-quality video datasets. However, training methods in the literature vary wi…
▽ More
We present Stable Video Diffusion - a latent video diffusion model for high-resolution, state-of-the-art text-to-video and image-to-video generation. Recently, latent diffusion models trained for 2D image synthesis have been turned into generative video models by inserting temporal layers and finetuning them on small, high-quality video datasets. However, training methods in the literature vary widely, and the field has yet to agree on a unified strategy for curating video data. In this paper, we identify and evaluate three different stages for successful training of video LDMs: text-to-image pretraining, video pretraining, and high-quality video finetuning. Furthermore, we demonstrate the necessity of a well-curated pretraining dataset for generating high-quality videos and present a systematic curation process to train a strong base model, including captioning and filtering strategies. We then explore the impact of finetuning our base model on high-quality data and train a text-to-video model that is competitive with closed-source video generation. We also show that our base model provides a powerful motion representation for downstream tasks such as image-to-video generation and adaptability to camera motion-specific LoRA modules. Finally, we demonstrate that our model provides a strong multi-view 3D-prior and can serve as a base to finetune a multi-view diffusion model that jointly generates multiple views of objects in a feedforward fashion, outperforming image-based methods at a fraction of their compute budget. We release code and model weights at https://github.com/Stability-AI/generative-models .
△ Less
Submitted 25 November, 2023;
originally announced November 2023.
