-
Better & Faster Large Language Models via Multi-token Prediction
Authors:
Fabian Gloeckle,
Badr Youbi Idrissi,
Baptiste Rozière,
David Lopez-Paz,
Gabriel Synnaeve
Abstract:
Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared m…
▽ More
Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared model trunk. Considering multi-token prediction as an auxiliary training task, we measure improved downstream capabilities with no overhead in training time for both code and natural language models. The method is increasingly useful for larger model sizes, and keeps its appeal when training for multiple epochs. Gains are especially pronounced on generative benchmarks like coding, where our models consistently outperform strong baselines by several percentage points. Our 13B parameter models solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models. Experiments on small algorithmic tasks demonstrate that multi-token prediction is favorable for the development of induction heads and algorithmic reasoning capabilities. As an additional benefit, models trained with 4-token prediction are up to 3 times faster at inference, even with large batch sizes.
△ Less
Submitted 30 April, 2024;
originally announced April 2024.
-
Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Data
Authors:
Yu Yang,
Aaditya K. Singh,
Mostafa Elhoushi,
Anas Mahmoud,
Kushal Tirumala,
Fabian Gloeckle,
Baptiste Rozière,
Carole-Jean Wu,
Ari S. Morcos,
Newsha Ardalani
Abstract:
Code datasets, often collected from diverse and uncontrolled sources such as GitHub, potentially suffer from quality issues, thereby affecting the performance and training efficiency of Large Language Models (LLMs) optimized for code generation. Previous studies demonstrated the benefit of using embedding spaces for data pruning, but they mainly focused on duplicate removal or increasing variety,…
▽ More
Code datasets, often collected from diverse and uncontrolled sources such as GitHub, potentially suffer from quality issues, thereby affecting the performance and training efficiency of Large Language Models (LLMs) optimized for code generation. Previous studies demonstrated the benefit of using embedding spaces for data pruning, but they mainly focused on duplicate removal or increasing variety, and in other modalities, such as images. Our work focuses on using embeddings to identify and remove "low-quality" code data. First, we explore features of "low-quality" code in embedding space, through the use of synthetic corruptions. Armed with this knowledge, we devise novel pruning metrics that operate in embedding space to identify and remove low-quality entries in the Stack dataset. We demonstrate the benefits of this synthetic corruption informed pruning (SCIP) approach on the well-established HumanEval and MBPP benchmarks, outperforming existing embedding-based methods. Importantly, we achieve up to a 3% performance improvement over no pruning, thereby showing the promise of insights from synthetic corruptions for data pruning.
△ Less
Submitted 4 December, 2023;
originally announced December 2023.
-
Large Language Models for Compiler Optimization
Authors:
Chris Cummins,
Volker Seeker,
Dejan Grubisic,
Mostafa Elhoushi,
Youwei Liang,
Baptiste Roziere,
Jonas Gehring,
Fabian Gloeckle,
Kim Hazelwood,
Gabriel Synnaeve,
Hugh Leather
Abstract:
We explore the novel application of Large Language Models to code optimization. We present a 7B-parameter transformer model trained from scratch to optimize LLVM assembly for code size. The model takes as input unoptimized assembly and outputs a list of compiler options to best optimize the program. Crucially, during training, we ask the model to predict the instruction counts before and after opt…
▽ More
We explore the novel application of Large Language Models to code optimization. We present a 7B-parameter transformer model trained from scratch to optimize LLVM assembly for code size. The model takes as input unoptimized assembly and outputs a list of compiler options to best optimize the program. Crucially, during training, we ask the model to predict the instruction counts before and after optimization, and the optimized code itself. These auxiliary learning tasks significantly improve the optimization performance of the model and improve the model's depth of understanding.
We evaluate on a large suite of test programs. Our approach achieves a 3.0% improvement in reducing instruction counts over the compiler, outperforming two state-of-the-art baselines that require thousands of compilations. Furthermore, the model shows surprisingly strong code reasoning abilities, generating compilable code 91% of the time and perfectly emulating the output of the compiler 70% of the time.
△ Less
Submitted 11 September, 2023;
originally announced September 2023.
-
Code Llama: Open Foundation Models for Code
Authors:
Baptiste Rozière,
Jonas Gehring,
Fabian Gloeckle,
Sten Sootla,
Itai Gat,
Xiaoqing Ellen Tan,
Yossi Adi,
**gyu Liu,
Romain Sauvestre,
Tal Remez,
Jérémy Rapin,
Artyom Kozhevnikov,
Ivan Evtimov,
Joanna Bitton,
Manish Bhatt,
Cristian Canton Ferrer,
Aaron Grattafiori,
Wenhan Xiong,
Alexandre Défossez,
Jade Copet,
Faisal Azhar,
Hugo Touvron,
Louis Martin,
Nicolas Usunier,
Thomas Scialom
, et al. (1 additional authors not shown)
Abstract:
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama…
▽ More
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.
△ Less
Submitted 31 January, 2024; v1 submitted 24 August, 2023;
originally announced August 2023.
