-
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Authors:
Srinivasan Iyer,
Xi Victoria Lin,
Ramakanth Pasunuru,
Todor Mihaylov,
Daniel Simig,
** Yu,
Kurt Shuster,
Tianlu Wang,
Qing Liu,
Punit Singh Koura,
Xian Li,
Brian O'Horo,
Gabriel Pereyra,
Jeff Wang,
Christopher Dewan,
Asli Celikyilmaz,
Luke Zettlemoyer,
Ves Stoyanov
Abstract:
Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diver…
▽ More
Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, the fine-tuning objectives themselves. In this paper, we characterize the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. To this end, we create OPT-IML Bench: a large benchmark for Instruction Meta-Learning (IML) of 2000 NLP tasks consolidated into task categories from 8 existing benchmarks, and prepare an evaluation framework to measure three types of model generalizations: to tasks from fully held-out categories, to held-out tasks from seen categories, and to held-out instances from seen tasks. Through the lens of this framework, we first present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT. OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG. Not only does it significantly outperform OPT on all benchmarks but is also highly competitive with existing models fine-tuned on each specific benchmark. We release OPT-IML at both scales, together with the OPT-IML Bench evaluation framework.
△ Less
Submitted 30 January, 2023; v1 submitted 22 December, 2022;
originally announced December 2022.
-
OPT: Open Pre-trained Transformer Language Models
Authors:
Susan Zhang,
Stephen Roller,
Naman Goyal,
Mikel Artetxe,
Moya Chen,
Shuohui Chen,
Christopher Dewan,
Mona Diab,
Xian Li,
Xi Victoria Lin,
Todor Mihaylov,
Myle Ott,
Sam Shleifer,
Kurt Shuster,
Daniel Simig,
Punit Singh Koura,
Anjali Sridhar,
Tianlu Wang,
Luke Zettlemoyer
Abstract:
Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open…
▽ More
Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.
△ Less
Submitted 21 June, 2022; v1 submitted 2 May, 2022;
originally announced May 2022.
-
PyText: A Seamless Path from NLP research to production
Authors:
Ahmed Aly,
Kushal Lakhotia,
Shicong Zhao,
Mrinal Mohit,
Barlas Oguz,
Abhinav Arora,
Sonal Gupta,
Christopher Dewan,
Stef Nelson-Lindall,
Rushin Shah
Abstract:
We introduce PyText - a deep learning based NLP modeling framework built on PyTorch. PyText addresses the often-conflicting requirements of enabling rapid experimentation and of serving models at scale. It achieves this by providing simple and extensible interfaces for model components, and by using PyTorch's capabilities of exporting models for inference via the optimized Caffe2 execution engine.…
▽ More
We introduce PyText - a deep learning based NLP modeling framework built on PyTorch. PyText addresses the often-conflicting requirements of enabling rapid experimentation and of serving models at scale. It achieves this by providing simple and extensible interfaces for model components, and by using PyTorch's capabilities of exporting models for inference via the optimized Caffe2 execution engine. We report our own experience of migrating experimentation and production workflows to PyText, which enabled us to iterate faster on novel modeling ideas and then seamlessly ship them at industrial scale.
△ Less
Submitted 12 December, 2018;
originally announced December 2018.
