-
Efficient Large Scale Language Modeling with Mixtures of Experts
Authors:
Mikel Artetxe,
Shruti Bhosale,
Naman Goyal,
Todor Mihaylov,
Myle Ott,
Sam Shleifer,
Xi Victoria Lin,
**gfei Du,
Srinivasan Iyer,
Ramakanth Pasunuru,
Giri Anantharaman,
Xian Li,
Shuohui Chen,
Halil Akin,
Mandeep Baines,
Louis Martin,
Xing Zhou,
Punit Singh Koura,
Brian O'Horo,
Jeff Wang,
Luke Zettlemoyer,
Mona Diab,
Zornitsa Kozareva,
Ves Stoyanov
Abstract:
Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning, we…
▽ More
Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning, we find MoEs to be substantially more compute efficient. At more modest training budgets, MoEs can match the performance of dense models using $\sim$4 times less compute. This gap narrows at scale, but our largest MoE model (1.1T parameters) consistently outperforms a compute-equivalent dense model (6.7B parameters). Overall, this performance gap varies greatly across tasks and domains, suggesting that MoE and dense models generalize differently in ways that are worthy of future study. We make our code and models publicly available for research use.
△ Less
Submitted 26 October, 2022; v1 submitted 20 December, 2021;
originally announced December 2021.
-
Beyond English-Centric Multilingual Machine Translation
Authors:
Angela Fan,
Shruti Bhosale,
Holger Schwenk,
Zhiyi Ma,
Ahmed El-Kishky,
Siddharth Goyal,
Mandeep Baines,
Onur Celebi,
Guillaume Wenzek,
Vishrav Chaudhary,
Naman Goyal,
Tom Birch,
Vitaliy Liptchinsky,
Sergey Edunov,
Edouard Grave,
Michael Auli,
Armand Joulin
Abstract:
Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages. However, much of this work is English-Centric by training only on data which was translated from or to English. While this is supported by large sources of training data, it does not reflect translation needs worldwide. In…
▽ More
Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages. However, much of this work is English-Centric by training only on data which was translated from or to English. While this is supported by large sources of training data, it does not reflect translation needs worldwide. In this work, we create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. We build and open source a training dataset that covers thousands of language directions with supervised data, created through large-scale mining. Then, we explore how to effectively increase model capacity through a combination of dense scaling and language-specific sparse parameters to create high quality models. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively to the best single systems of WMT. We open-source our scripts so that others may reproduce the data, evaluation, and final M2M-100 model.
△ Less
Submitted 21 October, 2020;
originally announced October 2020.
-
Capillary transport in low saturated sands: superfast non-linear diffusion model versus direct experimental observations
Authors:
Alex V. Lukyanov,
Vladimir Mitkin,
Theo G. Theofanous,
Mike Baines
Abstract:
We have established in a pilot study, that the spreading of liquids in sandy porous materials at low levels of saturation, typically less than ten percent of the available void space, has very distinctive features in comparison to that at higher saturation levels. In particular, it has been shown, on theoretical grounds, that the spreading is controlled by a special type of diffusional process, th…
▽ More
We have established in a pilot study, that the spreading of liquids in sandy porous materials at low levels of saturation, typically less than ten percent of the available void space, has very distinctive features in comparison to that at higher saturation levels. In particular, it has been shown, on theoretical grounds, that the spreading is controlled by a special type of diffusional process, that its physics can be captured by an equation of the super-fast diffusion class, and that these findings were supported by first-of-a-kind experiments. In this paper, we take these findings to the next level including deeper examination and exposition of the theory, an expanded set of experiments to address scaling properties, and systematic evaluations of the predictive performance against these experimental data, kee** in mind also potential practical applications.
△ Less
Submitted 11 December, 2018;
originally announced December 2018.
-
Superfast non-linear diffusion: Capillary transport in particulate porous media
Authors:
A. V. Lukyanov,
M. M. Sushchikh,
M. J. Baines,
T. G. Theofanous
Abstract:
The migration of liquids in porous media, such as sand, has been commonly considered at high saturation levels with liquid pathways at pore dimensions. In this letter we reveal a low saturation regime observed in our experiments with droplets of extremely low volatility liquids deposited on sand. In this regime the liquid is mostly found within the grain surface roughness and in the capillary brid…
▽ More
The migration of liquids in porous media, such as sand, has been commonly considered at high saturation levels with liquid pathways at pore dimensions. In this letter we reveal a low saturation regime observed in our experiments with droplets of extremely low volatility liquids deposited on sand. In this regime the liquid is mostly found within the grain surface roughness and in the capillary bridges formed at the contacts between the grains. The bridges act as variable-volume reservoirs and the flow is driven by the capillary pressure arising at the wetting front according to the roughness length scales. We propose that this migration (spreading) is the result of interplay between the bridge volume adjustment to this pressure distribution and viscous losses of a cree** flow within the roughness. The net macroscopic result is a special case of non-linear diffusion described by a superfast diffusion equation (SFDE) for saturation with distinctive mathematical character. We obtain solutions to a moving boundary problem defined by SFDE that robustly convey a time power law of spreading as seen in our experiments.
△ Less
Submitted 30 October, 2013;
originally announced October 2013.
