Mixtral of Experts
Authors:
Albert Q. Jiang,
Alexandre Sablayrolles,
Antoine Roux,
Arthur Mensch,
Blanche Savary,
Chris Bamford,
Devendra Singh Chaplot,
Diego de las Casas,
Emma Bou Hanna,
Florian Bressand,
Gianna Lengyel,
Guillaume Bour,
Guillaume Lample,
Lélio Renard Lavaud,
Lucile Saulnier,
Marie-Anne Lachaux,
Pierre Stock,
Sandeep Subramanian,
Sophia Yang,
Szymon Antoniak,
Teven Le Scao,
Théophile Gervet,
Thibaut Lavril,
Thomas Wang,
Timothée Lacroix
, et al. (1 additional authors not shown)
Abstract:
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected e…
▽ More
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.
△ Less
Submitted 8 January, 2024;
originally announced January 2024.
Mistral 7B
Authors:
Albert Q. Jiang,
Alexandre Sablayrolles,
Arthur Mensch,
Chris Bamford,
Devendra Singh Chaplot,
Diego de las Casas,
Florian Bressand,
Gianna Lengyel,
Guillaume Lample,
Lucile Saulnier,
Lélio Renard Lavaud,
Marie-Anne Lachaux,
Pierre Stock,
Teven Le Scao,
Thibaut Lavril,
Thomas Wang,
Timothée Lacroix,
William El Sayed
Abstract:
We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences o…
▽ More
We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses the Llama 2 13B -- Chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.
△ Less
Submitted 10 October, 2023;
originally announced October 2023.
