-
Code Llama: Open Foundation Models for Code
Authors:
Baptiste Rozière,
Jonas Gehring,
Fabian Gloeckle,
Sten Sootla,
Itai Gat,
Xiaoqing Ellen Tan,
Yossi Adi,
**gyu Liu,
Romain Sauvestre,
Tal Remez,
Jérémy Rapin,
Artyom Kozhevnikov,
Ivan Evtimov,
Joanna Bitton,
Manish Bhatt,
Cristian Canton Ferrer,
Aaron Grattafiori,
Wenhan Xiong,
Alexandre Défossez,
Jade Copet,
Faisal Azhar,
Hugo Touvron,
Louis Martin,
Nicolas Usunier,
Thomas Scialom
, et al. (1 additional authors not shown)
Abstract:
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama…
▽ More
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.
△ Less
Submitted 31 January, 2024; v1 submitted 24 August, 2023;
originally announced August 2023.
-
PLCMOS -- a data-driven non-intrusive metric for the evaluation of packet loss concealment algorithms
Authors:
Lorenz Diener,
Marju Purin,
Sten Sootla,
Ando Saabas,
Robert Aichner,
Ross Cutler
Abstract:
Speech quality assessment is a problem for every researcher working on models that produce or process speech. Human subjective ratings, the gold standard in speech quality assessment, are expensive and time-consuming to acquire in a quantity that is sufficient to get reliable data, while automated objective metrics show a low correlation with gold standard ratings. This paper presents PLCMOS, a no…
▽ More
Speech quality assessment is a problem for every researcher working on models that produce or process speech. Human subjective ratings, the gold standard in speech quality assessment, are expensive and time-consuming to acquire in a quantity that is sufficient to get reliable data, while automated objective metrics show a low correlation with gold standard ratings. This paper presents PLCMOS, a non-intrusive data-driven tool for generating a robust, accurate estimate of the mean opinion score a human rater would assign an audio file that has been processed by being transmitted over a degraded packet-switched network with missing packets being healed by a packet loss concealment algorithm. Our new model shows a model-wise Pearson's correlation of ~0.97 and rank correlation of ~0.95 with human ratings, substantially above all other available intrusive and non-intrusive metrics. The model is released as an ONNX model for other researchers to use when building PLC systems.
△ Less
Submitted 24 May, 2023;
originally announced May 2023.
-
INTERSPEECH 2022 Audio Deep Packet Loss Concealment Challenge
Authors:
Lorenz Diener,
Sten Sootla,
Solomiya Branets,
Ando Saabas,
Robert Aichner,
Ross Cutler
Abstract:
Audio Packet Loss Concealment (PLC) is the hiding of gaps in audio streams caused by data transmission failures in packet switched networks. This is a common problem, and of increasing importance as end-to-end VoIP telephony and teleconference systems become the default and ever more widely used form of communication in business as well as in personal usage. This paper presents the INTERSPEECH 202…
▽ More
Audio Packet Loss Concealment (PLC) is the hiding of gaps in audio streams caused by data transmission failures in packet switched networks. This is a common problem, and of increasing importance as end-to-end VoIP telephony and teleconference systems become the default and ever more widely used form of communication in business as well as in personal usage. This paper presents the INTERSPEECH 2022 Audio Deep Packet Loss Concealment challenge. We first give an overview of the PLC problem, and introduce some classical approaches to PLC as well as recent work. We then present the open source dataset released as part of this challenge as well as the evaluation methods and metrics used to determine the winner. We also briefly introduce PLCMOS, a novel data-driven metric that can be used to quickly evaluate the performance PLC systems. Finally, we present the results of the INTERSPEECH 2022 Audio Deep PLC Challenge, and provide a summary of important takeaways.
△ Less
Submitted 11 April, 2022;
originally announced April 2022.
-
AECMOS: A speech quality assessment metric for echo impairment
Authors:
Marju Purin,
Sten Sootla,
Mateja Sponza,
Ando Saabas,
Ross Cutler
Abstract:
Traditionally, the quality of acoustic echo cancellers is evaluated using intrusive speech quality assessment measures such as ERLE \cite{g168} and PESQ \cite{p862}, or by carrying out subjective laboratory tests. Unfortunately, the former are not well correlated with human subjective measures, while the latter are time and resource consuming to carry out. We provide a new tool for speech quality…
▽ More
Traditionally, the quality of acoustic echo cancellers is evaluated using intrusive speech quality assessment measures such as ERLE \cite{g168} and PESQ \cite{p862}, or by carrying out subjective laboratory tests. Unfortunately, the former are not well correlated with human subjective measures, while the latter are time and resource consuming to carry out. We provide a new tool for speech quality assessment for echo impairment which can be used to evaluate the performance of acoustic echo cancellers. More precisely, we develop a neural network model to evaluate call quality degradations in two separate categories: echo and degradations from other sources. We show that our model is accurate as measured by correlation with human subjective quality ratings. Our tool can be used effectively to stack rank echo cancellation models. AECMOS is being made publicly available as an Azure service.
△ Less
Submitted 27 January, 2022; v1 submitted 6 October, 2021;
originally announced October 2021.
-
Crowdsourcing approach for subjective evaluation of echo impairment
Authors:
Ross Cutler,
Babak Naderi,
Markus Loide,
Sten Sootla,
Ando Saabas
Abstract:
The quality of acoustic echo cancellers (AECs) in real-time communication systems is typically evaluated using objective metrics like ERLE and PESQ, and less commonly with lab-based subjective tests like ITU-T Rec. P.831. We will show that these objective measures are not well correlated to subjective measures. We then introduce an open-source crowdsourcing approach for subjective evaluation of ec…
▽ More
The quality of acoustic echo cancellers (AECs) in real-time communication systems is typically evaluated using objective metrics like ERLE and PESQ, and less commonly with lab-based subjective tests like ITU-T Rec. P.831. We will show that these objective measures are not well correlated to subjective measures. We then introduce an open-source crowdsourcing approach for subjective evaluation of echo impairment which can be used to evaluate the performance of AECs. We provide a study that shows this tool is accurate and highly reproducible. This new tool has been recently used in the ICASSP 2021 AEC Challenge which made the challenge possible to do quickly and cost effectively.
△ Less
Submitted 27 February, 2022; v1 submitted 25 October, 2020;
originally announced October 2020.
