-
Is Context Helpful for Chat Translation Evaluation?
Authors:
Sweta Agrawal,
Amin Farajian,
Patrick Fernandes,
Ricardo Rei,
André F. T. Martins
Abstract:
Despite the recent success of automatic metrics for assessing translation quality, their application in evaluating the quality of machine-translated chats has been limited. Unlike more structured texts like news, chat conversations are often unstructured, short, and heavily reliant on contextual information. This poses questions about the reliability of existing sentence-level metrics in this doma…
▽ More
Despite the recent success of automatic metrics for assessing translation quality, their application in evaluating the quality of machine-translated chats has been limited. Unlike more structured texts like news, chat conversations are often unstructured, short, and heavily reliant on contextual information. This poses questions about the reliability of existing sentence-level metrics in this domain as well as the role of context in assessing the translation quality. Motivated by this, we conduct a meta-evaluation of existing sentence-level automatic metrics, primarily designed for structured domains such as news, to assess the quality of machine-translated chats. We find that reference-free metrics lag behind reference-based ones, especially when evaluating translation quality in out-of-English settings. We then investigate how incorporating conversational contextual information in these metrics affects their performance. Our findings show that augmenting neural learned metrics with contextual information helps improve correlation with human judgments in the reference-free scenario and when evaluating translations in out-of-English settings. Finally, we propose a new evaluation metric, Context-MQM, that utilizes bilingual context with a large language model (LLM) and further validate that adding context helps even for LLM-based evaluation metrics.
△ Less
Submitted 13 March, 2024;
originally announced March 2024.
-
Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
Authors:
Duarte M. Alves,
José Pombal,
Nuno M. Guerreiro,
Pedro H. Martins,
João Alves,
Amin Farajian,
Ben Peters,
Ricardo Rei,
Patrick Fernandes,
Sweta Agrawal,
Pierre Colombo,
José G. C. de Souza,
André F. T. Martins
Abstract:
While general-purpose large language models (LLMs) demonstrate proficiency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only when specializing on a single task. In this paper, we propose a recipe for tailoring LLMs to multiple tasks present in translation workflows. We perform continued pretraining on a multilingual mixture of monolingual and pa…
▽ More
While general-purpose large language models (LLMs) demonstrate proficiency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only when specializing on a single task. In this paper, we propose a recipe for tailoring LLMs to multiple tasks present in translation workflows. We perform continued pretraining on a multilingual mixture of monolingual and parallel data, creating TowerBase, followed by finetuning on instructions relevant for translation processes, creating TowerInstruct. Our final model surpasses open alternatives on several tasks relevant to translation workflows and is competitive with general-purpose closed LLMs. To facilitate future research, we release the Tower models, our specialization dataset, an evaluation framework for LLMs focusing on the translation ecosystem, and a collection of model generations, including ours, on our benchmark.
△ Less
Submitted 27 February, 2024;
originally announced February 2024.
-
Unbabel's Participation in the WMT19 Translation Quality Estimation Shared Task
Authors:
Fabio Kepler,
Jonay Trénous,
Marcos Treviso,
Miguel Vera,
António Góis,
M. Amin Farajian,
António V. Lopes,
André F. T. Martins
Abstract:
We present the contribution of the Unbabel team to the WMT 2019 Shared Task on Quality Estimation. We participated on the word, sentence, and document-level tracks, encompassing 3 language pairs: English-German, English-Russian, and English-French. Our submissions build upon the recent OpenKiwi framework: we combine linear, neural, and predictor-estimator systems with new transfer learning approac…
▽ More
We present the contribution of the Unbabel team to the WMT 2019 Shared Task on Quality Estimation. We participated on the word, sentence, and document-level tracks, encompassing 3 language pairs: English-German, English-Russian, and English-French. Our submissions build upon the recent OpenKiwi framework: we combine linear, neural, and predictor-estimator systems with new transfer learning approaches using BERT and XLM pre-trained models. We compare systems individually and propose new ensemble techniques for word and sentence-level predictions. We also propose a simple technique for converting word labels into document-level predictions. Overall, our submitted systems achieve the best results on all tracks and language pairs by a considerable margin.
△ Less
Submitted 11 September, 2019; v1 submitted 24 July, 2019;
originally announced July 2019.
-
Unbabel's Submission to the WMT2019 APE Shared Task: BERT-based Encoder-Decoder for Automatic Post-Editing
Authors:
António V. Lopes,
M. Amin Farajian,
Gonçalo M. Correia,
Jonay Trenous,
André F. T. Martins
Abstract:
This paper describes Unbabel's submission to the WMT2019 APE Shared Task for the English-German language pair. Following the recent rise of large, powerful, pre-trained models, we adapt the BERT pretrained model to perform Automatic Post-Editing in an encoder-decoder framework. Analogously to dual-encoder architectures we develop a BERT-based encoder-decoder (BED) model in which a single pretraine…
▽ More
This paper describes Unbabel's submission to the WMT2019 APE Shared Task for the English-German language pair. Following the recent rise of large, powerful, pre-trained models, we adapt the BERT pretrained model to perform Automatic Post-Editing in an encoder-decoder framework. Analogously to dual-encoder architectures we develop a BERT-based encoder-decoder (BED) model in which a single pretrained BERT encoder receives both the source src and machine translation tgt strings. Furthermore, we explore a conservativeness factor to constrain the APE system to perform fewer edits. As the official results show, when trained on a weighted combination of in-domain and artificial training data, our BED system with the conservativeness penalty improves significantly the translations of a strong Neural Machine Translation system by $-0.78$ and $+1.23$ in terms of TER and BLEU, respectively. Finally, our submission achieves a new state-of-the-art, ex-aequo, in English-German APE of NMT.
△ Less
Submitted 29 June, 2019; v1 submitted 30 May, 2019;
originally announced May 2019.
-
LOCV calculation of the equation of state and properties of rapidly rotating neutron stars
Authors:
A. H. Farajian,
M. Bigdeli,
S. Belbasi
Abstract:
In this paper, we have investigated the structural properties of rotating neutron stars using the numerical RNS code and the equation of states which have been calculated within the lowest order constrained variational approach. In order to calculate the equation of state of nuclear matter, we have used UV$_{14}$ $+$TNI and AV$_{18}$ potentials. Here, we have computed the maximum mass of the neutr…
▽ More
In this paper, we have investigated the structural properties of rotating neutron stars using the numerical RNS code and the equation of states which have been calculated within the lowest order constrained variational approach. In order to calculate the equation of state of nuclear matter, we have used UV$_{14}$ $+$TNI and AV$_{18}$ potentials. Here, we have computed the maximum mass of the neutron star and the corresponding equatorial radius at different angular velocity. We have also computed the structural properties of Keplerian rotating neutron star for maximum mass configuration, $M_{K}$, $R_{K}$, $f_{K}$ and $j_{max}$.
△ Less
Submitted 11 May, 2018; v1 submitted 12 April, 2018;
originally announced April 2018.
-
Hydrogen Compounds of Group-IV Nanosheets
Authors:
L. C. Lew Yan Voon,
E. Sandberg,
R. S. Aga,
A. A. Farajian
Abstract:
The structural and electronic properties of the hydrides of silicene and germanene have been studied using ab initio calculations. The trend for the M-H (M=C, Si, Ge) bond lengths, and corresponding bond energies, is consistent with the atomic size trend, and comparable to those of MH_4 hydrides. Band structures were also obtained for the buckled configuration, which is the stable form for both si…
▽ More
The structural and electronic properties of the hydrides of silicene and germanene have been studied using ab initio calculations. The trend for the M-H (M=C, Si, Ge) bond lengths, and corresponding bond energies, is consistent with the atomic size trend, and comparable to those of MH_4 hydrides. Band structures were also obtained for the buckled configuration, which is the stable form for both silicene and germanene. Upon hydrogenation, both silicane (indirect gap) and germanane (direct gap) are semiconducting.
△ Less
Submitted 13 July, 2010;
originally announced July 2010.
-
Vacuum polarization in nanotubes
Authors:
K. Sasaki,
A. A. Farajian,
H. Mizuseki,
Y. Kawazoe
Abstract:
This paper has been withdrawn by the author due to a crucial error.
This paper has been withdrawn by the author due to a crucial error.
△ Less
Submitted 24 June, 2003; v1 submitted 5 July, 2002;
originally announced July 2002.
-
Effective Screening of Localized Charged Perturbations in Metallic Nanotubes: Roles of Massive Bands
Authors:
K. Sasaki,
A. A. Farajian,
H. Mizuseki,
Y. Kawazoe
Abstract:
The massive-band effects on screening behavior of metallic carbon nanotubes are theoretically investigated using two different methods; continuous and lattice quantum theories. Both approaches show screening of a localized external perturbation with an effective screening length of the order of the nanotube diameter. Calculating the nonlinear deformation of the local density of states near the c…
▽ More
The massive-band effects on screening behavior of metallic carbon nanotubes are theoretically investigated using two different methods; continuous and lattice quantum theories. Both approaches show screening of a localized external perturbation with an effective screening length of the order of the nanotube diameter. Calculating the nonlinear deformation of the local density of states near the charged perturbation, we show that the perturbative effects of the massive bands are effectively canceled by direct massive band interactions, such that a good agreement between the two methods can be achieved. The effective screening is important in nanoscale integration of nanotube-based electronic devices.
△ Less
Submitted 16 April, 2003; v1 submitted 30 May, 2002;
originally announced May 2002.
-
Nonlinear charging, and transport times in doped nanotubes junctions
Authors:
Keivan Esfarjani,
Amir A. Farajian,
Siu Tat Chui,
Yoshiyuki Kawazoe
Abstract:
The nonlinear capacitance in doped nanotube junctions is calculated self consistently. It decreases as a function of the applied bias when the latter becomes larger than the pseudogap of the nanotube. For this device, one can deduce a relaxation time of about 0.1 femtosecond. Because of its negative differential resistance (NDR), a switching time of less than a fs can also be deduced.
The nonlinear capacitance in doped nanotube junctions is calculated self consistently. It decreases as a function of the applied bias when the latter becomes larger than the pseudogap of the nanotube. For this device, one can deduce a relaxation time of about 0.1 femtosecond. Because of its negative differential resistance (NDR), a switching time of less than a fs can also be deduced.
△ Less
Submitted 10 September, 2003; v1 submitted 29 April, 2002;
originally announced April 2002.
