Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Models
Authors:
Abteen Ebrahimi,
Arya D. McCarthy,
Arturo Oncevay,
Luis Chiruzzo,
John E. Ortega,
Gustavo A. Giménez-Lugo,
Rolando Coto-Solano,
Katharina Kann
Abstract:
Large multilingual models have inspired a new class of word alignment methods, which work well for the model's pretraining languages. However, the languages most in need of automatic alignment are low-resource and, thus, not typically included in the pretraining data. In this work, we ask: How do modern aligners perform on unseen languages, and are they better than traditional methods? We contribu…
▽ More
Large multilingual models have inspired a new class of word alignment methods, which work well for the model's pretraining languages. However, the languages most in need of automatic alignment are low-resource and, thus, not typically included in the pretraining data. In this work, we ask: How do modern aligners perform on unseen languages, and are they better than traditional methods? We contribute gold-standard alignments for Bribri--Spanish, Guarani--Spanish, Quechua--Spanish, and Shipibo-Konibo--Spanish. With these, we evaluate state-of-the-art aligners with and without model adaptation to the target language. Finally, we also evaluate the resulting alignments extrinsically through two downstream tasks: named entity recognition and part-of-speech tagging. We find that although transformer-based methods generally outperform traditional models, the two classes of approach remain competitive with each other.
△ Less
Submitted 15 February, 2023;
originally announced February 2023.
AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages
Authors:
Abteen Ebrahimi,
Manuel Mager,
Arturo Oncevay,
Vishrav Chaudhary,
Luis Chiruzzo,
Angela Fan,
John Ortega,
Ricardo Ramos,
Annette Rios,
Ivan Meza-Ruiz,
Gustavo A. Giménez-Lugo,
Elisabeth Mager,
Graham Neubig,
Alexis Palmer,
Rolando Coto-Solano,
Ngoc Thang Vu,
Katharina Kann
Abstract:
Pretrained multilingual models are able to perform cross-lingual transfer in a zero-shot setting, even for languages unseen during pretraining. However, prior work evaluating performance on unseen languages has largely been limited to low-level, syntactic tasks, and it remains unclear if zero-shot learning of high-level, semantic tasks is possible for unseen languages. To explore this question, we…
▽ More
Pretrained multilingual models are able to perform cross-lingual transfer in a zero-shot setting, even for languages unseen during pretraining. However, prior work evaluating performance on unseen languages has largely been limited to low-level, syntactic tasks, and it remains unclear if zero-shot learning of high-level, semantic tasks is possible for unseen languages. To explore this question, we present AmericasNLI, an extension of XNLI (Conneau et al., 2018) to 10 indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. Additionally, we explore model adaptation via continued pretraining and provide an analysis of the dataset by considering hypothesis-only models. We find that XLM-R's zero-shot performance is poor for all 10 languages, with an average performance of 38.62%. Continued pretraining offers improvements, with an average accuracy of 44.05%. Surprisingly, training on poorly translated data by far outperforms all other methods with an accuracy of 48.72%.
△ Less
Submitted 16 March, 2022; v1 submitted 18 April, 2021;
originally announced April 2021.
