Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification
Authors:
Kushal Tatariya,
Heather Lent,
Johannes Bjerva,
Miryam de Lhoneux
Abstract:
Emotion classification is a challenging task in NLP due to the inherent idiosyncratic and subjective nature of linguistic expression, especially with code-mixed data. Pre-trained language models (PLMs) have achieved high performance for many tasks and languages, but it remains to be seen whether these models learn and are robust to the differences in emotional expression across languages. Sociolin…
▽ More
Emotion classification is a challenging task in NLP due to the inherent idiosyncratic and subjective nature of linguistic expression, especially with code-mixed data. Pre-trained language models (PLMs) have achieved high performance for many tasks and languages, but it remains to be seen whether these models learn and are robust to the differences in emotional expression across languages. Sociolinguistic studies have shown that Hinglish speakers switch to Hindi when expressing negative emotions and to English when expressing positive emotions. To understand if language models can learn these associations, we study the effect of language on emotion prediction across 3 PLMs on a Hinglish emotion classification dataset. Using LIME and token level language ID, we find that models do learn these associations between language choice and emotional expression. Moreover, having code-mixed data present in the pre-training can augment that learning when task-specific data is scarce. We also conclude from the misclassifications that the models may overgeneralise this heuristic to other infrequent examples where this sociolinguistic phenomenon does not apply.
△ Less
Submitted 5 February, 2024;
originally announced February 2024.
CreoleVal: Multilingual Multitask Benchmarks for Creoles
Authors:
Heather Lent,
Kushal Tatariya,
Raj Dabre,
Yiyi Chen,
Marcell Fekete,
Esther Ploeger,
Li Zhou,
Ruth-Ann Armstrong,
Abee Eijansantos,
Catriona Malau,
Hans Erik Heje,
Ernests Lavrinovics,
Diptesh Kanojia,
Paul Belony,
Marcel Bollmann,
Loïc Grobol,
Miryam de Lhoneux,
Daniel Hershcovich,
Michel DeGraff,
Anders Søgaard,
Johannes Bjerva
Abstract:
Creoles represent an under-explored and marginalized group of languages, with few available resources for NLP research.While the genealogical ties between Creoles and a number of highly-resourced languages imply a significant potential for transfer learning, this potential is hampered due to this lack of annotated data. In this work we present CreoleVal, a collection of benchmark datasets spanning…
▽ More
Creoles represent an under-explored and marginalized group of languages, with few available resources for NLP research.While the genealogical ties between Creoles and a number of highly-resourced languages imply a significant potential for transfer learning, this potential is hampered due to this lack of annotated data. In this work we present CreoleVal, a collection of benchmark datasets spanning 8 different NLP tasks, covering up to 28 Creole languages; it is an aggregate of novel development datasets for reading comprehension, relation classification, and machine translation for Creoles, in addition to a practical gateway to a handful of preexisting benchmarks. For each benchmark, we conduct baseline experiments in a zero-shot setting in order to further ascertain the capabilities and limitations of transfer learning for Creoles. Ultimately, we see CreoleVal as an opportunity to empower research on Creoles in NLP and computational linguistics, and in general, a step towards more equitable language technology around the globe.
△ Less
Submitted 6 May, 2024; v1 submitted 30 October, 2023;
originally announced October 2023.
