Hate Speech and Offensive Content Detection in Indo-Aryan Languages: A Battle of LSTM and Transformers.

Abstract

Social media platforms serve as accessible outlets for individuals to express their thoughts and experiences, resulting in an influx of user-generated data spanning all age groups. While these platforms enable free expression, they also present significant challenges, including the proliferation of hate speech and offensive content. Such objectionable language disrupts objective discourse and can lead to radicalization of debates, ultimately threatening democratic values. Consequently, social media platforms have taken steps to monitor and curb abusive behavior, necessitating automated methods for identifying suspicious posts. This paper contributes to Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages (HASOC) 2023 shared tasks track for Hate Speech Detection in Low-Resource Languages. We, team Z-AGI Labs, conduct a comprehensive comparative analysis of hate speech classification across five distinct languages—Bengali, Assamese, Bodo, Sinhala, and Gujarati—within the context of the HASOC competition. Our study encompasses a wide range of pre-trained models, including Bert variants, XLM-R, and LSTM models, to assess their performance in identifying hate speech across these languages. Results reveal intriguing variations in model performance. Notably, Bert Base Multilingual Cased emerges as a strong performer across languages, achieving an F1 score of 0.67027 for Bengali and 0.70525 for Assamese. At the same time, it significantly outperforms other models with an impressive F1 score of 0.83009 for Bodo. In Sinhala, XLM-R stands out with an F1 score of 0.83493, whereas for Gujarati, a custom LSTM-based model outshined with an F1 score of 0.76601. This study offers valuable insights into the suitability of various pre-trained models for hate speech detection in multilingual settings. By considering the nuances of each, our research contributes to an informed model selection for building robust hate speech detection systems.

1 Introduction

In the era of expanding global connectivity through social media, platforms such as Facebook, X (Formerly Twitter), YouTube, and Instagram have grappled with a disturbing surge in hate speech perpetuated by individuals and organized groups. With the surge of Influencers and Content Creators, also commonly known as the Creator Economy[1], there has been an alarming rise in incidents concerning targeted attacks on individuals based on their opinions, appearance, and ethnicity[2]. The consequences of such pervasive abusive language are far-reaching, often resulting in public humiliation and significant personal and professional consequences[3, 4, 5] for victims. The escalating prevalence of online hate speech, often characterized by anonymity, scale, and overwhelming volumes that challenge human moderators, underscores the pressing need for social media platforms to strike a balance between safeguarding freedom of expression and fostering an environment of inclusiveness and respect.

The need for content moderation[6] by detecting Hate and Offensive engagement has pushed organizations and research groups to develop systems and solutions at scale. Significant work has been done to identify toxic, profane, and offensive comments. However, a majority of contributions focus predominantly on Resource-heavy languages such as English[7, 8, 9, 10, 11, 12, 13]. This brings constraints to hate and offensive speech content detection and moderation in low-resource languages. The lack of large-scale corpus and pre-trained models makes it extremely difficult to tackle Natural Language Understanding (NLU) downstream tasks.

In response to these challenges, Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages (HASOC) presented four shared tasks as a part of its 5th edition, 2023. Out of these, Task 1 and Task 4[14] focus on detecting hate and offensive content in 5 low-resource languages as follows:- Bodo[15], Bengali[16], and Assamese[17] in Task 4[18], Gujarati and Sinhala in Task 1[19]. Each language has its corresponding dataset, evaluation metric, competition page, and leaderboard. The challenges present one problem statement, that is to classify a given content into one of the following classes:

• HOF (Hate and/or Offensive): This content is hate speech, offensive, and/or profane.

• NOT (Non-Hate and/or Offensive): This content is not hate speech, offensive, and/or profane.

In this paper, we describe our approach to tackle the challenges. Here’s the summary of our contribution:

• Adequate preprocessing techniques for datasets.

• Provide a strong LSTM with an attention head baseline model.

• Comparative analysis of pre-trained models in zero-shot and few-shot settings.

• Fine-tuning large multi-lingual models on the given datasets.

From here, the report continues in the following manner: In section 2, we highlight previous approaches as related work. In section 3, we give an overview of the dataset for each language and describe the challenge at hand. In section 4, we present our approach in detail, covering the nitty gritty of our experimental set-up, cross-validation strategy, models used, and intuition behind them. In section 5, we brief the results from the experiments section. Then, we conclude in section 6 with the final takeaways, our standings, and the scope of future work. The implementation details can be found in the following GitHub repository.

2 Related Work

Detecting hate speech poses a formidable challenge in the realm of research, with existing literature encompassing diverse methodologies, such as dictionary-based approaches[20], the utilization of distributional semantics[21], and the recent exploration of the efficacy of neural network architectures[22]. However, it is notable that a substantial portion of this research predominantly focuses on hate speech detection within the English language. Conversely, there is limited scholarly attention directed towards other foreign languages[23, 24, 25, 26] and the intricacies of code-switched text[27, 28]. Despite the significant impact of regional low-resource languages on online hate speech, this domain remains relatively uncharted, with recent investigations exploring the utility of transformers[29] and author profiling through the application of graph neural networks[30].

Historically, numerous strategies have been employed to address the challenge of identifying hate speech. Kwok and Wang[31] experimented with a straightforward bag of words (BOW) methodology to detect hate speech, but these lightweight models yielded subpar results characterized by elevated false positive rates. Enhancing these models with various fundamental natural language processing (NLP) components, such as part-of-speech tags[32] and N-gram graphs, contributed to improved performance. Lexical techniques employing TF-IDF in conjunction with Support Vector Machines (SVM) as a classification model surprisingly achieved commendable results[33].

The advent of embedding words into distributed representations marked a pivotal shift, as researchers harnessed word embeddings like Glove[34] and FastText[35] to project discrete text into a latent space, surpassing the performance of conventional BOW and lexical approaches.

Recurrent Neural Networks (RNNs) remained the go-to method for tackling various natural language challenges over an extended period. For instance, the winning approach in the 2020 HASOC competition for Hindi[36] employed a one-layer Bidirectional Long Short-Term Memory (BiLSTM) model with FastText embeddings to discern hate speech. Likewise, the most accurate model for English[37] adopted an LSTM architecture with Glove embeddings to represent textual inputs. Mohtaj et al.[38] also embraced a character-based LSTM, aligning with this prevailing trend.

In recent times, self-attention-based transformer models[29], and their derivatives such as BERT[39], derived from extensive corpus-trained encoders, have exhibited greater potential than traditional RNNs across a multitude of NLP tasks. BERT-like models have garnered substantial attention due to their remarkable transfer learning capabilities, outperforming alternative approaches consistently[40].

Despite the substantial body of research on hate speech detection, experiments dedicated to low-resource languages remain relatively scarce. Notably, simple logistic regression using LASER embeddings demonstrated superior performance to BERT-based models[41], underscoring the necessity for more precise multilingual base language models. Consequently, we have witnessed the ascendancy of multilingual language models like XLM-Roberta[42]. Following the trend, region-specific low-resource language models are developed. Some of the notable contributions are MuRIL[43], SinBERT[44] for Sinhala, BanglaBERT[45], Indic-BERT[46], and XLM-Indic[47] variants. Authors in [48] provide a detailed study on performance of mono-lingual and multi-lingual in the context of cross-lingual evaluation for hate speech identification. Previous editions of HASOC[49, 50] have witnessed a significant effort towards improving performance in low-resource languages such as Hindi[51], Marathi[52], etc. In the ensuing sections, we will elucidate our approach, which leverages several multilingual models for hate speech identification, accompanied by an exhaustive comparative analysis against alternative methodologies.

3 Dataset Description

3.1 Sinhala

Task 1 consists of 2 sub-tasks, one for Sinhala, and the other for Gujarati. Sinhala, one of the two official languages in Sri Lanka, is spoken by over 17 million people. This edition of HASOC brings the first-ever shared task for the aforementioned Indo-Aryan low-resource language. The train and test sets for Sinhala are based on the SOLD: Sinhala Offensive Language Detection dataset[53]. The SOLD consists of 10,000 manually annotated tweets divided into two classes: Offensive and Not offensive, both at the token level and the sentence level. However, the dataset provided for the task contains 7500 samples in the train set, each labeled as HOF or NOT. The test set contains 2500 samples.

Figure 1: Train-Test Split for the respective Tasks

3.2 Gujarati

Gujarati is one of the 22 official languages of India with over 50M native speakers. The train set for this task contains 200 tweets whereas the test set contains 1196 tweets. This is also a coarse-grained binary classification problem but in a few-shot setting. The train data frame is made up of 5 columns, named as follows: tweet_id, created_at, text, user_screen_time, and label. The test set contains only tweet_id and the text column.

3.3 Assamese, Bodo, and Bengali

Task 4 consists of 3 Kaggle competitions, each corresponding to one of the following languages: Assamese, Bodo, and Bengali. The primary sources for data collection are X (formerly Twitter), Facebook, and YouTube Comments. Each train set contains tweet-label pairs, with only tweets in the test set to predict the targets.

Figure 2: Target Distribution for the respective tasks.

4 Experimental Set-up

In this section, we discuss our approach and explain the experimental set-up details. We start with creating a validation strategy for each dataset. As all the datasets are fairly balanced (refer to figure 2), we opt for K-Fold cross-validation with 5 folds. And while creating the splits, we set the random seed to 2023.

4.1 Preprocessing

The preprocessing step involves cleaning the contents for feature extraction. We notice that the Bodo tweets do not contain any emoji or repetitive punctuation marks that need attention. However, there are instances where some of the content contains English words in a code-mixed manner. Note that, such instances are seen across all the datasets. For the other two datasets in Task 4, though the majority of the content is cleaned, some of them contain emojis and a few repetitive punctuation characters. The repetition is removed, and the emojis are converted into their respective textual description using the emot library. The Task 1 datasets for Sinhala and Gujarati contain usernames as @USER and the usernames are made available in a separate column. Along with that, the datasets also contain code-mixed English words, repetitive punctuation characters, emojis, and hashtags. Note that, there are no URLs or hyperlinks associated with any of the content. The usernames are removed along with repetitive punctuation characters. The hashtags are further processed with Ekphrasis tokenizer to segment them into meaningful tokens. The emoji2vec[54] embeddings are used on top of emojis, as it has shown competitive results in previous works.

4.2 Modeling

To start with, we create an LSTM-attention-based baseline with 2 bi-direction LSTM layers followed by an attention block. The attention head is further connected through 2 dense layers with sigmoid activation in the last layer. The model is trained with Adam optimizer and Binary Cross Entropy as the loss function. The hyperparameters involved such as the batch size, number of epochs, learning rate, vocab size, embedding dimension, and maximum length of the input sequence are varied and tuned on a case-to-case basis.

As the available datasets have less number of samples per language (refer to figure1), we also leverage Transformer-based language models for the downstream task at hand. The available training data are used to fine-tune the encoder layers of the transformer-based models, leaving the embedding layers frozen. Note that to incorporate emojis semantics, we concatenate the embedding layers with the emoji2vec-generated embeddings. We experiment with various multi-lingual transformer-based models for fine-tuning such as Bert-Base-Multilingual (Cased and Uncased), DistilBert-Base-Multilingual-Cased, XLM-Roberta-Base, Muril-Base, and XLM-Indic-Base (UniScript and Multi-Script). Other than that, we also present the results from a couple of language-specific models such as Bangla-BERT and SinhalaBERTo.

Table 1: Model aliases used a reference in the result graphs.

Model Name	Alias Name
LSTM Baseline	LSTM
Bert Base Multilingual Cased	mBert C
Bert Base Multilingual Uncased	mBert U
DistilBert Base Multilingual Cased	mDistil C
XLM Roberta Base	XLM-R
Muril Base Cased	MuRIL
XLM Indic Base Multiscript	XLM-I M
XLM Indic Base Uniscript	XLM-I U

Note that, the Huggingface implementation for the models is used via TFAutoModelForSequenceClassification with corresponding hyperparameters for each. All the training and inference are done using Kaggle runtime and MacBook Air M2 with 24GB unified memory.

For inference, we ensemble the models from each fold with equal weightage on the logits, then take a threshold of 0.5 to classify into HOF or NOT. The labels are mapped to numbers as follows: HOF is mapped to 0, and NOT is mapped to 1.

5 Results

Figure 3: Leaderboard macro-F1 scores of different models for respective languages in Task 4.

The competitions for Task 4 are evaluated on macro-f1 metrics, whereas the Task 1 challenges are evaluated on macro-f1, Precision, and Recall.

Table 2: Leaderboard macro-f1 scores in Task 4.

Models	Bodo	Assamese	Bengali
LSTM Baseline	0.81291	0.67616	0.60856
Bert Base Multilingual Cased	0.83009	0.70525	0.64294
Bert Base Multilingual Uncased	0.82962	0.66398	0.61561
DistilBert Base Multilingual Cased	0.77606	0.67399	0.643
XLM Roberta Base	0.78668	0.376	0.38476
Muril Base Cased	0.35085	0.68985	0.70467
XLM Indic Base Multiscript	0.78186	0.633	0.59989
XLM Indic Base Uniscript	0.62257	0.376	0.37743
csebuetnlp/banglabert	-	-	0.75625

It is evident from the results matrix (refer to table 3) that, the LSTM baseline poses a strong competition in performance for all the languages. Even going a step further for Gujarati, the LSTM-based Model scores highest amongst XLM-Indic-Base-MultiScript and Bert-Base-Multilingual. This results in the highest F1-Score of 0.76601 and a Recall of 0.79704, with a marginal gain of 0.001 for F1 and 0.03 for Recall against the second-best performing model for the task. For Sinhala, XLM-Roberta seems to be the winner beating our LSTM Baseline and Sinhala Bert with a considerable margin. Due to time constraints and run submission limits, we experimented with a handful of BERT-based and Roberta-based models for fine-tuning along with the LSTM-with-Attention baseline Model.

For Task 4, we have varied candidate models for experimentation as mentioned in section 4 (refer to table 1). Our LSTM baseline poses as one of the top performers for Bodo and Assamese by yielding the third-highest F1-Score, beating XLM-Roberta-based models. For Bengali, however, BanglaBert beat the rest of the models with a staggering 0.75625 f1-score. The best-performing model for Bodo and Assamese is Bert-Base-Multilingual-Cased with an f1-score of 0.83009, and 0.70525 respectively. Note that, all the scores mentioned above (refer to table2) are the performance on the hidden test set, and directly taken from the system-run report provided by the Organizers after a finalized leaderboard. A visual representation of comparative performance for Task 4 is shown in figure 3.

Table 3: Leaderboard macro-f1 scores in Task 1.

Models	Gujarati	Sinhala
LSTM Baseline	0.7660	0.7530
Bert Multilingual base Cased	0.7656	0.8095
XLM Roberta Base	-	0.8349
XLM Indic Base Multiscript	0.7235	-

The obtained results help us climb to 3rd/20 for Bengali, 5th/16 for Sinhala, 7th/17 for Gujarati, 8th/20 for Assamese, and 12th/19 for Bodo.

6 Conclusion

This work has been submitted to the CEUR-2023 Workshop Proceedings for Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages (HASOC) track. In this report, we entail our approach to solving two tasks for the Track on classifying a given content into Hate and Offensive (HOF) or NOT. We experiment with various models starting from simple LSTM-based architecture to pretrained transformer-based multilingual models. Though the transformer shows sheer dominance in the majority of the tasks, our LSTM baseline emerges as a strong competitor with promising results, even resulting in the highest f1-score amongst our candidate models for Gujarati. We plan to further extend our work to other low-resource indic languages, and explore the possibilities of zero-shot, few-shot, and cross-lingual transfer learning scenarios. We also aim to develop a unified language model to incorporate all the languages such as NLLB[55] for similar downstream tasks to strengthen content moderation.