2.1 Traditional Approaches

In this scenario, conventional classification methods depend on feature engineering, wherein texts are transformed into feature vectors before classification using standard algorithms like SVM and NB. Examples of conventional approaches are outlined briefly. The authors in [⁸] underscore the significance of utilizing datasets from multiple platforms to enhance the generalizability of the classifier in detecting offensive language.

They experimented with SVM and TF-IDF, and achieved F1 score of 84%. Besides, authors in [⁹] explore the influence of preprocessing steps on offensive language and hate speech classification. They demonstrate that thorough preprocessing techniques have notable effects on detection rates. The best experimental outcomes were achieved using SVM and BoW, attaining F1 scores of 95% and 89% for hate speech and offensive language, respectively.

Likewise, authors in [¹⁰] use the Arabert embedding with Deep Forest, the best experimental results showed acceptable macro-averaged and weighted-average F1-score results of 63% and 80%, respectively.

Furthermore, authors in [¹¹] employ BoW and TF-IDF to categorize tweets as offensive or normal. Their findings indicate that ensemble classifier (Bagging) outperforms single classifier, achieving F1 score of 0.88. In similar work, the authors in [¹²] categorize tweets into those of normal, hate, and abusive.

They utilize NB and SVM classifiers with trigrams, bigrams, and unigrams. The best experimental outcomes are achieved with NB, resulting F1 score of 0.896 and 0.744% for (abusive and hate vs. normal) and (normal vs. hate vs. abusive) tasks, respectively.

In a distinct approach, researchers in [¹³] use Social Graph, tweet-based and profile features to differentiate non-abusive Twitter accounts from abusive ones. The best-achieved F1 score was 85% using the NB classifier.

2.2 Deep Learning Approaches

In this context, these methodologies utilize a neural network capable of automatically learning representations of input tweet texts by varying the level of the abstraction. These learned representations are then leveraged to execute the classification task. Commonly employed embedding models include Mazajk and AraVec.

The prevailing DL architectures utilized for Arabic hate speech classification encompass BERT, LSTM, and GRU. Below, we briefly outline some examples of DL approaches. Authors in [¹⁴] categorize tweets into religious, general-hate, racial, sexism, or normal.

They employ an embedding layer randomly initialized to learn the word embedding from the training data. The best experimental outcomes are achieved using the Hybrid CNN-LSTM model, resulting in an F1 score of 73%. In a similar study, the authors in [¹⁵] evaluate two AraVec models to categorize tweets into normal and hateful. The experiment with Hybrid CNN-LSTM achieved F1 score results of 71.68%.

Likewise, authors in [¹⁶] explore the influence of word embedding and neural networks on the performance rates across various classification tasks. They train multiple embedding models and subsequently employ these models to train several neural networks for different classification task.

The best experimental outcomes are observed with Skip-gram and CNN, resulting in F1 scores of 70.80%, 75.16%, and 87.22% for the 6-class, 3-class, and 2-class classifications, respectively. Besides, the authors in [¹⁸] use ensemble CNN and Bidirectional LSTM (BiLSTM) classifiers based on the AraBERT. The best outcome is obtained using the average-based ensemble approach, yielded F1 score of 80.23%(BiLSTMs), 84.01% (CNNs), and 91.12% (CNNs) for 6 class, 3 class, and binary classification tasks, respectively.

In a similar, authors in [¹⁷] use AraVec and AraBERT to categorize tweets as being normal, abusive, or hateful. The best performance was achieved using CNN, yielding F1 score of 0.721. In a similar, the authors in [¹⁹] use CNN with Multilingual BERT embedding model, yielding F1 score of 75.51%, 78.9%, and 87.03% for 6 class, 3 class, and binary classification tasks, respectively.

Furthermore, authors in [²⁰] utilize a bidirectional GRU enhanced by an attention layer alongside the AraVec to identify offensive language and hate speech. Moreover, they examine the effect of different oversampling techniques and pre-processing techniques on the performance results.

The best outcomes consist of F1 score results of 0.859% and 0.75 for offensive and hate speech, respectively.

In a distinct strategy, researchers in [²¹] fine-tune the pre-trained AraBERT [⁴⁷] for classifying tweets being offensive, vulgar, hate speech, or clean. They achieved an F1 score result of 83.2%.

2.3 Gaps and Contributions

After reviewing the current studies, we can realize that some current pre-trained language models have not yet been evaluated for hate speech classification from Arabic tweets. Moreover, there is no existing study where Monolingual, Multilanguage, and Multidialectal pre-trained language models based on transformers are compared to demonstrate their validity for classifying Arabic hate speech. In this work, we rely on transfer learning models due to their major advantages:

They can capture long-term dependency in language, while it does not need a large dataset. Additionally, we conduct a comparative analysis between deep learning approaches and conventional approaches as our baselines.

2.4 Dataset Description

The dataset used in this paper was published in [¹⁴]. The basic characteristics of the corresponding dataset are presented in Table. 1. The tweets were collected using a curated list of hashtags known to elicit hateful content on Twitter. Subsequently, the retrieved tweets were manually annotated. The racial hate speech class constitutes a minor subset of the tweets, whereas the majority belong to the non-hate class.

2.5 Features Representation

The efficacy of a classification system depends on how it represents the text. Specifically, for tasks such as tweet classification, it is essential to convert the tweet's textual content into an appropriate representation for learning a classifier.

Hence, in this work, we adopt three distinct representations, which are outlined briefly below. The Bag of Words (BoW) [²²] method stands as one of the foremost techniques employed for information retrieval. BoW centers on counting the occurrences of words within a given text corpus.

This approach generates a vocabulary comprising unique words found across all tweets and utilizes these as feature vectors to indicate the absence or presence of such words within the vocabulary. Term Frequency Inverse Document Frequency (TF–IDF) weighting scheme that combines Inverse Document Frequency (IDF) with Term Frequency (TF).

This technique is commonly used for Text Mining and Information Retrieval, which converts the tweet to a matrix of integer producing sparse matrices of the counts [²³]. Word Embedding (WE) [²⁴, ²⁵] stands as an effective technique that has seen considerable success in recent years. A feature vectors space consists of unsupervised word embedding vectors.

These vectors represent the semantic spaces of each word in a real-valued space. Word embedding vectors offer a dense representation of word meaning, where the word is characterized as a real-valued features vector. Word embedding models can be produced using static pre-trained models like word2vec [²⁵], GloVe [²⁶], and fastText [²⁷], or by employing contextual pre-trained embedding models like BERT [²⁸].

2.6 Model Description

This subsection presents the classification models. We outline the conventional learning systems, the deep learning architectures, and the transfer learning models we have used.

Traditional machine learning. We evaluate two models most commonly used the Multinomial Naïve Bayes and Support Vector Machine, which predict classes based on a combination of features.

Support Vector Machine (SVM) [²⁹] most well-known classifiers since it is highly accurate and effective in text classification. This classifier offers the advantage of typically performing well even when trained with a limited amount of data [³⁰]. For the hyperparameter optimization, we experiment with various values: 'C' = [¹, .01, 10, 0.1], 'class_weight' = [balanced, None], 'penalty' = [l2, l1]. Following the optimization, we utilize the linear SVM classifier with its default configuration.

Multinomial Naïve Bayes (MNB) [³¹] is one of the most well-known classifiers since it is highly accurate and effective in text classification.

It operates by considering the frequency of such word to generate in a multinomial fashion the data distribution. For the hyper-parameter optimization, we evaluate the values of: 'Alpha' = [0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1] and 'fit_prior' = [True, False]. After optimization, the default parameters of the classifier were used. To derive feature vectors for inclusion for such classifiers, we utilize the three types of tweet representation previously discussed (i.e. Word Embedding, TF-IDF, and BoW).

Deep Learning. The deep learning methods explored in this study are briefly described in the following. Convolution Neural Network (CNN) stands as the most effective neural network model, offering an alternative approach to traditional feedforward neural networks. In CNN architecture, different layers are sparsely connected, linking a local region of an input layer with neurons in the subsequent layer.

The work by [³²] was the first that applied CNN to text classification, wherein words are transformed into numerical values via word embedding. A 2-dimensional matrix is formed from the tweet text, where each row is a word vector in that tweet. The typical CNN architecture encompasses several stages, including a fully connected layer, a pooling layer, and a convolutional layer.

Recurrent Neural Network (RNN) [³³] is another class of neural network to address the challenge of sequential learning faced by the conventional neural network. The connections among nodes construct the directed graph over the temporal sequences, enabling the model to highlight the dynamic temporal behaviors.

Long Short-Term Memory (LSTM) represents the most widely recognized variant, as introduced by [³⁴], and trained via backpropagation through time. LSTM networks are equipped with memory blocks, enabling them to learn the temporal sequence and their long-terms dependency effectively. On the other hand, Bi-directional Long Short-Terms Memory (BiLSTMs) facilitate the two-way information flow. This architecture involves training two LSTM network simultaneously, one for the forward and one for the reverse direction [³⁵].

In this scenario, we use word embedding as a feature representation. We particularly use Aravec [³⁶], which consists of 300-dimensional vectors for each word.

Transfer Learning. In this scenario, the process is to adjust a pre-trained language model to a new dataset through the transfer of the learned features. In other words, a technique to improve learning of a new task by transferring knowledge from the learned task [³⁷].

The transformer operates as an attention mechanism, enabling the learning of contextual relationships between words within a text. It includes two primary components: the encoder, which processes the textual input; and the decoder generates estimates for the corresponding task [³⁸].

Unlike directional models that sequentially process textual input (e.g., right-to-left or left-to-right), the encoder of the transformer simultaneously processes the entire sequence.

This approach enables model to capture the word context based on their surrounding context as a whole. The authors in [³⁸] achieved an enhancement in the translation task with the use of the attention mechanism avoiding relying on RNN, paving the way for additional transformer architectures.

Bidirectional Encoder Representations from Transformers (BERT) is the first transformer-based language model introduced by Google. The model is pre-trained on large unsupervised text data based on two self-supervision tasks:

Masked Language Modeling and Next Sentence Prediction. In the first task, approximately 0.15 of the words in such sentences were masked at random, and the model forecast the masked words.

The second task involves the classification of two sentences, wherein the model was tasked with discerning the original orders between the two sentences, thereby enhancing document-levels understanding.

Alternatively, the authors in [³⁹] proposed a cross-lingual language model refer as XLM, improving BERT while attaining remarkable achievements across different machine translation and cross-lingual classification tasks.

Unlike BERT, which is not adjusted for multi-lingual tasks due to limited shared vocabulary across languages, XLM tackles this challenge by processing all languages using a shared vocabulary generated based on a preprocessing method called Byte Pair Encoding [⁴⁰, ⁴¹]. Additionally, XLM uses the dual-language training mechanism alongside BERT so as to learn inter-language word relationship effectively.

3.1 Tweet Preprocessing

The preprocessing stage plays a pivotal role in natural language processing systems, particularly in text classification [⁴²]. They contain elongated words, hashtags, user mentions, and expressions that make tokenization difficult. To mitigate these challenges, we implement the following steps:

3.2 Hyperparameter Optimization for DL Models

The DL architectures evaluated in this work contain numerous hyperparameter, which necessitate estimation to achieve optimal results. To achieve this, we used the performance of a validation dataset to select the most suitable hyperparameter for the test dataset. For the hyperparameter optimization, we conduct 10-folds cross-validation using the corresponding dataset.

We employ the test data to make predictions while evaluating the predictions based on the optimized hyperparameters.

Table 2 illustrates the optimal hyperparameter for the corresponding model (LSTM, CNN, and BiLSTM). To avoid over-fitting during the supervised training of a neural network, we utilize early stopping by ending the training procedure before the converging of the weights.

3.3 Transfer Learning Fine-Tuning

The transformers-based models used in this work are pre-trained trained based on formal general corpora (Arabert, XLM-RoBERTa, and Multilanguage-BERT) and based on informal corpora (i.e. AraBERT-Twitter).

Thus, it is important to study the contextual information derived from the pre-trained layers while fine-tuning it for our interested downstream task. The fine-tuning consists of updating weights using the annotated dataset. BERT takes a sequence of 512 tokens as input and outputs 12 self-attention heads and a 768-dimensional vector.

For the optimization, we use the Adam optimizer [⁴⁴,⁴⁵] which performs well for natural language processing and the BERT model specifically. Additionaly, we evaluate other multilingual model, we chose the xlm-roberta-base and xlm-roberta-large checkpoints which include 100 languages.

For the purposes of fine-tuning, authors in [⁴⁶] have recommended choosing from the values of the following parameters: number of epochs, batch size, learning rate, and maximum sequence. We fine-tuned the corresponding models by evaluating different parameters as presented in Table 3. We set the maximum sequence length as 64 for all the experiments.

3.4 Evaluation Metrics

To assess the effectiveness of the corresponding approaches, we will use various evaluation measures capable of accurately assessing the model's performance. Given that hate speech classification poses the imbalanced learning challenge, we will particularly emphasize the Macro-average and Weighted-average metrics to compute comprehensive performance metrics.

These metrics are presented as follow: Precision (P), referred to as positive predictive value, indicates the proportion of correctly classified positive instances out of all instances classified as positive. For example, the Precision of the Normal class is estimated as follows:

where CC_Normal is the Correctly Classified as Normal and TC_Normal is the Total Classified as Normal. Recall (R) (also known as sensitivity), is the division of correctly predicted positive instances to the total positive instances. For example, the Recall of the Normal class is calculated as follows:

where CC_Normal is the number of Correctly Classified as Normal, and TN_Normal is the Total number of Normal instances. F1 measure (F1) is the harmonic mean between the Precision and Recall. For example, the F1 of the Normal class is estimated as follows:

3.5 Results Analysis

In this subsection, we explore the achieved results of the approaches we evaluated. We employed the tenfold cross-validation method to evaluate performance metrics. This method involved splitting the dataset into 10 equally sized parts while maintaining a balanced representation of each class from the original dataset. One part was designated for testing, while the remaining parts were utilized for training.

This process was repeated 10 times, and the performance metric scores were averaged across the 10 iterations of cross-validation. Table 4 presents the prediction performances attained for both conventional machines leaning and DL classifiers. We utilized word embedding as an input feature vector in all models.

In the case of traditional machine learning approaches, also we tested also the statistical-features BoW and TF–IDF including SVM-BoW, SVM-TF-IDF, MNB-BoW, and MNB-TF–IDF. In most traditional classifiers, BoW achieves better performance results than the TF-IDF and word embedding features.

The baseline experiments (DL and traditional learning approaches) did not perform satisfactorily due to an insufficient number of training instances. The DL-based approaches achieve better performance results than conventional machine learning approaches.

These findings are in line with the majority of related works, where DL based approaches found to have comparable accuracy results to traditional learning algorithms on the corresponding task.

As highlighted in Table 5, the pre-trained language model before fine-tuning substantially outperforms the baseline systems. It is important to note that multidialectal-based transformer models achieve better results than monolingual and multilingual-based models, yielding F1 score results of 63.38% and 80.85% for Macro averaged and Weighted averaged, respectively.

Table 6 shows the accuracy results of the pre-trained language model after fine-tuning. It is important to note that the fine-tuning of the pre-trained language model improves the accuracy results. Additionally, the multidialectal-based models outperform monolingual and multilingual ones, yielded F1 score results of 70.76% and 84.69% for Macro averaged and Weighted averaged, respectively.

Subsequently, we compare the performance of fine-tuned multidialectal-based model base-arabertv02-twitter (O) with three existing conventional machine learning approaches:

Unlike CNN and LSTM, our method does not necessitate a substantial quantity of labeled dataset to achieve promising performance result; this is a common requirement in many DL approaches. Moreover, in contrast to NB and SVM, our model eliminates the need to extract and design handcrafted features.

Given the nature of social media data, characterized by frequent usage of slang, abbreviations, and informal language, our method effectively processes the input words while considering their contextual surroundings.

Experimental results demonstrated that our proposition outperforms existing approaches by a difference between [% and 21%] and [% and 44 %] in weighted averaged and macro averaged F1 scores respectively. Comparing these results, we highlight the significance of multidialectal-based models trained on Twitter data since those models achieve the best results.

Furthermore, we highlight the significance of parameter tuning to discover the optimal hyperparameter values. Subsequently, we conduct an error analysis on the best pre-trained language models. For each model, we scrutinize tweets that were misclassified.

Furthermore, we examine tweets that were misclassified by all four models. Table 8 showed the number of tweets misclassified by such pre-trained language model.

The four best accurate models (bert-base-arabertv02-twitter (X), bert-large-arabertv02-twitter (Y), bert-large-arabic (W), and bert-base-arabert (Z)) predicted the same wrong labels 726 times out of 5217.Regarding the best system bert-base-arabertv02-twitter, the Non-HS instances mislabeled are biased towards General-HS; the Sexism-HS, Religious-HS, Racial-HS, and General-HS instances mislabeled are biased towards Non-HS. Fig. 1 shows the confusion matrix of the optimized system.

Fig. 1 Confusion matrix of our proposition