3.1 Feature Extraction

3.1.1 Pre-Processing and Feature Selection:

From data collections, preprocessing involves removing unnecessary information, such as special characters, numbers, URLs, and punctuation. After the user text was preprocessed, different features were selected to represent the user, described below:

1. Sentiment Analysis: This characterization was performed using two approaches. The first one employs NRC Emotion Lexicon (EmoLex)^fn.

The EmoLex lexicon contained eight basic emotions (Anger, Sadness, Fear, Anticipation, Trust, Surprise, Joy, and Disgust).

The second approach employs CNN^fn to identify four emotions (Sadness, Fear, Anger, and Joy).

2. Demographic Information: For this characterization, we focused on inquiring about two attributes (Gender and Age); using a lexical resource available at World Well-Being Project^fn.

The prediction was performed at the publication level. So, the weights of each one are added up, and the final sum represents the result of the prediction in the user’s history.

The gender is indicated by the sign obtained from the result, a positive result indicates a female user, and a negative result indicates a male user. While age is represented by the sum of the weights.

3. Polarity: For polarity, each publication in the user’s history, was analyzed to calculate polarity with the help of the lexical resource SentiWordNet^fn, assigning the polarity (positive or negative) to the words.

The positive and negative scores found for each term were added separately to obtain two different scores for each publication: the positive (S+) and the negative (S−).

This was carried out for each of the publications of depressive and non-depressive users.

4. Personality: For personality prediction, we used the model developed in [¹⁴].

Which predicts the Big5 (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) personality scores, the model integrates psycholinguistic features, including five lexical and LIWC features, along with features of BERT language model, using a Multilayer Perceptron (MLP).

5. Thematic: Finally, a vectorization of the users’ histories was made using a weighted TF-IDF, which assigns weights to the words to discriminate between classes (depressive and non-depressive)

Emotion traits are included based on Chen et al. showing that emotion-based features have a positive impact on depression detection.

In addition to emotions, polarity analysis, associated with the vocabulary shared by users on social networks, may contain relevant information.

For example, when words associated with different everyday situations, are mentioned in positive or negative contexts.

In addition, profile information, such as demographic attributes, has been associated with depressive indicators. In addition, several studies conducted by psychologists have associated neurotic personality with emotional stability.

In addition to these, other results on the detection of mental disorders have concluded that personality is a relevant trait for discriminating those suffering from a mental disorder [¹⁰].

Finally, a thematic analysis provides a structured and systematic approach to understanding sentiment and allows the detection of patterns.

Fig. 1 General Model Architecture 

Once obtained, the GMU module is passed through a fully connected hidden layer, a dropout layer, and finally, an output layer with sigmoid activation, for final classification.

4.1 Corpora and Evaluation Metrics

To evaluate the proposed model, we considered two datasets related to depression on two social networks (Reddit^fn and Twitter [²²]).

These two collections were built for the English language. In the case of Reddit, the dataset was constructed from a collection of depression-related posts.

This dataset was named e-Risk2018 and was employed throughout the CLEF evaluation forum.

On the other hand, the Twitter dataset was collected via API, and for this collection, users were considered depressed if any of their posts contained a self-reported clinical diagnosis of depression, such as: ”I am/am/was/was/have been diagnosed with depression”.

In Table 1 we can see some statistics for these datasets. Since the objective is to identify depressed users, and we do not have much positive sample data to work with, we proposed the use of a penalized model.

To do this, we pass the weights of each class in order to try to balance out the depressive class and thereby get the model to ”pay more attention” to samples from the underrepresented class.

In the case of Twitter, as there is no test set, a 5-fold cross-validation was performed for this dataset.

Here, we show different evaluation metrics to evaluate the performance of our model: Precision, Recall and F1-score of the depressive class. Note that in this work, we focused on enhancing the F1-score of the depressive class.

4.2 Data Fusion Baselines

We considered four data fusion methods as a baseline:

— Concatenation: Different works have concluded that a simple concatenation of representation could be good [²].

— Features-Union [¹⁷]: This method is quiet similar to concatenation method; the principal difference between both is that this method assign the same weight to each modality and it is useful to combine several feature extraction mechanisms into a single.

— Multi-kernel: This method shows good performance among heterogeneous data. We implemented two Multi-Kernel learning styles with SVM: 1) MKL(Average) [¹¹].

This is a simple wrapper that defines the combination as average base kernels; and 2) MKL (GRAM) [¹²].

Gradient-based RAdius-Margin optimization, this method focuses on finding the combination of the kernel that simultaneously maximizes the margin between classes while minimizing the resulting kernel’s radius.

— EmbraceNet: This neural network-based method ensures compatibility with any learning model and correctly handles different modalities [⁸].

Furthermore, this model has been compared with several neural network fusion techniques, achieving better performance.

4.3 Experimental Results

As we mentioned before, we explored the individual performance of each of the previously described features using a Support Vector Machine with a linear kernel as a classification method.

Table 2 shows the evaluation metrics on the depressive class of the seven characterizations extracted from the users’ posts on Reddit and Twitter. The Table above shows that the different characterizations behave differently for each dataset.

This behaviour might be related to the data and the way the models were built to extract the features, as a few of these methods were built for different types of datasets.

Once obtaining these results, we calculated the Coincident Failure Diversity (CFD) measure, to assess the diversity and complementarity between each fusion of the seven characterizations in the tow datasets (Reddit and Twitter).

This measure^fn is employed to determine the probability that members of the same system commit mistakes coincidentally.

Reddit. In Table 3, at the top, we can see the diversity between the fusion of each characterization for Reddit, as we can see the highest CFD of 0.87 was obtained with the fusion of the characterizations Emo-Lex, CNN-Emotions, thematic information, and gender.

This suggests that for the Reddit dataset, these four characterizations provide the highest diversity and complementarity. Since the table is so large, we only show the highest CFD obtained with the four characterizations mentioned previously and the CFD of the fusion of the seven characterizations.

Twitter. In Twitter’s case, the characterizations with the highest Coincident Failure Diversity are Emo-Lex, CNN-Emotions, and Thematic Information. As shown in Table 3 at the bottom, these characterizations achieve a CFD of 0.59.

Again, we only show the highest CFD achieved with the three above-mentioned characterizations and the CFD of the fusion of the seven characterizations.

Table 3 shows that the characterizations with the highest CFD, change throughout the datasets. Although, in both cases, emotions and thematic information have a relevant impact on obtaining the highest diversity.

With this in mind, if we choose an appropriate fusion method, we can achieve a model with better performance. To evaluate the GMU module properly, we considered the use of four different fusion methods, previously described.

All the methods presented here were both trained and tested with the same erisk2018 and Twitter datasets. For this experiment, we expect to confirm the hypothesis presented in this paper, and we also seek to determine the effectiveness of using GMU over traditional fusion methods.

Reddit. In the case of Reddit, we trained and tested the fusion methods with the four characterizations that obtain the highest CFD. In this dataset, the results can be observer at the first part of Table 4.

Twitter. In the second part of the Table 4, we can see the performance of each fusion method on the Twitter dataset, using the three characterizations with the highest CFD for this dataset.

For both the Reddit and Twitter datasets 4, the neural network-based fusion methods are the highest F1 scores, of which the proposed model outstood by obtaining the best performance.

This is because the GMU module achieves better learning of the diversity of characterizations compared to the other fusion methods for both cases.

4.4 Comparison with the State of Art

Reddit. In the case of Reddit, we compare ourselves with various works presented in the CLEF 2018 evaluation forum, which make use of fusion techniques, where they use the same dataset as in this work (eRisk2018). The results can be observed in Table 5.

Twitter. On Twitter, we compare ourselves with other works that use the same dataset described above. These works only report the F1-score. The results can be observed in Table 6.

For both cases, the proposed method outper-formed the state-of-the-art works. Thus, of the results obtained, the following stand out:

1. The implementation of a GMU in a simple neural architecture, together with an adequate selection of characterizations, outperformed various traditional fusion techniques implemented as a baseline, indicating that fusing the different characterizations at a deeper level is indeed relevant for depression detection.

2. Our approach outperformed the eRisk2018 winner approaches and the state-of-the-art works for Twitter. It is key to note that some of the works presented in the state of the art tried different complex models with a wide range of features using traditional fusion methods such as late and early fusion.

In contrast, the one presented here was only based on the use of 4 characterizations for Reddit and 3 characterizations for Twitter, and a GMU module as a fusion method.

5.1 GMU Error Analysis

Based on the results obtained, we decided to evaluate and determine why some users were misclassified and if there is any common reason between the data sets. For both datasets the errors were related to the amount of information.

Very short publication histories do not provide sufficient information for the model. Whereas very long histories the features may vary over a long period of time. Therefore, in both cases, the signs of depression are not so clear to the model, so it cannot adequately discriminate between classes.

5.2 Diversity and Model Performance

Looking at Figure 3, we can see that in the case of Reddit the Pearson [⁵]^fn correlation between the diversity obtained with the CFD measure and the F1-score of the depressive class in the GMU architecture is 0.71, which indicates that there is a strong correlation between both variables, so the more diversity, the higher the performance for the detection of depressive users.

Fig. 3 Pearson correlation between GMU performance and CFD 

The same for the case of Twitter where we obtain a correlation of 0.64 between diversity and the F1-score of the depressive class of the GMU model.

5.3 Differences between Reddit and Twitter

In this analysis, we decided to compare some elements of both datasets to provide information on the differences between the relevance of the characterizations in the datasets.

To address this, we decided to assess the similarity between the classes, depressives and non-depressives, in both datasets (Reddit and Twitter).

As we can see in Table 9, the first column indicates the vocabulary overlap between the classes with Jaccard coefficient.

This coefficient is a statistical measure used in natural language processing to compare the similarity between documents [¹³].

In our case, we can see that the Jaccard coefficient is higher on Reddit than on Twitter. This tells us that in the Reddit dataset the vocabulary used by both classes is close enough.

While the one used by Twitter differs significantly between both classes, which could explain the main reason why the thematic characterization is more relevant for Twitter than for Reddit.

On the other hand, in column 2 we show the degree of imbalance between classes where, again, Redditt is higher than Twitter.

The degree of imbalance between classes is an important feature to take into account when working with corpora, since depending on the degree of imbalance there may be different levels of difficulty [³]. Therefore, we may find it more difficult to classify depressive users on Reddit than on Twitter.