2.1 Text Analysis in SNS

Text analysis tools and techniques have been increasingly used to get insights into the users’ internal states (e.g., mood and emotions), or other psychological traits [³⁸, ¹⁸]. Text analysis has been particularly used to explore and predict different mental disorders through posts on SNSs [⁹, ¹¹].

Previous works have shown that certain keywords in SNS posts can be used to identify individuals with mental disorders. For instance, these types of posts include hashtags such as #depression, #anxiety or #suicide, among other words that may be related to mental disorders such as the names of antidepressants [¹⁴] or the name of the disorder itself [⁹, ¹⁰, ²⁷].

For instance, in [¹⁰] it was proposed a lexicon of depressed users on Twitter and found that some recurrent themes were related to symptoms (e.g., anxiety, withdrawal, severe, delusions), disclosure (e.g., fun, play, helped, god), treatment (e.g., medication, side-effects, doctor, doses), and relationships (e.g., home, woman, she, him). Authors from [³⁴] showed that college students with depression often use more personal singular pronouns at the moment of writing. Although this study was carried out with written essays, these findings could potentially be extrapolated to SNS posts.

Another approach often used is sentiment analysis since keywords and words used by users with mental disorders are often charged with negative emotions [¹¹, ⁴³]. For example, it has been reported that some words such as issues, bad, or anxiety could be used to predict the jump from depression to suicidal thoughts [¹¹].

Some of these words had high frequency like other less negative ones such as make, around, time, when, where, and others. However, this could raise some concerns as people might not be completely honest on SNS by purposely undermining their own negative feelings so others do not feel bad for them [¹²] or by expressing more positive emotions than they are actually experiencing since it might attract more attention to the post [³⁷].

Text analysis per se can be challenging, but it can be more difficult if analysis is carried out without further context. For a more precise interpretation of the users’ moods, more data associated with the users moods at the time of posting is required.

2.2 Image Analysis on SNS

The content of shared images can include data about the user’s interests and potentially about their mental state or mood. Examples of these types of content include the number of faces (i.e., individuals) in the image, the predominant color, and the types of objects, among others.

The content of Instagram images has been analyzed and linked to different aspects of users. For example, when studying the relationship between personality traits and gender, and the images posted, researchers found a link between extraversion and gender of users [²¹].

Likewise, [³⁶] extracted objects using Microsoft Azure Cognitive Services with the aim of classifying images into thousands of categories, such as car, city, interior, and others, and used them to determine the age and gender of users. The relationship between mental health and shared photos has also been studied.

For example, in [²⁷] studied the relationship between several visual attributes of the images such as color, themes or emotions, and self-disclosures of Instagram users related to their mental health. Images shared on SNSs have also been used for detecting depression.

In [³¹] analyzed the content of images and data such as the number of posts per day or the number of likes to compare nondepressed and depressed individuals. Features such as the number of faces in the photos and color properties were related to depression.

However, the dataset entries were observed and labeled through crowdsourcing with Amazon’s Mechanical Turk, which could have introduced a bias since emotions can be interpreted differently by third parties.

Moreover, the user who posted the image might have a salient emotion at the moment of posting, which could have been missed by annotators since they are only looking at the image.

Then, having third parties annotate the images of others can be a challenge to derive adequate findings, since different cultures or experiences can shape or bias the annotations.

In another study, [⁴⁶] used a deep regression network (deemed DepressNet) to analyze faces since they might indicate a depressive disorder. However, apart from selfies, users on Instagram typically post different types of images, such as landscapes, artwork, or pets, which makes it difficult to deploy in the wild.

Finally, in [⁸] used multimodal data from Instagram posts, including the content of images, text, and user’s behavior, to detect users with depressive moods. For the image analysis, used the AlexNet Convolutional Neural Network for transfer learning to get a prediction score of depressive images.

Afterward, they merged individual predictions for image, text, and behavior for an overall prediction. Although the dataset was obtained from real users, they retrieved the dataset by searching specific depression keywords on Instagram users’ profiles, which could have biased the results toward individuals who self-describe as depressed.

In general, there is a need for conducting studies that analyze data coming from individuals with their typical behavior on SNS, i.e., ecologically valid findings. Moreover, research must consider a wider range of posts from users [²⁹] as opposed to selecting posts with specific hashtags (e.g., #depression), which can bias the results and our understanding of the manner in which these aspects take place in the real world.

3.1 Participants

We used a convenience sampling method to recruit participants. The invitations to participate in the study were sent through electronic media such as WhatsApp or Facebook Messenger.

We recruited 50 individuals from Northwest Mexico, from which 35 participants (13 male) remained until the end of the study.

Our participants were, on average, 23.51 years old (SD = 3.36), ranging 19-40 years old. Sixteen (16) of them (45.71%) were university students.

34% of the participants said they were regular users of Instagram. All our participants were native speakers of Spanish. All participants signed an informed consent. No monetary incentive was given.

3.2 Instruments

We used the following instruments to obtain data in the wild.

3.2.3 Web-Based App for Data Collection

We developed a web-based app (Figure 2) to retrieve the users’ latest posts using their Instagram (IG) handle. Every time the users posted on IG, our web-based app retrieved the posted image, text, and date.

Fig. 2 Snapshots of the web-based app (In spanish) 

Below the retrieved image from the IG post, the app displayed the PANAS-X questionnaire so that our participants could rate the types of emotions felt at the moment of posting that particular image.

This strategy differs from previous approaches since it provides not only the data linked to the post (e.g., image, text, timestamp) but also about their emotions at the time of posting.

3.3 Data Collection Protocol

Thirty-five individuals participated in this study who received a demographic questionnaire and Beck’s depression inventory (BDI) in Spanish [³⁵], which they responded to online without supervision.

First, we explained to the participants the general purpose of the study. Also, we asked them about their frequency of use of IG and IG handle. Of the 35 participants, 17 obtained a Beck score that falls in the category of minimal depression (score 0-13), 5 in mild depression (score 14-19), 7 as moderate depression (score 20-28), and 6 with severe depression (score 29 or more). The procedure was as follows:

4.1 Text Processing

For studying the link between the text of an IG post and the emotions reported by the participants, we used two different tools that identify the general emotion from the given text and also provide additional information that can be related to depression, such as the amount of singular or plural pronouns, which has been reported to be relevant in depressed students [³⁴].

The first tool was the Google AutoML Natural Language [¹³], which delivers the magnitude and value score of the identified emotion in the text, pronoun count, first-person pronouns, first-person singular pronouns, plural pronouns, and first-person plural pronouns.

The second tool used was the Spanish version of SentiStrength [⁴⁰], original version by [³⁹], which delivers the negative and positive scores from the identified emotion in the text. In the case of SentiStrength, we removed emoticons since the tool is unable to detect them and can only interfere the analysis.

Finally, we also computed the number of characters and words, the ratio of the number of pronouns over the number of words, the ratio of first-person pronouns over total pronouns, and the ratio of plural pronouns over total pronouns. In total, we obtained 14 features from the text.

4.2 Image Processing

To extract features from the image dataset, we used the state-of-the-art Automated Machine Learning (AutoML) Vision by Google Cloud Platform^fn, which is an implementation of AutoML for image classification and object detection.

It consists of an Application Programming Interface (API) that offers machine learning models that assign labels and detects objects in images, and it can also be used to train personalized models of machine learning. In total, we obtained 9 features from images. Color perception has been suggested as a marker of mood [⁵], where grayer and darker colors are related to depressive moods.

We included the dominant Red (R), Green (G), and Blue (B) colors of each photo provided by the Google Cloud service. Also, the levels of hue and saturation have been of interest to researchers as they could possibly indicate levels of sadness or depression [³¹].

Drawing on this, the mean values of Hue, Saturation, and Value (HSV) were retrieved from the images through the scikit-image library^fn.

We obtained relevant object labels from images that can be used to relate to a depressive mood. For example, the label face, since according to []reece2017instagram, the number of faces reflects a greater social interaction, which can be related to less tendency to depression. Table 1 and Table 2 show data related to faces and their emotions as detected by Google Cloud Vision API.

Since the dataset features images with humans, pets, landscapes, drawings or anything the user felt like posting, we also obtained labels animal, if there were animals in the photo, and sketch, since we identified that several images were drawings, cartoons or sketches. Table 3 shows the data with this type of content.

4.3 Behavior Data Processing

Mental status can be related to posting behavior, such as the time or frequency of posting activity. In Figure 4, we show a heatmap with the posting activity from all our participants.

Fig. 4 Posting activity from all users throughout the duration of the study (x-axis = days; y-axis = participants) 

A darker color indicates more publications during each day. For the purposes of this figure, we grouped the users by Beck’s class in the Y axis (A_xx = minimal depression, B_xx = mild, C_xx = moderate, D_xx = severe).

Figure 5 shows the time of day in which participants posted the most, grouped by the participants’ severity of depression.

Fig. 5 Posts binned per the hour of day (x axis = time of day; y axis = number of posts) 

It can be seen that there is a sharp increase of activity from 10 PM to 5 AM in our participants, particularly those categorized as severely depressed. Most posts occurred from midnight to noon across all groups. From behavior processing, we obtained 3 features: date, time, and time between consecutive posts in minutes.

5.1 Inferential Statistics Approach

5.1.1 Relating the Severity of Depression and the Emotions Linked to a Post

Detecting depressive moods in IG posts where there are no clear signals of sadness, such as gloomy pictures or certain keywords, can be challenging. For this, we first need to identify the types of emotions related to a particular post, i.e., the emotions experienced by the participants at the moment of posting.

For this, we used t-student tests to explore the relationship between the level of depression as approximated by the Becks’ inventory (administered at the beginning of the study), and the general positive/negative emotions through the PANAS-X (administered each time the user posted).

As shown in Figure 6, for the General Positive Affect (GPA), participants with depressive moods (i.e., mild, moderate, and severe) had lower scores (GPA=20.98, SD=8.84) when compared with users with minimal depressive moods (GPA=25.00, SD=9.22), which was statistically significant (t=4.01, df=312, P<0.0001).

Fig. 6 General negative affect (GNA) and general positive affect (GPA) from PANAS-X across the Beck’s classes 

As for the General Negative Affect (GNA), participants with depressive moods had a significantly higher score (GNA=17.74, SD=6.51) when compared with users with minimal depressive moods (GNA=14.76, SD=8.09), which was statistically significant (t=−3.67, df=321, P<0.0001). Also, participants with severe depression had a mean GPA of 18.64 (SD=7.30), while the rest of the classes had an average GPA score of 23.72 (SD=9.35) (t=−4.49, df=97, P<0.0001).

On the other hand, participants with severe depression also reported higher scores of the GNA (mean=21.62; SD=8.41) when compared with the average GNA (mean=15.26; SD=6.87) of the rest of the participants (t=5.3, df=71, P<0.0001). This suggests that the PANAS-X helps discriminate negative and positive emotions associated with a particular post. Even more, these feelings seem to concur with Beck’s classes.

5.2 A Classic Machine Learning Approach

We used a classic machine learning approach to classify the participants’ Beck’s class based solely on the data shared in their posts. For this, we represented each participant with a vector that we used for training models. Afterward, we evaluated the models on unseen vectors. Since we have a small dataset, we used N-fold cross-validation for the evaluation.

5.3 A Deep Learning Approach

We also utilized deep learning (DL) algorithms to learn about the potential to discriminate between depressed and nondepressed individuals based on images. We used the same aforementioned classes: Predominantly depressive and Barely depressive. Due to a small dataset, we used transfer learning, which is a DL approach where learned features in a pre-trained model with larger datasets are transferred to a second network with other target tasks and data.

The transfer learning approach has proven to be powerful when there are data in a target network [⁴⁵]. The base model used for the transfer learning was pre-trained with the ImageNet dataset^fn, which contains more than 14 million images within 1, categories.

The architecture used is ResNet50 [¹⁵], from which the weights of the pre-trained model were transferred to our target classification task, removing the output layer of ResNet50 and adding the connected layers of classification by image. We used the library Tensorflow^fn for the experiments. We performed four different experiments in this DL approach.

For the 4 experiments, we used transfer learning with the model with ResNet50 architecture. The key difference among the 4 experiments was the datasets used. Since we have a small dataset, we sought to augment it with similar datasets labeled from people with depression or labeled as positive or negative images.

As mentioned, our participants shared images with artistic content and drawings besides photos of people and landmarks. We found datasets with these types of content that are labeled as positive and negative, which we next describe.

In total, we used 3 datasets for these four experiments (E1, E2, E3, E4), merging some of them for the experiments: a) the dataset we collected (i.e., IG dataset); b) a dataset from abstract paintings from the MART museum from Italy (i.e., MART dataset); c) and a dataset from artistic images showing emotions (i.e., Art photo dataset). We next list the experiments:

During these experiments, the training across the 4 experiments was made through 45 epochs. We tested with dropout regularization to avoid overfitting.

In the following sections, we describe each of the following experiments with mode details. The results of those will be shown in the following pages.

5.3.5 Classification Results with DL

Figure 7 and Figure 8 show the training and validation accuracy from the four experiments. As we can see, the results are lower than 0.60 accuracy across all the experiments in both the training and validation sets. This experiment provides an understanding of the potential of using the algorithms in this problem using all the data for training and validation.

Fig. 7 Training accuracy through 45 epochs 

Fig. 8 Validation accuracy through 45 epochs 

For further exploration, we trained and validated using only the MART dataset, not including the Instagram data. The accuracy results are 1 in training and 0.75 in validation, showing overfitting but, most importantly, showing that adding the Instagram datasets significantly decreased results. This can be caused due to different reasons, such as the image distributions from both datasets being different or the labels positive or negative not aligning well with the classes from the Beck’s depression inventory.

In general, just like most available datasets, both the MART dataset and the Art photo dataset were labeled a posteriori by third parties and do not properly reflect the emotions when a photo is taken or posted by an individual.

On the contrary, our Instagram dataset was collected in the wild, where our participants were categorized according to the Becks’ depression inventory, who might be privately dealing with their condition, showing subtle signs through their posts while not being truly explicit about it. The results obtained through deep learning are not conclusive.

We believe that these results can be improved by combining more information from the posts, such as the image captions. However, we wanted to show these results, which can illustrate the difficulty of this problem. More studies of this kind are needed to deepen the understanding of this problem and the way the DL can help tackle it.

6.1 Ecological Validity

The in-the-wild nature of this study can be double-edged. On the one side, the findings derived from the data can be ecologically valid [⁴, ⁷], since the dataset we collected is similar to what young adults are posting on Instagram in their day-to-day lives.

Using hashtagged posts can cause filter bias during analysis [²⁹]. To avoid this, in this work, we considered all types of posts by users during a four-week window, which can be difficult for algorithms to discriminate due to the noise derived from heterogeneous content published by users.

One of the challenges in this work is the diversity in the data across various levels of depression, as categorized by Beck’s depression inventory. For instance, an image shared by a person with severe depression might be similar in terms of colors or lighting to an image shared by a person without any particular sign of depression.

At the same time, people with the same severity of depression might share unrelated or opposite images. To illustrate the heterogeneity of the dataset, Figure 9 shows the HSV levels and GPA/GNA data related to four posts from different users who were categorized either as minimal or severe, according to Beck’s depression inventory.

Fig. 9 Posts from four participants: (a) Severe 1: GPA=11, GNA=43; (b) Severe 2: GPA=29, GNA=36; (c) Minimal 1: GPA=46, GNA=15; (d) Minimal 2: GPA=41, GNA=10. The maximum value for GPA and GNA is 50. Faces have been blurred for privacy 

We also show the actual images, and the caption associated, which have been translated to English by the authors. Following, Figure 9a and Figure 9b, we can see that Hue and Value (ie., brightness) are similar, but not in terms of Saturation. Also, both GPA are relatively low (GPAa=11, GPAb=29) and both GNA are high (GNAa=43, GNAb=36).

We want to highlight that both images were posted by participants categorized as severely depressed. Interestingly, these images yielded similar values to Figure 9c in terms of hue, saturation, and value (HSV). In other words, images Figure 9a and Figure 9b look alike to their polar opposite, Figure 9c.

Presumably, one expected type of image with high GNA and low GPA from our severely depressed participants is Figure 9c, which is dark and essentially colorless. This image strongly differs from Figure 9d, even when both have similar values of GPA (GPAc=46; GPAd=41) and GNA (GNAc=15, GNAd=10). Taking all of these aspects into account, using machine learning approaches for classifying these sorts of images is not trivial.

In most related work, the challenge associated with using complete datasets from the day-to-day lives of users has not been accounted for since datasets generally originate from homogeneous groups (e.g., online communities for depression support) or commonly used hashtags on SNS (e.g., #depression).

Since posts can have similar characteristics stated from the moment that they are posted, much more homogeneous data within target classes can be obtained, and higher variations between classes can have a significant positive impact on the performance of machine learning approaches. As we have seen in this work, having a more diverse dataset has a negative impact on the performance, especially due to the size of the dataset.

6.2 Limitations and Scope

One limitation of the present study is the quantity of data collected. For the statistical and machine learning approaches, it is highly beneficial to have large datasets (in this case, posts and users) to be able to yield better results. Therefore, this study could have benefited from a larger sample of users over a longer period.