1 Introduction

During recent years, there have been a lot of research in the area of Natural Language Processing (NLP) related to sentiment analysis [¹², ¹³, ¹¹, ¹⁰].

Stance detection can be viewed as a subtask of opinion mining, similar to sentiment analysis. In sentiment analysis, systems determine whether a piece of text is positive, negative, or neutral. Stance detection goes even further and tries to detect whether the author of the text is in favor or against a given target. The main difference to sentiment analysis is that in stance detection, systems are to determine the author’s favorability towards a given target and the target may not even be explicitly mentioned in the text. Moreover, the text may express positive opinion about an entity contained in the text, but one can also infer that the author is against the defined target (an entity or a topic). It has been found difficult to infer stance towards a target of interest from tweets that express opinion towards another entity [⁸].

There are many applications which could benefit from the automatic stance detection, including information retrieval, textual entailment, or text summarization, in particular opinion summarization.

The same stance towards a target may be expressed by positive or negative language. This phenomenon has not yet been thoroughly investigated. The pioneer work in English Tweets [⁹] annotated stance dataset with additional sentiment labels and show that knowing the sentiment label is beneficial for stance detection, however they also state that “even though sentiment can play a key role in detecting stance, sentiment alone is not sufficient”.

Our goal is to examine how stance and sentiment influence each other in Czech language and either confirm or reject the hypothesis that sentiment labels are beneficial for stance detection.

The rest of this paper is organized as follows. Section 2 presents the related work. The dataset is described in Section 3. The annotation of sentiment is covered in Section 4. Our approach is presented in Section 5.

Conducted experiments are described in Section 6. Finally, we conclude in Section 7.

2 Related Work

The SemEval-2016 task Detecting Stance in Tweets¹ [⁸] had two subtasks: supervised and weakly supervised stance identification.

The goal of both subtasks was to classify tweets into three classes (In favor, Against, and Neither). The performance was measured by macro-averaged F1-score of two classes (In favor and Against) denoted F1_ma2 and by micro-averaged F1-score for the same two classes denoted F1_mi2. This evaluation measure does not disregard the Neither class, because falsely labelling the Neither class as In favor or Against still affects the scores. We use the same evaluation metrics F1_ma2, accuracy, and the F1-score of all classes (F1_ma3).

The supervised task (subtask A) tested stance towards five targets: Atheism, Climate Change is a Real Concern, Feminist Movement, Hillary Clinton, and Legalization of Abortion. Participants were provided with 2814 labeled training tweets for the five targets.

A detailed distribution of stances for each target is given in Table 1. The distribution is not uniform and there is always a preference towards a certain stance. The distribution reflects the real-world scenario, in which a majority of people tend to take a similar stance [²].

For the weakly supervised task (subtask B), there were no labeled training data but participants could use a large number of tweets related to the single target: Donald Trump.

The best results (F1_ma2 56.0%, F1_mi2 67.8%) for subtask A were achieved by an advanced baseline using SVM classifier with unigrams, bigrams, and trigrams along with character n-grams (2, 3, 4, and 5-gram) as features.

Wei et al. [¹⁵] present the best result for subtask B and they ranked close second in subtask A of the SemEval stance detection task. They used a convolutional neural network (CNN) designed according to Kim [⁴].

They initialized the embedding layer with pre-trained word2vec embeddings. The main difference from Kim’s network is the used voting scheme. During each training epoch, several iterations were selected to predict the test set. At the end of each epoch, the majority voting scheme was applied to determine the label for each sentence. This was done over a specified number of epochs and finally the same voting was applied to the results of each epoch. The train and test data were separated according to the stance targets.

Mohammad et al. [⁹] annotated the SemEval-2016 task Detecting Stance in Tweets dataset [⁸] with sentiment labels and whether the opinion is expressed towards the given stance target. They performed a detailed analysis of the dataset and conducted several experiments. They showed that sentiment label is beneficial for stance detection however it is not sufficient (F1_ma2 56.1%, F1_mi2 59.6%).

3 Dataset

The dataset for the target entity “Miloš Zeman” was annotated by one annotator and then 302 comments were also labeled by a second annotator to measure inter-annotator agreement. The dataset for the target entity “Smoking Ban in Restaurants” was independently annotated by two annotators (2,203 comments) and then the majority voting scheme was applied to the gold label selection (third annotator was used to resolve conflicts). The inter-annotator agreement (Cohen’s κ) is 0.579 for “Miloš Zeman” and 0.423 for “Smoking Ban in Restaurants”.

The inter-annotator agreement for “Smoking Ban in Restaurants” was quite low, thus they selected a subset of the dataset, where the original two annotators assigned the same label as the gold dataset (1,388 comments).

4 Annotation

We annotated the Czech president - “Miloš Zeman” stance detection dataset with sentiment labels (positive, negative, and neutral).

The whole dataset was annotated by one annotator and then a second annotator was used to calculate inter-annotator agreement (Cohen’s κ) on 131 comments. The annotators should assign the strongest sentiment to each comment or neutral label when the comment is factual (non-subjective) without anticipating further information (context). The inter-annotator (Cohen’s κ) is 0.524% (see the confusion matrix Table 4) and accuracy is 71.8%.

Table 3 shows the distribution of sentiment and stance labels in the extended dataset. While most comments are against the target, the sentiment of most comments is neutral and only a small portion of the dataset is positive. Most of the comments that are in favor of the target are neutral which means that the comments are non-subjective, however the comments against the target are mostly negative and almost none is positive. The comments neither for nor against the target are mostly neutral as expected. For positive sentiment the comment is mostly in favor of target. Negative sentiment most of the time means against the target and neutral sentiment is almost uniformly distributed across stance labels.

We also labeled the comments for the presence of the “Miloš Zeman” entity and the “president” entity. The distribution of entities by stance and sentiment labels is shown in Table 5. The presence of these entities was detected by regular expressions⁴.

The extended corpus annotated with sentiment labels and marked for the presence of entities “Miloš Zeman” and “president” is available for research purposes at http://nlp.kiv.zcu.cz/ research/sentiment#stance.

5 The Approach Overview

For all experiments we use Maximum Entropy classifier from Brainy machine learning library [⁵]. We evaluate using 20-fold cross-validation to allow comparison with previous work [²].

6 Experiments

For all experiments we report the macro-averaged F1-score of two classes F1_ma2 (In favor and Against) - the official metric for the SemEval-2016 stance detection task[⁸], accuracy, and the macro-averaged F1-score of all three classes (F1_ma3).

Table 6 shows results of all our experiments. We performed experiments with using the gold sentiment labels as features for stance detection and using the gold stance labels as features for sentiment analysis (i.e. using the Oracle feature). The results show that the Oracle feature improves results in all cases. The Oracle feature combined with unigrams and character n-grams also outperforms the previous state-of-the-art results for stance detection by 3.0% F1_ma3, 2.6% F1_ma2, and 2.2% Acc.

Another experiment included using features that indicate the presence of the “Miloš Zeman” entity and the “president” entity combined with the gold labels as in Oracle feature. Our expectation was that this should improve the results (as it did in English), however the results show that in fact the information about the presence of the target entity does not lead to better results.

We further performed an ablation study for the combination of features (ChN_5,7 + E + FW + LW + TL + Sh + WN_1,2,3 + O + TP + TZ). In Table 6 the bold numbers denote the best results for the given column.

The ablation study shows that the FW feature present little to no information gain for the classifier. We further experimented with combinations of features and that lead to the best feature sets for both stance detection and sentiment analysis (see the last two lines in Table 6). Both of these sets contain emoticons, word shape, oracle and target entities.

7 Conclusion

We presented the first Czech dataset annotated for both stance and sentiment labels including the presence of target entities. We have shown that stance and sentiment can be mutually beneficial and confirmed our initial hypothesis. Moreover, we have outperformed the state-of-the-art results for stance detection in Czech and set a new state-of-the-art results for the sentiment analysis part of the dataset.

Our best result outperformed the previous stance detection state of the art by 4.9% F1_ma3, 3.9% F1_ma2, and 4.2% Acc. The sentiment analysis unigram baseline was outperformed by 6.9% F1_ma3, 7.1% F1_ma2, and 8.0% Acc.

In the future we plan to extend this analysis on other target entities and explore the usefulness of labels assigned by trained models instead of using gold labels for the Oracle feature.