nueva página del texto (beta)
Inglés (pdf)
Artículo en XML
Referencias del artículo
Traducción automática
Enviar artículo por email
Citado por SciELO 
Similares en
SciELO 
Permalink
1 Introduction
Social blogging platforms allow authors to share information, their personal experiences and opinions on a wide variety of topics. These platforms also allow readers to leave their comments on the blog. The readers might agree or disagree with the author, provide suggestions for improving the blog or might have some questions. The authors need to make sure that the blogs they write cater to the taste of their audience and are understood by them so that they remain engaged. The primary mechanism for an author to understand how her audience are reacting to the blogs is to understand the comments they write for the blog. For the popular blogs, the number of comments can be large. Hence, if the author has created many blogs, reading all the comments, understanding the commenter's reaction, and which parts of the blog (called blog segment) need attention is a tedious and time-consuming task. Understanding what parts of the blog the audience did not understand (and had questions about), what parts did they agree/disagree with, and where did they feel the need to make changes to the blog can enable the authors to write the blogs in a more engaging way in future or to improve the existing blogs. In absence of a tool support, the authors do not fully benefit from insightful comments made by the readers to improve the content of their future blogs (or make changes to existing ones).
In this work, we present a method for automatically associating the comments to the part of the blog they refer to and classifying the comments as Suggestion, Agreement, Disagreement or Question. Additionally, we provide a visual representation of this information, so that authors can quickly understand the type and strength of user reactions generated by the various parts of their blogs. This analysis can also help a reader in determining what blogs (and which parts of these blogs) should they read depending on their interests and roles. In today's world, where information overload is a major challenge, such insights can help them become more productive.
In Section 2 we detail the prior explorations in the field of comment association and classification. In Section 4 we describe the methodology in detail and present the results in Section 5 followed by the conclusion.
2 Related Work
To the best of our knowledge, there is no existing work which talks about comment association and classification simultaneously. Moreover, there is no work which classifies comments in the set of categories that we are considering, though there are different works which have looked at these categories separately.
2.1 Classification
Classification of user comments is a well-researched area. People have come up with various classification schemes for YouTube video comments [17], product reviews [3], tweets [5], etc. Most of these schemes talk about the type of sentiment, emotion [16] or mood expressed in the comment. There are a few works on identification of spam, off-topic, obscene, toxic and abusive [2] comments made on online blogs or YouTube videos 1.
There are also a few works which focus on the semantics or content aspects of the short texts. There are prior explorations [21,22] on advice mining from the web forums which introduce various linguistic features which can be used to identify advices.
We leverage these features to classify sentences as suggestion. [22] proposed a hidden Markov model for labelling sequential sentences as advice revealing or not and use syntactic, semantic and contextual features for their task. Our task is different from theirs since their task involves independent comments and not sequential sentences.
With respect to the agreement and disagreement categories, there have been previous work on recognizing disagreement in informal political arguments [1] and identifying agreement and disagreement in the social media dialogues [11]. Among these work, [1] show that use of contextual and dialogue features improve accuracies as compared to unigrams. Topic independent features improve the performance of agreement-disagreement classification over unigrams as demonstrated by [11].
Apart from this, stance detection in tweets is also similar to identifying agreement and disagreement in text with respect to a part of the blog. A set of structural, contextual, sentiment and label-based features for predicting stance towards a mentioned target are defined in [9]. However, above mentioned approaches will not directly work in our case because of the differences in domain and also the dataset under consideration.
While there has been significant work on classifying short text, some of which also address comment classification, we are not aware of any work which classifies the comments in multiple classes, and in particular, the classes we are taking into consideration (agreement, disagreement, questions and suggestions).
We hypothesize that comments belonging to these classes will provide constructive and valuable insights about the blog to its author and other readers.
2.2 Association of Comments
There have been some research in the area of associating comments to a part of the news story and aligning comments to the news topics. An unsupervised technique is proposed in [18] which takes cosine similarity of LDA, SS-PLSA and BOW features of both comment and the segment of the news article to align comments with the segments. [19] propose a supervised technique for the task of alignment and show that the structured learning approach performs better than the other unsupervised and binary classification approaches. Frameworks for aligning comments to news topics by automatically extracting topics from a given news article and its associated comments are described in work like [7,6]. However, in this work we position the comment association task as a 'question-answering' task where comment is considered as a query and the different parts of the blogs as the answer.
3 Problem Definition
The problem that we are trying to solve can be stated as: How to help authors and readers in extracting useful insights from the comments on a blog to:
— Help authors in understanding the audience reaction and improve upon her writing in future
— Help readers in understanding the blog through comments, possibly to prioritize the reading
In this work, we aim to solve the following sub-problems which define the useful insights that we will be presenting to the authors/readers.
Understanding the scope of the comment with respect to the blog. By this we mean associating the comment to the segment(defined at the level of a paragraph) of the blog it is referring to.
-
Understanding the type of the comment. We are considering the following types:
— Agreement: comments that support (parts of) the content of the blog.
— Disagreement: comments that contradict/disapprove of (parts of) content in the blog.
— Suggestion: comments that advise or suggest changes to the content or suggests some alternatives to what's present in the blog.
— Question: comments that capture queries or doubts about (parts of) the content in the blog.
Visual representation of the comments related information for better insights
We hypothesize that the classes under consideration are exhaustive (after the removal of irrelevant comments) because there could be two scenarios in which reader can comment (1) reader does not understand the blog, and (2) the reader understands the blog. In scenario (1) he/she will have doubts with respect to the content of the blog and thus his/her comments will belong to the 'Question' category. In case (2) he/she will either have some suggestions for improving the content or will agree/disagree with the content. There might also be some comments, which go off-topic or are general point of views like "I don't like traveling alone" but we are not considering them since they can always be preprocessed and filtered out.
4 Methodology
In this section, we describe the dataset, the features, and the techniques used for the classification and the comment association task.
4.1 Data Preparation
We are not aware of previous work which simultaneously tackles the tasks of comment classification and its association with the part of the blog. Therefore, we curated a dataset to suit our purpose with the help of human annotators.
We collected a total of 90 blogs from different online blogging websites and asked the annotator to write comments on the presented blog indicating which part of the blog it corresponds to and the category (suggestion, question, agreement, disagreement) it belongs to.
We had a total of 271 comments with 95 in Question, 70 in Disagreement, 674 in Agreement and 39 in Suggestion category. Since the size of our corpus was small we used auxiliary datasets for the training the classifiers individually. We used Open Domain Suggestion Mining dataset [12] for the suggestion classification task. This dataset consists of tagged sentences from various domains such as electronics reviews, hotel reviews, customer service reviews, and travel forums. We used a subset of Internet Argument Corpus (IAC) [20] for the agreement-disagreement classification task. This dataset consists of pairs of the kind (quote, response), where quote is the base sentence and the response either agrees or disagrees with the quote. There was a huge class imbalance in this data and thus we downsample sentences with disagreement label to get a balanced dataset.
4.2 Classification
We build separate classifiers for Suggestion, Question and Agreement/Disagreement for the training purpose and propose a hierarchical approach to classify the given test comment into one of the possible categories. The categories under consideration are mutually exclusive in our case and thus each comment can belong to only one category.
4.2.1 Suggestion Classifier
We train a binary SVM classifier for classifying the comment as suggestion or non-suggestion. Our main contribution is in feature engineering. We consider the following set of features for this classification task and show the performance improvement results in the next section.
Clue Words (Clue): We curated our own list of clue words (such as suggest, recommend, advice, urge, request, etc.) which were selected by investigating the training data. A binary feature vector of dimension equal to the number of clue words is created where the value corresponding to each dimension denotes the presence or absence of a clue word in the comment.
Modal Verbs (MV): [21] has shown that advice revealing sentences often expresses modality which are expressed using the modal verbs (such as can, could, might, should, would, etc.). We define a set of modal verbs and a binary feature corresponding to each of the modal verb, indicating the presence or absence of the modal verb in the comment.
Imperative Mood Expressions (IME): [21] found that sentences containing imperative mood expressions (such as 'do not bring mobile phones', 'it is a good idea to add more experimental results') result in the actions in certain ways. We also used this feature in order to characterize the suggestions. We used the same heuristic method as defined by [21] for finding value of this feature. The heuristic says that if the verb present in the comment is not preceded by a subject, then most likely the comment contains an imperative mood expression.
Typed Dependencies (TypDep): We leveraged this feature as defined by [22]. We considered only conjunct, clausal subject, and nominal subject relations, which are denoted by "conj","csubj", and "nsubj", respectively in the comment's parse tree obtained using Stanford Dependency Parser [4].
-
Informativeness Score (InfScore): It is the summation of the tf-idf score of all the words in the comment:
where C is the comment and wi is the word in the comment.
The training was done using the auxiliary dataset mentioned in above subsection. We used radial basis function (rbf) kernel while training the SVM.
4.2.2 Question Classifier
We consider identification of question as a binary classification task. We use Stanford parser [10] for obtaining the parse tree of the given sentence and check for the presence of either of the two tags, namely SBARQ and SQ in the tree.
SBARQ: Presence of this tag indicates the presence of a direct question introduced by a wh-word or a wh-phrase.
SQ: Presence of this tag indicates the presence of an inverted yes/no question, or main clause of a wh-question, following the wh-phrase in SBARQ.
We mark the given sentence as belonging to a Question category if either of these tags is present in the parse tree of the comment.
4.2.3 Agree-Disagree Classifier
We train a binary SVM classifier in order to classify the sentences into agreement and disagreement category. Presence of agreement or disagreement depends upon the context and thus for this classification task we also consider the segment of the blog that the comment is associated with. Each of the features we experimented with is explained below.
LIWC [14]: These features are used by [1] for recognizing disagreement in political arguments. We hypothesize that use of linguistic features will help in identifying the agreement and disagreement. We calculate the LIWC features for both the comment and the part of the blog it belongs to.
Glove Embedding (Glove) [15]: This feature represents the semantics of the comment. The feature value is a 50-dimensional vector obtained by the summation of the 50 dimensional embeddings of all the words present in the comment.
N-grams: We curated our own list of n-grams after investigating the data. These n-grams characterizes the presence or absence of agreement in the comment. For each n-gram, a binary value is assigned depending upon the presence or absence of that n-gram in the comment.
Positive and Negative sentiment words (Pos-Neg): We leverage the positive and negative sentiment words curated by [8] for identifying the polarity of product reviews to classify the comment as agreement or disagreement. We consider the difference in the number of positive and negative sentiment words present in the comment as the feature.
Positive Sentiment words (Pos): This feature denotes the number of positive sentiment words present in the comment.
Negative Sentiment words (Neg): This feature denotes the number of negative sentiment words present in the comment.
Afinn score (Affin) [13]: It gives a word polarity score between -5 to +5. The feature value is the summation of the afinn score of all the words in the comment. The same feature is also calculated for the part of the blog that the comment belongs to.
The training was done using the auxiliary dataset mentioned in the previous subsection. We used radial basis function (rbf) kernel while training the SVM. The model performed best with just the N-gram features.
4.2.4 Hierarchical Classifier
Given a comment, we propose a hierarchical approach to classify it into one of the classes we are taking into consideration. We use the individually trained classifier models as described in the previous section with the features as per the best performing models.
Figure 1 shows the workflow of this classifier. The comments first go through the Suggestion classifier which if predicted as a suggestion, are classified as suggestion and not passed through any of the other classifiers. All the comments which are not classified as suggestions are fed to the Question classifier. If the comment is predicted as a question, then we label it as a question and pass the rest of the comments through the Agree-Disagree classifier.
This classifier finally classifies all the remaining comments into agreement or disagreement.
We believe that some suggestions like "Did you try deep Learning approaches?" are written in question form and will get mis-classified if they are first passed through the question classifier. Thus, we chose this hierarchy.
4.3 Association of Comments with Segment of the Blog
We model the association task as a 'question-answering' problem and use Learning to Rank models to rank different segments given a comment. We consider comment as the query and the segments of the blog as the answers.
Given a query the model ranks the answers. We used RankLib library's ListNet2model for this purpose.
We use the following lexical and semantic features for our purpose:
Lexical Features:
1. Segment Length (SegLen): Number of terms in the segment
2. Segment Position (SegPos): Relative position of the segment with respect to the blog
3. Exact Match (EMatch): It is a binary feature indicating whether the comment is a substring of the segment
4. Term Match (TMatch): Number of terms that are common in the comment and the segment
5. Synonym Match (SMatch): It is the fraction of comment's terms whose synonym is present in the segment
6. Language Model(LM): It is a score which is computed as the log likelihood of the comment being generated from the segment
Semantic Features:
5 Evaluation and Results
We evaluated comment association task on the following two metrics.
-
Mean Reciprocal Rank (MRR): It is given by:
where C is the set of the comments that are queried for association and ranki, refers to the rank position of the correctly associated segment.
Percentage accuracy: It is the ratio of comments correctly associated to the total number of comments queried for the association.
Table 1 shows the results of the task of associating the comments with the segment of the blog. It can be inferred from the results that use of Word2Vec embeddings (W2V) is bringing down both MRR and accuracy values as compared to the case when it is not used. However, use of Universal Sentence Encoding (USE) feature along with all the other lexical features improves both MRR and accuracy values. There is an appreciable improvement in the metric values when only Term match (TMatch) feature is used.
Table 1 Results of the Comment Association Model
| Features | MRR | Accuracy |
|---|---|---|
| SegLen+SegPos+EMatch+TMatch+SMatch+LM+W2V | 0.745 | 0.631 |
| SegLen+SegPos+EMatch+TMatch+SMatch+LM | 0.763 | 0.675 |
| SegLen+SegPos+EMatch+TMatch+SMatch+LM+USE | 0.769 | 0.692 |
| TMatch | 0.849 | 0.798 |
We evaluated our individual classifier models and the hierarchical classifier model on Precision, Recall and F1 score metrics.
Table 2 presents the results from the Suggestion classifier. We can clearly see that there is a significant improvement in recall and f1-score when clue words (Clue) are used along with modal verbs (MV) and imperative mood expressions (IME). When clue words (Clue) and modal verbs (MV) features are used along with typed dependency (TypDep) and informativeness score (InfScore) features the precision score increases at the expense of recall. Finally using all the features together shows significant increase in the recall and f1 score values, indicating that all these features together help in identifying the suggestive characteristics of the comments.
Table 2 Results for Suggestion Classifier
| Features | Precision | Recall | F1 score |
|---|---|---|---|
| Clue | 0.44 | 0.18 | 0.25 |
| Clue+MV+IME | 0.46 | 0.64 | 0.54 |
| Clue+MV+TypDep+InfScore | 0.48 | 0.59 | 0.53 |
| Clue+MV+IME+TypDep+InfScore | 0.47 | 0.62 | 0.53 |
Our model was able to identify the question with a precision of 0.86 and recall of 0.73 with F1-score being 0.79.
Table 3 presents the results of the agree and disagree classifier. We can see that using the N-grams provides good performance trade-off.
Table 3 Results for Agree-Disagree Classifier
| Features | Precision | Recall | F1 score | |||
|---|---|---|---|---|---|---|
| Agree | Disagree | Agree | Disagree | Agree | Disagree | |
| LIWC | 0.54 | 0.55 | 0.43 | 0.65 | 0.48 | 0.60 |
| Glove+N-grams+PosNeg | 0.63 | 0.65 | 0.63 | 0.65 | 0.63 | 0.65 |
| Glove+N-grams+Pos+Neg | 0.63 | 0.66 | 0.64 | 0.65 | 0.64 | 0.66 |
| Glove+N-grams+Affin | 0.67 | 0.66 | 0.60 | 0.72 | 0.63 | 0.69 |
| N-grams | 0.65 | 0.79 | 0.84 | 0.58 | 0.73 | 0.67 |
Table 4 presents the results of the hierarchical classifier. It is evident from the metric values that with the hierarchical classification the precision, recall and f1 score improves since once classified by one classifier as positive, the comment is not passed to the next classifier.
Table 4 Results for Hierarchical Classifier
| Class | Precision | Recall | F1 score |
|---|---|---|---|
| Suggestion | 0.47 | 0.62 | 0.53 |
| Question | 0.95 | 0.75 | 0.84 |
| Agreement | 0.63 | 0.85 | 0.72 |
| Disagreement | 0.76 | 0.50 | 0.60 |
As one can see, the results of classifier are satisfactory to provide useful insights, even though there is room for improving these classifiers.
5.1 Visualization
We built a mobile app to present these insights about the comments and the blog to the author and the readers.
Figure 2a shows the landing page of the app where the author/reader can see the list of the blogs he/she has written or can read. The doughnut chart on the right side of each blog shows the category-wise distribution of the comments made on that blog and the number inside the chart is the total number of comments made on that blog. The legends in the top bar shows the class represented by each colour in the chart.
Figure 2b presents the view when a particular blog is chosen. The blog is partitioned into the segments demarcated by the blocks. The number in parenthesis besides each category is the count of comments on the blog that belongs to the category. The scroll bar is segmented according to the segments in the blog and the colour represents the dominant comment category for that segment.
Figure 2c shows the view of blog when the author/reader click on the scroll bar which takes him/her to the corresponding segment where the list of the comments and the category they belong to can be seen.
The author(or reader) can also look at the comments on the blog belonging to only a particular category by clicking on that category from the top legend bar. It can be seen from Figure 2d that all the segments coloured yellow have comments from question category.
As one can see, our tool can help the authors and readers by providing insights about what type of reactions/comments a given blog is attracting, and which parts of the blogs are responsible for those reactions. One can consider enabling several features using such information, for example, sorting the blogs based on most or least number of comments of a particular type (agreement/disagreement/question/suggestion). Also, one may directly find the segments of blogs which attract specific type of comments (and need not open the blogs individually to see them). One can also create summaries of comments of a given type for a particular segment of blog.
Such summaries are more useful as they would be talking about the same thing in same manner, and hence the possibility of creating a coherent summary is higher.
6 Conclusions
This paper presents a way to leverage the information present in the comments and deliver insights to the authors about the audience's reaction to the content at a granular level. These insights will help the author in understanding the audience and improving upon the future content. Readers can also benefit from the comments as they help in understanding the blog and possibly help in prioritizing which blog (and which parts of it) they should read.
