Measuring Similarity of Opinion-bearing Sentences (2021)

Abstract: For many NLP applications of online reviews, comparison of two opinion-bearing sentences is key. We argue that, while general purpose text similarity metrics have been applied for this purpose, there has been limited exploration of their applicability to opinion texts. We address this gap in the literature, studying: (1) how humans judge the similarity of pairs of opinion-bearing sentences; and, (2) the degree to which existing text similarity metrics, particularly embedding-based ones, correspond to human judgments. We crowdsourced annotations for opinion sentence pairs and our main findings are: (1) annotators tend to agree on whether or not opinion sentences are similar or different; and (2) embedding-based metrics capture human judgments of “opinion similarity” but not “opinion difference”. Based on our analysis, we identify areas where the current metrics should be improved. We further propose to learn a similarity metric for opinion similarity via fine-tuning the Sentence-BERT sentence-embedding network based on review text and weak supervision by review ratings. Experiments show that our learned metric outperforms existing text similarity metrics and especially show significantly higher correlations with human annotations for differing opinions.

## introduction online reviews are an integral part of e-commerce platforms. consumers utilize these reviews to make purchasing decisions, and businesses use this feedback to improve products or services. with the ever-growing number of reviews, nlp research has focused on methods to make sense of this vast data resource, including applications for opinion summarization @xcite @xcite @xcite and opinion search @xcite . a key characteristic of text in this domain is that it contains opinion-bearing sentences (hereafter, "opinion sentences"). as in preceding work @xcite , we view an opinion as having an aspect (e.g., the feature of a product or dimension of a service) and an appraisal (e.g., a positive or negative sentiment). in many applications, one needs to determine if two related opinion sentences are comparable in meaning. from an applied viewpoint, one might think of two opinions being comparable if they support the same recommendation, with respect to the relevant aspect. to compare two opinions, prior work has employed text similarity metrics, where cosine similarity based on tf-idf @xcite or embedding representations @xcite ) is used to measure opinion sentence similarity. we group existing text similarity metrics broadly into two types: lexical-and embedding-based approaches. the lexical-based approaches, such as bleu @xcite and rouge @xcite , evaluate text by capturing the overlap in surface forms, such as n-grams of tokens. however, they are often ineffective when texts employ paraphrases or synonyms. embedding-based metrics, such as word mover's distance (wmd) @xcite , moverscore @xcite , and sentence-bert (sbert) @xcite , typically relax the restriction of strict string matching by comparing continuous representations for words and sentences. such approaches have shown to work well for various nlp applications, including areas involving comparison of sentence meaning; for example, paraphrase detection, question answering, or summarization @xcite @xcite . however, a detailed study investigating their relationship to corresponding human judgments of opinion sentences is lacking. in other text similarity settings, such as summary evaluation @xcite , caption evaluation @xcite and machine translation evaluation @xcite , embedding-based metrics out-perform lexical-based metrics, as demonstrated by its increased correlation with human judgment scores. however, embedding-based metrics have not yet been evaluated on opinion texts. the success of these embedding-based metrics in other types of text (such as news) cannot be guaranteed for opinion texts. this is because opinion text can be associated with a sentiment polarity. opinion bearing words that are opposite in sentiment polarity are semantically related. yet, many of these embedding-based metrics are often trained on semantic relatedness but not specifically sentiment polarity. we address the gap of lacking research on similarity for opinion-bearing texts in the literature with the following research questions: (1) how do humans evaluate similarity of two opinion sentences? (2) how well do existing metrics capture similarity in a way similar to humans? and if not well, (3) how do we develop metrics to more effectively measure similarity for opinion sentences? we address the first question by conducting a crowdsourcing task that collects human annotations for the degree of similarity of two opinion sentences. foot_0 for the second research question, we examine the correlation of the text similarity metrics against our crowdsourced annotations. for the third question, we explore approaches to fine-tune embedding-based metrics for similarity of opinion texts. we make several contributions: (1) we collect and release a dataset of 1635 sentence pairs with similarity scores; (2) we show that annotators broadly agree on whether an opinion sentence pair is "similar" or "different"; (3) we demonstrate that text similarity metrics have weak correlation to human judgments of opinion similarity, and that they perform poorly with differing opinions in particular; (4) we conduct an analysis of differing opinions to characterize the limitations of such approaches when dealing with opinion sentences; and, ( 5 ) we propose to learn a metric for similarity of opinion texts by fine-tuning sbert via weak supervision by review ratings. our experiments show that the fine-tuned sbert model outperforms existing metrics for distinguishing different opinions and for measuring similarity of opinion sentences. ## related work our research is related to text similarity metrics, which broadly include lexical-based metrics, embedding-based metrics and learned metrics. lexical-based metrics rouge @xcite ) is a commonly used metric for opinion summary evaluation. it measures similarity between texts by counting the overlaps of n-grams. bleu @xcite is the default metric for machine translation evaluation. similar to rouge, it also relies on counting overlaps in n-grams. such lexical matching methods face the same limitation in evaluating texts that are similar in meaning but expressed with different words @xcite . meteor @xcite is proposed to relax the exact n-gram matching to allow matching words with its synonyms. embedding-based metrics embedding-based metrics are proposed to overcome the limitations of lexical-based metrics @xcite @xcite . word mover's distance (wmd) @xcite and moverscore @xcite measure how similar two texts are by accumulating the distance between word embeddings and contextual embeddings, respectively. sentence-bert (sbert) @xcite is a sentence encoder that can be used with cosine similarity to capture similarity of meaning between sentences. ## on metrics and human judgments the following baseline metrics are chosen for our investigation: (1) rouge variants, rouge-1, rouge-2 and rouge-l without stemming and without stopword removal, using rouge-2.0 @xcite ; (2) spice @xcite , an image captioning evaluation metric that compares the scene graph of one text against the reference text; and, (3) wmd, using the implementation of wmd from gensim @xcite with the normalized 300-dimension word2vec trained on google news. we follow @xcite to transform wmd scores to a similarity score using exp -(w m d) ; (4) sbert, using sentence-transformers library in python; and (5) moverscore, using the authors' implementation. these metrics are representative of the different types of metrics. rouge is a lexical-based metric, spice is a metric that incorporates representations of sentence meaning, and wmd, sbert and moverscore are embedding-based metrics. pearson and kendall correlations are reported in ## sos: sentence-bert for opinion similarity our earlier analysis shows that existing embeddingbased metrics have low correlation with human judgments for sentence pairs of opposite sentiment polarity. we hypothesize that the performance of embedding-based metrics can be improved with sentiment polarity information. a straightforward approach to improving embedding-based metrics for opinion similarity is to use sentimentally trained word embeddings. wmd is based on the word2vec word embeddings trained on google news. for opinion similarity, the sentiment specific word embeddings trained on tweets and the sentiment information associated with emoticons @xcite can be used. we call this baseline approach wmd-sswe. motivated by the observation that bert-based sentence embeddings has shown superior performance for measuring sentence similarity, we propose to learn a metric for opinion similarity based on sbert. our metric is a siamese network of sbert that takes in two sentences as inputs and outputs a similarity score. to overcome the problem of costly human annotated similarity score for training, we propose to train the metric through weak supervision based on review ratings. we call our model sos (sbert for opinion similarity). online reviews usually contain a review text and review rating. the review text contains opinion texts and the review rating provides an overall sentiment polarity of review text. for popular review platforms, the review rating typically spans a score of 1 to 5. a review rating of 1 is negative and 5 is positive. we can draw the connection that a review text associated with higher rating is positive. similarly, a review text associated with lower rating is negative. in our work, we consider positive reviews to have a star rating of 4 and 5, while negative reviews to have a star rating of 1 and 2. we omit reviews of star ratings of 3. for the same product, review texts with the same sentiment polarity (both positive or both negative) are deemed to be similar and review texts with different sentiment polarity (one positive and one negative) are different. this forms the basis of creating the training datasets for fine-tuning our model. we explore both siamese networks and triplet networks for training the opinion similarity model. the siamese network for fine-tuning sos, sos s , formulates opinion similarity as a classification task with a learning objective to classify a pair of sentences as similar or otherwise. the supervision is a binary signal that a pair is either similar or different. this approach is used by an unsupervised metric for a sentence similarity, where a siamese network of sbert is fine-tuned on snli and mnli dataset which is a classification task @xcite . for this work, we create training, development and test datasets of review pairs. each pair contain reviews from the same product. the pair is either similar (either both positive or both negative) or different (one positive and one negative). we also ensure that the dataset is balanced with similar and different pairs. the training objective is cross entropy loss. the second variant of our metric is to fine-tune with a triplet network for sos, sos t . each training instance is a triplet of an anchor example, positive example and negative example. the learning objective is triplet loss, which is to score the distance between anchor example and positive example to be smaller than the distance between anchor example and negative example by a margin. each triplet is constructed from reviews for the same product. we randomly select a review to form the anchor, and randomly selected another review review that have the same sentiment polarity as the anchor example as the positive example. we then select another review with opposite sentiment polarity to the anchor example as the negative example. for this work, we create training, development and test datasets of review triplets. both siamese and triplet networks are effective for metric learning, but sos t performs better than sos s as the training triplets provide context that helps modeling the similarity more effectively. this finding is consistent with the literature for semantic text similarity @xcite . ## experiments and results we examine the performance of our model variants sos s and sos t on measuring similarity of sentences. specifically, we compare which training network, the siamese or triplet, is more appropriate to fine-tune our model for our task. apart from comparing the networks, we also included variations in constructing the training pairs or triplets. we constructed pairs and triplets with the entire review text, first sentence of review text or random sentence of review text. we choose to include sentence variations because our task is at a sentence level therefore training examples at sentence level is an appropriate consideration. we train four variants of sos: (1) sos-siamese-pc (sos s p c ) and sos-triplet-pc (sos t p c )trained with reviews from amazon pc dataset foot_1 ; and, (2) sos-siamese-yelp (sos s y elp ) and sos-triplet-yelp (sos t y elp )-trained with reviews from the yelp academic dataset foot_2 . these two review datasets are selected with the intention to train our models on domain related reviews. the models on amazon pc reviews roughly parallel the laptop dataset, and models on yelp reviews roughly parallel the restaurant dataset. we use python using the code from sentence-transformers library. we use sbert (stsb-bert-base) model in our metric, and fine-tuned with 10% warm up steps, one epoch and a batch size of eight. we run each model three times and report the average correlation on our opinion similarity evaluation dataset. the accuracy on the development datasets determines which models to choose. for sos s models, sos s p c and sos s y elp are both trained with 8000 training examples of entire reviews. our best sos t models are sos t p c , fine-tuned with 1000 training triplets of entire reviews and margin of 1, and sos t y elp , fine-tuned with 3000 training triplets of entire reviews and a margin of seven. ## error analysis we examined possible reasons why metrics have difficulty assessing differences in opinion. although our sos models have higher correlation than sbert for "broadly different" pairs, correlations are still not at the same level for "broadly similar" pairs. however, sos models are still not able to do well for opinions that contain implicit aspects or implied opinions. this is a possible reason for the lower correlation in "broadly different" pairs. we leave addressing these two challenges to future research. ## discussion our work have implications for the automatic evaluation of review summaries. rouge and its variants are the default automatic metrics for review summary evaluation @xcite @xcite . however, rouge is shown to be ineffective at evaluating summary pairs with opposite sentiment polarity @xcite . the sentiment agreement is recognized to be important dimension of a review summary and has been included in the human evaluation component of review summaries @xcite . this calls for new automatic metrics that considers the agreement of sentiment polarity between summary pairs. our work fits into this area because we demonstrated that our sos metric captures the sentiment agreement at sentence level. future work in this area is to extend the sos metric to review summary evaluation. while we focus on investigating text similarity for opinion sentences in this work, an equally interesting direction is to approach this from an inference perspective where one opinion sentence entails the other. we leave this line of investigation to future work. ## conclusions we investigate how humans make similarity judgments over opinion sentences. we contribute a dataset of crowdsourced similarity judgments for opinion sentences. the agreement amongst annotators for judgments is moderate. we study the limitations of current text similarity methods when they are adopted for this task and our analysis show that this is likely due to the inability of current metrics to model differing opinions. by fine-tuning siamese sentence-bert using weak supervision, we increase the pearson correlation with human judgments to 0.606 and 0.794 on laptop and restaurant respectively of our opinion similarity dataset.