Pixie: Preference in Implicit and Explicit Comparisons (2022)

Abstract: We present Pixie, a manually annotated dataset for preference classification comprising 8,890 sentences drawn from app reviews. Unlike previous studies on preference classification, Pixie contains implicit (omitting an entity being compared) and indirect (lacking comparative linguistic cues) comparisons. We find that transformer-based pretrained models, finetuned on Pixie, achieve a weighted average F1 score of 83.34% and outperform the existing state-of-the-art preference classification model (73.99%).

## introduction online user reviews contain a cornucopia of information on user expectations about a product. users often express their opinions on a product by comparing it against competitors. understanding preferences in natural language is crucial in capturing user's opinions and expectations. previous studies show that app reviews include rich insights about user expectations and problems of mobile apps that are valuable for app developers @xcite @xcite . we found that app reviews often include comparative sentences, from which we can determine a reviewer's preferences. identifying the preferred entity from an app review involves (1) comparative sentence identification (csi) @xcite , i.e., identifying sentences that contain a comparison, and (2) comparative preference classification (cpc) @xcite , i.e., identifying the preferred entity in a comparative sentence. we focus on the second task. prior work on cpc focuses on explicit comparisons, where all compared entities are explicitly mentioned. extracting comparative sentences by matching keywords or patterns @xcite @xcite over-sentence app s1 bye uber, hello lyft . uber s2 does this app really need to be 260 mb when the marriott app is only 47 mb? hilton honors s3 beats the pants off pandora. spotify s4 i think that it's a lot more fun than temple run. ## related work comparative sentence structures have been a subject of syntactic and semantic theories @xcite @xcite . early studies in computational linguistics include syntactic and semantic handling of comparative constructions @xcite , comparative structures in question answering @xcite , using quantifiers to identify comparisons @xcite , and semantic interpretation of comparatives @xcite . @xcite present a binary classification dataset containing comparative and noncomparative sentences. they present a classifier based on class sequential rules (csr) and leverage comparative keywords to identify comparative sentences. @xcite extend this work by annotating comparative sentences with the preferred entity. @xcite annotate comparative sentences by identifying comparison predicates, entities being compared, aspect of comparison, and comparison type (gradable or non-gradable). however, they focus on reviews of only one product type (digital cameras) to create their dataset. hence, their dataset lacks diversity in topics. @xcite create compsent-19, a cross-domain dataset for comparative argument mining. they propose a gradient boosting model based on pretrained sentence embeddings to identify the preferred entity. @xcite propose a model called entity-aware dependency-based deep graph attention network (ed-gat) that consists of a multihop graph attention network with dependency relations to identify the preferred entity. the ed-gat model achieves a micro f1-score of 87.43% on the compsent-19 dataset. previous work on preference classification has overlooked implicit and indirect comparisons common in user-generated text such as app reviews. further, existing datasets are either too small with a few comparative sentences or have a skewed distribution. for example, ganapathibhotla and liu's dataset contains only 837 comparative sentences, 84% of which have the first mentioned entity in the text as preferred. only 15% of kessler and kuhn's dataset constitutes comparative sentences. only 27% of the sentences in compsent-19 @xcite contain a preference, 70% of which prefer the first mentioned entity in the sentence. further, existing datasets consider the order of the appearance of compared entities in a sentence to annotate the preferred entity. for instance, annotations for compsent-19 @xcite and ganapathibhotla and liu's dataset are both determined based on the order of appearance of the entity in a sentence (i.e., is the first appearing entity in the sentence preferred or the second). ## method we introduce the essential concepts below. comparative sentence : a sentence that contains information on similarity, dissimilarity, or preference between two entities. pixie includes (1) comparative sentences that lack comparative quantifiers, adjectives, or keywords, i.e., indirect comparisons, (2) implicit comparisons where only one of the compared entities is mentioned, and (3) explicit comparisons which mention both (including pronominal references). preferred entity : an entity that is chosen over another based on a stated or implied preference. a preferred entity can be the current app (e.g., s 1p in ## results segment embeddings enhanced bert and xl-net model's performance in terms of weighted average f1-scores, but a slight decline for deberta's and albert's performance. the none and current classes consistently achieve the lowest and the highest f1-scores, respectively, for all models. the none class was also the most ambiguous class to annotate manually. recall for the none class is lower than precision for all models except ed-gat. all transformer-based models achieve a higher recall than precision for the current class except for albert (without segment embeddings) and ed-gat. ed-gat @xcite trained on pixie achieves a weighted average f1-score of 73.99%, with the highest f1-score (80.57%) for the cur-rent class and lowest (51.54%) for none. upon further analysis, we found that most of the incorrect classifications in transformer-based models are for the none class (71.64%), whereas, for ed-gat, only 8.77% of the misclassified sentences belong to the none class. ed-gat yielded most misclassifications for the current class (55.27% of misclassified instances) while only 14.93% of misclassifications for the transformer-based models belong to the current class. original sentence masked sentence 1 cnn should leave journalism to the pros at fox news. <@xmath0. 2 way better than pandora by a long shot!!!! way better @xmath1: results for cross-dataset experiments. the values are in %. ## conclusion masking compared entities ensure that pixie can be used to train a general-purpose preference classification model. additional analysis is needed to claim the domain generality of our dataset-that is, whether a model trained on pixie can identify the preferred entity in texts from other domains such as scientific papers and news. comparative sentences in pixie are limited to user-generated text and may not generalize well over more formal texts. both bert and xlnet show improvements with segment embeddings, suggesting that the demarcation of the other app helps the model identify the preferred entity. the traditional machine learning models perform worst and the transformer-based pretrained models fine-tuned on pixie achieve a substantially better performance than the state-ofthe-art approaches for preference classification. identifying preferences in user reviews can aid developers in understanding user expectations about mobile apps. users often express their likes and dislikes about an app or feature by comparing it with alternative apps and features. understanding user preferences can be particularly valuable in enhancing the functionality as well as security and privacy features of apps. a user's preferences regarding apps would depend not only on how well the app is constructed relative to its competitors but also on how easily the app is used by end-users. for example, security concerns may be signaled by descriptions of steps to access sensitive financial or medical data @xcite expressed in association with comparisons. a follow-on direction is to extract and prioritize user expectations by identifying the specific features of an app of greatest influence on the indirect or direct comparisons in a review.