SubjQA : A D ataset for S ubjectivity and R eview C omprehension (2020)

Abstract: Subjectivity is the expression of internal opinions or beliefs which cannot be objectively observed or verified, and has been shown to be important for sentiment analysis and word-sense disambiguation. Furthermore, subjectivity is an important aspect of user-generated data. In spite of this, subjectivity has not been investigated in contexts where such data is widespread, such as in question answering (QA). We develop a new dataset which allows us to investigate this relationship. We find that subjectivity is an important feature in the case of QA, albeit with more intricate interactions between subjectivity and QA performance than found in previous work on sentiment analysis. For instance, a subjective question may or may not be associated with a subjective answer. We release an English QA dataset (SubjQA) based on customer reviews, containing subjectivity annotations for questions and answer spans across 6 domains.

## introduction subjectivity is ubiquitous in our use of language @xcite @xcite , and is therefore an important aspect to consider in natural language processing (nlp). for example, subjectivity can be associated with different senses of the same word. boiling is objective in the context of hot water, but subjective in the context of a person boiling with anger @xcite . the same applies to sentences in discourse contexts @xcite . while early work has shown subjectivity to be an important feature for low-level tasks such as word-sense disambiguation and sentiment analysis, subjectivity in nlp has not been explored in many contexts where it is prevalent. in recent years, there has been renewed interest in areas of nlp for which subjectivity is important, and a specific topic of interest is question answering (qa). this includes work on aspect extraction @xcite , opinion mining @xcite and community qa @xcite . many qa systems are based on representation learning architectures (e.g. @xcite ; @xcite ) that are typically trained on factual, encyclopedic knowledge such as wikipedia or books. it is unclear if these architectures can handle subjective statements such as those that appear in reviews. the interactions between qa and subjectivity are even more relevant today as users' natural search criteria in many domains, including products and services, have become increasingly subjective. according to @xcite , around 20% of product queries were labeled as being 'subjective' by workers. their questions can often be answered by online customer reviews, which tend to be highly subjective as well. although qa over customer reviews has gained traction recently with the availability of new datasets and architectures @xcite @xcite @xcite , these are agnostic with respect to how subjectivity is expressed in the questions and the reviews. furthermore, the datasets are either very small (< 2000 questions) or have target-specific question types (e.g., yes-no). most qa datasets and systems focus on answering questions over factual data such as wikipedia articles and news @xcite @xcite @xcite . in this work, on the other hand, we focus on qa over subjective data from reviews on product and service websites. in this work, we investigate the relation between subjectivity and question answering (qa) in the context of customer reviews. as no such qa dataset exists, we construct subjqa. 1 in order to capture subjectivity, our data collection method builds on recent developments in opinion extraction and matrix factorisation, instead of relying on linguistic similarity between questions and reviews @xcite . subjqa includes over 10,000 english examples spanning 6 domains that cover both products and services. we find a large percentage of subjective questions and answers in subjqa, as 73% of the questions are subjective and 74% of the answers are subjective. experiments show that existing qa systems trained to find factual answers struggle with subjective questions and reviews. for instance, fine-tuning bert @xcite , a state-of-the-art qa model, yields 92.9% f 1 on squad @xcite , but only achieves a mean score of 74.1% f 1 across the domains in subjqa. to demonstrate the importance of subjectivity in qa, we develop a subjectivity-aware qa model that extends an existing qa model in a multi-task learning paradigm. it is trained to predict the subjectivity label and answer span simultaneously, and does not require subjectivity labels at test time. this qa model achieves 76.3% f 1 on an average over different domains of subjqa. ## subjectivity written text, as an expression of language, contains information on several linguistic levels, many of which have been thoroughly explored in nlp. 2 for instance, both the semantic content of text and the (surface) forms of words and sentences, as expressed through syntax and morphology, have been at the core of the field for decades. however, another level of information can be found when trying to observe or encode the so-called private states of the writer @xcite . examples of private states include the opinions and beliefs of a writer, and can concretely be said to not be avail- opinion extraction topic and review selection reviews question generation q: how is the writing? r: i believe this is the author's first novel and i was not expecting much. but both the story and character development were quite impressive. neighborhood model construction selected review selected topic @xmath0, @xmath1 figure 1: our data collection pipeline able for verification or objective observation. it is this type of state which is referred to as subjectivity @xcite . whereas subjectivity has been investigated in isolation, it can be argued that subjectivity is only meaningful given sufficient context. regardless, much previous work has focused on annotating words @xcite , word senses @xcite @xcite , or sentences @xcite , with notable exceptions such as @xcite ; @xcite , who investigate subjectivity in phrases in the context of a text or conversation. there is limited work that studies subjectivity using architectures that allow for contexts to be incorporated efficiently @xcite . as subjectivity relies heavily on context, and we have access to methods which can encode such context, what then of access to data which encodes subjectivity? we argue that in order to fully investigate research questions dealing with subjectivity in contexts, a large-scale dataset is needed. we choose to frame this as a qa dataset, as it not only offers the potential to investigate interactions in a single contiguous document, but also allows interactions between contexts, where parts may be subjective and other parts may be objective. concretely, one might seek to investigate the interactions between an objective question and a subjective answer, or vice-versa. ## data collection we found two limitations of existing datasets and collection strategies that motivated us to create a new qa dataset to understand subjectivity in qa. first, data collection methods @xcite often rely on the linguistic similarity between the questions and the reviews (e.g. information retrieval). however, subjective questions may not always use the same words or phrases as the review. consider the examples below. the answer span 'vegan dishes' is similar to the question q 1 . the answer to the more subjective question q 2 , however, is linguistically quite different. example 1 q 1 : is the restaurant vegan friendly? review: [...] many vegan dishes on its menu. q 2 : does the restaurant have a romantic vibe? review: amazing selection of wines, perfect for a date night. secondly, existing review-based datasets are small and not very diverse in terms of question topics and types @xcite . we, therefore, consider reviews about both products and services from 6 different domains, namely tripadvisor, restaurants, movies, books, electronics and grocery. we use the data of @xcite for tripadvisor, and yelp 3 data for restaurants. we use the subsets for which an opensource opinion extractor was available @xcite . we use the data of mcauley and yang (2016b) that contains reviews from product pages of amazon.com spanning multiple categories. we target categories that had more opinion expressions than others, determined by an opinion extractor. figure 1 depicts our data collection pipeline which builds upon recent developments in opinion extraction and matrix factorization. an opinion extractor is crucial to identify subjective or opinionated expressions, which other ir-based methods cannot. on the other hand, matrix factorization helps identify which of these expressions are related based on their co-occurrence in the review corpora, instead of their linguistic similarities. relying on opinions instead of factual information to construct the dataset makes it inherently subjective. furthermore, pairing questions and reviews based on related opinion expressions provides us some control over the diversity in the dataset. to the best of our knowledge, we are the first to explore such a method to construct a subjective qa dataset. example 2 review: [...] character development was quite impressive. e 1 :‹'impressive', 'character development'› review: 3 stars for good power and good writing. e 2 :‹'good', 'writing'› in the next (neighborhood model construction) step, we characterize the items being reviewed and their subjective extractions using latent features between two items. in particular, we use matrix factorization techniques @xcite to construct a neighborhood model n via a set of weights w e,e , where each corresponds to a directed association strength between extraction e and e . for instance, e 1 and e 2 in example 2 could have a similarity score 0.93. this neighborhood model forms the core of data collection. we select a subset of extractions from n as topics (topic selection) and ask crowd workers to translate them to natural language questions (question generation). for each topic, a subset of its neighbors from n and reviews which mention them are selected (review selection). in this manner, question-review pairs are generated based on the neighborhood model. finally, we present each question-review pair to crowdworkers who highlight an answer span in the review. additionally, they provide subjectivity scores for both the questions and the answer span. ## dataset analysis in this section, we analyze the questions and answers to understand the properties of our subjqa dataset. we present the dataset statistics in section 4.1. we then analyze the diversity and difficulty of the questions. we also discuss the distributions of subjectivity and answerability in our dataset. additionally, we manually inspect 100 randomly chosen questions from the development set in section 4.3 to understand the challenges posed by subjectivity of the questions and/or the answers. ## subjectivity modelling we now turn to experiments on subjectivity, first investigating claims made by previous work, and whether they still hold when using recently developed architectures, before investigating how to model subjectivity in qa. ## related work we are witnessing a substantial rise in usergenerated content, including subjective information ranging from personal experiences to opinions about specific aspects of a product. however, subjectivity has largely been studied in the context of sentiment analysis @xcite mining @xcite , with a focus on text polarity. there is renewed intereste in incorporating subjective opinion data into general data management systems @xcite and for querying subjective data. in this work, we revisit subjectivity in the context of review qa. @xcite ; @xcite also use review data, as they leverage question types and aspects to answer questions. however, no prior work has modeled subjectivity explicitly using end-to-end architectures. furthermore, none of the existing reviewbased qa datasets are targeted at understanding subjectivity. this can be attributed to how these datasets are constructed. large-scale qa datasets, such as squad @xcite , newsqa @xcite , coqa @xcite , mlqa @xcite are based on factual data. we are the first to attempt to create a review-based qa dataset for the purpose of understanding subjectivity. recent work has corroborated our findings on the benefits of modelling subjectivity qa, and highlights the differences in the distributions of hidden representation between subjqa and the factual squad data @xcite . ## conclusion in this paper we investigate subjectivity in qa by leveraging end-to-end architectures. we release subjqa, a question-answering dataset which contains subjectivity labels for both questions and answers. the dataset allows i) evaluation and development of architectures for subjective content, and ii) investigation of subjectivity and its interactions in broad and diverse contexts. we further implement a subjectivity-aware model and evaluate it, along with 4 strong baseline models. we hope this dataset opens new avenues for research on querying subjective content, and into subjectivity in general.