Analyzing Coreference and Bridging in Product Reviews (2022)

Abstract: Product reviews may have complex discourse including coreference and bridging relations to a main product, competing products, and interacting products. Current approaches to aspect-based sentiment analysis (ABSA) and opinion summarization largely ignore this complexity. On the other hand, existing systems for coreference and bridging were trained in a different domain. We collect mention type annotations relevant to coreference and bridging for 498 product reviews. Using these annotations, we show that a state-of-the-art factuality score fails to catch coreference errors in product reviews, and that a state-of-the-art coreference system trained on OntoNotes does not perform nearly as well on product mentions. As our dataset grows, we expect it to help ABSA and opinion summarization systems to avoid entity reference errors.

## introduction to help consumers and businesses make sense of high volumes of product reviews, the nlp community has developed techniques for aspect based sentiment analysis (absa) @xcite , and, more recently, opinion summarization @xcite . these techniques have developed mostly without addressing challenges in coreference @xcite or bridging @xcite . in aspect based sentiment analysis (absa), aspect categories and associated polarities are extracted @xcite . in one subtask of se-meval 2016 task 5, this is done on a per-sentence basis without awareness of the product being reviewed. in the other, the full review is available, but entity comparisons are not explicitly peformed. this approaches poses a danger when a customer mentions a competing product or interacting product in the review, because aspects pertaining to the * work performed at nec laboratories america. competing product may be falsely associated with the main product. as a multi-document summarization task with extractive @xcite and abstractive @xcite approaches, opinion summarization may create coreference errors by quoting a pronoun out of context (extractive) or hallucinating a sentence with entities confused (abstractive). factuality checking @xcite promises more correct summaries, either by postprocessing outputs judged to be logically inconsistent @xcite , or by providing a training signal for contrastive learning @xcite . as we show in section 4, a state-of-the-art natural language inference (nli)-based factuality score often fails to capture coreference errors. because existing absa and factuality scores do not learn to catch coreference or bridging errors adequately, a new resource is necessary. de clercq and hoste (2020) released coreference annotations on restaurant reviews, but this domain mostly lacked the mentions of competitors and interacting products found in product reviews. in this paper, we define a mention classification task for product reviews which simplifies the coreference and bridging resolution tasks. our simplified task reduces labeling burden compared to labeling all pairs of mentions. minimally trained crowdworkers are able to assign our labels with good agreement. we collected labels for 8,894 mentions in 498 reviews already, and plan to continue collecting labels from 3,000 reviews. the size of the dataset currently may be adequate only for evaluation, but we plan to collect more data which will make it useful for development. our contributions are: (1) simplifying coreference and bridging for product reviews into a task for which we can obtain quality labels from crowdworkers, (2) constructing a dataset for this task, (3) showing the weakness of a state-of-the-art factu-ality score on detecting confused entity mentions in product reviews, and (4) preliminary analysis of an existing coreference system applied to our annotated data. once enough data for training is collected, we envision that absa or nli systems might use predicted mention types as features, so that e.g. an absa system would recognize a sentence discussing an attribute of a competing product and not report it as an aspect of the product being reviewed, or a factuality score would catch entity inconsistency between source and generated text. ## dataset we annotated 498 electronics reviews from the amazon review dataset @xcite , consisting of reviews posted from may 1996 to july 2014. we use the electronics category as we expect the reviews in this category to include competing products and interacting items frequently. the rating for each review is given, and we retrieved the product name from the rainforest api. foot_0 3 annotation ## do factuality scores detect coreference errors? using our dataset, we can construct examples that test a factuality score's ability to accept coreference-consistent substitution of entities and reject inconsistent substitions. for nli-based factuality checking, we apply the summac zero shot (zs) system @xcite in which the whole review document is used as a single premise for implication. although the original paper suggested that sentence-level granularity could be beneficial, the document-level granularity may have a better chance of following coreference and bridging relations across sentences. both versions are trained on mnli @xcite plus vitamin c @xcite . we test the summac-zs score on our annotated product reviews as follows. for the mention categories "main product," "competing product," and "interacting product," we take sentences that contain the second or subsequent mentions of these categories (so that coreference antecedents are likely), and construct one sentence in which we replace that mention with the main product name, or the first mention of a competing product, or the first mention of an interacting product. the task is to determine whether this generated sentence is factually correct or not. one consistent replacement and one inconsistent replacement was generated from each of 60 reviews. replacements whose type agrees with the original mention are usually expected to be correct and replacement across categories are expected to be incorrect, but in case they are not, muc b3 ceaf4 p r f1 p r f1 p r f1 avg f1 ontonotes 85.9 85.5 85.7 79.0 78.9 79.0 76.7 75.2 75.9 80.2 main 68.3 59.5 63.6 63.1 48.3 54.7 50.5 68.1 58.0 58.8 competing 37.1 27.4 31.6 43.7 28.8 34.7 57.7 40.6 47.7 38.0 generic 22.2 11.8 15.4 32.3 14.0 55.0 19.6 18.8 28.0 21.0 the ground truth is manually checked by an author and disagreements are filtered out. examples of the replacements are shown in appendix c. the summac-zs models were tested on the manually verified nli pairs. everything in the generated text but the entity mention exactly matches the source text. hence, there are no semantic challenges apart from the entity resolution. therefore, this result shows significant room for improvement in distinguishing non-coreferent entities. ## evaluating pre-trained coreference we evaluate the coreference clusters output by the system of @xcite against the clusters consisting of all mentions of three types: main, competing, and generic. generally these mention types will consist of a union of coreference clus-ters. to associate coreference clusters output by the system to one of these mention types, we take the union of all the clusters intersecting the mention type. therefore recall failures will occur only when a mention fails to be detected or is not recognized as an anaphor to be linked to anything. good recall means that the mentions of the category were recognized as potential anaphors. a precision failure with respect to these mention types indicates an error in which the coreference system links a mention with an antecedent of a different type. we use the coreference model in @xcite with the spanbert-large encoder trained on ontonotes 5.0 foot_3 and set all parameters as in the original paper. the model achieves lower avg f1 in competing and generic compared to main. from the mention distribution in ## conclusion we presented a new corpus of 498 electronics product reviews with a relaxed form of coreference and bridging annotation. we tested an ontonotesbased coreference system on the reviews, and used the annotations to measure how much a factuality score failed to detect coreference errors on product reviews. as more data is collected, we hope the resource will be useful to help absa and opinion summarization systems avoid entity reference errors in analyzing product reviews.