Enhancing Aspect Term Extraction with Soft Prototypes (2020)

Abstract: Aspect term extraction (ATE) aims to extract aspect terms from a review sentence that users have expressed opinions on. Existing studies mostly focus on designing neural sequence taggers to extract linguistic features from the token level. However, since the aspect terms and context words usually exhibit long-tail distributions, these taggers often converge to an inferior state without enough sample exposure. In this paper, we propose to tackle this problem by correlating words with each other through soft prototypes. These prototypes, generated by a soft retrieval process, can introduce global knowledge from internal or external data and serve as the supporting evidence for discovering the aspect terms. Our proposed model is a general framework and can be combined with almost all sequence taggers. Experiments on four SemEval datasets show that our model boosts the performance of three typical ATE methods by a large margin.

## introduction aspect term extraction (ate) is a fundamental subtask in aspect-based sentiment analysis. given a review sentence, ate aims to extract all aspect terms that users have expressed opinions on. for example, from the review "the bombay style bhelpuri is very palatable.", ate aims to extract "bhelpuri". ate has been widely studied in the last twenty years. early researches are devoted to design rulebased @xcite and feature engineering-based @xcite methods. with the development of deep learning techniques, recent researches mostly regard ate as a sequence labeling task and focus on developing various types of neural models @xcite @xcite to generate a tag sequence for the review. though achieving impressive progress, current sequence taggers mentioned still face a serious challenge: the taggers may converge to an inferior state due to the lack of samples for tail words. as shown in figure 1 , about 80% aspect terms and context words (i.e., non-aspect terms) appear no more than five times in the commonly-used semeval datasets. without enough sample exposure, neural models can hardly achieve an optimal performance @xcite . to tackle this challenge, correlating samples with each other may offer helping hands. for example, if we correlate the rare aspect term "bhelpuri" with a frequent one like "food", there will be more abundant samples for "bhelpuri" than ever. the problem then becomes how to build such a tokenlevel correlation. retrieving synonyms is an intuitive approach to this problem, but it has two limitations. firstly, synonyms only exist for a small number of words in the vocabulary. this will make the correlations incomplete. though we can calculate the nearest neighbors for a certain word based on the pre-trained word embeddings, it is not guaranteed that they have a similar semantic meaning. secondly, in ate, the existence of an aspect term depends on whether there are opinions on it. that is to say, we need to build a dynamic correlation for a certain word based on its entire contexts rather than the word itself. indeed, if the retrieval is conducted based on an individual token, the above two limitations always exist. in this paper, we propose a soft retrieval method to build the token-level correlation for both aspect terms and context words. rather than conducting a hard retrieval for individual tokens, we turn to retrieve the tokens' counterparts according to their contexts. as shown in figure 2 , after conducting the soft retrieval, we can obtain a generated sample strictly corresponds to the input sample in every position. we name the generated sample "soft prototype" since it is actually a simplified prototype that can build a reference point for guiding the tagging process for the input sample. we resort to the language models (lms) to implement the soft retrieval and generate high-quality soft prototypes. as a self-supervised task, language modeling needs no extra annotations and can absorb data-specific global knowledge. moreover, lms tend to generate frequent outputs @xcite , which exactly meets our needs for correlating a rare word in the input sample with a frequent one in the soft prototype. specifically, we first pretrain bi-directional lms using the given training samples on ate datasets. alternatively, we can take advantage of large-scale unlabeled data like yelp and amazon reviews to pre-train lms. then, after fixing the pre-trained lms, we can infer each token's prototype according to its contexts for both the training and testing samples. we regard the generated soft prototypes as the supporting evidence for tagging aspect terms, and design a simple and effective gating mechanism to fuse the knowledge embedded in both samples before sending them to a sequence tagger. the soft prototypes can be combined with almost all existing sequence taggers. to demonstrate the effectiveness of our proposed model, we conduct experiments on four semeval datasets by adding the generated soft prototypes on three existing sequence taggers. the results prove that our soft prototypes significantly boost the performance of their original counterparts. in this paper, we focus on the supervised scenario. different from the aforementioned supervised models, we develop a novel model to enhance ate. by automatically generating and utilizing soft prototypes, we correlate samples with each other, which greatly enhances the learning process of sequence taggers. moreover, the decoupling of soft prototypes from taggers makes our model flexible and general, i.e., it can be combined with almost all neural sequence taggers. prototypes in neural networks the idea of prototypes (or templates) originates from information retrieval (ir) approaches for sentence matching tasks like response generation @xcite . they aim to retrieve a related sample from the dataset as the counterpart of the input sample. more recently, several studies shed new light in this domain by deeply fusing prototypes with neural networks. many of them use the taskdependent metrics @xcite , common metrics such as jaccard similarity @xcite @xcite , or existing tools like lucene @xcite to retrieve prototypes, and then input the prototypes into a neural model for generating outputs. @xcite follows another line, where the prototype (the target words related to a source word in machine translation) is generated using a pretrained seq2seq model. the approach of generating words via lms is inspired by a recent study @xcite . however, the method in @xcite is developed for text classification and is not suitable for the ate task here. concretely, their method randomly replaces a small percentage (typically 10%) of original training words with the generated ones and then discard the original words. this operation may work well for text classification tasks which only require sentence-level information. for token-level tasks like ate, the original words are however necessary for tagging each token correctly. moreover, the small percentage of replacement implies that the generated knowledge cannot be fully incorporated into the new sample. in contrast, we generate a prototype for each word in the sentence, and then deeply fuse the original word with its corresponding prototype to make good use of their embedded knowledge for ate. to the best of our knowledge, we are the first to introduce the retrieval method to handle the data deficiency problem in ate. to this end, we propose a new approach to generate and utilize soft prototypes that can build the token-level correlation for aspect terms and context words. ## methodology in this section, we first illustrate the overall framework for enhancing ate with soft prototypes. we then detail the generation and utilization of soft prototypes. lastly, we describe the objective function and the training procedure. ## conclusion in this paper, we present a general softproto framework to enhance the ate task. rather than designing elaborated sequence taggers, we turn to correlate samples with each other through soft prototypes. for this purpose, we resort to the language models for automatically generating soft prototypes and then design a gating conditioner for utilizing them. the performance of softproto can be further improved after introducing the large-scale external unlabeled data like yelp and amazon reviews. extensive experiments on four semeval datasets demonstrate that softproto greatly boosts the performance of the typical ate methods and introduces small computational cost.