Constituency Lattice Encoding for Aspect Term Extraction (2020)

Abstract: One of the remaining challenges for aspect term extraction in sentiment analysis resides in the extraction of phrase-level aspect terms, which is non-trivial to determine the boundaries of such terms. In this paper, we aim to address this issue by incorporating the span annotations of constituents of a sentence to leverage the syntactic information in neural network models. To this end, we first construct a constituency lattice structure based on the constituents of a constituency tree. Then, we present two approaches to encoding the constituency lattice using BiLSTM-CRF and BERT as the base models, respectively. We experimented on two benchmark datasets to evaluate the two models, and the results confirm their superiority with respective 3.17 and 1.35 points gained in F1-Measure over the current state of the art. The improvements justify the effectiveness of the constituency lattice for aspect term extraction.

## introduction in aspect-based sentiment analysis, aspect term extraction (ate) serves as a fundamental task that aims to identify aspect terms from review texts @xcite ). an aspect term can be a word or a phrase that describes certain attribute of an entity (e.g., restaurant). for example, in the sentence "the food is very average, the thai fusion stuff is a bit too sweet", the terms "food" and "thai fusion stuff " are the aspect terms to extract in this task. one of the challenges for ate is to extract uncommon phrase-level aspect terms, which may contain complicated structures of words. one line of the research work on ate leverages the syntactic structure of a sentence explicitly. early research in this direction focuses on rule-based @xcite or feature engineering approaches @xcite that require manually designed rules based on part-ofspeech tagging or dependency structure. they may also rely on features from predefined lexicons or syntactic analysis from annotated corpus. these approaches have unavoidable limitations in that they are labor-intensive and it takes extensive efforts to construct the rules and features. besides, there are neural network approaches @xcite that encode the dependency parse structure of a sentence in neural models. these work primarily focuses on syntactic dependency information of sentence, given that the dependency structure may indicate the position of a potential aspect term. nevertheless, their tree-dependent nature of the encoding process makes the optimization inconvenient. another line of work resorts solely to deep learning techniques such as lstm @xcite , cnn @xcite , attention mechanism @xcite and transformer @xcite to automate the semantic representations and have achieved promising results. among them, @xcite applies an additional pre-training for bert @xcite with in-domain corpora before fine-tuning, which is the current state of the art for this task. although deep learning methods can produce increasingly appealing results, they suffer from phraselevel aspect term extraction without explicit inclusion of sentence syntactic information @xcite . particularly, constituency parsing, which generates constituency trees with hierarchical constituents that may indicate the span of a targeted aspect term, is rarely exploited in recent models. take the sentence "other guests enjoyed pizza, santa fe chopped salad and fish and chips" in figure 1 for example. the intended aspect terms are "santa fe chopped salad" and "fish and chips". in this example, constituency parsing provides us with constituents that match the two aspect terms exactly, showing that the constituents can be used to identify the spans of potential aspect terms, especially phrase-level aspect terms. in this paper, we argue that the span annotations of constituents are the most critical information in incorporating the constituency structure for an ate model. that means we only need to pay attention to the constituents rather than the whole hierarchical tree structure. to this end, we first propose to construct a constituency lattice structure that contains both the original sentence and the corresponding constituents. the idea is partially motivated by the work on chinese named-entity recognition (ner) @xcite , in which a lattice structure is also used to help identify the boundaries of target named entities in the sequence labeling process. we then introduce two methods to effectively encode the constituency lattice structure for aspect term extraction, leading to our two models, namely cl-bilstm and cl-bert. for the first method, we propose to encode the constituency lattices into bilstm-crf @xcite to enhance the representations of sentence words. for the second, we opt for bert as the base model @xcite , which is a large pre-trained language model structured by a stack of transformer blocks. since transformers have a different architecture than lstms, it is non-trivial to modify the input embedding layer of bert like lstms. to address this issue, we propose to encode the sentence and its constituents separately for several initial layers, and then apply attention to gather constituency spans information from the constituents to enhance the sentence encoding. experimental results on two benchmarks show that the two proposed methods achieve substantial improvements over the baselines, demonstrating the effectiveness of our constituency lattice encoding methods. ## approach in this section, we first formally define the problem of aspect term extraction (ate) and then provide the details of our constituency lattice encoding methods. ## conclusion this paper presents our efforts to address the problem of aspect term extraction, especially phrase-level aspect term extraction, in aspect-based sentiment analysis. motivated by the recent process in sequence labeling problems such as chinese named-entity recognition, we propose to incorporate constituency lattices in neural models to leverage the syntactic information explicitly for aspect term extraction. we demonstrate how to acquire the constituency lattices from constituency parse trees and encode them in two existing neural models, namely bilstm-crf and bert. extensive experiments are conducted on two benchmark datasets to evaluate the two models, and the experimental results confirm their effectiveness with respective 3.17 points and 1.35 points gained in f1-measure over the current state of the art. since constituency parsing plays a critical role in our constituency lattice encoding, we show that the two models can be further improved once better parsers are provided. finally, although our models are shown to deal with most of the phrase-level aspect term extractions, the error analysis shows that there are still long aspect terms that cannot be extracted completely by either our models or the baselines. but we believe this issue can be relieved as soon as there are more such samples provided in the training set.