An Empirical Study on Leveraging Position Embeddings for Target-oriented Opinion Words Extraction (2021)

Abstract: Target-oriented opinion words extraction (TOWE) (Fan et al., 2019b) is a new subtask of target-oriented sentiment analysis that aims to extract opinion words for a given aspect in text. Current state-of-the-art methods leverage position embeddings to capture the relative position of a word to the target. However, the performance of these methods depends on the ability to incorporate this information into word representations. In this paper, we explore a variety of text encoders based on pretrained word embeddings or language models that leverage part-of-speech and position embeddings, aiming to examine the actual contribution of each component in TOWE. We also adapt a graph convolutional network (GCN) to enhance word representations by incorporating syntactic information. Our experimental results demonstrate that BiLSTM-based models can effectively encode position information into word representations while using a GCN only achieves marginal gains. Interestingly, our simple methods outperform several state-of-the-art complex neural structures.

## introduction target-oriented opinion words extraction (towe) @xcite ) is a fine-grained task of targetoriented sentiment analysis @xcite aiming to extract opinion words with respect to an opinion target (or aspect) in text. given the sentence "the food is good but the service is extremely slow", towe attempts to identify the opinion words "good" and "extremely slow" corresponding respectively to the targets "food" and "service". towe is usually treated as a sequence labeling problem using the bio tagging scheme @xcite to distinguish the beginning, inside and outside of a span of opinion words. the food is good but the service is extremely slow. true labels for target 'food': the/o food/o is/o good/b but/o the/o service/o is/o extremely/o slow/o. true labels for target 'service': the/o food/o is/o good/o but/o the/o service/o is/o extremely/b slow/i. towe extraction results: {(food, good), (service, extremely slow)} learning effective word representations is a critical step towards tackling towe. traditional work @xcite @xcite has used hand-crafted features to represent words which do not often generalize easily. more recent work @xcite @xcite has explored neural networks to learn word representations automatically. previous neural-based methods @xcite has used word embeddings @xcite @xcite to represent the input. however, towe is a complex task that requires a model to know the relative position of each word to the aspect in text. words that are relatively closer to the target usually express the sentiment towards that aspect @xcite . @xcite employ long short-term memory (lstm) networks @xcite to encode the target position information in word embeddings. @xcite transfer latent opinion knowledge into a bidirectional lstm (bilstm) network that leverages word and position embeddings @xcite . @xcite have proposed ong, a method that combines bert (bidirectional encoder representations from transformers) @xcite , position embeddings, ordered neurons lstm (on-lstm) @xcite ), foot_0 and a graph convolutional network (gcn) @xcite to introduce syntactic information into word representations. while this model achieves state-of-the-art results, previous studies have shown that the on-lstm does not actually perform much better than lstms in recovering latent tree structures @xcite . besides, on-lstms perform worse than lstms in capturing short-term dependencies @xcite . since opinion words are usually close to targets in text, on-lstm risks missing the relationship between the aspect and any information (e.g. position) relating to the opinion words. in this paper, we empirically evaluate a battery of popular text encoders which apart from words, take positional and part-of-speech information into account. surprisingly, we show that methods based on bilstms can effectively leverage position embeddings to achieve competitive if not better results than more complex methods such as ong on standard towe datasets. interestingly, combining a bilstm encoder with a gcn to explicitly capture syntactic information achieves only minor gains. this empirically highlights that bilstmbased methods have an inductive bias appropriate for the towe task, making a gcn less important. ## methodology given sentence @xmath0, . . . , w n } with aspect w @xmath1, our approach consists of a text encoder that takes as input a combination of words, partof-speech and position information for towe. we further explore enhancing text encoding by incorporating information from a syntactic parse of the sentence through a gcn encoder. ## conclusion we presented through extensive experiments that by employing a simple bilstm architecture that uses input representations from pre-trained word embeddings or language models, post embeddings and position embeddings, we can obtain competitive, if not better results than the more complex current state-of-the-art methods @xcite . the bilstm succeeds in exploiting position embeddings to improve performance. by adapting a gcn to incorporate syntactic information from the sentence we achieve further gains. in future work, we will explore how to improve existing towe models by effectively leveraging position embeddings.