Trading Syntax Trees for Wordpieces: Target-oriented Opinion Words Extraction with Wordpieces and Aspect Enhancement (2023)

Abstract: State-of-the-art target-oriented opinion word extraction (TOWE) models typically use BERT-based text encoders that operate on the word level, along with graph convolutional networks (GCNs) that incorporate syntactic information extracted from syntax trees. These methods achieve limited gains with GCNs and have difficulty using BERT wordpieces. Meanwhile, BERT wordpieces are known to be effective at representing rare words or words with insufficient context information. To address this issue, this work trades syntax trees for BERT wordpieces by entirely removing the GCN component from the methods’ architectures. To enhance TOWE performance, we tackle the issue of aspect representation loss during encoding. Instead of solely utilizing a sentence as the input, we use a sentence-aspect pair. Our relatively simple approach achieves state-of-the-art results on benchmark datasets and should serve as a strong baseline for further research.

## introduction target-oriented opinion word extraction (towe; @xcite ) is a subtask in aspect-based sentiment analysis (absa; @xcite ), which aims to identify words that express an opinion about a specific target (or aspect) in a sentence. for instance, in the sentence "such an awesome surfboard.", a towe model should identify "awesome" as the opinion word for the given aspect surfboard. towe provides explicit aspectopinion pairs which can be used to improve results in downstream tasks such as opinion summarization @xcite and information extraction @xcite @xcite . currently, many towe methods @xcite @xcite @xcite use pretrained bert @xcite , to encode the input 1. sentence: such an awesome surfboard wordpieces: 'such', 'an', 'awesome', 'surf', '##board' ## task formalization the towe task aims to identify an opinion word in a sentence @xmath0, . . . , w ns } with respect to an aspect w @xmath1. the sentence is typically tokenized into a sequence of tokens at different levels of granularity (e.g. subwords or whole words), @xmath2, . . . , t nt }, with t @xmath3. the goal is to assign one of three tags (i, o, or b) to each token using the iob format @xcite , which indicates whether the word is at the inside, outside or beginning of the opinion word relative to the aspect. ## syntax-aware approaches to towe typically, syntax-aware approaches to towe @xcite @xcite employ a text encoder that utilizes pretrained bert @xcite and position embeddings @xcite (or category em-beddings @xcite ) to learn whole word representations that are aware of the aspect's location in text. these approaches also include a gcn that operates on a syntax tree in order to incorporate syntactic information into the model. ordered-neuron lstm gcn (ong): veyseh et al. (2020) combine an ordered neuron lstm (on-lstm; shen et al. (2018)) and a gcn for towe. the on-lstm layer is an lstm variant that considers the order of elements in the input sequence (including bert and position embeddings) when modeling dependencies between them. the gcn encodes syntactic structural information into the representations obtained by the on-lstm layer. bert+bilstm+gcn: mensah et al. (2021) replaces the on-lstm of the ong model with a bilstm to better capture short-term dependencies between aspect and opinion words. attention-based relational gcn (argcn): jiang et al. ( 2021 ) combine contextualized embedding obtained using bert with a category embedding (i.e., iob tag embedding) to incorporate aspect information. they subsequently use a relational gcn @xcite and bil-stm to respectively incorporate syntactic and sequential information for towe classification. ## experiments and results we experiment with the following baselines: argcn, bert+bilstm+gcn and ong. we use the suffixes (s) or (s,a) to indicate whether the modified versions of these methods uses a wordpiece sentence or wordpiece sentence-aspect pair as input, respectively. we used the publicly available code and optimal hyperparameter settings from the authors of argcn foot_0 and bert+bilstm+gcn. 2 we have implemented ong model variants ourselves using the suggested hyperparameter configurations from the authors. 3 following previous work @xcite , we use the same experimental setup and evaluate on the laptop dataset (lap14) and the restaurant datasets (res14, res15, res16) @xcite @xcite . the result reported for each dataset is the average over micro f1 scores obtained from five different runs. each run uses a different random seed to ensure the stability of our results. ## impact of bpe subword representations we previously examined the use of wordpiece representations derived from pretrained bert for towe models. in this section, we look into using byte pair encoding (bpe) @xcite as an alternative method for subword representation, which is inspired by data compression techniques @xcite . it is worth noting that bpe representations are generally not obtained from pretrained bert. however, since roberta is pretrained using bpe, and roberta is a variant of bert, we can still explore the impact of using bpe representations in towe models. to do this, we replace the bert component in our best model, bert+bilstm(s,a), with roberta, developing the model roberta+bilstm(s,a). the results of roberta+bilstm(s,a) and its variations are shown in ## conclusion we demonstrated that replacing gcns with bert wordpieces while enhancing the aspect representation achieves sota results in syntax-aware towe approaches. the aspect enhancement method serves as a "prompt" for the model. we intend to explore prompt-based learning @xcite to further improve the aspect representation.