Generating Complement Data for Aspect Term Extraction with GPT -2 (2022)

Abstract: Aspect Term Extraction (ATE) is the task of identifying the word(s) in a review text toward which the author express an opinion. A major challenges for ATE involve data scarcity that hinder the training of deep sequence taggers to identify rare targets. To overcome these issues, we propose a novel method to better exploit the available labeled data for ATE by computing effective complement sentences to augment the input data and facilitate the aspect term prediction. In particular, we introduce a multistep training procedure that first obtains optimal complement representations and sentences for training data with respect to a deep ATE model. Afterward, we fine-tune the generative language model GPT-2 to allow complement sentence generation at test data. The REINFORCE algorithm is employed to incorporate different expected properties into the reward function to perform the fine-tuning. We perform extensive experiments on the benchmark datasets to demonstrate the benefits of the proposed method that achieve the state-of-the-art performance on different datasets.

## introduction aspect term extraction (ate) is one of the fundamental tasks in aspect-based sentiment analysis (absa). its goal is to recognize the terms upon which a sentiment opinion is expressed in text. for instance, in the sentence "the staff of the restaurant were good but the quality of the food was terrible", an ate system should recognize the two aspect terms (targets) "staff " and "quality of food". ate finds its applications in absa systems to identify targets toward which sentiment analysis is done. a major challenge for ate is the lack of enough training data. for instance, the widely-used se-meval datasets, e.g., res15 @xcite or res16 @xcite , contain less than 2,000 training samples with only 20% of the words appearing more than five times @xcite . this small size of training data hinders the deep sequence taggers to achieve optimal performance, especially for the tail targets (i.e., targets with few examples in the dataset) @xcite . in order to alleviate this issue, prior work has resorted to data augmentation techniques to exploit additional training signals from different sources, including data from related tasks, e.g., absa (performing multi-tasking learning @xcite ), new labeled data for ate produced by pre-trained sequence-to-sequence models @xcite , and soft prompts that are generated by pre-trained language models @xcite . as such, the critical requirements for such prior methods involve annotation for related tasks of ate (e.g., absa), or large in-domain corpora to train the sequence-to-sequence/language models for data generation (called external data). unfortunately, these requirements might be unavailable or very expensive to obtain in different domains, making it less applicable for various scenarios in practice. to this end, this work aims to solve the issue of data scarcity for ate without relying on annotated data for related tasks and large in-domain corpora. in particular, our main proposal is to fine-tune existing large-scale language models so they can generate complement sentences for input sentences in existing labeled datasets for ate (i.e., not using external data as in prior works). here, the motivation is that data scarcity might present a challenge for ate models, especially on tail examples with rare aspect terms and context patterns @xcite . the complement sentence thus aims to provide supporting evidence and facilitate the recognition of aspect terms for the input sentences. as such, our method first seeks to obtain complement sentences for all the sentences in a given ate dataset via a multi-step training procedure. in the first step, we train a base ate model on a labeled training dataset to encode the available knowledge about aspect terms in the dataset. however, due to data scarcity, the base model might not be exposed sufficiently to aspect term patterns, thus limiting the ability of the produced representations for the input sentences to fully capture relevant information/features for ate. to achieve complement sentences for ate datasets, in the second step, we thus propose to learn optimal representation vectors/word embeddings that can be combined (e.g., via adding) to improve the representation vectors from the base model for ate (called complement representations). our motivation is that the insufficient coverage of aspect term patterns in the representations would cause the base model to exhibit poor performance (i.e., high loss) on the validation dataset. to this end, we propose to infer the complement representations for each validation sentence by incorporating them into the base model as additional parameters and minimizing the loss of the augmented model on the validation data. in the implementation, we divide the training data for an ate dataset into k folds. by choosing one fold as validation data and treating the k -1 remaining folds as training data, the aforementioned procedure can return the optimal complement representations for each sentence in the validation fold. as such, we repeat this process for k possible choices of the validation fold that in all produce complement representations for each sentence in the training data. to employ the complement representations for training data, we can introduce them into the base model for retraining. however, this will cause a mismatch in the test time where labels for sentences are not available and the complement representations cannot be obtained. to this end, instead of directly using the learned complement representations, we propose to first transform them into complement sentences based on the glove word embeddings @xcite . this is done by introducing constraints to encourage the learned representation vectors to belong to the same space with glove word embeddings. the complement representations can then be mapped into complement sentences by finding the words whose glove embeddings are closest to the complement representations. in this way, each original training sentence for ate can be associated with a complement sentence. using pairs of original and complement sentences as training data, in the next step, we propose to train a generative model that can trans-form the original sentences into their complement versions. as such, in the test time, we can apply the generative model to generate complement sentences for test data and use glove embeddings to produce complement representation vectors for data augmentation. in this work, we propose to fine-tune the language model gpt-2 @xcite on the original and complement sentence pairs to obtain the generative model. our motivation stems from the small number of the pairs for the original and complement sentences (due to the small size of ate datasets) that might not be sufficient to train a generative model well. by leveraging the pretrained gpt-2 model, we expect that its language priors can compensate for the data scarcity issue and bootstrap the learning process from complement data. finally, we use reinforce @xcite to fine-tune gpt-2 to facilitate the enforcement of expected properties for the generated sentences (i.e, the similarity or the length comparability with respect to the complement sentences produced in prior step). we perform extensive evaluations for the proposed method on different benchmark datasets for ate. our experiments reveal the superior performance and demonstrate the effectiveness of the proposed method. ## experiments datasets & parameters: to evaluate the effectiveness of the proposed method, we employ the commonly used semeval datasets for ate. specifically, we use the datasets of semeval 2014 task 4 @xcite , i.e., lap14 and res14 reviews for laptops and restaurants, and semeval 2015 task 12 @xcite and semeval 2016 task 5 @xcite , i.e., res15 and res16 with reviews for restaurants. following prior work (chen and qian, 2020a), we employ the official train/test splits and randomly select 150 samples from training data as validation data for those datasets. in our experiments, we use the development set of the lap14 dataset to tune the hyper-parameters. based on our experiments, the following values are selected: 200 dimensions for the bilstm layer and the feed-forward layers; 1 layer for bilstm and 2 layers for the feed-forward networks; the bert base version for the encoding layer with fixed parameters; gpt-2 small for sentence generation; 0.3 for the λ in the word vector augmentation; 0.1 for the trade-off parameters α reg and α glov e in the complement embedding optimization; 0.1 and 0.05 for the trade-off parameters β and γ in the reward for gpt-2 fine-tuning; 0.3 for the learning rate for the adam optimizer; and 50 for batch-size. baselines: following prior work, we compare our model with: (1) the winners of the semeval tasks: ihs-rd @xcite , dlirec @xcite , @xcite , nlangp @xcite ; (2) deep joint models, i.e., jointly train ate with opinion term extraction (ote) or aspect-based sentiment analysis (absa): rncrf @xcite , min @xcite , cmla @xcite , hast @xcite , racl (chen and qian, 2020b), dual-mrc @xcite ; (3) deep models trained on ate datasets augmented with external in-domain corpora and resources: bilstm-crf @xcite , seq2seq @xcite , bert @xcite , de-cnn @xcite , bert-pt @xcite , softproto @xcite . we also compare with cl-bert @xcite which employs constituency trees for ate. for the evaluation metric, following prior work, we report the f1 score for aspect term prediction. a prediction is counted as correct if its boundaries match the gold aspect term. we name our model ateoa which stands for aspect term extraction with optimal augmentation. results: the main results are shown in ## related work ate has been first approached with rule-based @xcite or featurebased @xcite @xcite methods. recently, ate methods have focused on neural networks such as lstm @xcite , cnn @xcite or transformer @xcite ). an ate system can be used in downstream applications such as sentiment analysis @xcite @xcite or opinion term extraction @xcite . one of the challenges for this task is the scarcity of training data which hinders the training of large neural networks. to alleviate this issue, two major directions have been explored in the literature: (i) joint training, i.e., jointly solving ate task with another related task such as absa @xcite @xcite or opinion term extraction (ote) @xcite ; and (ii) data augmentation, i.e., augmenting ate models with in-domain unlabeled data @xcite @xcite generated complement sentences targets however , there are major issues with although , exist some problems with the touchpad touchpad the touchpad which render the device nearly useless . and mouse makes touchpad useless and touchpad useless way too much money for such a terrible meal . food costs so much for such a bad meal . meal in this work, we demonstrate the viability of this technique for aspect term extraction. ## conclusion we introduce a new training procedure for aspect term extraction. in the proposed procedure, the available ate dataset is employed to train a deep model which is further used to find complement representations for input sentences in training data. later, to obtain the complement sentences at the inference time, we tine-tune the pre-trained language model gpt-2 to generate sentences similar to the complement sentences found in the previous steps (with glove mapping). our extensive experiments on benchmark datasets reveal the superiority of the proposed model, leading to the state-of-the-art performance for the datasets. moreover, our analysis show that all steps of the proposed procedure are necessary and effective for the ate task. in future, we will explore the application of this procedure in other related task such as ote and absa.