Word Substitution with Masked Language Models as Data Augmentation for Sentiment Analysis (2023)

Abstract: This paper explores the use of masked language modeling (MLM) for data augmentation (DA), targeting structured sentiment analysis (SSA) for Norwegian based on a dataset of annotated reviews. Considering the limited resources for Norwegian language and the complexity of the annotation task, the aim is to investigate whether this approach to data augmentation can help boost the performance. We report on experiments with substituting words both inside and outside of sentiment annotations, and we also present an error analysis, discussing some of the potential pitfalls of using MLM-based DA for SSA, and suggest directions for future work.

## introduction one important challenge in sentiment analysis, like for most areas of nlp approached as supervised learning tasks, is that of limited availability of labeled training data. as annotation is typically a manual process requiring human expertsthereby incurring a high cost in terms of time, effort, and money -the creation of labeled training data represents a major bottleneck, especially for smaller languages like norwegian. at the same time, we know that the amount of training examples is the most important driver for increasing model performance. this paper reports on preliminary results with using a pre-trained masked language model (mlm) for data augmentation (da), applying the mlm to generate alternative substitutions for different words in the training data. more specifically we investigate the role of mlmbased da for the task of so-called structured sentiment analysis (ssa), i.e. predicting not just the positive/negative polarity of a text, but also the spans of polar expressions, targets and holders @xcite . we start by briefly discussing related work, before presenting the resources used in our experiments. we then discuss the details of the data augmentation strategy, before presenting our experimental results, including an error analysis. ## related work previous work on data augmentation in nlp can be viewed as either augmenting the feature space (e.g. interpolation techniques, addition of noise) versus augmentation of the (input) data space, where augmentation can take place at character, word, phrase or document-level @xcite . while rule-based approaches to data augmentation have been studied, a majority of current da-research within nlp falls within a modelbased approach to augmentation of the input data @xcite . examples of these techniques range from da through back-translation @xcite via seq2seq-based generation of paraphrases @xcite to the current widespread use of pre-trained lms to generate text augmentations @xcite . the latter approach is also the one adopted in this paper. within the field of sentiment analysis, a specific challenge has been the issue of label preservation @xcite , i.e. avoiding a switch of polarity following augmentation. both lexicon-based @xcite and embedding-based (feng et al., 2019) methods for substitution, or combinations of the two, have been proposed to address this challenge. the previous work that is perhaps closest to that of the current paper is that of @xcite , both in terms of da approach and application task. in the context of the 2022 semeval shared task (task 10) on structured sentiment analysis @xcite , @xcite augment the training datasets by masking out each token in turn and generating replacements with xlm-roberta-large, only accepting the top 5 most confident word predictions. chen et al. (2022) augments sentiment-bearing sentences only, but exclude replacements for tokens that are within the span of polar expressions. ## sentiment dataset and models the norwegian review corpus -norec @xcite -is a corpus of professional reviews gathered from multiple norwegian news sources and spanning a wide range of different domains (books, music, movies, games, restaurants, various consumer goods, and more). a subset of this corpus, norec fine @xcite , comprises 11 437 sentences that have been annotated for so-called structural or fine-grained sentiment information. the annotations indicate the span of polar expressions, their corresponding holderand target expressions, and the associated polarity (positive/negative) and intensity (on a three-point scale), see figure 1 for an example. our experiments are performed on the pre-defined splits of the norec fine dataset. all our experiments are based on the classification architecture described by @xcite , which adapts the graph-based semantic parser perin by @xcite for the task of structured sentiment analysis, achieving state-of-art results by directly predicting sentiment graphs from text. we will refer to this architecture as ssa-perin. 1 a norwegian instance of the bert (base, cased) architecture @xcite dubbed norbert2 and released by the norlm initiative @xcite is used both for training our ssa-perin models and also for generating substitutions in our mlm-da experiments. it features a 50 000 wordpiece vocabulary and was our augmentation approach comprises three steps: our da approach is similar to that of @xcite , but differs in that we exhaustively consider replacements for all tokens in all sentences in the training data (not just the top 5 most probable replacements for sentiment-bearing sentences only), and we also test the effect of allowing replacements inside spans annotated as polar expressions. for the latter, we experiment with constraining the candidates by only allowing replacements that do not represent a switch of polarity with respect to the sentiment lexicon norsentlex foot_2 @xcite . in addition, we present separate experimental results for replacing tokens only inside or only outside annotation spans (target, holder, and polar expressions), and both, while also testing different values for the confidence threshold. ## experimental results and discussion below we first describe the tested model configurations and evaluations measures, before discussing the results and presenting an error analysis, including suggestions for future work. ## conclusion we have presented a suite of experiments with using a pre-trained masked language model for augmenting the annotated training data for structured sentiment analysis. while we find that the described augmentation strategy often leads to improvements, the effects are modest, and high variance makes it difficult to draw definite conclusions. our analysis of the results, based on a dataset of annotated norwegian review data, point to several directions for further research on this topic, such as filtering candidate tokens for substitution with respect to frequency, pos, polarity, negation cues, and more.