Towards Reversal-Based Textual Data Augmentation for NLI Problems with Opposable Classes (2020)

Abstract: Data augmentation methods are commonly used in computer vision and speech. However, in domains dealing with textual data, such techniques are not that common. Most of the existing methods rely on rephrasing, i.e. new sentences are generated by changing a source sentence, preserving its meaning. We argue that in tasks with opposable classes (such as Positive and Negative in sentiment analysis), it might be beneficial to also invert the source sentence, reversing its meaning, to generate examples of the opposing class. Methods that use somewhat similar intuition exist in the space of adversarial learning, but are not always applicable to text classification (in our experiments, some of them were even detrimental to the resulting classifier accuracy). We propose and evaluate two reversal-based methods on an NLI task of recognising a type of a simple logical expression from its description in plain-text form. After gathering a dataset on MTurk, we show that a simple heuristic using a notion of negating the main verb has a potential not only on its own, but that it can also boost existing state-of-the-art rephrasing-based approaches.

## introduction in natural language processing (nlp), the high performance of a machine learning solution often depends on the quality and quantity of training data, but its collection is not always trivial @xcite . for some tasks, such as sentiment analysis, extensive corpora already exist, and can be used at least as a starting point. however, in the area of natural language interfaces (nlis), tasks are often very specific, and training data often has to be collected from scratch, which can be a major limiting factor. data augmentation is a family of techniques that take an initial dataset (often limited in size) and automatically generate more examples, with the hope that they will introduce some realistic variability, thus reducing reliance on possibly costly and time-consuming data collection. in some areas, good data augmentation approaches are available and widely used. for instance, to augment an image, one can scale it @xcite or crop it @xcite . data augmentation is not limited just to computer graphics. for instance, @xcite proposed an augmentation approach for a more narrow area of schema matching. however, in nlp, usage of data augmentation is somewhat limited @xcite . currently, it is a very active area of research, and multiple different approaches are used. almost all of them, however, rely on the same fundamental principle: to augment a sentence, generate sentences that have the same meaning, but use slightly different phrasing. one of the limitations of such "rephrasing-based" approaches is that when applied to a sentence labelled with a certain class label, they can only generate sentences that belong to the same class. for instance, @xcite proposed to replace certain words in a sentence with their synonyms. applying such an approach to a positive review "this is a good movie" can result in "this is a fantastic movie", "this is a good film" and so on, but all of them will still be positive reviews. in this paper, we propose a different approach: instead of preserving the meaning of the sentence intact, we attempt to reverse its meaning, so that the "opposite" class can also benefit from data augmentation. to the best of our knowledge, limited attention has been paid to investigating such approaches, with the exception of few papers in the domain of adversarial learning @xcite @xcite . reversing polarity of a free-text snippet, such as a movie review or a tweet, can be challenging and even impossible, if done automatically. in this preliminary investigation we are limiting ourselves to short, relatively technical sentences, which are very common in nli space. using reversal of meaning for data augmentation can be useful for the tasks which have "naturally opposable" classes, for instance, in voice assistant dealing with opposite actions ("set the alarm for 7am" vs. "unset the alarm for 7am" or "turn the volume down" vs. "turn the volume up"). our motivating example is understanding a logical statement, expressed as a short sentence in natural language. we were dealing with the first step of this task, recognising one of six statement types, which constitute three "naturally opposable" pairs. one of such pairs is "equal"/"not equal". by taking an example that belongs to "equal" class such as "x is equal to two", we can reverse its meaning to "x is not equal to two", and in such way augment "not equal" class as well. as far as we know, it is the first attempt to use such a technique for textual data augmentation in text classification tasks. the main contributions of this paper are the following: the rest of the paper is structured as follows. in section 2, we outline the task and describe data collection. section 3 follows with an overview of related research. section 4 describes our approaches in detail, and is followed by section 5 outlining our evaluation methodology. experimental results are discussed in section 6, while section 7 concludes the paper and proposes directions for future work. the motivational application behind our experiments is the task of translating a sentence in natural language into a logical expression that otherwise would have to be specified using complex and not very user-friendly language, such as sql (or a formula editor in a tool such as excel). this task can be solved in different ways, and some of them rely on inferring statement type (or its intent) as the first step. we were concerned with six such types that were divided into three pairs, where classes in each pair are exact opposites. it means that if the meaning of a sentence from one pair class is reversed, we get a sentence that belongs to the second class in the same pair. an overview of all classes with examples is given in a variety of free-text phrases can be used to describe the same logical expression. for instance, "x is n u ll" can be phrased as "x is blank", "x should not be empty", "x should be populated in all cases" and so on. we wanted to get a reliable expression type detector that would be robust to such examples of language variability. we collected the data on amazon mechanical turk platform. a single hit presented a logical statement (such as "len gt h(x) > 0" or "@xmath0, and a turker had to provide four significantly different phrases, that describe the logic in natural language. first, we performed a quick pilot collection to make sure that our instructions and setup make sense, involving only master turkers. no problems were encountered, so we proceeded with the main data collection, involving only turkers with acceptance rate on previous tasks of at least 60%. each hit was completed by at least 10 workers, and we paid $0.20 for every successful completion 1 . we had to reject about 35% of submitted hits due to imprecise or spammy answers. additionally, some of the accepted answers were same, for instance, expression "x is n u ll" was often transcribed as "x must be empty". after eliminating such duplicates, we got a dataset of 851 sentences, achieving good balance inside each pair of classes. the resulting dataset is publicly available online 2 . next section describes existing data augmentation methods. we discuss methods that try to preserve the meaning of the original sentence (majority of them), as well as few exceptions. ## related research textual data augmentation was used in a number of different areas, including text classification @xcite or visual question answering @xcite , reading comprehension systems @xcite and machine translation @xcite . in fields like computer vision or speech, there are well-established methods that work across many applications, such as introducing random noise into an audio clip or cropping an image. however, according to @xcite , in the field of nlp, it is very difficult to come up with an approach that would be easily applicable to various tasks. it can explain existence of a variety of augmentation strategies, most of which can be broadly categorised into two big categories: strategies that rephrase a sentence preserving its original meaning, and strategies that deliberately change the meaning of a sentence. 1 median completion time for a hit was 204 seconds. 2 https://github.com/ alexey-tarasov-irl/acl2020_nli_workshop ## our approaches approach by @xcite , described in the previous section, is not directly applicable to our task, as replacing nouns and numbers won't result in effective negation of sentences in our dataset. the only useful aspect-antonyms of adjectivesis also present in paper by @xcite , which also contains other useful insights. we propose two approaches, that enhance add negation and antonym by @xcite described in section 3.2: it might seem that both add negation and antonym by @xcite can result in a dataset of much lower quality, compared to the original. however, their intention was to pollute a sentence enough to change its meaning in some way, or make it incomprehensible, to deliberately confuse dialogue systems. however, we are trying to achieve something much more complicated. in our task, it's not enough to break the meaning of a sentence. we aim for coming up with valid sentences that properly negate the source sentence. this is why our main assumption is that not producing a sentence at all is better than producing a sentence that doesn't make sense. we only replace adverbs and adjectives, and only if it's the only adverb/adjective in the sentence, and it is directly connected to the root. for each such sentence, we produce a new sentence for each antonym found in wordnet, and preserve comparative/superlative adjective forms. ## experiment methodology the goal of our experiments was to investigate whether reversal-based data augmentation can boost accuracy in text classification tasks. we were concerned with two questions: ## results in experiment #2 ( overall, the results allow us to recommend main verb inversion as a promising direction for textual data augmentation in classification tasks. com-pared to no augmentation baseline, it improved macro f1 by 7.53-27.54pp (or by 13.42-99.15%), depending on how balanced the dataset is. main verb inversion was especially beneficial when used to imbalanced datasets. when used on top of eda, main verb inversion was able to improve the f1 score by 2.04-5.93pp (or 2.92-11.47%). ## conclusions and future work in this paper we addressed a problem of detecting a type of a logical statement from its textual description. we gathered our own task-specific dataset on mturk, and then tried to boost the accuracy of the resulting classifier by applying textual data augmentation. we used two approaches (add negation and antonym) by @xcite from the current research in adversarial learning. in our experiments, neither of them could show stable improvement over a no-augmentation baseline. even worse, often they had a detrimental effect on the macro f1 score of the resulting classifier. we proposed two approaches: adjective and adverb antonymy and main verb inversion. the former failed to expose any benefit in our experiments; however, the latter consistently performed better than baselines. despite its simplicity, it could achieve significantly better results than the no-augmentation baseline (improvement was ranging from 13% to 99%). main verb inversion also showed the capability to introduce information into the training set, which is not available to state-ofthe-art rephrasing-based approaches. a combination of eda and main verb inversion was 3-11% better than eda on its own. main verb inversion showed promising performance, but it might be difficult to use it to negate sentences, expressed in less technical language (such as tweets or movie reviews). this is why enhancing its capabilities to other application areas seems to us like the primary direction for future work. it's unlikely that it can become a widely used cross-application approach in its current shape, but we hope that our findings will be thought-provoking for researchers who want to pursue reversal-based augmentation further. one of the possible improvements is to rely on a black-box model instead of heuristic rules (e.g. a sequence-tosequence model that takes a sentence and returns the corresponding inverted sentence). while verb negation/inversion showed good performance, approaches based on directly seeking antonyms proved to be ineffective. it might be interesting to investigate the reasons of their failure in more detail.