An Empirical Study of Contextual Data Augmentation for J apanese Zero Anaphora Resolution (2020)

Abstract: One critical issue of zero anaphora resolution (ZAR) is the scarcity of labeled data. This study explores how effectively this problem can be alleviated by data augmentation. We adopt a state-of-the-art data augmentation method, called the contextual data augmentation (CDA), that generates labeled training instances using a pretrained language model. The CDA has been reported to work well for several other natural language processing tasks, including text classification and machine translation. This study addresses two underexplored issues on CDA, that is, how to reduce the computational cost of data augmentation and how to ensure the quality of the generated data. We also propose two methods to adapt CDA to ZAR: [MASK]-based augmentation and linguistically-controlled masking. Consequently, the experimental results on Japanese ZAR show that our methods contribute to both the accuracy gainand the computation cost reduction. Our closer analysis reveals that the proposed method can improve the quality of the augmented training data when compared to the conventional CDA.

## introduction in pro-drop languages, such as japanese and chinese, the arguments of a predicate are frequently omitted in sentences. automatic recognition of such omitted arguments plays an important role in understanding the predicate-argument structure of a sentence. this task is referred to as zero anaphora resolution (zar). figure 1a depicts an example of the task. in the sentence, the nominative argument of the predicate was acquitted is omitted. such an omitted argument is referred to as a zero pronoun often represented as φ. in figure 1a , φ refers to the man. one critical issue of zar is the scarcity of labeled data. to compensate for the scarcity, several studies have explored the direction of exploiting unlabeled data @xcite @xcite @xcite @xcite . another promising direction is to automatically augment labeled data (data augmentation). one state-of-the-art data augmentation method is contextual data augmentation (cda), which augments labeled data by replacing an arbitrary token(s) with another token(s) predicted to be plausible in the surrounding context by a pretrained language model (lm). cda works well in several natural language processing (nlp) tasks, such as text classification @xcite and machine translation @xcite . however, unlike the direction of exploiting unlabeled data, the potential of data augmentation in the context of zar has largely remained underexplored. this study investigates how the idea of cda can be effectively employed in zar, taking the task of japanese zar as a case study. figure 1 illustrates a straightforward approach of applying the idea of cda to zar. for each given training instance (i.e., input sentence (a) paired with its gold labels (l)), a standard cda method would use a pretrained lm to generate a variant(s) of the original sentence, say sentence (b). the new sentence (b) is then paired with the original gold labels (l) and is added to the training set (detailed in section 4.1). this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . this straightforward approach raises two critical issues. one is that the computational cost of cda can be non-trivial because it runs a huge pretrained lm for each input sentence in the training set. note that even with a relatively small training set, this cost can be enormous when one wants to repeatedly conduct experiments with a large variety of settings. the second issue is that cda may produce improper instances. figure 1b shows that the original noun man is properly replaced with another noun criminal. by saying that a word is properly replaced with another, we mean that the replacement does not affect the anaphoric relation labels of the original sentence. this requirement is critical because we want to use produced instances with the original zar signals to train our model. however, if not appropriately controlled, cda may disrupt the original anaphoric relations, as in figure 1c , where replacing the verb sued with struck affects the anaphoric relation between φ and boy. these issues can be crucial when applying cda to zar, but they have not been addressed in previous works with cda for other nlp tasks. to address these two issues, we newly design a cda variant with two directions of extension: (i) [mask]-based augmentation and (ii) linguistically-controlled masking. unlike a standard cda method, which replaces tokens with other tokens, the [mask]-based augmentation replaces tokens with the [mask] token (section 4.2). this simple extension halves the cost of running a pretrained lm. our experiments show that it also improves the performance of zar. the second extension, which is linguistically-controlled masking, provides a means of controlling the types of tokens to be replaced to inhibit improper token replacements (section 4.3). the token replacement quality must be considered even in the [mask]-based augmentation scheme because replacing a token with the [mask] token may also affect the anaphoric relations. our experiments demonstrate that controlling token replacement by part-of-speech (pos) tags effectively inhibits improper token replacements and improves the performance of zar (see section 6 for details). through extensive experiments, we reveal the relationship between performance gain and replacement of each token type with various masking probabilities. we deepen our understanding of the change of antecedents through a detailed analysis on actual augmented data. in summary, our main contributions are as follows: ## method in this section, we first review an existing standard cda method (section 4.1), which is our starting point of data augmentation for zar. we then discuss the quantitative and qualitative aspects of cda: (i) computational inefficiency of symbolic replacement and (ii) change of antecedents. considering these aspects, we improve a standard cda method with two methods, namely [mask]-based augmentation (section 4.2) and linguistically-controlled masking (section 4.3). ## experiments in this section, we first investigate the effectiveness of [mask]-based augmentation by comparing it with the baseline cda (section 4.1) on the validation set (section 5.2). we then investigate the relationships between performance gain and each pos category to be replaced with some masking probability to find the best configuration of linguistically-controlled masking over the validation set (section 5.3). finally, we compare the best configured model with the other existing models over a test set (section 5.4). ## analysis of augmented data in this section, we analyze the actual instances augmented by linguistically-controlled masking. the model built by all-but-verb masking s all\verb achieved the best zar f 1 ; hence, we further investigated the negative effect of verb replacement. we did this by observing the actual instances produced by s all\verb and verb-only masking s verb . figure 4 depicts the three types of sentences of these two models, that is, the original ones x, the masked ones x , and those with some symbolically replaced tokens x . note that although our method did not use any symbolic substitutions x for predicting the of zero anaphora (section 4.2), they were produced only for the analysis. 8 looking at the symbolic token generated from each mask provides us an intuition on the feature vector of each mask. for example, if the token exist is generated from a mask, we can interpret that its corresponding vector approximately represents the meaning of exist. in the original sentence x, the nominative argument of the target predicate noticeable was realized as a zero pronoun φ, and its antecedent was the scratches. in the sentence generated by s verb , the symbolic token exist filled the [mask] (x 1 ), and the semantic structure of the sentence, especially the original relation between the two predicates (verbs) be fixed and noticeable, was changed. the antecedent can be interpreted as null, which was changed from the original one the scratches. in contrast, in the sentence generated by s all\verb (x 2 ), the antecedent of the zero pronoun was not changed, even though half of the tokens in the sentence has been replaced. the antecedent was that, which aligned with the original token (position), scratches. the original relation between the two predicates, be fixed and noticeable, still remained. in addition, the pretrained mlm generated the tokens surrounding these predicates, such that the context drastically changed from the original one the scratches can't be repaired, but . . . to if that gets fixed, . . . . such a contextual variation plays an important role for improving the robustness and the coverage of a zar model, even if the meaning is slightly unnatural. this can be a reason for the results in section 4.3, indicating that the setting of a high masking probability (approximately 0.5) and all-but-verb masking s all\verb achieves a better performance.