Data Augmentation via Subtree Swapping for Dependency Parsing of Low-Resource Languages (2020)

Abstract: The lack of annotated data is a big issue for building reliable NLP systems for most of the world’s languages. But this problem can be alleviated by automatic data generation. In this paper, we present a new data augmentation method for artificially creating new dependency-annotated sentences. The main idea is to swap subtrees between annotated sentences while enforcing strong constraints on those trees to ensure maximal grammaticality of the new sentences. We also propose a method to perform low-resource experiments using resource-rich languages by mimicking low-resource languages by sampling sentences under a low-resource distribution. In a series of experiments, we show that our newly proposed data augmentation method outperforms previous proposals using the same basic inputs.

## introduction data sparsity has been a problem since the beginning of natural language processing. neural networks have not solved it and have made it even more visible with their hunger for data. hence a revived interest in data augmentation, since artificially created annotation can prove very useful to tackle the lack of manually annotated training data. many approaches have been proposed to perform data augmentation. some rely on external resources, such as unannotated raw text in order to iteratively increase their training data with automatically annotated examples @xcite . however, this is an error-prone method and useful annotations must be separated from harmful ones. other proposals instead rely solely on available annotated data in order to generate new examples @xcite . in this paper, we present a language-agnostic approach to the automatic generation of dependencyannotated sentences. our method essentially swaps compatible subtrees from different sentences in order to generate new annotated sentences. by enforcing a number of constraints on the subtrees to be swapped, we avoid to generate too ungrammatical sentences. furthermore, contrary to previous work that kept the syntactic structure of sentences or impoverished it, our method injects structures from other sentences, so it can introduce more syntactic complexity in the generated sentences. in order to assess the potential of our new data augmentation method for low-resource languages in a way that is independent from the specific sentences in low-resource treebanks, we propose a method to mimic low-resource language data using high-resource language data. by sampling sentences from highresource languages using a low-resource language distribution (over sentence length), we can perform the same experiment several times in a more faithful low-resource setting, while dampening the role played by each individual sentence on the final results, which can be prominent in very low-resource settings. while there are many methods to parsing low-resource languages, such as unsupervised parsing or cross-lingual transfer, in this paper we consider a very restrained setting where one has only access to mono-lingual parsing data and nothing more. this very strict setting addresses two research problems. ## data augmentation in this section we present a language-agnostic method to automatically generate high-quality dependency annotated data from a small amount of (manually) annotated sentences. beside being as languageagnostic as possible, our method should (1) create new structures and (2) create grammatically sound sentences. while the cropping operation of @xcite creates new sentences, it has two main issues. the first one is that sentences created this way may be ill-formed. for example, french finite verbs always need an overt subject, and removing it creates ungrammatical sentences. the second one is that it creates new sentences by impoverishing existing ones. cropped trees are by essence smaller and simpler than their source trees. this may be a problem since small treebanks already have less material and simpler structures than bigger ones. to avoid these issues, we propose to create new sentences by swapping subtrees between sentences under constraints we describe below. swapping subtrees between sentences creates both simpler and more complex sentences, and strong constraints ensure that these sentences are as grammatical as possible. figure 1 : illustration of tree swapping operation between sentences. sentences c and d result from swapping the direct objects (represented with an incoming dashed arrow) from sentences a and b. sentence c is more complex than a since its new direct object has an adjective. figure 1 shows an example of tree swapping that creates both a more complex and a simpler sentence. in sentence c, the new direct object has an adjective modifying the noun, while in a, the noun had just a determiner. swapping subtrees can introduce other new complexities in otherwise simple sentences, such as relative clauses. sentences d and b show the inverse phenomenon. ## low-resource experiments with high-resource languages to be able to perform extensive experiments, we chose to work with resource-rich languages and use them to mimic low-resource conditions. the most basic way to do so is to sample a small number of sentences from a big treebank to artificially create a small one. this has the big advantage that we can sample many different small treebanks and therefore dampen the influence of any given training sentence on parsing results. this is especially important in a very low-resource setting where each sentence can have a strong impact on the actual parsing results, while not being very representative of its actual source language. however, the difference between high-resource treebanks and low-resource treebanks goes beyond the mere number of sentences or words. bigger treebanks tend to have more varied constructions than their smaller counterparts foot_0 . this means that for a set number of sentences or tokens, a sample from a resourcerich treebank may contain more information than a sample from a resource-poor one. therefore, if we want to mimic low-resource languages with high-resource ones, we have to take sentence complexity into consideration. in this paper we use sentence length as a surrogate for sentence complexity, under the assumption that more complex constructions require more words 2 . other complexity markers could be taken into account such as dependency relation distribution. but annotators do not choose sentences according to their internal structures or complexity, thus the relation distribution of a small treebank is as contingent as the presence of any given sentence in it. so we keep this open for future work. ## experiments in order to assess the potential of tree swapping as a data augmentation method for low-resource languages, we ran a series of experiments on 10 languages from ud 2.4, representing various families and various syntactical and morphological typologies. for each language, we sampled 8 times 40 sentences with the smoothed distribution of low-resource language sentence lengths depicted in figure 2 . while 40 training sentences might seem very low, in ud 2.4, buryat has 19 training sentences, kurmanji 20, upper sorbian 23 and kazakh 31. ## conclusion in this paper, we have presented a new language-agnostic data augmentation technique based on dependency subtree swapping for creating new dependency trees. we also presented a method for performing more faithful low-resource experiments using high-resource languages by sampling training data under a distribution that favors shorter sentences to mimic the sentence length distribution of low-resource languages. we have shown that our newly proposed tree swapping method consistently outperforms previously proposed augmentation techniques based on tree morphing. we have also shown that our method can create many new sentences and that they are useful for parser training as the score increases with the number of sentences. this is important since contrary to previous tree morphing techniques, the number of sentences created by tree swapping is potentially unbounded. then, we have shown that relaxing the strong swapping constraints on a per language basis further improves the results. but that the language/constraints relation is not necessarily clear. finally, we saw that despite being useful for training parsers, the created sentences are not diverse enough to be useful for model validation. previous works have demonstrated the possibility of training parsers with incomplete annotation @xcite . as a few generated sentences may be odd sounding or slightly ungrammatical, it would be interesting to see how parsers fare when trained with sound trees over ungrammatical sentences. we keep it as a future work. we also need to further investigate the three-way interaction between languages, augmentation techniques and parsing algorithms, as apparently not all augmentation techniques fare as well for all languages. mixing data augmentation policies might also have a positive impact. more generally, it would be interesting to see how far a parser can go with only a handful of annotated sentences.