T ree S wap: Data Augmentation for Machine Translation via Dependency Subtree Swapping (2023)

Abstract: Data augmentation methods for neural machine translation are particularly useful when limited amount of training data is available, which is often the case when dealing with low-resource languages. We introduce a novel augmentation method, which generates new sentences by swapping objects and subjects across bisentences. This is performed simultaneously based on the dependency parse trees of the source and target sentences. We name this method TreeSwap. Our results show that TreeSwap achieves consistent improvements over baseline models in 4 language pairs in both directions on resource-constrained datasets. We also explore domain-specific corpora, but find that our method does not make significant improvements on law, medical and IT data. We report the scores of similar augmentation methods and find that TreeSwap performs comparably. We also analyze the generated sentences qualitatively and find that the augmentation produces a correct translation in most cases. Our code is available on Github.

## introduction most natural language processing (nlp) problems are formulated as supervised learning tasks, where large amounts of data is required to train models. collecting annotated datasets is often timeconsuming and laborious, so this motivated a lot of work in nlp to create methods for generating synthetic data that improves the dataset used for training in both size and variety, ultimately leading to more performant models @xcite . these data augmentation (da) methods not only help in resource-constrained scenarios, but can also improve class imbalance @xcite , mitigate bias @xcite , make the model more robust to out of distribution inputs @xcite or simply improve model accuracy. an efficient data augmentation method for any nlp task has two main objectives, which need to be balanced: the augmented data should be diverse enough, that it provides new information during training, but it should also be label-preserving to avoid injecting unwanted noise into the model. in machine translation, this means that our aim is to generate diverse sentence pairs from existing data such that the parallelism holds. in this paper, we propose treeswap, a data augmentation method for neural machine translation (nmt) using dependency parsing. the core idea of treeswap is to find corresponding subtrees in the dependency parse trees of a translation pair and swap these to generate new data. as our augmentation procedure is based on dependency parsing with some additional rules to improve grammatical and morphological correctness, the generated sentence pairs are semantically nonsensical in many cases. using such nonsensical or nonce, but syntactically correct sentences as training data has been studied before and shown to perform well even when models cannot rely on semantic or lexical cues @xcite . to demonstrate the effectiveness of treeswap, we perform resource-constrained experiments on 4 language pairs in both directions. we also train models on domain-specific corpora and evaluate on both in-domain and out-of-domain test sets. we compare our results to other common augmentation methods in nmt using standard machine translation metrics. to study the quality of the generated sentences and understand the possible errors in the augmentation, we also perform a qualitative analysis on the synthetic sentence pairs. ## related work in the context of machine translation, backtranslation @xcite has been the most dominant da method. it uses monolingual data in the target language to generate new training samples. backtranslation and its variants were shown to boost translation quality at multiple scales @xcite and demonstrate sota results on many language pairs @xcite . @xcite select rare words in the corpus and replace these in new contexts simultaneously in the source and target sentences. @xcite introduce reward augmented maximum likelihood (raml), which replaces words in the target sentence with other words from the target vocabulary. switchout @xcite is an extension of raml, where the augmentation is performed on both the source and target sentences. instead of selecting words from the vocabulary for replacement, seqmix @xcite randomly combines two sentences from the input. @xcite introduce soft contextual da, where they replace the embedding of a random word with a weighted combination of other semantically similar, related words. @xcite use the depth of tokens in the dependency tree for weighting the selection probabilities of tokens for blanking, dropout and replacement. @xcite augment by merging the predictions of multiple forward and backward models with the original dataset. @xcite improve the translation of entities and terminological expressions using knowledge graphs for augmentation. @xcite apply simple transformations that are used as auxiliary tasks in a multi-task learning framework with the aim of providing new contexts during prediction. @xcite propose continuous semantic augmentation (csanmt), which augments each training instance with an adjacency semantic region to cover synonymous representations. syntax-based augmentation methods have been shown effective in a number of nlp tasks. @xcite use the directionality of relationships in a dependency tree to improve relation classification models. s ¸ahin and steedman (2018) generate augmented data for part-of-speech tagging by morphing the dependency tree through cropping edges and performing rotations around the root. @xcite extend this method for dependency parsing and also apply another augmentation called nonce sentence generation, inspired by @xcite . @xcite extends the subtree swapping method to augment data for dependency parsing. they perform the swapping in a more generic setting, not only on subjects and objects, but apply a wide range of morphological and structural constraints to ensure grammatical correctness. @xcite see improvements in few-shot constituency parsing by dependency subtree substitution. @xcite present a generalization of the previous methods and perform experiments on multiple nlp tasks. for reference, preliminary results of treeswap have been published prior to this paper @xcite . ## experiments we conduct experiments on 4 language pairs, english to german, hebrew, vietnamese and hungarian in both directions. we selected corpora that are considered low-resource and widely used in the community to evaluate data augmentation approaches for machine translation. we also perform domain-specific experiments in three domains, evaluating the effectiveness of the da method on both in-domain and out-of-domain setups. we ran all experiments 3 times with different seeds for robust results. datasets for english-german and english-hebrew we use the iwslt 2014 text translation track @xcite datasets for training data as done by @xcite , @xcite and @xcite . for development and testing we use the tst2013 and tst2014 datasets. for english-vietnamese, we use the iwslt 2015 text translation track @xcite dataset with the tst2012 and tst2013 datasets used for development and testing as done by @xcite and @xcite . for hungarian-english, we produce a subsample comparable in size to the iwslt datasets using the hunglish2 corpus @xcite . as lowresource datasets are usually composed of a few sources and they generally are not linguistically diverse, we decided to only sample from the modern literature subcorpus of hunglish2 and discard the others. this should still be considered as a highresource experiment with withheld data, although we try to mimic a low-resource scenario as much as possible. following @xcite and @xcite , we use the it, law and medical domain-specific datasets published by @xcite . the statistics of the datasets are summarized in ## results in order to measure the effectiveness of treeswap, we used common evaluation metrics such as the bleu and the meteor scores. these scores were computed for both the augmented and baseline models to enable a comparative analysis of the proposed method against previous augmentation approaches. the results of these analyses are presented in ## conclusion in this paper we presented a new data augmentation method for nmt that we call treeswap. our method generates new samples by swapping compatible subtrees of the dependency parse trees of translation-pairs. more precisely we swap objects and subjects simultaneously in the source and target sentences between two translation pairs to generate new parallel translations. experiments on 4 language pairs in both directions have shown that models trained with data augmented using treeswap can consistently outperform baseline models. we also compared treeswap to other augmentation methods used in nmt and found that treeswap achieves compatible performance to other methods. however, with domain-specific corpora, treeswap brought little to no performance gains in terms of quantitative metrics, which suggests that the type of corpora used for augmentation heavily influences the success of our method. our qualitative analysis has shown that the generated sentences are predominantly correct translations, but also revealed that treeswap can induce certain undesired grammatical errors. it is an interesting future direction to explore how these issues could be fixed either via heuristics or fixing the morphosyntactic errors with another model. the improvements by treeswap (like many other augmentation methods) seem to depend on finding a good balance between distorting the translation distribution and enriching the model with synthetic translation pairs. it would be interesting to study the change in translation distributions induced by treeswap.