Data Augmentation for Inline Tag-Aware Neural Machine Translation (2022)

Abstract: Despite the wide use of inline formatting, not much has been studied on translating sentences with inline formatted tags. The detag-and-project approach using word alignments is one solution to translating a tagged sentence. However, the method has a limitation: tag reinsertion is not considered in the translation process. Another solution is to use an end-to-end model which takes text with inline tags as inputs and translates them into a tagged sentence. This approach can alleviate the problems of the aforementioned method, but there is no sufficient parallel corpus dedicated to such a task. To solve this problem, an automatic data augmentation method by tag injection is suggested, but it is computationally expensive and augmentation is limited since the model is based on isolated translation for all fragments. In this paper, we propose an efficient and effective tag augmentation method based on word alignment. Our experiments show that our approach outperforms the detag-and-project methods. We also introduce a metric to evaluate the placement of tags and show that the suggested metric is reasonable for our task. We further analyze the effectiveness of each implementation detail.

## introduction while most machine translation studies are focused on plain text, the textual information that we encounter every day on the internet contains words with different styles and links within the sentence. various styling of any part of the text is called inline formatting and is represented by markup and markdown tags. the inline formatting not only improves the readability of documents but also provides additional information with tags; so it is important to correctly translate sentences including tag information. in addition, the widespread use of formatting tags in the computer-based document system makes it inevitable to increase the demand for translating web text or structured documents containing inline tags. there are two main approaches to translating segments with inline tags. one solution is the detagand project method @xcite . it first strips tags from the source sentence and translates only the plain text. then, the removed tags are reinserted into the translation results using word alignments, which can be induced from attention weight in the model or an external aligner such as simalign @xcite . this method does not take into account the re-insertion of tags in the translation process, making it difficult to restore tags at the proper positions. another way is to use an end-to-end model which takes sentences including tags as inputs and generates translation results with tags. since tag information is considered, the translation can be performed with more context, thus this method potentially improves the quality of translation and the placement of tags. to train end-to-end models, a parallel corpus, where both source target sentences contain aligned tags, is required. even though a parallel corpus with markup tags was released by @xcite , their data is limited to the domain of online help and there is still not many of such data available to train a high-quality model. to address this lack of tagged parallel corpus, @xcite introduces a data augmentation approach using tag injection. their method is to insert tags into corresponding fragments in the source and the target. in their approach, the aligned phrases are identified by an exhaustive search by matching all translated source fragments with all target fragments. this method has two drawbacks by its nature. the first is that their approach requires a high computational cost because it requires computing translation for all possible phrases for at least millions of parallel sentences to train a model. secondly, only constrained tags can be augmented because they find corresponding pairs with out-of-context translation. in this paper, we propose an efficient and effec-tive tag augmentation method using word alignments @xcite to overcome the above shortcomings. our method uses an external word aligner to compute correspondence between the source and target words, and find aligned fragments by phrase extraction algorithm @xcite . then tags are inserted according to the phrasal alignments. the tag-augmented parallel corpus by this method can train a model that translates sentence containing tags in an end-to-end way. for comparisons, we implement competitive baselines and propose a metric to automatically evaluate the placement of tags. through experiments, we show that our approach is superior to the detag-and-project methods and demonstrate the effectiveness of each implementation detail. ## method in this section, we propose an efficient and effective method to insert inline tags into an existing parallel corpus. in augmented data, the position of tags in the source segment must be preserved in the target segment. the word "preserved" means that tags in the target sentence must surround spans with the same role and meaning as the corresponding source spans. in other words, the source and target fragment in the same tag has to correspond with each other. moreover, inline tags can contain not only a word but also a phrase or even any consecutive words. therefore, how to find corresponding phrase foot_0 pairs for each parallel sentence is the key to synthesizing tag-aligned parallel data. this makes our method focus on finding aligned phrase pairs. our proposed augmentation method consists of three steps. we first generate word alignments for the parallel corpus using external word aligners (section 2.1). then we extract aligned phrase pairs for each sentence pair with the word alignments (section 2.2). lastly, for each parallel sentence and the aligned phrase pairs, since each sentence usually has a lot more aligned pairs than the number of words in the sentence, we randomly select some of the pairs and insert tags to surround the phrases (section 2.3). figure 1 presents the whole process of our methods. the example is from philipp koehn's lecture foot_1 . ## evaluation in this section, we describe several metrics to evaluate our methods and present experimental results. ## conclusion in this paper, we have presented an efficient and effective inline tag augmentation method to insert tags into existing parallel corpora using a word aligner and the phrase extraction algorithm. our approach injects inline tags economically and accurately. we also introduced a reasonable metric for the automatic and accurate evaluation of the placement of tags and analyzed the effectiveness of the detailed methods used in our approach. the experiment results show that the model trained on data augmented by our method outperforms the previous detag-and-project methods.