T rans I ns: Document Translation with Markup Reinsertion (2021)

Abstract: For many use cases, it is required that MT does not just translate raw text, but complex formatted documents (e.g. websites, slides, spreadsheets) and the result of the translation should reflect the formatting. This is challenging, as markup can be nested, apply to spans contiguous in source but non-contiguous in target etc. Here we present TransIns, a system for non-plain text document translation that builds on the Okapi framework and MT models trained with Marian NMT. We develop, implement and evaluate different strategies for reinserting markup into translated sentences using token alignments between source and target sentences. We propose a simple and effective strategy that compiles down all markup to single source tokens and transfers them to aligned target tokens. A first evaluation shows that this strategy yields highly accurate markup in the translated documents that outperforms the markup quality found in documents translated with popular translation services. We release TransIns under the MIT License as open-source software on https://github.com/DFKI-MLT/TransIns . An online demonstrator is available at https://transins.dfki.de .

## introduction in mt research, models are usually trained and evaluated on plain text parallel data. but such models do not translate complex formatted documents created, e.g., with ms office. translating such documents comes with several challenges. text content has to be separated from formatting and other code and made available as input to mt. this requires a parser that handles the document format at hand and provides access to the embedded text content. after translation, the translated text must be placed into the target document and a further component is needed to create the translated version of the document reflecting the formatting and layout of the original document. furthermore, mt has to be able to handle inline sentence markup, i.e. to make sure that markup in the source sentence is correctly transferred to the appropriate parts of the target sentence. it is possible to train markup-aware mt models, e.g. by replacing tags with unique mask tokens in training and translation, as described in (zhechev and van genabith, 2010), but in order to use an existing mt model that is unaware of markup, the only option is to remove markup from the source sentence and to reinsert it at proper positions in the target sentence after translation. du et al. (2010) describe a reinsertion strategy based on the phrase segmentation indicated by the decoder. this is refined by hudik and ruopp (2011) who use word alignments instead of phrase segmentation. joanis et al. (2013) propose a hybrid approach combining phrase segmentation with word alignments. building on these, müller (2017) evaluates different markup handling strategies and provides implementations as part of the zurich nlp mtrain foot_0 framework. in order to utilize state-of-the-art mt technology and obtain alignments at the same time, we use marian foot_1 nmt (junczys-dowmunt et al., 2018). marian allows transformer models to be trained using guided alignment so that the decoder produces translations together with alignments between source and target tokens. the opus-mt foot_2 project (tiedemann and thottingal, 2020) provides pre-trained marian models for many language pairs, mostly trained with guided alignment based on eflomal foot_3 word alignments (östling and tiedemann, 2016), but unaware of markup. below, we describe transins, a translator for non-plain text documents. we use the okapi 5 framework to process such documents and extend okapi in order to query marian for translations and alignments. we study different alignment-based markup reinsertion strategies, starting with the one implemented in mtrain. we identify deficits and present improved strategies. finally, we evaluate different strategies and compare the markup quality between documents translated by transins and popular translation services. ## okapi framework okapi is a free open-source framework designed to support localization and translation processes. it includes a collection of filters providing access to the translatable content for many file formats. a workflow in okapi is modelled as a pipeline of steps that pass events through the pipeline. events are associated with resources, e.g. text units, and are created when using a filter on a source document. a typical okapi pipeline for translating documents consists of four steps: the raw document to filter events step reads the source document with an associated filter configuration and sends the filter events with the associated text units down the pipeline. the segmentation step breaks down the text units into sentences, using rules specified in segmentation rules exchange format (srx) foot_5 , a standard describing how to segment text. the leveraging step sends each sentence to a translation service and stores the generated translation with the sentence. translation services are accessed via connectors. okapi provides connectors for popular translation services, but a connector for marian is not included. the filter events to raw document step creates the target document in the original format from the translated text content coming in as filter events. okapi handles global document markup, but not inline sentence markup. this has to be dealt with by the translation service. @xcite 3 transins system description below, we describe how we build transins (translation with markup reinsertion) based on the okapi translation pipeline by adding marian specific components and setting up an additional okapi pipeline to support efficient processing. marian comes with a web-socket server that loads a model once at start time and then listens for single or batch translation queries. the server can be run remotely, supporting distributed setups with multiple marian servers. in order to use marian as a translation service from the leveraging step, we provide a marian connector that implements the okapi connector interface. most mt models are trained on parallel data where sentences are preprocessed, e.g. tokenized. sentences to be translated need to be preprocessed in the same way. also, the translation provided by the mt model might require postprocessing, e.g detokenization. with marian, pre-/postprocessing often resorts to perl scripts written for the moses statistical mt system (koehn et al., 2007). for transins, we use a python reimplementation provided by sacremoses foot_7 . transformer mt models often apply subword tokenization in preprocessing. in postprocessing, subword tokenization has to be undone in the translated sentence. for transformer models, byte-pair encoding (bpe) (sennrich et al., 2016) and sentencepiece (kudo and richardson, 2018) are popular subword tokenizers. we use publicly available implementations of both subword tokenizers. foot_8 , foot_9 undoing the subword tokenization in the translated sentence in postprocessing is straightforward by applying simple string replacements. transins wraps the steps described above in a web service that provides corresponding endpoints for pre-/postprocessing single sentences or batches. the steps to apply can be configured separately for each translation direction. the marian connector for okapi calls this web service to preprocess a sentence before translation and again afterwards to postprocess the translated sentence. the standard okapi translation pipeline is somewhat inefficient: extracted sentences arrive at the leveraging step one-by-one, i.e. only single sentences are sent for pre-/postprocessing and translation to the corresponding services, even though the services support batch processing. the accumulated overhead of connecting and disconnecting slows throughput significantly. for efficient batch processing, we set up another okapi pipeline, the sentence collector pipeline, consisting of a raw once the batch translator has collected all sentences it sends batches to preprocessing, translation and postprocessing. the batch translator also serves as cache: for each source sentence, it stores the resulting target sentence. in a second run, we process the source document with the translation pipeline, but we adapt the marian connector so that it queries the batch translator cache. this avoids connections to remote services, resulting in increased throughput. figure 1 shows the transins workflow with both okapi pipelines. foot_10 ## markup reinsertion okapi provides sentences in a generic format. sentence internal tags are encoded using characters from the unicode private use area (pua). each tag consists of two such characters. the first encodes the type of the tag, which is either opening (e.g. <@xmath0, closing (e.g. </@xmath1, or isolated (e.g. <br/>). the second character encodes a running tag index. tags provided by okapi are always wellformed and balanced, i.e. for each opening tag, there is a corresponding closing tag, and tag pairs are properly nested. tag indices are unique, and so is each tag pair. the opus-mt models we use with transins are unaware of markup, so the only option is to remove tags before translation and to reinsert them afterwards. the general workflow of all transins markup reinsertion strategies is shown in figure 2 . alignments provided by marian foot_11 refer to token indices. as tokenization is decided by preprocessing, we can only map and remove tags from the source sentence after preprocessing. postprocessing like detokenization might change target tokenization. therefore tag reinsertion has to be done into the raw target sentence before postprocessing. as reinserted tags may end up anywhere and in any order in the target sentence, we do a cleanup step after tag reinsertion that takes care of: tags in subword token sequences: if a tag is inserted within a sequence of subword tokens, e.g. as created by bpe, that tag has to be moved so that the subword token sequence can be properly merged in postprocessing. we move opening and isolated tags to the front of the sequence and closing tags to the end. improper tag order: reinserted tags in the target sentence might occur in incorrect order, i.e. closing tag before corresponding opening tag. in this case, we swap the tags. furthermore, tag pairs might be improperly nested. we reorder and, if required, insert additional tags to restore a proper nesting. e.g., a target sentence with two "overlapping" tag scopes (1) <@xmath2, we describe the markup reinsertion strategies implemented in transins. for some strategies the cleanup step has to be adapted or extended. ## evaluation to the best of our knowledge, there are no public standard evaluation data sets for markup transfer yet. we collect 10 complex web pages from spiegel.de, a well-known german news provider, that contain a total of 378 sentences with inline markup. we convert the web pages to ms office @xcite unaligned isolated tags are still moved in front of the following aligned token. @xcite tag interpolation produces a markup error if a gap target token is correctly aligned with a source token without tags. for both translation directions mtrain produces most errors. using perfect alignments yields only a minor improvement. mtrain++ reduces the number of errors by about two-thirds. surprisingly, the quality for both translation directions is slightly better when using marian alignments instead of perfect alignments. this is due to the fact that a single change in alignment, even if it is a correction, can potentially change the markup of multiple tokens. with cms, a change in alignment only effects a single target token, making this strategy less volatile. for both types of alignments, cms produces the smallest number of errors. the results are similar for both translation directions. for de -> fr, using marian alignments with a maximum gap size of 3, we find 8 errors caused by incorrect alignments and 4 errors caused by incorrect tag interpolation. reducing the maximum gap size to 2 and 1 decreases the number of interpolation errors, as tag interpolation is applied to fewer gaps. with a gap size of 0, i.e. with no tag interpolation, the number of errors caused by incorrect alignments increases by 2. these errors are now no longer corrected by tag interpolation. with perfect alignments, only the errors caused by incorrect tag interpolation remain. the main focus of our evaluation is the comparison of cms with popular translation services. these services are able to handle markup, but the details are unknown to us. ## conclusion in this paper, we present transins, an opensource system implementing several alignment based strategies for markup reinsertion in translated documents. mtrain constitutes a baseline, while mtrain++ can handle more complex configurations. cms correctly handles all problem cases discussed and outperforms the markup transfer in documents translated with popular translation services.