Automatic Bilingual Markup Transfer (2021)

Abstract: We describe the task of bilingual markup transfer, which involves placing markup tags from a source sentence into a fixed target translation. This task arises in practice when a human translator generates the target translation without markup, and then the system infers the placement of markup tags. This task contrasts from previous work in which markup transfer is performed jointly with machine translation. We propose two novel metrics and evaluate several approaches based on unsupervised word alignments as well as a supervised neural sequence-to-sequence model. Our best approach achieves an average accuracy of 94.7% across six language pairs, indicating its potential usefulness for real-world localization tasks.

## introduction machine translation (mt) has two primary use cases: fully automatic mt and assistance for human translators. fully automatic translation is used widely by consumers, while professional human translation with machine assistance remains the preferred method for translations that require a guarantee of publication quality. in both use cases, markup of particular spans of the source text that encodes formatting, hyperlinks, and other extralinguistic information must be transferred to corresponding spans of the target translation. prior work on neural machine translation has focused on the problem of simultaneous translation and markup for the fully automatic use case @xcite . this work describes approaches to the complementary problem for the assistance use case. a common and effective workflow for professional translators is to first produce the text of a translation, then transfer markup into this text. this paper describes approaches to automating the second step in this workflow by automatically transferring source markup into a fixed reference translation. this fixed reference may not be preferred by a machine translation model, for example because it was written by a human, and therefore the correspondence between source and target may be challenging to infer. in this way, markup transfer is similar to word alignment, which is typically applied to authentic human translations rather than machine translations. indeed, @xcite describe an algorithm for using word alignments to perform markup transfer. this work contains three novel contributions: in experiments across six language pairs, we find that neural word alignments increase markup transfer accuracy over fastalign by 5.2% using prior markup transfer methods, our improved transfer algorithm increases accuracy by an additional 7.3%, and our supervised approach further increases accuracy by 9.9%. our best approach has an average accuracy of 94.7%, compared to a baseline of 72.3% from applying the markup transfer algorithm of @xcite to word alignments from fastalign @xcite . this improved performance indicates potential usefulness in a professional localization setting. nlp practitioners may also benefit from this reliable method of transferring span annotations to new languages. ## related work recent work in neural word alignment has indicated that markup transfer is a downstream task, though evaluations of word aligners have not included an explicit evaluation of markup transfer @xcite @xcite . experiments in this paper are the first to quantify the amount by which the improved alignment quality of a neural aligner compared to fastalign also improves markup transfer accuracy. markup can be represented using xml tags @xcite . previous work describes two approaches to markup transfer for fully automated machine translation, where the goal is to place each xml tag from the source into the target translation in a way that produces well-formed xml. the first approach is to include markup while training the translation model, such that the translation model takes as input a source sentence with xml markup and directly generates a translation that includes xml tags. a translation training set that includes markup can either be created by human translators @xcite or synthesized by adding markup to an existing unformatted bitext @xcite . a translation model that generates both text and markup may prefer an output sequence for which the xml markup is invalid (e.g. there might be an opening tag that is not closed). this problem can be addressed through xml-constrained beam search @xcite . this approach requires training data that contains xml markup. the second approach is to train the translation model without markup, separately train a word aligner, and then transfer format using an inference pipeline. after the translation model has generated a text translation, the alignment model aligns the tokens of the source segment to the generated translation. finally, a deterministic algorithm (labeled min-max in section 4.2) transfers the markup from the source segment into the translation via the word alignments @xcite . this approach does not require training data that contains xml markup. past work has not measured markup transfer accuracy directly, because when a system generates both a translation and its markup, the translation differs from the reference by more than just markup. instead, automatic metrics such as xml accuracy check that all source tags appear in the target and are properly nested. xml-based bleu splits the translation at every formatting tag both for the reference and the translation and calculates the bleu score @xcite on the resulting subsegments @xcite . past work has also included manual evaluation of the transferred markup information @xcite , since transfer accuracy could not be assessed directly. in contrast, our goal is to transfer markup directly into the reference translation. evaluation of markup accuracy is therefore straightforward: a tag is placed correctly if it appears at the correct character position within the reference translation. ## bilingual markup transfer in this section we introduce tag pairs, the data structure with which we represent markup information, and define two evaluation metrics. ## unsupervised markup transfer for unsupervised markup transfer, we apply a twostep process. first we use an unsupervised aligner to infer the alignments between source and target subwords. the second step uses a deterministic algorithm to place tag pairs based on these alignments. two advantages of this unsupervised approach are that it does not require training data with markup, and it can leverage any word aligner. ## supervised markup transfer when a bitext annotated with markup is available, it is possible to train a supervised markup transfer system. we implement a sequence-to-sequence model using the transformer @xcite architecture that learns to generate the target sequence with tags given input of the source with tags and the target without tags. to perform well in this task, the model must learn to copy the target text, infer the correspondence between source and target tokens, and place the tags present in the source text at corresponding positions in the target. to encourage the model to learn the correspondence between source and target subwords, we pretrain it for machine translation, translating a source segment without tags into a target without tags. afterwards, we train the model to project the markup tags into a given target sentence. the input of the model during this stage of training (and during inference) is the source segment with tags, a separator token, and then the target segment without tags. figure 4 provides an input-output example. after training we can project markup tags into the target sentence by searching for the most likely output sequence under the model, which will be a target sentence containing markup. we first consider greedy search. while a well-formed output results most of the time, the model does not always generate the same target sentence that appeared in the input. it also does not always reproduce all tags that appeared in the source segment. to circumvent these issues, we can constrain the search towards a consistent output. during output sequence generation, we keep track of the text of the produced hypothesis, and constrain the next target token to be either a prefix of the remaining target text or a markup tag. when producing a markup tag, we make sure that only markup tags that appeared in the source segment can be opened, and we track their counts. to enforce a valid tag structure, we ensure that only the most recent opening tag without a corresponding closing tag can be closed. we additionally ensure that all tags appearing in the source are produced in the target exactly once. these constraints can be implemented efficiently using a bias vector that prevents invalid tokens by setting their bias to a large negative value. during every decoding step this bias vector is added to the logits before retrieving the most likely token. during development of this model we noticed that the output of the unconstrained search provides a signal about its quality. if unconstrained greedy search does not copy the target text or does not reproduce all tags in a well-formed structure, typically the constrained search produces output with incorrect markup tag positions. therefore, we evaluate an additional method which uses the output of unconstrained greedy search from the sequenceto-sequence model, but with a fallback to unsupervised markup transfer if either the text or tags of the output are inconsistent with the input-the two failure modes described above. ## evaluation ## conclusion we introduced the task of bilingual markup transfer into a fixed reference translation. using two novel metrics, tag accuracy and average character distance, we evaluated both unsupervised and supervised approaches to this task. both may be useful, depending on the availability of training examples with markup. our supervised approach provides higher tag accuracy, but at the expense of higher average character distance. combining supervised and unsupervised approaches corrects for this problematic behavior and provides a reliable and accurate method for markup transfer.