Tag Assisted Neural Machine Translation of Film Subtitles (2021)

Abstract: We implemented a neural machine translation system that uses automatic sequence tagging to improve the quality of translation. Instead of operating on unannotated sentence pairs, our system uses pre-trained tagging systems to add linguistic features to source and target sentences. Our proposed neural architecture learns a combined embedding of tokens and tags in the encoder, and simultaneous token and tag prediction in the decoder. Compared to a baseline with unannotated training, this architecture increased the BLEU score of German to English film subtitle translation outputs by 1.61 points using named entity tags; however, the BLEU score decreased by 0.38 points using part-of-speech tags. This demonstrates that certain token-level tag outputs from off-the-shelf tagging systems can improve the output of neural translation systems using our combined embedding and simultaneous decoding extensions.

## introduction neural machine translation (nmt) uses neural networks to translate unannotated text between a source and target language, but without additional linguistic information certain ambiguous inputs may be translated incorrectly. consider the following examples: the titanic is fighting between good and evil 2) titanic struggles to stay afloat. the titanic is struggling not to sink 침몰하지 않기 위한 엄청난 투쟁. ## related work very early work addressed named entity translation by treating automatically identified named entities with a special translation system, usually a transliterator @xcite . this work did not attempt to integrate the translation models for one to benefit from information learned by the other. later, especially with neural machine translation (nmt) systems, source-side feature augmentation research studied the inclusion of linguistic feature information into the source-side token embeddings, usually by adding in or concatenating additional learned feature vectors to the token embedding vectors, as we do in this work @xcite @xcite @xcite . this approach can also be adopted on the target-side, as presented here or in @xcite @xcite . however, these methods only add linguistic feature information to the input, without encouraging the system to model that information in any particular way. factored translation systems, under both statistical and neural machine translation, instead explore the addition of externally supplied linguistic features to the raw text at both input and output. these features include part-of-speech (pos) tags, word lemmatizations, morphological analysis, and semantic analysis @xcite @xcite . factored translation models map feature-augmented input into feature-augmented output, however outputs include only an underlying lemma together with the predicted features. these systems also use a rule-based morphology toolkit in post-processing to generate the output surface forms from predicted output features, requiring knowledge of appropriate rule systems for the output language. an additional tagged architecture @xcite predicted syntax-tagged surface forms, but did so by appending the tags to the surface form tokens directly, rather than predicting separate factors. in general, the focus of factored models has been to increase vocabulary coverage, for example of highly agglutitanative languages with rich morphologies, rather than our goal of disambiguating polysemous of polysyntactic words or otherwise handling named entities in a more nuanced way. finally, one previous work does consider a fully tagged (both source and target) factored neural model predicting tags with surface forms with independent layers in much the same way as presented here @xcite . this work showed negative results for various syntactic tag types on iwslt'14 shared task data @xcite , whereas this work presents ner and pos tags on film subtitles data. ## tagged seq2seq we implemented two extensions to the standard seq2seq encoder-decoder architecture for neural machine translation to use token-level tags to improve translation results. 1 by combining token and tag embeddings in the input and simultaneously predicting tokens and tags in the output, the nmt system learned to translate tagged source sentences to tagged target sentences (figure 1 ). we used a transformer encoder and decoder for the base seq2seq model @xcite . tags are added to the data as a preprocessing step. ## experiments we used a transformer encoder and decoder @xcite for the base seq2seq system, each with 6 layers and 8 attention heads, and layer and embedding dimensions 512. training was done for 40 epochs at half precision with the optimizer known as adam (kingma and ba, 2015) with β = (0.9, 0.98) and an inverse square root learning schedule with maximum learning rate 5 × 10 -4 after 500 updates and decay 1 × 10 -4 . parameter updates occurred after every 8,192 token-tag pairs at most (rounding off to complete sentences), with 30% dropout and label smoothing of 0.1 on the training loss. at inference time, a beam of 5 candidates was maintained, and the models were evaluated with their bleu score on the token sequence only (tagging accuracy was not evaluated due to the difficulty of establishing alignment). ## results bleu scores from untagged and tagged translation experiments show an improvement from the use of ner tags ( we also evaluated a model trained with pos tags, but found a decrease in bleu score ( ## model limitations it should not go unnoticed that the typical inference algorithms for sequence labeling, particularly the bilstm-crf inference employed by most ner systems, are incompatible with the autoregressive sequence decoding algorithms (greedy decoding and beam search) used for inference by seq2seq models. that the beam decoding algorithm (and autoregressive likelihood model) used here for tags was unable to account for (be conditioned on) the as-yet uncomputed right context was cause for much apprehension before experimental results became available. these positive results notwithstanding, future work could explore how to better incorporate the full tagging context in tag de-coding, perhaps, for example, by predicting the sequence more wholistically with non-autoregressive decoding @xcite . we also imagine that the design of the underlying seq2seq architecture may lend itself to certain types of sequence labeling. for example, the bidirectional context modeled by a bilstm-based translation model may be more suitable for certain types of sequence labeling tasks than the transformer's attentional activations. because our contributions are agnostic to the type of sequence labeling (ner or part-of-speech tagging or any other kind) as well as to the design of the encoder and decoder, future experiments should also explore these possibilities. ## conclusion we implemented extensions to existing neural machine translation models that allow the use of offthe-shelf token-level tagging systems to improve translation accuracy. translation inputs and training outputs were tagged with pre-trained sequence labeling systems. a standard encoder-decoder architecture was extended to include tag embeddings and tag prediction at each token position. at model input, token and tag embedding vectors were added to produce a combined embedding. at model output, the final decoder layer used separate softmax layers to predict tokens and tags. during training, a combined loss function encouraged the model to learn token and tag information jointly. this tag assisted translation system was tested against baseline token-only systems on a german to english film subtitle corpus with both word and subword tokenization. subword tokenization reduced the size of the effect, suggesting the need for specialized subword tokenization to prioritize the integrity of important word categories. however, on word tokenized data, the 1.61 point increase in bleu score using named entity tags demonstrates that the proposed architecture is useful for improving translation outputs with automatic named entity recognition, while the 0.38 point decrease using part-of-speech tags indicates more difficulty in utilizing that tag information. further examination of the cross-entropy showed that adding tags reduced the token cross-entropy thereby improving token modeling. future experiments can explore the use of other types of tag data as well as other decoding paradigms.