Incorporating External Annotation to improve Named Entity Translation in NMT (2020)

Abstract: The correct translation of named entities (NEs) still poses a challenge for conventional neural machine translation (NMT) systems. This study explores methods incorporating named entity recognition (NER) into NMT with the aim to improve named entity translation. It proposes an annotation method that integrates named entities and inside–outside–beginning (IOB) tagging into the neural network input with the use of source factors. Our experiments on English→German and English→ Chinese show that just by including different NE classes and IOB tagging, we can increase the BLEU score by around 1 point using the standard test set from WMT2019 and achieve up to 12% increase in NE translation rates over a strong baseline.

## introduction the translation of named entities (ne) is challenging because new phrases appear on a daily basis and many named entities are domain specific, not to be found in bilingual dictionaries. improving named entity translation is important to translation systems and cross-language information retrieval applications @xcite . conventional neural machine translation (nmt) systems are expected to translate nes by learning complex linguistic aspects and ambiguous terms from the training corpus only. when faced with named entities, they are found to be occasionally distorting c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. location, organization or person names and even sometimes ignoring low-frequency proper names altogether @xcite . this paper explores methods incorporating named entity recognition (ner) into nmt with the aim to improve ne translation. ner systems are often adopted as an early annotation step in many natural language processing (nlp) pipelines for applications such as question answering and information retrieval. this work explores an annotation method that integrates named entities and inside-outside-beginning (iob) @xcite tagging into the neural network input with the use of source factors. in our experiments, we focus on three ne classes: organization, location and person, and use the state-of-the-art encoder-decoder transformer network. we also investigate how the granularity of ne class labels influences ne translation quality and conclude that specific labels contribute to the ne translation improvement. further, we execute an extensive evaluation of the mt output assessing the influence of our annotation method on ne translation. our experiments on @xmath0, we can increase the bleu score by around 1 point using the standard test set from wmt2019 and achieve up to 12% increase in ne translation rates over a strong baseline. belfast -gi@@ ants won thanks to patri@@ ck d@@ w@@ yer en fine-grained belfast 2 -0 gi@@ 3 ants 3 won 0 thanks 0 to 0 patri@@ 1 ck 1 d@@ 1 w@@ 1 yer 1 en coarse-grained belfast 1 -0 gi@@ 1 ants 1 won 0 thanks 0 to 0 patri@@ 1 ck 1 d@@ 1 w@@ 1 yer 1 en iob tagging belfast b -o gi@@ b ants i won o thanks o to o patri@@ b ck i d@@ i w@@ i yer i en inline ann. ## nmt with ne tagging we explore incorporating ne information as additional parallel streams (source factors) to signal ne occurrence in the fashion described in @xcite . source factors provide additional word-level information, are applied to the source language only, and take form of supplementary embeddings that are either added or concatenated to the word embeddings. this is illustrated with the following formula: further, we also explore whether the ne class granularity may influence translation quality and help decrease word ambiguity. for this purpose, we define a "fine-grained" case, where we use specific ne class labels (e.g. person, location, organization) and also a "coarse-grained" case, where we use two different source factor values only: (0) as default and (1) to denote a named entity in a generic manner. additionally, we investigate whether inside-outside-beginning (iob) tagging @xcite used to signalize where a ne begins and ends as a second input feature may guide models to translate compound named entities better. in iob tagging, (b) indicates the beginning, (i) the inside and (o) the outside of a ne (a regular word or a sequence of words). we annotate source sentences with an external ner system. examples for the different annotation strategies (that we experiment with) are presented in as our goal resembles that of @xcite , we compare our approach against their inline annotation method with xml boundary tags. @xcite use specific ne class labels, which correspond to the "fine-grained" case in our work. we refer to their approach as "inline ann. (finegrained)" and present this annotation method in for @xmath3.0. the latter dataset is shortened to match the size of the training dataset for @xmath4, see also as ne recognition is an active research field and the search for best recognition methods continues, the quality of ner systems may vary under different research scenarios and domains @xcite . incorrect ne annotation in the data may influence the results of this work negatively. therefore, we focus on three well-researched ne classes: person, location and organization, limiting, thus, the possibility of incorrect annotation. we use spacy named entity recognition (ner) system foot_2 to recognize named entities in the source sentences. the ratio of sentences in the training data with at least one named entity occurrence (based on three ne classes) in the source sentence amounts to 50.44% for en-de and 53.95% for en-zh. we tokenize the english and german corpora using the spacy tokenizer foot_3 , and use the open-nmt tokenizer foot_4 (mode aggressive) on the chinese side. further, we perform a joint source and target byte-pair encoding (bpe) @xcite for @xmath5, both with 32,000 merge operations. for every source sentence in the training data (after applying bpe), we generate two files with source factors: i. marking named entities (either the coarse-grained or the fine-grained case), ii. marking iob tagging. the baseline model is trained with no external annotation. ## conclusion our work focused on establishing if annotating named entities with the use of source factors leads to their more accurate translation. we can state that the general translation quality with the annotated models improves (improvements in bleu score). additionally, in-depth automatic and human named entity evaluation prove that the same holds true for ne translation. the accuracy of named entity annotation plays a crucial role during the annotation of named entities in the training data as well as during evaluation (automatic hit/miss analysis). by establishing spacy's f1-score on random300 during the hu-man hit/miss analysis to amount to approx. 85%, we conclude that the accuracy of any ner system greatly influences the practicability of our approach. therefore, the improvement of named entity translation is closely related to the improvement of ner systems.