Fast-Paced Improvements to Named Entity Handling for Neural Machine Translation (2022)

Abstract: In this work, we propose a Named Entity handling approach to improve translation quality within an existing Natural Language Processing (NLP) pipeline without modifying the Neural Machine Translation (NMT) component. Our approach seeks to enable fast delivery of such improvements and alleviate user experience problems related to NE distortion. We implement separate NE recognition and translation steps. Then, a combination of standard entity masking technique and a novel semantic equivalent placeholder guarantees that both NE translation is respected and the best overall quality is obtained from NMT. The experiments show that translation quality improves in 38.6% of the test cases when compared to a version of the NLP pipeline with less-developed NE handling capability.

## introduction ne play a crucial role in many downstream nlp tasks. there is extensive research showing that properly handling ne improves the performance of systems performing question answering (talmor and berant, 2019), summarization @xcite , and information retrieval @xcite . in this paper, we focus on nmt, another task that benefits from ne modeling @xcite . nmt models are prone to disturb ne, leading to critical quality issues in the translation. overcoming such problems is challenging since it is hard to have good coverage of all possible entities in the training data. this is due to the open-ended nature of ne as well as their domain specificity. for example, for the organization (org) category, new entities appear daily in a variety of domains. moreover, ne are linguistically complex structures that can occur in ambiguous contexts. this impairs the ability of models to generalize and instead learn unwanted biases @xcite . this causes ne to be hallucinated towards frequent realizations, omitted, or incorrectly translated. figure 1 shows some examples of this issue in the output translation of an @xmath0. this occurs despite the model having 65 × 10 6 parameters and being trained with 100 million sentence pairs. the nmt community has long been familiar with the ne handling problem @xcite . this has spurred research on how to address such model limitations. invariably, all works resort to either incorporating new modeling features in existing nmt architectures @xcite or integrating with external knowledge sources to bridge the ne gap @xcite . in spite of the achievements of the previously mentioned works, they have the drawback of requiring a model-specific solution. in a commercial setting, this is problematic since ne handling, at least for some categories, might come only as an afterthought. having the nlp pipeline already in place entails that rolling out changes can be slow due to the high number of existing models. it should be noted that there are also time and budget constraints regarding the model size and volume of training data in order to make a nmt system economically viable. this blocks translation quality improvements related to ne handling. ## related work the standard approach to ne handling within a nlp pipeline corresponds to introducing ne information and forwarding it to the nmt component. the end goal is to allow the model to improve the ne translation quality. in previous work, there are different approaches to make use of this ne information, which we summarize below. a possible approach is the placeholder method @xcite , where source sentences are masked by a generic entity token, exposed to the nmt model during training. after translation, the masks are placed back into the target sentence, based on an index or alignment. @xcite extend this approach to overcome the limitation of dealing with rare words in this setting. this is done with a dedicated character-level sequence-to-sequence model for ne translation. a ne recognition step is also added to enable the use of categoryspecific entity tokens. nes are crawled from the training data and their translation extracted from wikipedia. the ne translation pairs are then used to train both the character-level and nmt models. another line of research uses entity embeddings to convey word-level ne category information to guide the nmt model. an example is source factors @xcite , which take the form of supplementary embeddings that are added or concatenated to existing word embeddings in the model. @xcite combines this with an additional long short-term memory (lstm) @xcite layer to better handle ne. this contrasts with the work from @xcite , where better translation quality is achieved by directly combining source factors in a transformer network @xcite . semkge (moussallem et al., 2019) take a similar approach but construct the embeddings differently. these map subject-relation-object triples from a knowledge graph (kg) @xcite into a continuous vector space to obtain knowledge graph embeddings (kge) @xcite . to this end, a supervised fasttext @xcite classifier determines a set of referring expressions of ne from the kg and uses them to initialize the embedding weights of the nmt model. @xcite use a similar methodology but focus on dealing with the drawback of only taking into account ne that appear in both the kg and the training dataset. to leverage the remaining relevant information in the kg, phrase translation pairs are first extracted from the training data. the pairs that appear in the kgs are considered seed pairs in a kges semantic space. this semantic space is then used to compare new ne with the seed pairs. if kge are close, then a synthetic sentence pair is generated by replacing the original ne with the new ones. continuing the entity embedding research line, @xcite take it a step further and provide a generic recipe to achieve a single end-to-end ne-aware nmt model, which avoids the overhead of separate ne handling steps. moreover, there is no extra cost at inference time since the ne components can be disabled. to achieve this, an enhanced encoder and decoder are trained in a multitask framework by combining translation and ne recognition in a focal loss @xcite . given the current state-of-the-art, we conclude that previous approaches introduce coupling to the nmt architecture by either changing it or jointly training new embeddings. while this brings advantages in many scenarios, we argue that it is also valuable to address the use case where a large nlp pipeline already exists and fast incremental improvements to ne need to be delivered by means of new categories. in this context, we build upon the placeholder approach, where we are willing to sacrifice translation quality for a translation guarantee that certain words are perfectly translated. we extend this approach to better reconcile these two competing aspects as well as study the more complex case where the ne require translation. ## named entity handling in our approach, we first start by performing a ne recognition pre-processing step (section 3.1). then, we obtain the corresponding translation for a given target language (section 3.2). we forward all the previous information to a ne handler step that obtains the best possible quality from the existing nmt model while guaranteeing that the expected translation appears in the output translation. ## experiments we carry experiments in all ne handling steps, namely: recognition (section 4.1), ne translation (section 4.2), and nmt integration (section 4.3). ## conclusions and future work in this work, we presented a ne handling process, with the ultimate goal of bootstrapping an existing nlp pipeline to improve translation quality. this problem was tackled from a perspective of allowing such improvements to be delivered without having to change one of the main components of the pipeline, the nmt. by having this decoupling, the improvements can be delivered fast, enhancing the user experience in situations where ne translation errors can lead to catastrophic communication errors. our process is based on dedicated recognition and ne translation steps. integration into the existing nmt is done through semantic equivalent masking and an entity fallback mechanism. to evaluate ne recognition, we compared our domain custom model against two out-of-thebox models. the results show that the trade-off between recognition performance and data acquisition costs justifies a custom model for our use case. to evaluate our overall approach, we compared the translation quality of ne of the existing pipeline with the improved version. it was possible to observe that we achieved translation quality improvements while affording translation guarantee at the same time, validating our approach. we also want to highlight that our approach allows us to easily anonymize personally identifiable information (pii) data by exposing the ne mask rather than its original text. this is a concern for us since our nlp pipeline supports a feed-back loop between nmt and human post-edition. the semantic equivalent mask is advantageous in this scenario since it allows editors to review more natural-looking sentences and without the cognitive overhead of processing a generic placeholder. regarding future work, one of the concerns is how to extend the generation of semantic equivalent ne to categories other than person. the main obstacle is identifying the necessary linguistic properties for the generation in all necessary target languages. another concern is the scalability of the ne recognition component. thus far, our solution has been efficient since we have an overarching domain that ties in otherwise different topics. when moving to a completely different domain, we want to investigate how to keep this efficiency in collecting new data while leveraging the existing model.