A Case Study on the Importance of Named Entities in a Machine Translation Pipeline for Customer Support Content (2022)

Abstract: This paper describes the research developed at Unbabel, a Portuguese Machine-translation start-up, that combines MT with human post-edition and focuses strictly on customer service content. We aim to contribute to furthering MT quality and good-practices by exposing the importance of having a continuously-in-development robust Named Entity Recognition system compliant with General Data Protection Regulation (GDPR). Moreover, we have tested semiautomatic strategies that support and enhance the creation of Named Entities gold standards to allow a more seamless implementation of Multilingual Named Entities Recognition Systems. The project described in this paper is the result of a shared work between Unbabel ́s linguists and Unbabel ́s AI engineering team, matured over a year. The project should, also, be taken as a statement of multidisciplinary, proving and validating the much-needed articulation between the different scientific fields that compose and characterize the area of Natural Language Processing (NLP).

## introduction customer support professionals deal with multiple issues and problems arising from humaninteraction, from answering questions or responding to customer complaints, to processing orders and returns, as well as sharing information and services. they are, in a sense, a direct line between customers and service providers, so they must be efficient, fast, and overall understandable, all while working remotely. unbabel enhances customer support abilities through the combination of a machine translation (mt) layer, coupled with human post-edition, allowing to combine the speed and scale of mt with the quality of human editing. to that end, we focus on named entity recognition processes that compose a vital part of the automatic translation pipeline, since they promote an increase in translation quality, and ensure 2018 data protection regulation compliance. to promote high mt performances, a named entity recognition system (ner) was applied, enabling the identification of nes in context, e.g., prediction of nes according to its surroundings, while simultaneously categorizing the ne. the identified nes are then automatically blocked for translation or automatically annotated as ne of interest for further processes such as localization. this step ensures a decrease in mt "hallucinations" (inadequate translations) @xcite , since nes are often responsible for these severe mt mistranslations, which negatively impacts the overall translation quality, considered mostly as critical errors in terms of severity. there is a second step associated with the ne pipeline, the anonymization process. the anonymization guarantees that all the nes corresponding to personal identifiable information (pii) are either replaced by an adequate placeholder, for example email; phone number; reference number; or replaced by a semantic equivalent @xcite in case the ne is a person's name. in the latter, the real name is replaced with a fictitious name that agrees in gender with the original one. this step has a fourfold goal: i) ensures customer sensitive data protection and prevents mt learning with pii information; ii) prevents mt mistranslation; iii) ensures gender agreement (specifically in the case of the replacement of names for semantic equivalents), and iv) guarantees document readability, which is particularly relevant for posteditors. in short, the application of ner is fundamental for enhancing translation quality and preventing personal data breaches, which can lead to fines for non-compliance cases. despite the aforementioned importance that nes represent within a mt pipeline, their definition seems to be somehow elusive. the fact that there is not a unique definition of what constitutes a ne in the literature can be directly associated with the fact that they are structures with the needed plasticity and adaptability to be applied to different tasks. at the muc, @xcite , named entities were defined as "unique identifiers"; in 2003 connl shared task: language-independent named entity recognition, they were described as "phrases that contain the names of persons, organizations and locations." @xcite , and for @xcite they are "textual units corresponding to predefined semantic categories". despite the different definitions, they all seem to agree that a named entity functions as a referent @xcite 1); a linguistic object carrying relevant information in a document, needed, according to @xcite , to allow the computer system to "understand" documents. considering the importance of such structures within a document, we investigate an alternative approach to semi automatically generate training data (still requires manually annotation source language) for named entity recognition (ner) models from parallel corpus @xcite . this is important for the use case where we want to expand ner language coverage within a particular domain. the goal is to only require ne annotated data on the source side and automatically determine the correspondence in the translation. this avoids the time-consuming and high-priced human annotations necessary to train ner for a new language. to achieve our goals, we benchmark different alignment models, and use their output to project nes annotations from source to target text. we will show their impact in the english-german and english-brazilian portuguese language pairs as well as in the domains of tourism and technology. ## related work in the last few years, machine learning systems have been predominantly used to achieve state-ofthe-art ner results and much has been developed since the early message understanding conferences (muc) initiatives. a continuous flow of proceeding works in the field, both in the industry and in a more academic environment, has yielded significant changes that go from new, high performance computational technologies related to the ner subtask itself, to new different applications and goals. these frameworks have been developed to accommodate particular objectives for particular domains, such as in the case of the healthcare industry @xcite , where ner models were used, for example, to extract structure information from unstructured electronic health records (ehr). despite all technological advances, commonly used frameworks still heavily rely on human intervention to provide modeling features or heuristics to solve downstream nlp tasks. while solutions have been proposed to overcome the need for these handcrafted features @xcite 1), the need of labeled data is still an obstacle. in cross-lingual applications, this problem is further aggravated with the cardinality of the number of necessary language pairs. when expanding ner language coverage, this problem can be tackled using named entities word alignment within parallel corpora. this information allows the transfer of ne annotations from a source sentence and its translation @xcite . recent work has shown impressive results with the application of new deep learning models, e.g., transformers, based on an encoder/decoder architecture, mapping sentences to vectors, which result in a representation of the input sequence of words in the source language @xcite . this has boosted the quality of ner and word alignment models. @xcite propose contextual string embeddings for the ner. the embeddings are pre-trained on large unlabeled corpora without any explicit notion of words and thus, fundamentally, model words as sequences of characters, contextualized by their surrounding text. therefore, the same word will have different embeddings depending on its contextual use. this allows the embeddings to properly represent polysemic words, language specific prefixes and suffixes, and handle misspelled words. the approach achieved state-of-the-art results in the conll 2003 ner shared task. wang et al. (2019) propose the use of the m-bert, multilingual bidirectional encoder representations from transformers, for crosslingual transfer without the need of a dedicated cross-lingual training objective and with no aligned data. experiments were carried out in three different languages (spanish, hindi, and russian) and showed that m-bert generalizes well across languages for a variety of downstream tasks @xcite modrzejewski et al. (2020) explores an approach to improve translation quality by conveying ne information through source factors in a machine translation model. the method showed an increase of 1% in the bleu score, when using the wmt2019 standard test, and an increase of 12% when compared with a strong baseline for ne translation. as stated above, several ner models have been proposed, some with the main goal of allowing off-the-shelf usage, such as stanza, google cloud natural language, and spacy. in all systems, a wide variety of nes are taken into account, that range from address; date-time; e-mail, payment/credit-cards in case of google, or location; facilities; law; language, inter alia, in case of spacy. nevertheless, performing ner in a specific domain remains a challenge. in our case, we target the customer-support domain, where the previous tools underperform or lack necessary ne types. we resort to training custom models with in domain data. scaling this approach to many different languages is expensive due to the cost of obtaining labeled data. by using a word alignment-based approach @xcite to project existing ne annotations to a new language in parallel corpora we can address this issue. ## dataset annotation to validate the word alignment-based ne projection, we manually annotated two datasets: tourism-dataset, and technology-dataset. for the tourism-dataset, we used parallel data (bitext) in en (source) and in de. the datasets, comprising 2500 sentences each, were annotated by two linguists, one responsible for the en data set annotation, whilst a second one was responsible for the de version. for de two different translations were annotated, one from machine translated only (mt), and the other with an extra post-edition layer (pe). the technology-dataset consists of 360 post-edited sentences for the en-pt/br language pair and was fully annotated by one of the previous linguists. all datasets went to a preprocessing stage, where the data sets were divided into sentences, allowing the annotation to be made sentence by sentence using prodigy foot_0 , an annotation platform. both annotators used unbabel´s internal ne annotation guidelines. the annotators also had access to online information, namely dictionaries, maps, and other relevant sources of information that could facilitate the task. ## named entity projection to understand the impact of using an alignment approach in building a multilingual ner system, we tested four state-of-the-art aligners: fastalign 3 , the current aligner used by unbabel; eflomal 4 ; simalign foot_1 , and awesomealign foot_2 . each aligner had available different sets of configurations that, when combined, amounted to a total of 53 different alignment possibilities for 3 https://github.com/dgel/fastalign 4 github.com/robertostling/eflomal each ne category. the different configuration for aligners ranged from: using the output word alignments, ne identified in the source sentence were projected in the target based on a min-max algorithm. this means that we consider the target entity span to range the lowest to highest word alignments. model ranking for ne projection task results were presented for assessment using an online software, developed by unbabel ś ai team, that showed all alignment results for the four aligners used, together with their configurations. the alignment results were displayed from best (number 0) to worst alignment result (number 53). moreover, the developed interface also allowed us to compare two models (figure 1 ), giving a panorama over the alignment quality for each category (figure 2 ). with access to the information displayed by the above-mentioned interface, we were able to understand the differences in alignments that generated nes spans between the en source dataset and its de counterpart. moreover, we were also able to compare the de dataset with and without an extra post-edition layer, as to determine if such a task does interfere positively or negatively in the ne projection results. also, we were able to evaluate the aligner settings that showed better performance within the 53 possible combinations and benchmark the current aligner used by unbabel. the ne projection task was evaluated using a classification setting with the following standard performance metrics: precision, recall and f1 @xcite , in order to have a more fine-grained performance perspective of the applied model results: the precision value is defined as the number of positive ne predictions (true positives) divided by the sum of true positives and false positives. this formula is used to understand the classifier exactness. the question that the concept of precision answers is, of all the nes retrieved by the ne projection algorithm, how many were actually correct. lower values of precision indicate a higher number of false positives. the recall value is defined as the ratio of correctly predicted true positive nes, divided by the sum of true positives and false negatives. the question recall answers is, of all the nes in the test dataset, how many were retrieved correctly by the ne projection algorithm. the f-value, also known as f1, is defined as the harmonic mean of the precision and the recall, being appropriate to identify the desired average rate. ## experimental results our study yields very promising results, showing the devised approach to be trustworthy for building multilingual gold standards for ner training when the correct alignment system coupled with specific correct configurations is implemented.