Domain Informed Neural Machine Translation: Developing Translation Services for Healthcare Enterprise (2020)

Abstract: Neural Machine Translation (NMT) is a deep learning based approach that has achieved outstanding results lately in the translation community. The performance of NMT systems, however, is dependent on the availability of large amounts of in-domain parallel corpora. The business enterprises in domains such as legal and healthcare require specialized vocabulary but translation systems trained for a general purpose do not cater to these needs. The data in these domains is either hard to acquire or is very small in comparison to public data sets. This is a detailed report of using an open-source library to implement a machine translation system and successfully customizing it for the needs of a particular client in the healthcare domain. This report details the chronological development of every component of this system, namely, extraction of data from in-domain healthcare documents, a pre-processing pipeline for the data, data alignment and augmentation, training and a fully automated and robust deployment pipeline. This work proposes an efficient way for the continuous deployment of newly trained deep learning models. The deployed translation models are optimized for both inference time and cost.

## introduction the emergence of neural machine translation (nmt) was sparked by the use of recurrent neural networks (rnn) for machine translation. the rnn encoder in this approach is responsible for encoding the source language phrase into a fixedlength vector. this vector is then decoded into the target language @xcite . some approaches also use long short term memory (lstm) @xcite for this task @xcite . sequence to sequence models with attention @xcite started coming to fruition with the advent of the idea that the lstm layers stacked on top of each other stopped improving at about a depth of 4. the attention-based models were essentially of two types, local and global. the attention models proved to be much more efficient in translating long sentences as compared to nonattention models @xcite . but long sentences still exhibited "exposure bias" which led to the emergence of attention models that attend over the input and generated outputs separately @xcite . despite these remarkable advances in nmt, one of the major problems still faced by some enterprises is domain-specific translation where the generalpurpose translators do not perform well. leveraging in-domain corpora to skew a general purpose translator successfully has been an area of interest in nmt in recent years. one of the successful techniques to do this has been mixed fine-tuning @xcite which suggests to train an nmt model on out-of-domain corpora until model convergence and then resume training from step 1 on a mix of in-domain and outof-domain data. this paper tackles the challenge of translating indomain correspondence letters for one such healthcare enterprise. the documents that these enterprises send out to their clients contain confidential information, legal/medical clauses that need to be translated in an enterprise-specific manner. some parts of text like entity names, addresses, text in a language different from the language of the document do not need to be translated. in addition, the human-translated data that we leverage is present in sources such as html pages, or .doc/.docx forms, so intelligent parsing is required to extract parallel text. in this paper, we illustrate the construction and deployment of a translation system from english to spanish. first, we describe the training from the ground up of a general-purpose translator using an open-source library, openseq2seq @xcite . then, we describe an iterative process to customize this model for two use cases: medical handwritten text and formal medical correspondence letters. we use various resources from our clients at our disposal. first, we scanned the human-translated documents that were sent to the end-users to extract domain-specific sentences in the source and reference languages, cleaned the data, and built a parallel corpus. we also utilized another resource obtained from the enterprise, a translation memory, which is an xml to xml mapping that human translators used to refer to for the translation of specific sentences and form sections of a letter. we used these domain-specific parallel corpora to fine-tune the general-purpose model while making sure it did not overfit. this study also proposes continuous deployment architecture for these models that is highly efficient at inference time and seamlessly deploys newly trained models with zero downtime. the rest of the report is organized as follows: section 2 describes all the data sets used in this work, section 3 outlines how the data from different sources was made suitable for training, section 4 describes the architecture and evaluation procedures used. section 5 explains the experiments and customization of the model towards the target domain, section 6 describes an efficient and inference-optimized architecture for our model and we conclude in section 7. ## data sets our data sets included two sources from the public domain, one used for training and one for evaluation for the general domain. for the customization part of the training process, various resources internal to the customer were leveraged. the customer sends medical claim correspondence letters which are manually translated in spanish. we used the translations from these documents as a major part of our in-domain data set. also, human translators use a translation memory to refer to the correct translation of some phrases or sentences. translation memory consists of xml mapping files that contained source xml in english and reference xml in spanish. data fragment source sentences paracrawl foot_0 public 38m wmt-news public 14k correspondence letters (m&r) customer internal 492k translation memory (m&r) customer internal 15k table (1) data sets used in this work and corresponding source and number of sentences in each. ## data preparation while our baseline experiment used the raw version of the public data, we cleaned and aligned all the data sets to ensure the quality of data on which the model is trained. this step was especially critical to the customer documents, as they were word document files (docx) containing tables and forms and not only plain text. this needed special attention as described in the following sections. ## experiments and results the following experiments are done on the transformer model foot_1 with fixed training hyperparameters using openseq2seq @xcite . then we eliminate address lines, augment the data with random punctuation and casing. this reduces the amount of training data but enhances the quality of it. ## nmt inference service deployment we built a scalable, performance-oriented, and cost-optimized deployment pipeline targeting a cloud-native environment. we separated all text processing from neural model inference. text processing and clients serving rest apis were implemented as light-weight microservices that run on cpu. neural models are served by tensorrt inference server containers (nvidia, 2019), which are provisioned with gpu. models are placed on persistent storage accessible to the tensorrt inference servers (figure 3 ). we chose tensorrt inference server because of the following features that it provides: each model can have one or more versions available in the model repository. each version is stored in its own, numerically named subdirectory where the name of the subdirectory corresponds to the version number of the model. the server monitors all changes in the model store and adds or removes models or model versions from serving without any restarts of tensorrt inference server. ## conclusion our experiments demonstrate how a generalpurpose neural machine translation framework can be customized to a specific use case for a specialized domain enterprise. they also show how different versions of the same model architecture can serve different needs. for instance, experiment 2 yields a model that is suitable for a manually typed general language data but not suitable for medical claim correspondence letters for the customer. experiment 3, however, yields a model that performs exceedingly well in the given customer scenario and has a bleu score of 0.8 which would be very hard to manifest for a general-purpose translator. in this work, we also describe an architecture for the deployment of deep learning models (specifically neural machine translation) optimized for inference using tensorrt. we explain how the models can be automatically deployed and changed at run time following the customer's needs. for instance, the translation model of experiment 2 is served for a human typing interface, whereas the fine-tuned model from experiment 3 is provided for medical correspondence letters.