Analyzing Challenges in Neural Machine Translation for Software Localization (2023)

Abstract: Advancements in Neural Machine Translation (NMT) greatly benefit the software localization industry by decreasing the post-editing time of human annotators. Although the volume of the software being localized is growing significantly, techniques for improving NMT for user interface (UI) texts are lacking. These UI texts have different properties than other collections of texts, presenting unique challenges for NMT. For example, they are often very short, causing them to be ambiguous and needing additional context (button, title text, a table item, etc.) for disambiguation. However, no such UI data sets are readily available with contextual information for NMT models to exploit. This work aims to provide a first step in improving UI translations and highlight its challenges. To achieve this, we provide a novel multilingual UI corpus collection ( \sim 1.3M for English \leftrightarrow German) with a targeted test set and analyze the limitations of state-of-the-art methods on this challenging task. Specifically, we present a targeted test set for disambiguation from English to German to evaluate reliably and emphasize UI translation challenges. Furthermore, we evaluate several state-of-the-art NMT techniques from domain adaptation and document-level NMT on this challenging task. All the scripts to replicate the experiments and data sets are available here. ˆ{,}

## introduction there is a rapid increase in access to technology for people from around the globe. for software to be used by everybody, it is essential to provide user interface (ui) texts in their native languages for monolingual speakers. however, many applications are created in english and later localized to various languages. to decrease the translation time, localization companies take the help of machine translation (mt). human annotators use the mt system for generating initial translations and postedit the system's output to increase efficiency in computer-assisted translation tools @xcite . when producing high-quality translations, costs can be saved by decreasing the annotator's time on editing and making the localization process cheaper. thus, enabling more companies to localize in several languages at low cost. neural machine translation (nmt) @xcite @xcite is the current state-of-the-art (sota) approach for generating the initial "draft" for the translations. although conventional nmt models are sufficient in many cases, they use only the source sentence alone to predict the target translation (locality assumption) @xcite . however, this is problematic when translating several types of concise ui texts that need more information. it has been a decade since muntés @xcite have shown the need for integrating additional context into mt models for software localization and have pointed out several issues. for example, consider translating the english source word "login" into german. here, the translation depends on where the text is. if it is on a button, the correct translation is the verb form "anmelden". in the case of a title text (faq, documentation, etc.), the translation might rather be the noun form "anmeldung". depending on the application, the translation can also be simply "login" even on a button. another common phenomenon to deal with when translating these short ui texts is semantic ambiguity. for example, if we want to translate the word "home", we need to know the context where the sentence is present. although the translation is often the german word "haus", it is not correct in the context of linux applications. in this case, we would need to translate "home" as "benutzerverzeichnis" ($home directory). apart from the issues mentioned above, gender (the gender of the item in a table depends on the gender of the table title text), and consistency in terminology and style (e.g formal v/s informal) are problematic when translating ui. however, sentence-level nmt models lack the contextual information to generate suitable target translations. several approaches were proposed to integrate contextual information into nmt models. domain adaptation and multi-domain models using tags are common techniques in nmt to inform the models about the type of content it is translating @xcite @xcite . knowing the domain/topic of the text enables nmt to select and generate appropriate translations in a particular setting. another recent and growing field that tries to make models context-aware is document-level nmt (doc-nmt) @xcite @xcite @xcite . the surrounding source and/or target sentences usually can provide significant cues and hints for the nmt model to understand the domain and the context in which the source sentence is occurring. these approaches are promising and could benefit localizing ui segments, but the lack of sufficient annotated ui data with contextual information is the main hindrance to applying them. although localization files of software like gnome, ubuntu, and kde are present in the opus corpus @xcite , they do not provide the document structure or any other metadata. also, the segments in the corpora do not come from a vast amount of domains. neither large amounts of ui data are publicly available nor a targeted test set to measure the model's ability to use context for ui translations. these constraints limit the potential to improve nmt for software localization and are necessary as a first step. this work addresses the limitations mentioned above by creating a corpus for ui data with document-level information extracted from multiple domains, a targeted test suite, and baselines using current sota methods. our main contributions in this paper are the following • we present a task of translating ui texts and show their unique properties requiring additional context. furthermore, we provide a novel multilingual ui corpus covering multiple domains with contextual information to ## addressing data scarcity for ui for improving nmt for ui segments, the first challenge to address is the lack of parallel data with contextual information. to achieve this, we present a novel ui corpus that we assemble from software localization files of publicly available repositories. first, we briefly explain the structure and contents of the localization files which we search and collect to create our data set. then, we highlight what types of contextual information are available in these files that are useful for nmt. finally, we describe how we scraped large amounts of these po files for multiple languages. ## realistic & targeted evaluation the previous section described how we collected the data and its characteristics. the next step is to design a realistic evaluation framework to showcase challenges in ui translation and estimate the nmt model's quality reliably. in this section, first, we propose two test sets considering different use cases while evaluating. then, to highlight one fundamental phenomenon when dealing with ui strings, we create a challenge set for disambiguation based on heuristics and human annotation. we explain how we curated the different test sets below. ## baselines to evaluate the current nmt approaches with contextual information on ui translations, we provide baselines (@xmath0. we build context-aware nmt models on our ui corpus and measure their ability to use context and highlight their shortcomings. first, we describe how we integrate multiple types of contextual information in nmt. then, we present the results and analyze the performance of the models on the multiple test sets. ## conclusion we presented a multilingual ui corpus with additional meta-information for researchers in the community to exploit and build context-aware nmt models for software localization. we also have proposed two evaluation setups to replicate conditions occurring in localization companies and show the difficulty in tackling new applications. furthermore, we experiment using domain adaptation and doc-nmt techniques to provide baselines and present the benefit of using different types of context in intra-applications while showing its ineffectiveness in cross-application scenarios. moreover, we suggest an automatic procedure to create a targeted disambiguation test set where context is necessary to generate the correct target word translation. finally, we show that the baseline systems fail in such challenging settings and call for sophisticated context-aware nmt models to improve the process of software localization.