On R omanization for Model Transfer Between Scripts in Neural Machine Translation (2020)

Abstract: Transfer learning is a popular strategy to improve the quality of low-resource machine translation. For an optimal transfer of the embedding layer, the child and parent model should share a substantial part of the vocabulary. This is not the case when transferring to languages with a different script. We explore the benefit of romanization in this scenario. Our results show that romanization entails information loss and is thus not always superior to simpler vocabulary transfer methods, but can improve the transfer between related languages with different scripts. We compare two romanization tools and find that they exhibit different degrees of information loss, which affects translation quality. Finally, we extend romanization to the target side, showing that this can be a successful strategy when coupled with a simple deromanization model.

## introduction neural machine translation (nmt) has opened up new opportunities in transfer learning from highresource to low-resource language pairs @xcite @xcite . while transfer learning has shown great promise, the transfer between languages with different scripts brings additional challenges. for a successful transfer of the embedding layer, both the parent and the child model should use the same or a partially overlapping vocabulary @xcite . it is common to merge the two vocabularies by aligning identical subwords and randomly assigning the remaining subwords from the child vocabulary to positions in the parent vocabulary @xcite @xcite . this works well for transfer between languages that use the same script, but if the child language is written in an unseen script, most vocabulary positions are replaced by random subwords. this significantly reduces the transfer from the embedding layer. @xcite argue that romanization can improve transfer to languages with unseen scripts. however, romanization can also introduce information loss that might hurt translation quality. in our work, we study the usefulness of romanization for transfer from many-to-many multilingual mt models to low-resource languages with different scripts. our contributions are the following: ## romanization romanization describes the process of mapping characters in various scripts to latin script. this mapping is not always reversible. the goal is to approximate the pronunciation of the text in the original script. however, depending on the romanization tool, more or less information encoded in the original script is lost. we compare two tools for mapping our translation input to latin script: uroman foot_0 @xcite is a tool for universal romanization that can romanize almost all character sets. it is unidirectional; mappings from latin script back to other scripts are not available. uconv foot_1 is a command-line tool similar to iconv that can be used for transliteration. it preserves more information from the original script, which is expressed with diacritics. uconv is bi-directional for a limited number of script pairs. below is an example of the same chinese sentence romanized with uroman and uconv: 她到塔皓湖去了 uroman: ta dao ta hao hu qu le uconv: tā dào tǎ hào hú qù le "she went to lake tahoe." the two tools exhibit different degrees of information loss. uroman ignores tonal information and consequently collapses the representations of 塔 (pinyin tǎ; 'tower') and 她 (pinyin tā; 'she'). romanization with uconv retains this distinction but it still adds ambiguity and loses the distinction between 她 (pinyin tā; 'she') and 他 (pinyin tā; 'he'), among others. while uconv exhibits less information loss, its use of diacritics limits subword sharing between languages. we measure character-level overlap between english and romanized arabic, russian and chinese with chrf scores @xcite ) and find they are much higher for uroman (9.6, 18.8 and 13.3) compared to uconv (6.8, 18.1 and 7.2 respectively). ## deromanization romanization is not necessarily reversible with simple rules due to information loss. therefore, previous work on romanized machine translation has focused on source-side romanization only @xcite @xcite @xcite . we argue that romanization can also be applied on the target side, followed by an additional deromanization step. this step can be performed by a character-based transformer @xcite that takes data romanized with uroman or uconv as input and is trained to map it back to the original script. we provide more details on our deromanization systems in appendix a.2. ## conclusion we analyzed the value of romanization for transferring multilingual models to low-resource languages with different scripts. while we cannot recommend romanization as the default strategy for multilingual models and transfer learning across scripts because of the information loss inherent to it, we find that it benefits transfer between related languages that use different scripts. the uconv romanization tool outperforms uroman because it preserves more information encoded in the original script and consequently causes less information loss. furthermore, we demonstrated that romanization can also be successful on the target side if followed by an additional, learned deromanization step. we hope that our results provide valuable insights for future work in transfer learning and practical applications for low-resource languages with unseen scripts.