Effective Architectures for Low Resource Multilingual Named Entity Transliteration (2020)

Abstract: In this paper, we evaluate LSTM, biLSTM, GRU, and Transformer architectures for the task of name transliteration in a many-to-one multilingual paradigm, transliterating from 590 languages to English. We experiment with different encoder-decoder combinations and evaluate them using accuracy, character error rate, and an F-measure based on longest continuous subsequences. We find that using a Transformer for the encoder and decoder performs best, improving accuracy by over 4 points compared to previous work. We explore whether manipulating the source text by adding macrolanguage flag tokens or pre-romanizing source strings can improve performance and find that neither manipulation has a positive effect. Finally, we analyze performance differences between the LSTM and Transformer encoders when using a Transformer decoder and find that the Transformer encoder is better able to handle insertions and substitutions when transliterating.

## introduction the world's written languages collectively represent hundreds of different writing systems. transliteration is the process of converting a word's written representation in one language to its equivalent in a target language and is a key component of machine translation and cross-lingual information extraction and retrieval. it is especially relevant for recovering named entities, which often cannot be "translated" in the traditional sense, and linking names across texts in different languages. in this work, we apply existing architectures used for machine translation to the task of name transliteration, the task of converting the written citation form of a name in one language to another language. we use an existing massivelymultilingual resource of aligned parallel names in 591 languages @xcite and evaluate four neural architectures for transliteration. while the use of neural machine translation architectures for transliteration is not novel, to the best of our knowledge a comparative study of the performance of multiple models and preprocessing strategies in the many-to-one transliteration paradigm has not been previously performed. the contributions of this paper are as follows: first, we provide a performance comparison of several different neural architectures on the task of name transliteration in a many-to-one paradigm. second, we evaluate the effectiveness of various methods of manipulating input sequences for name transliteration. finally, we present a multilingual transliteration model with a 1-best accuracy of 73%, a 4-point improvement over the previous best model for this task. ## related work the 2018 named entities workshop included a shared task on named entity transliteration @xcite that established several useful metrics to evaluate various elements of transliteration quality. we use their mean f-score metric in this work. as part of the shared task, @xcite compared rnn and cnn architectures on the task of name transliteration for 13 language pairs. they found that their character-level rnn model performed best and achieved state-of-the-art results for one language pair. notably, the authors focused on only a few higher-resource languages, whereas our study uses a much larger set of languages, including many lower-resourced ones. merhav and ash (2018) released a set of bilingual name dictionaries mined from wikipedia for english to russian, hebrew, arabic, and japanese katakana transliteration, and compare traditional weighted fst models with more modern neural techniques on bilingual transliteration tasks. their models collapse all names written in latin scripts under the english label. but very different languages may share a script; we aim to create models sensitive to these differences. @xcite recently released a much more comprehensive name corpus covering 3 million names across 180 languages derived from wikipedia, geonames, and the dataset released by @xcite . regarding lower-resourced transliteration, upadhyay et al. (2018) proposed a bootstrapping method wherein a "weak" generation model is used to guide discovery of possible transliteration candidates for a given word. le and sadat (2018) used a combination of g2p, neural networks and word embeddings to improve english-vietnamese transliteration. @xcite introduced a now-popular technique for invoking transfer learning in a manyto-one multilingual translation framework. a single model is trained with a mixed source vocabulary and a single target vocabulary. an artificial "flag" token is appended to each source sequence to help the model identify languages, circumventing the need for training a separate encoder and decoder for each language pair. we use this approach to perform many-languages-to-one transliteration. wu and yarowsky (2018) explored approaches to extremely low-resource name transliteration, using a multiparallel corpus of biblical names across 591 languages @xcite . the authors also provided a cursory report on the results of an experimental multilingual transliteration model, using the technique proposed by @xcite . they found that a character-level rnn trained on concatenated data from all source languages significantly outperformed individual language-pair models. using the same data, we aim to provide a more comprehensive study of the performance of various neural architectures and the applicability of potential performance-boosting preprocessing techniques for the task of transliteration in a many-to-one framework. ## discussion our findings demonstrate that using a transformer architecture for the encoder and decoder results in reliably strong performance for this task. attempting to improve the model through simple "tweaks" to the input does not improve performance and may hurt it. however, one may wonder why the transformer does so much better at this task than an lstm encoder and a decoder with attention when the transliteration problem is relatively simple compared to sentence translation. there is only limited reordering in transliteration, and there may be fewer long-range dependencies. while we have manually examined the output of the transliteration system across architectures, it is difficult to identify obvious patterns, and attempting to do so puts us at risk of overgeneralizing our observations that are based on the relatively few writing systems that we can read. to further analyze performance, for every item in the test set we computed the levenshtein distance-expressed as insertions, deletions, and substitutions-between the source name and target name. we then analyzed the relationship between these edit distance metrics and 1-best accuracy by predicting whether the system correctly transliterated each item of the test set using logistic regression. we analyze two systems: the lstm and transformer encoders, each paired with the transformer decoder. 4 we examined the contribution of each type of edit (insertions, etc.) on the performance of each system. we fit a logistic regression model to the 689,960 predictions across all items and random seeds for the models we are comparing. for each item, we used the number of source-target edit distance operations to predict whether the model made any errors. we employed interactions between the encoder type and each predictor to explicitly test for differences between the lstm and transformer encoders. in other words, our model tested whether the two encoders differed in how each edit operation required affected their ability to correctly transliterate. we found that there were significant differences in the interaction terms of the model for insertions and substitutions, but not for deletions. for the transformer encoder, with each additional insertion in the edit distance, it was 1.1% less likely to produce an error than the lstm was (@xmath0.0006), for substitutions, 1.8% (@xmath1.0001). this helps further characterize the performance differences between these models. while they are equally capable of handling deletions, the transformer encoder can better handle insertions and substitutions, and the advantage is larger for substitutions. ## conclusion we conclude that using a transformer architecture for both the encoder and decoder leads to the best performance on the many-languages-to-english transliteration task that we evaluate. however, using macrolanguage codes and pre-romanizing the input do not improve performance. our best multilingual transliteration model achieves a 1-best accuracy of 73%, a 4-point improvement over the baseline provided by @xcite . when using a transformer encoder, our model demonstrates an improved ability to handle substitution edits in source-target pairs compared to an lstm encoder. while an off-the-shelf mt system provides a strong starting system for the task of many-toone transliteration, future improvements for lowerresourced settings will likely require a greater level of sophistication, possibly using monolingual pretraining to better model the source language given few training examples. additionally, while using the transformer for the encoder and decoder gives the strongest results, it may be possible to further simplify the model and achieve similar results. figure 5: cer (lower is better) across encoder and decoder architectures uroman. in proceedings of acl 2018, system demonstrations, pages 13-18, melbourne, australia. association for computational linguistics. melvin johnson, mike schuster, quoc v. le, maxim krikun, yonghui wu, zhifeng chen, nikhil thorat, fernanda viégas, martin wattenberg, greg corrado, macduff hughes, and jeffrey dean. 2017. google's multilingual neural machine translation system: enabling zero-shot translation. transactions of the association for computational linguistics, 5:339-351. guillaume klein, yoon kim, yuntian deng, jean senellart, and alexander rush. 2017. opennmt: opensource toolkit for neural machine translation. in proceedings of acl 2017, system demonstrations, pages 67-72, vancouver, canada. association for computational linguistics. soumyadeep kundu, sayantan paul, and santanu pal. 2018. a deep learning based approach to transliteration. in proceedings of the seventh named entities workshop, pages 79-83, melbourne, australia. association for computational linguistics. ngoc tan le and fatiha sadat. 2018. low-resource machine transliteration using recurrent neural networks of asian languages. in proceedings of the seventh named entities workshop, pages 95-100, melbourne, australia. association for computational linguistics.