E nglish-to- C hinese Transliteration with Phonetic Auxiliary Task (2020)

Abstract: Approaching named entities transliteration as a Neural Machine Translation (NMT) problem is common practice. While many have applied various NMT techniques to enhance machine transliteration models, few focus on the linguistic features particular to the relevant languages. In this paper, we investigate the effect of incorporating phonetic features for English-to-Chinese transliteration under the multi-task learning (MTL) setting—where we define a phonetic auxiliary task aimed to improve the generalization performance of the main transliteration task. In addition to our system, we also release a new English-to-Chinese dataset and propose a novel evaluation metric which considers multiple possible transliterations given a source name. Our results show that the multi-task model achieves similar performance as the previous state of the art with a model of a much smaller size.

## introduction transliteration, the act of mapping a name from the orthographic system of one language to another, is directed by the pronunciation in the source and target languages, and often by historical reasons or conventions. it plays an important role in tasks like information retrieval and machine translation @xcite . over the recent years, many have addressed transliteration using sequence-to-sequence (seq2seq) deep learning models @xcite @xcite , enhanced with several nmt techniques @xcite . however, this recent work neglects the most crucial feature for transliteration, i.e. pronunciation. to english ipa chinese pinyin a /"ei./ 艾 ài my /mi/ 米 mǐ depending on the specific language, the written form of a word reveals its pronunciation to various extents. for alphabetical languages such as english and french, a letter, or a sequence of letters, usually reflects the word pronunciation. for example, the word amy (in the international phonetic alphabet, ipa, /"ei.mi/) has the sub-word a corresponding to /"ei./ and my corresponding to /mi/. in contrast, characters in a logographic foot_0 writing system for languages like chinese or japanese do not explicitly indicate sound @xcite . in this paper, we give a treatment to the problem of transliteration from english (alphabet) to chinese foot_1 (logogram) using an rnn-based mtl model with a phonetic auxiliary task. we transform each chinese character to the alphabetical representation of its pronunciation via the official phonetic writing system, pinyin, foot_2 which uses latin letters with four diacritics denoting tones to represent the sounds. for example, the chinese transliteration for amy is 艾米 and the associated pinyin representation is ài mǐ. we summarize the correspondences occurring in this example in due to the similarity between the source name and the pinyin representation, @xcite proposed a sequential transliteration model that uses pinyin as an intermediate representation before transliterating a chinese name to english. in contrast, our idea is to build a model with a shared encoder and dual decoders, that can learn the mapping from english to chinese and pinyin simultaneously. by jointly learning source-to-target and source-to-sound mappings, the encoder is expected to generalize better @xcite and pass more refined information to the decoders. transliteration datasets are often extracted from dictionaries, or aligned corpus generated from applying named entity recognition (ner) system to parallel newspaper articles in different languages @xcite . we use two datasets for our experiments, one taken from news machine transliteration shared task @xcite and the other extracted from a large dictionary. we evaluate the transliteration system using both the conventional word accuracy and a novel metric designed for english-to-chinese transliteration (see section 5). our contributions are as follows: we report accuracy and f-score of 0.299 and 0.6799, respectively, on the news dataset, with a model of size 22m parameters, compared to the previous state of the art @xcite , which achieves accuracy and f-score of 0.304 and 0.6791, respectively, with a model of size 133m parameters. on the dict dataset, for source (x) target (y) pinyin (p) caleigh 凯莉 kai li ## problem formulation we use the word vocabulary to describe the set of characters for the purpose of our task specification. let v src and v tgt denote the source and target vocabularies, respectively. for a source word x of length i and a target word y of length j, we have: we formulate the task of transliteration as a supervised learning problem: given a collection of n training examples, {(x (i) , y (i) )} n @xmath0, the objective is to learn a predictor function, f : @xmath1, of which the parameter space maximizes the following conditional probability: for our multi-task transliteration model, the predictor becomes f mtl : @xmath2, p), where p denotes the written representation of the pronunciation of the target word y. for decoding, we maximize the conditional probabilities, p (p|x, ỹ) and p (y|x, p), where ỹ and p refers to the implicit information channeled by one task to the other. the phonetic information we use for our task refers to the pinyin version of the name in chinese, without tone marks, foot_3 because they are often removed for spelling chinese names in an alphabetical language. we present an example data point in the form of (x, y, p) in ## dataset preparation we experiment with two different english-to-chinese datasets. for simplicity, we denote the one taken from news machine transliteration shared task @xcite as "news," and the one extracted from the dictionary (xinhua news agency, 2007) as "dict." ## model our model is intent on solving english-to-chinese transliteration through joint supervised learning of source-to-target (main) and source-to-pinyin (auxiliary) tasks. training closely related tasks together can help the model to learn information that is often ignored in single-task learning, thus obtaining a better representation in the shared layers (in our case, encoder). moreover, the auxiliary task implicitly provides the phonetic information that is not easily learned through the single main task given the characteristics of chinese (see section 1). our model has a sequence-to-multiple-sequence (seq2multiseq) architecture that contains a shared encoder and dual decoders. between the encoder and decoder is a bridge layer 12 that transforms the 12 we call it "bridge" because it connects the shared encoder to each decoder. it allows flexible choices of the hidden sizes of the encoder and decoder and serves as the intermediate "buffer" before passing the encoder final state to each decoder. the encoder has an embedding layer with dropout @xcite , followed by a 2layer bilstm @xcite . the bridge layer consists of a linear layer followed by tanh activation. the shared encoder passes its final state to the main-task decoder and the auxiliarytask decoder via separate bridge layers. in each decoder, we use additive attention @xcite to compute the context vector (weighted sum of the encoder outputs according to the attention scores), then concatenate it with the target embedding to form the input of the subsequent 2-layer feed-forward lstm. the prediction is made by feeding the concatenation of the lstm's output, the context vector and the target embedding into a linear layer followed by log-softmax. our model is expected to simultaneously maximize the conditional probabilities mentioned in section 2. to achieve this goal, we use the linear combination of the main-task decoder's loss foot_10 (negative log likelihood; l y ) and the auxiliary-task decoder's loss (l p ) as the model's objective function: ## adaptive evaluation metrics we evaluate the transliteration system using word accuracy (acc) and its variants on the 1-best output: source target (f) target (m) med mona 莫娜 莫纳 1 colina 科莉娜 科利纳 2 we use acc and acc+ to denote the original accuracy and its variant with multiple references. the drawback of acc is that it may underestimate the quality of the system because it neglects the possibility of having more than one transliteration for a given source name, as is the case for english-to-chinese transliteration. for example in based on the knowledge of a native chinese speaker, we analyze the english-to-chinese dataset and summarize the key observations for source names with multiple target transliterations as follows: the minimum edit distance (med) between any two target @xmath3, and the lengths are the same; for any two such target names, distinct characters occur in the same position, and they often indicate the gender of the name (see to use act in accord with the above observations, we propose the following criterion for the accuracy indicator function (we refer to it as acc-act). let subscript t denote the position of a character, then i criterion(ŷ,y) = 1 if either med(ŷ, y) = 0 (which covers all the cases for acc) or the following conditions are met in order: there is no guarantee that characters that are interchangeable according to act can replace each other in every scenario. but since we only apply enc dec-m dec-a emb. h 256 256 128 δ 0.1 0.1 0.1 rnn h 512 512 128 δ 0.2 0.2 0.1 ## experimental setup recall from section 4 that we use λ to denote the weighting of the two tasks we train. we set the single-main-task (λ = 1) and the single-auxiliarytask (λ = 0) models as the baselines, and compare the multi-task models of different weightings (λ ∈ { 1 6 , 1 4 , 1 2 , 2 3 , 5 6 , 8 9 }) against them. we conduct experiments on both the news and dict datasets and select the best model for each of them to compare to the previous state of the art. ## discussion furthermore, we present some typical examples in which the multi-task model generates better predictions than the single-task in still, the multi-task model does not consistently handle all names better than the single-task modelespecially for exceptional names that do not have a regular transliteration. for instance, the name fyleman is transliterated into 法伊尔曼, but the character 伊 does not have any source-word correspondence if we consider the pronunciation of the source name. finally, our model can be generalized to other transliteration tasks by replacing pinyin with other phonetic representations such as ipa for english and rōmaji for japanese. in addition, acc-act can be extended to alphabetical languages by, for instance, constructing the alternating sub-word table which stores lists of interchangeable subsequences. another possible future work is to redesign the objective function by treating λ as a trainable parameter or including the correlation information @xcite . ## related work previous work has demonstrated the effectiveness of using mtl on models through joint learning of various nlp tasks such as machine translation, syntactic and dependency parsing @xcite @xcite . in most of this work, underlies a similar idea to create a unified training setting for several tasks by sharing the core parameters. besides, machine transliteration has a long history of using phonetic information, for example, by mapping a phrase to its pronunciation in the source language and then convert the sound to the target word @xcite . there is also relevant work that uses both graphemes and phonemes to various extents for transliteration, such as the correspondence-based @xcite and g2p-based (le and sadat, 2018) approaches. our work is inspired by the intu-itive understanding that pronunciation is essential for transliteration, and the success of incorporating phonetic information such as pinyin @xcite and ipa @xcite , in the model design. ## conclusion we argue in this paper that language-specific features should be used when solving transliteration in a neural setting, and we exemplify a way of using phonetic information as the transferred knowledge to improve a neural machine transliteration system. our results demonstrate that the main transliteration task and the auxiliary phonetic task are indeed mutually beneficial in english-to-chinese transliteration, and we discuss the possibility of applying this idea on other language pairs.