Neural Transduction for Multilingual Lexical Translation (2020)

Abstract: We present a method for completing multilingual translation dictionaries. Our probabilistic approach can synthesize new word forms, allowing it to operate in settings where correct translations have not been observed in text (cf. cross-lingual embeddings). In addition, we propose an approximate Maximum Mutual Information (MMI) decoding objective to further improve performance in both many-to-one and one-to-one word level translation tasks where we use either multiple input languages for a single target language or more typical single language pair translation. The model is trained in a many-to-many setting, where it can leverage information from related languages to predict words in each of its many target languages. We focus on 6 languages: French, Spanish, Italian, Portuguese, Romanian, and Turkish. When indirect multilingual information is available, ensembling with mixture-of-experts as well as incorporating related languages leads to a 27% relative improvement in whole-word accuracy of predictions over a single-source baseline. To seed the completion when multilingual data is unavailable, it is better to decode with an MMI objective.

## introduction translation matrices, i.e. concept-aligned word lists across the world's languages @xcite @xcite @xcite @xcite , enable several avenues of exploration in computational linguistics and human language technologies. they strengthen word alignment models, which can in turn be useful for machine translation @xcite , robust projection of morphosyntactic information across alignments @xcite , and interlinear glossing. further, fuller word lists enable neogrammarians to better explore phylogeny and phonology across languages @xcite . this work is motivated by the tremendous capacity for humans to generalize during translation, producing forms for words that have not been seen before. this becomes valuable especially for lowerfrequency words, which may not have been observed in training data but could be inferrable through regular processes such as cognate relationships with related languages @xcite , borrowing from neighboring or other influential languages, and even esoteric features like temporal similarity @xcite or image similarity @xcite . in this work, we focus on these cognate relationships, because cognates form a large amount of both core vocabulary @xcite and technical language @xcite . 1 unlike conventional bilingual lexicon induction @xcite , we do not wish to limit the predictions to words that have been previously seen in a corpus. automated methods to induce plausible translations for lexical translation would significantly reduce the human effort needed for both elicitation @xcite and building machine translation systems for less heavily supported languages. our approach to the problem of translation matrix completion is a neural model (parameterized as a character-level sequence-to-sequence network) that handles multiple language pairs, along with an objective function that maximizes both forward and backward probabilities. by leveraging both probabilities, we try to maximize the flow of information between both source and target languages, leading to more accurate model predictions. we find that by both leveraging information about a concept's form from related languages and carefully combining language-pair-wise predictions of an unknown target word, we can improve accuracy by 27% relative to our baseline multilingual neural model. ## related work the task of translation matrix completion, the filling-out of a universal conceptual inventory, has been approached by three broad classes of methods. the first is to manually construct concept inventories, as in @xcite and followup work. the next is to automatically identify cognate relationships, e.g. in word lists @xcite @xcite or raw text @xcite . the third, which is our focus, is to generate putative cognates by performing transduction in the form of sound or orthographic shifts. in this vein, @xcite generate cognates by a pipeline of dictionary lookup and probabilistic orthographic shifts. @xcite uses an svm to perform cognate generation. ciobanu (2016) uses a crf with reranking to the same end. @xcite and @xcite perform translation matrix completion with extracted cognate lists in 6 and 60 language families respectfully, using character-level statistical machine translation systems trained on separate source-target language pairs. wu et al. (2020) performed the same cognate transliteration task with a multi-source multi-target character-level variant of @xcite . we adopt the single system multilingual setup of @xcite , which allows sharing information across language pairs. we also take inspiration from recent successes in other generation tasks. @xcite 's multi-source missing data problem used multiple encoders and a single decoder to leverage multiple source language inputs, which we build on to employ the multiple sources simultaneously during inference. further, we introduce introduce a maximum mutual information (mmi) objective to the problem, motivated by the translational equivalence of cognates @xcite . mmi has been explored in speech recognition @xcite and dialog @xcite . besides mmi, there are a few existing methods for incorporating backward probabilities into the task of translation. @xcite and @xcite follow a noisy channel approach, using bayes' rule to integrate forward, backward and target language model probabilities. we follow @xcite 's approach and implement both the mmi objective and an ensemble mmi objective. ## translation matrix completion the core of this work is the translation matrix t (partially realized in ## mmi reranking the mmi objective provides a principled way to rerank predicted cognates. we motivate its use with the notion of translational equivalence @xcite )-the idea that two words (and particularly when we constrain our focus to cognates) should translate to each other, regardless of direction. particularly, cognacy is a symmetric relationship. the surface form in each language is a view into the (interlingual) concept. this is explicitly modeled in the mmi objective, which simultaneously optimizes for both the forward and backward direction translations. in the context of translation matrix completion, when filling in a single concept across multiple languages, the translations-and particularly cognate relationshipsshould all be equivalent across all languages (as opposed to sentence level translation, where a sentence in one language can have multiple interpretations in another language). predicting cognates de novo is a sequence transduction task akin to transliteration. to perform this task, it is common to use probabilistic models p θ (t | s) of target sequences t given source sequence s, controlled by a set of parameters θ. (we will omit the parameter subscript for brevity.) in our case, t and s are entries in the translation matrix, character sequences from alphabets σ t and σ s . the typical decoding objective is to maximize the conditional log-likelihood, which we will use as a baseline: foot_0 @xmath0 however, to encourage the symmetric relationship of translational equivalence, we can instead use the mmi objective: @xmath1: and then generalize it with a hyperparameter λ to control the weight of each term: @xmath2 because we wish to predict a word from its several cognates, we can further generalize @xcite . that is, given source words s i for languages @xmath3..n -1, we can find the target translation that maximizes the mutual information between each source-target language pair: however, in our reformulated objective function, each term log p(s i |t ) in equation ( 3 ) is intractable during decoding. it requires knowledge of the complete prediction t , which is unavailable until decoding has finished. thus, we approximate this second term by rescoring k-best lists generated by the forward model p(t |s i ). this approximation has previously been used successfully by @xcite . ## experimental setup we formulate lexical translation as sequence-to-sequence character translation, using one model capable of translating between any pair of languages. the input comprises the characters of the word along with tokens to identify the source and target language. including the target language token in the input conditions the multilingual model to generate text in the target language @xcite . the output is a character sequence in the target language. we continue to use this single-model architecture even when using multiple known entries in a row of the translation matrix (that is, many translations of the word to be predicted). we do this by combining distributions from each source language in either a mixture or product of experts @xcite . we also compare the two decoding objectives: conditional likelihood and maximum mutual information. dataset cognate relationships among european languages are well-studied and broadly verifiable. to this end, we use a data set from @xcite which contains cognates in six languages: french, spanish, italian, romanian, portuguese, and turkish. all use the latin alphabet plus language specific diacritics. except for turkish, these are romance languages. (in fact, turkish is not even in the indo-european family. turkish is included because many french and turkish words were imported into romanian, as well as french words into turkish, leading to cognates between these languages. the comparable performance thereon shows that the method is not limited to linguistic cognates.) like @xcite , we group words by cognate cluster using an unsupervised clustering algorithm @xcite , which results in 18k clusters covering 16k english concepts. note that a single concept may have more than one cognate cluster. we hold out 500 concepts for a validation set and 500 concepts for a test set. the validation and test data are then used globally for our scenarios defined below. experimental scenarios broadly, the test inputs available to our models at training and test time define three scenarios, illustrated in figure 1 . in novel, only a single form is present in the row. the model must predict a cognate for a novel concept-one whose forms it has never seen in training. (this would be a first step to filling a completely new row.) in the other two scenarios, only a single entry is missing. in single, the model has seen other entries in the row during training, including the single source word, and it must generate the missing form which is a form of indirect supervision. during inference, we only test for the single (directly supervised) language pair. in multi, generation is conditioned on all known forms of the concept. comparing novel to single addresses whether exposure to the concept's forms is beneficial (they differ in data availability), and comparing single to multi shows whether the standard single-input, single-output sequence transduction framework is sufficient for cognate prediction (they differ in conditioning during inference). in all scenarios, during training the model has seen all extant entries in the rows from the training set and no target-language words from the test set. while not all slots in the cartesian product of data scenarios (novel, single, multi), decoding objectives (conditional log-likelihood or mmi), and ensembling methods (mixture or product of experts) are plausible, this product subsumes the experiments we run. ## results how much does seeing the concept help? (novel vs. single) in does using multiple inputs help? (single vs. multi) the standard encoder-decoder architecture used for sequence transduction does not lend itself well to simple integration of multiple input sequences. is it enough to pick one source language and use this model as-is (single), or does the invested effort in ensembling predictions conditioned on multiple sources (multi) pay off? on average, the multi model, with its final predictions determined by highest sum of log-probabilities, n @xmath4, outperforms both novel and single ( how should we weight the ensemble? (multi vs multi-lse) we observe one shortcoming in the multi model: in some cases, it chooses to ignore an answer very highly rated among many source one possible solution is to instead sum the probabilities to form a mixture of experts, which is equivalent to taking the logsumexp (lse) of the log-probabilities. unlike a product of experts, a single model's low probability cannot 'veto'. on average, using lse improves overall accuracy, at worst it does nothing, and overall it results in our strongest model, multi-lse (results in how should we decode? (log-likelihood vs. mmi objective) above, we described two decoding strategies: maximum conditional log-likelihood and maximum mutual information. the latter infuses a bias toward translational equivalence. because the model's training objective is the conditional loglikeihood, mmi amounts to a rescoring method of the objective. we can decode novel, single, and multi with either strategy. target on average, using rescoring helps novel, but gives mixed results for single ( looking at results from novel-mmi, performance on most languages pairs is unchanged, but a few select language pairs have very large jumps in accuracy ( here, the multi-mmi model correctly chooses telefonist as the correct translation. despite the forward model not generating it as a candidate, the backward model scored it high enough to overcome the forward model. in addition, it is interesting that due to the nature of the backward model, the predictions that are towards the top of the list are more similar to the target candidate. we confirm this by computing the levenshtein distance between the top 10 candidates and the target word ( in the case of multi-mmi with lse, we find that the gains over multi-mmi are greater than the gains between multi and multi-lse. we believe that this is due to the backward model acting as a form of regularization that helps flatten the distribution so that one "expert" does not overpower the rest. despite these gains, it still does not make up for the overall loss in accuracy due to the mmi objective. finally, we show the effects of λ on accuracy on a per-language basis (figure 2 ). in most cases, a higher λ coincides with decreased performance. however, this does not hold for romanian, french, and turkish. first, for both italian and portuguese, we see changing λ does not greatly affect accuracy, implies that for these languages, the backward model is only acting as additional noise. in this case the backward distributions are too flat, so adding the backward term is the same as adding a constant to every candidate's score. for turkish and french, as we increase λ, the model accuracy increases. this might lead one to believe that the backward model is doing all the work, and the forward model is not helping at all. however, this cannot be true: accuracy plummets when only using the backward term. we conclude that the forward model instead acts as a base reference, from which the backward model can then fine tune the results. in the case of turkish, there seems to be a clear optimal λ of 0.6. ## future work the work we presented here has particular applications to low-resource languages. as it is misguided to claim that our system is language-agnostic without verifying @xcite , we plan to expand this work to other language families, such as the austronesian phonological cognate dataset of @xcite . another direction involves experimenting with non-uniform mixing weights that can adaptively give preference to certain languages, as in @xcite . we would also like to extend this work to generate cognates of inflected forms, rather than lemmas and without explicit lemmatization and inflection subcomponents. unlike existing cross-lingual morphological inflection tasks @xcite the source and target are in different languages, rather than relying on transfer. finally, to assess the downstream value of this linguistic tool, future work could populate a statistical translation model's phrase table with predictions from the model. ## conclusion we present a single neural model to handle multilingual many-to-many translation of single words. in addition, by indirectly leveraging multi-lingual information in sequence-to-sequence models, we can improve accuracy in the matrix completion task (novel vs single). by allowing knowledge of concepts that will eventually be tested on between non-target language pairs, the model indirectly learns the correct way to translate into an unseen word in the target language. in addition, directly leveraging multiple source languages improves accuracy on average (by 10% relative to our single model, which is equivalent to @xcite ). a flaw in the ensemble scoring method is remedied in part by using lse, and is also inherently corrected for in the multi-mmi model. in addition we show that the mmi objective is a feasible learning object and in some scenarios gives substantially better than baseline performance. one such scenario is when multilingual data is unavailable. when such data is available, our multi-lse model tends to give best performance overall.