Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations (2020)

Abstract: Sparse language vectors from linguistic typology databases and learned embeddings from tasks like multilingual machine translation have been investigated in isolation, without analysing how they could benefit from each other’s language characterisation. We propose to fuse both views using singular vector canonical correlation analysis and study what kind of information is induced from each source. By inferring typological features and language phylogenies, we observe that our representations embed typology and strengthen correlations with language relationships. We then take advantage of our multi-view language vector space for multilingual machine translation, where we achieve competitive overall translation accuracy in tasks that require information about language similarities, such as language clustering and ranking candidates for multilingual transfer. With our method, we can easily project and assess new languages without expensive retraining of massive multilingual or ranking models, which are major disadvantages of related approaches.

## introduction recent surveys consider linguistic typology as a potential source of knowledge to support multilingual natural language processing (nlp) tasks @xcite . linguistic typology studies language variation in terms of their functional processes @xcite . several typological knowledge bases (kb) have been crafted, from where we can extract categorical language features @xcite . nevertheless, their sparsity and reduced coverage present a challenge for an end-to-end integration into nlp algorithms. for example, the world atlas of language structure (wals; @xcite encodes 143 features for 2,679 languages, but their mean coverage per language is barely around 14%. dense and data-driven language representations have emerged in response. they are computed from multilingual settings of language modelling @xcite and neural machine translation (nmt) @xcite . however, the language diversity in the corpus-based representations is limited. the language coverage could be broadened with other knowledge, such as that encoded in wals, to distinguish even more language properties. therefore, to obtain the best of both views (kb and task-learned) with minimal information loss, we project a shared space of discrete and continuous features using a variant of canonical correlation analysis @xcite . for our study, we fuse language-level embeddings from multilingual machine translation with syntactic features of wals. we inspect how much typological knowledge is present by predicting features for new languages. then, we infer language phylogenies and inspect whether specific relationships are induced from the task-learned vectors. furthermore, to demonstrate that our approach has practical benefits in nlp, we apply our language vectors in multilingual nmt with language clustering @xcite and adapt the ranking of related languages for multilingual transfer @xcite . as a side outcome, we identify that there is an ideal setting to encode language relationships in language embeddings from nmt. finally, we are releasing a simple tool to allow everyone to fuse their own representations for clustering, ranking and more. ## multi-view language representations our primary goal is to fuse parallel representations of the same language in one shared space, and canonical correlation analysis (cca) allows us to find a projection of two views for a given set of data. with cca, we look for linear combinations that maximise the correlation of the two sources in each coordinate iteratively @xcite . after training, we can apply the transformation learned on a new sample from any view to obtain a cca-based language representation. 1 cca considers all dimensions of the two views as equally important. however, our sources are potentially redundant: kb features are mostly onehot-encoded, whereas task-learned ones inherit the high dimensionality of the embedding layer. moreover, few samples and sparsity could make the convergence harder. for the redundancy issue, singular value decomposition (svd) is an appealing alternative. with svd, we factorise the source data matrix to compute the principal components and singular values. furthermore, to deal with sparsity, we adopt a truncated svd approximation, which is also known as latent semantic analysis in the context of linear dimensionality reduction for termcount matrices @xcite . the two-step transformation of svd followed by cca is called singular vector canonical correlation analysis @xcite in the context of understanding the representation learning throughout neural network layers. that being said, we use svcca to get language representations and not to inspect a neural architecture. 2 ## methodology and research questions to embed linguistic typology knowledge in dense representations for a broad set of languages, we employ svcca ( §2) with the following sources: kb view. we employ the language vectors from the uriel and lang2vec database @xcite . precisely, we work with the k-nn vectors of the syntax feature class (u s ; 103 feats.), that are composed of binary features encoded from wals @xcite . secondly, we take the many-to-english language embeddings learned for the language clustering task on multilingual nmt (l w ; 256 dim.) @xcite , where they use 23 languages of the wit foot_0 corpus @xcite . one main difference for the latter is the use of factors in the architecture, meaning that the embedding of every input token was concatenated with the embedded pseudo-token that identifies the source language. the second difference is the neural architecture used to extract the embeddings: the former use a recurrent neural network, whereas the latter a small transformer model @xcite . finally, we train a new set of embeddings (l t ) that we extracted from the 53 languages of the ted corpus (many-to-english) processed by @xcite , using the approach of @xcite . 3 what knowledge do we represent? each source embeds specialised knowledge to assess language relatedness. the kb vectors can measure typological similarity, whereas task-learned embeddings correlates with other kinds of language relationships (e.g. genetic) @xcite . to analyse whether each kind of knowledge is induced with svcca, we assess the tasks of typological feature prediction ( §4) and reconstruction of a language phylogeny ( §5). what is the benefit for multilingual nmt (and nlp)? language-level representations can evaluate the distance between languages in a vector space. we then can assess their applicability on multilingual nmt tasks that require guidance from language relationships. therefore, language clustering and ranking related partner languages for (multilingual) transfer are our study cases ( §6). ## prediction of typological features an example of a typological feature is a word order specification, like whether the adjective is predominately placed before or after the noun (features #24 and #25 of u s ). our task consists in predicting syntactic features (u s ) leaving one-language and one-language-out one-family-out single svcca #f. one-language-family out to control phylogenetic relationships @xcite . previous work has shown that task-learned embeddings are potential candidates to predict features of a linguistic typology kb @xcite , and our goal is to evaluate whether svcca can enhance the nmt-learned language embeddings with typological knowledge from their kb parallel view. experimental setup. we use a logistic regression classifier per u s feature, which is trained with the nmt-learned or svcca representations in both one-language-out and one-language-familyout settings. for prediction, we use the original embedding or its svcca projection as inputs. ## language phylogeny analysis according to @xcite , there is a positive correlation between the language distances in a phylogenetic tree and a pairwise distance-matrix of task-learned representations. our goal therefore is to investigate whether fusing linguistic typology with svcca can preserve or enhance the embedded relationship information. for that reason, we examine how well a language phylogeny can be reconstructed from language representations ( §5.1), and also study the correlation (appendix b). ## application in multilingual nmt with multilingual nmt, we can translate several language-pairs using a single model. low-resource languages usually benefit through multilingual transfer, which resembles a simultaneous training of the parent(s) and child models. therefore, we want to take advantage of a language-level vector 7 in further analysis, we confirmed that the inferred tree with only 12 languages of svcca (without projection of extra entries) is comparable or better against the rest of the baselines. language clustering. the main idea is to obtain smaller multilingual nmt models as an intermediate point between maintaining many pairwise systems and a single massive multilingual model. with limited resources, it is challenging to support the first scenario, whereas the advantages for the massive setting are also very appealing (e.g. simplified training process, translation improvement for low-resource languages or zero-shot translation @xcite ). therefore, to address the task, @xcite trained a factored multilingual nmt model of 23 languages from @xcite , where the language embedding is concatenated in every input token. then, they performed hierarchical clustering with the representations, and selected a number of clusters guided by the elbow method. finally, they compared the systems against individual, massive and language family-based cluster models. in a practical multilingual nmt system, it is not only necessary to choose the right clustering, the ability to easily add new languages is also important. with this in mind, we apply our multi-view representations to compute a set of clusters, and we also address the question: do we need to train the massive model again if we want to add one or more new languages to our setting? language ranking. the original goal of lan-grank is to choose a parent language to perform transfer learning in different tasks, nmt included. to achieve this, @xcite trained a model based on the performance of several hundred pairwise mt systems using the dataset of @xcite . for the input features, they considered linguistically-informed vectors from lang2vec @xcite and corpus-based statistics, such as word/sub-word overlapping and the ratio of the token-types or the data size between the target child and potential candidates, where the latter features were some of the most relevant. considering the transfer capabilities within multilingual nmt and the possibility to obtain a ranked list of candidates from langrank, we propose an adapted task of choosing k-related languages for multilingual transfer. we then use our multi-view representations to rank related languages from the vector space, as they embed information about typological and lexical relationships. this is similar to the features that @xcite consider, but without training a ranking model fed with scores from pairwise mt systems. ## factors over initial pseudo-tokens we additionally argue that the configuration used to compute the language embeddings impacts what relationship they can learn. for the analysis, we extract an alternative set of 53 language embeddings (l t * ) but using the initial pseudo-token setting instead of factors. then, we perform a silhouette analysis to identify whether we can build cohesive and well-separated clusters of languages. figure 4 shows the silhouette analysis for the aforementioned embeddings (l t * ) together with the bible embeddings (l b ) that were trained with the same configuration. we observe that the silhouette score never exceeds 0.2, and the curve keeps degrading when we examine a higher number of clusters, which contrast the trend shown in figure 2 . the pattern proves that the vectors are not suitable for clustering (the hierarchies are shown in figure 6 in the appendix), and they might only encode enough information to perform a classification task in the multilingual nmt training and inference. for that reason, we consider it essential to use language embeddings from factors for extracting language relationships. ## related work for language-level representations, uriel and lang2vec @xcite allow a straightforward extraction of typological binary features from different kbs. @xcite @xcite exploits them to build latent language representations with independent binary variables. language features are encoded from data-driven tasks as well, such as nmt @xcite or language modelling @xcite @xcite with complementary linguistic-related target tasks @xcite . our approach is most similar to @xcite , as they build a generative model from typological features and use language embeddings, extracted from factored language modelling at character-level, as a prior of the model to extend the language coverage. however, our method primarily differs as it is mainly based in linear algebra, encodes information from both sources since the beginning, and can deal with a small number of shared entries (e.g. 23 from l w ) to compute robust representations. there has been very little work on adopting typology knowledge for nmt. there is not a deep integration of the topics @xcite , but one shallow and prominent case is the ranking method @xcite that we analysed in §6. finally, cca and its variants have been previously used to derive embeddings at word-level @xcite @xcite . @xcite also used svcca but to inspect sentence-level representations, where they uncover relevant insights about language similarity that are aligned with our results in §5. however, as far as we know, this is the first time a cca-based method has been used to compute language-level representations. ## takeaways and practical tool we summarise our key findings as follows: ## conclusion we compute multi-view language representations with svcca using two sources: kb and nmtlearned vectors. with a typological feature prediction task and the inference of phylogenetic trees, we showed that the knowledge and language relationship encoded in both sources is preserved in the combined representation. moreover, our approach offers important advantages because we can evaluate projected languages with entries in only one of the views and can easily extend the language coverage. the benefits are noticeable in multilingual nmt tasks, like language clustering and ranking related languages for multilingual transfer. we plan to study how to deeply incorporate our typologically-enriched embeddings in multilingual nmt, where there are promising avenues in parameter selection (sachan and neubig, 2018) and generation @xcite . last but not least, we thank the organisers and participants of the first workshop of typology for polyglot nlp, and the members of the statistical machine translation group at the university of edinburgh, whose provided relevant feedback in an early stage of the study. for computational linguistics (volume 1: long papers), pages 530-540, vancouver, canada. association for computational linguistics. maithra raghu, justin gilmer, jason yosinski, and jascha sohl-dickstein. 2017. svcca: singular vector canonical correlation analysis for deep learning dynamics and interpretability. in advances in neural information processing systems 30, pages 6076-6085. curran associates, inc. peter j rousseeuw. 1987. silhouettes: a graphical aid to the interpretation and validation of cluster analysis. journal of computational and applied mathematics, 20:53-65. devendra sachan and graham neubig. 2018. parameter sharing methods for multilingual self-attentional translation models. in proceedings of the third conference on machine translation: research papers, pages 261-271, belgium, brussels. association for computational linguistics. rico sennrich, orhan firat, kyunghyun cho, alexandra birch, barry haddow, julian hitschler, marcin junczys-dowmunt, samuel läubli, antonio valerio miceli barone, jozef mokry, and maria nadejde. 2017. nematus: a toolkit for neural machine translation. in proceedings of the software demonstrations of the 15th conference of the european chapter of the association for computational linguistics, pages 65-68, valencia, spain. association for computational linguistics. rico sennrich, barry haddow, and alexandra birch. 2016. neural machine translation of rare words with subword units. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1715-1725, berlin, germany. association for computational linguistics. m. serva and f. petroni. 2008. indo-european languages tree by levenshtein distance. epl (europhysics letters), 81(6):68005. xu tan, jiale chen, di he, yingce xia, tao qin, and tie-yan liu. 2019. multilingual neural machine translation with language clustering. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 963-973, hong kong, china. association for computational linguistics. yulia tsvetkov, sunayana sitaram, manaal faruqui, guillaume lample, patrick littell, david mortensen, alan w black, lori levin, and chris dyer. 2016. polyglot neural language models: a case study in cross-lingual phonetic representation learning. in proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1357-1366, san diego, california. association for computational linguistics. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, ł ukasz kaiser, and illia polosukhin. 2017. attention is all you need. in i. guyon, u. v. luxburg, s. bengio, h. wallach, r. fergus, s. vishwanathan, and r. garnett, editors, advances in neural information processing systems 30, pages 5998-6008. curran associates, inc. xinyi wang and graham neubig. 2019. target conditioned sampling: optimizing data selection for multilingual neural machine translation. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 5823-5828, florence, italy. association for computational linguistics. joe h ward jr. 1963. hierarchical grouping to optimize an objective function. journal of the american statistical association, 58(301):236-244.