Language Embeddings for Typology and Cross-lingual Transfer Learning (2021)

Abstract: Cross-lingual language tasks typically require a substantial amount of annotated data or parallel translation data. We explore whether language representations that capture relationships among languages can be learned and subsequently leveraged in cross-lingual tasks without the use of parallel data. We generate dense embeddings for 29 languages using a denoising autoencoder, and evaluate the embeddings using the World Atlas of Language Structures (WALS) and two extrinsic tasks in a zero-shot setting: cross-lingual dependency parsing and cross-lingual natural language inference.

## introduction recent efforts to leverage multilingual datasets in language modeling @xcite and machine translation @xcite highlight the potential of multilingual models that can perform well across various languages, including ones for which training sets are scarce. most of the current multilingual research focuses on learning invariant representations or removing language-specific features after training @xcite . despite recent advances, there are still limitations. previous work has shown that similar languages can benefit from sharing parameters, but less similar languages do not help @xcite . however, in spite of some interests in typology @xcite , identifying similar languages is nontrivial, especially for less studied ones. in addition, as @xcite suggest, learning invariant representations can actually harm model performance. therefore, in order to leverage language agnostic and language specific information effectively, we propose to generate language representations and examine the interactions among different language representations. one way to represent language identity within a multilingual model is the use of language codes, or dense vectors representing language embeddings. if languages are represented with vectors that capture cross-lingual similarities and differences across different dimensions, this information can guide a multilingual model regarding what and how much of the information in the model should be shared among specific languages. much of the previous research focused on generating language embeddings using prior knowledge such as word order @xcite , using a parallel corpus @xcite , and using language codes as an indicator to distinguish input and output words in a shared vocabulary into different languages @xcite . in contrast, our work focuses on generating and using language embeddings more effectively as softsharing @xcite of parameters among various languages in a single model. furthermore, we are motivated by a more difficult setting where the properties of each language are not known in advance, and no parallel data is available. we investigate whether we can generate language embeddings to represent typological information derived solely from corpora in each language without the use of any parallel text, translation models, or prior knowledge. inspired by the findings that structural similarity, especially word ordering, is crucial in large pretrained multilingual language models (k et al., 2020), we propose an unsupervised method leveraging denoising autoencoders @xcite to generate language embeddings. we show that our ap-proach captures typological information by comparing the information in our language embeddings to language-specific information in the world atlas of language structures (wals, @xcite . in addition, to address the question of whether the learned language embeddings can help in downstream language tasks, we plug-in the language embeddings to cross-lingual dependency parsing and natural language inference (xnli, @xcite in a zero-shot learning setting, obtaining performance improvements. ## related work previous related research approached language representations by using prior knowledge, dense language embeddings with multilingual parallel data, or no prior knowledge about languages but having language embeddings primarily as a signal to identify different languages. ## generating language embeddings we first present the data used to generate language embeddings, then introduce our approach inspired by denoising autoencoders @xcite . ## experiments to examine the quality of the typological information captured by the language embeddings, we perform intrinsic and extrinsic evaluations. our intrinsic evaluation consists of predicting linguistic typology and language features from the world atlas of language structures (wals, @xcite . our extrinsic evaluations are based on cross-lingual dependency parsing and cross-lingual natural language inference (xnli, @xcite in a zero-shot learning setting, where a trained model makes predictions on a language not seen during training, but for which a language embedding has been learned from plain monolingual text. in contrast with previous research which applies learned typology to cluster similar languages and train machine translation tasks in clusters @xcite , we explore if we can apply the learned embeddings directly into downstream tasks. we compare three different sets of embeddings based on our approach with three sets of embeddings from previous work: spe. lang_emb represents language embeddings from our proposed denoising autoencoder trained with language specific muse embeddings, using commoncrawl text. eng. lang_emb represents language embeddings trained with english muse embeddings after mapping words from different languages to english, using commoncrawl text. wiki lang_emb represents language embeddings trained with english muse embeddings using wikipedia. we use the same data selection and preprocessing process as detailed in section 3.1. we use these embeddings to show the impact of training data. in addition, we use these embeddings to compare with xlm embeddings trained with wikipedia. malaviya represents language embeddings from @xcite , trained with a many-to-one machine translation model using bible parallel data. it has 26 languages in common with our 29 languages except english, hebrew, and norwegian. we use these embeddings to represent previous methods of learning language representations from parallel data. 5 xlm mono represents language embeddings trained with xlm model using the same monolingual data as wiki lang_emb on 29 languages. xlm parallel represents language embeddings trained with xlm using monolingual and parallel data from 15 xnli languages. we extract the embeddings from the publicly available model. ## results and analysis we show results of our proposed language embeddings in comparison to the baselines and language vectors generated from previous work on linguistic typology, wals, cross-lingual parsing, and xnli. we report results with eng. language embeddings. detailed comparison to other language embeddings on each task can be found in appendix c. ## conclusion language embeddings have the potential to contribute to our understanding of language and linguistic typology, and to improve the performance of downstream multilingual nlp applications. our proposed method to generate dense vectors to capture language features is relatively simple, based on the idea of denoising autoencoders. the model does not require any labeled or parallel data, which makes it promising for cross-lingual learning in situations where no task-specific training datasets are available. we showed that the trained language embeddings represent typological information, and can also benefit the downstream tasks in a zero-shot learning setting. this is an encouraging result that indicates that task-specific annotated data for various languages can be leveraged more effectively for improved task performance in situations where language-specific resources may be scarce. at the same time, our results indicate that the effectiveness of our approach is sensitive to the set of languages used, highlighting the importance of using a more balanced variety of languages than is current practice, our work included. we will pursue an investigation of the impact of language selection in multilingual and cross-lingual models as future work, to our understanding of these methods and their broader applicability.