M o SEC ro T : Model Stitching with Static Word Embeddings for Crosslingual Zero-shot Transfer (2024)

Abstract: Transformer-based pre-trained language models (PLMs) have achieved remarkable performance in various natural language processing (NLP) tasks. However, pre-training such models can take considerable resources that are almost only available to high-resource languages. On the contrary, static word embeddings are easier to train in terms of computing resources and the amount of data required. In this paper, we introduce MoSECroT (Model Stitching with Static Word Embeddings for Crosslingual Zero-shot Transfer, a novel and challenging task that is especially relevant to low-resource languages for which static word embeddings are available. To tackle the task, we present the first framework that leverages relative representations to construct a common space for the embeddings of a source language PLM and the static word embeddings of a target language. In this way, we can train the PLM on source-language training data and perform zero-shot transfer to the target language by simply swapping the embedding layer. However, through extensive experiments on two classification datasets, we show that although our proposed framework is competitive with weak baselines when addressing MoSECroT, it fails to achieve competitive results compared with some strong baselines. In this paper, we attempt to explain this negative result and provide several thoughts on possible improvement.

## introduction the emergence of plms and their multilingual counterparts (mplms) @xcite have proven effective for various nlp tasks @xcite . however, such models are mostly limited to no more than a hundred languages, as the pre-training requires considerable data that is only available to these languages, leaving the majority * equal contribution. our contribution is summarized as follows: (i) we introduce mosecrot, a novel and challenging task for (especially low-resource) languages where static word embeddings are available. @xcite we propose a solution that leverages relative representations to construct a common space for source (english in our case) and target languages and that allows zero-shot transfer for the target languages. ## related work aligned crosslingual word embeddings enable transfer learning by benefiting from a shared representation space for the source and target languages. such embedding pairs are typically either trained jointly @xcite or obtained through post-alignment @xcite . our work applies a transformation in the manner of the latter to align two embedding spaces where the source embeddings are derived from a plm and target embeddings are static word embeddings. based on a recent consensus that similar inner representations are learned by neural networks regardless of their architecture or domain @xcite , @xcite propose an approach to align latent spaces with respect to a set of samples, called parallel anchors. they transform the original, absolute space to one defined by relative coordinates of the parallel anchors, and denote all the transformed samples in the relative coordinates as relative representations. model stitching was proposed as a way to com-bine (stitch together) components of different neural models. trainable stitching layers are first introduced by @xcite , with a series of subsequent works demonstrating the effectiveness of the approach @xcite . ## mosecrot task setting the task setting is straightforward: given a plm of a high-resource language (regarded as the source language) and static word embeddings of another language (low-resource and regarded as the target language), the goal is to achieve zero-shot transfer by using the target language embeddings directly with the source language model via embedding layer stitching. this can be done by first applying an alignment between the source and target embedding spaces and subsequently swapping the embedding matrices of the plm. we propose a novel method that leverages relative representations for embedding space mapping. in the following, we describe our methodology in more detail. ## methodology parallel anchor selection we first extract bilingual parallel lexica between the source and the target language. for most high-resource languages, large bilingual lexica are available from muse foot_0 . for low-resource languages, we crawl translations of source language vocabulary from panlex foot_1 and google translate foot_2 . then we derive a subset of the lexica as the parallel anchors a for our method: we only keep those parallel lexica which exist in the embeddings of source and target languages foot_3 . following moschella et al. (2023) , we build relative representations (rrs) for each token in the embedding space based on their similarities with anchor tokens in the respective language. specifically, we compute the cosine similarity of the embedding of each token with the embedding of each anchor token. this computation is done in the embedding spaces of the source and target languages respectively. for example, in the source language, the similarity between token x i and anchor a j is calculated as follows: ## analysis in this section, we want to propose possible reasons for the suboptimal results obtained by our framework tackling the mosecrot task. anchor selection the quality of the parallel anchors largely relies on the quality of the bilingual lexica, which may contain, among others, polysemous words, that may influence the alignment quality. normalization can also be a source of ambiguity. for example, muse converts all words into lowercase, so the word sie can have three meanings in the german-english lexicon: you, she, and they. we (1) only consider one translation (if there are multiple) for each target language word, which may not be the most accurate one; and (2) treat all target language words whose translations are in the source language vocabulary as anchors, which increases the frequency of noisy translation pairs. we try to decrease the influence of potentially noisy anchor pairs by reducing the number of anchors to 3000 and 500 (the original anchor set used during the preliminary experiments contains 6731 anchors, see §4) through random sampling, following the observation by @xcite that uniform selection from an anchor set is both straightforward and has good performance. we also remove stop words, whose translations are more unstable, from the anchor set. neither of the two modifications shows an improvement over the full anchor set (see §b for the comparison). one possible explanation is that the translation qualities vary across anchors and thus we cannot predict the quality of sampled anchors. translation quality we find that a large portion of translations retrieved from panlex are of low quality. this is partly due to panlex using intermediate languages when direct translation is unavailable for the language pair. we filter the translations by empirically setting a threshold to the translation quality scores, available through the api for every translation. nevertheless, we note that a high translation quality score does not guarantee the translation is perfect, and many translations are good despite having low translation quality scores. we believe the lack of high-quality parallel lexica is a possible reason that rrs do not reach their full potential on low-resource languages. reinitialized embedding space our method requires swapping the original plm embeddings with the transformed english rrs before finetuning on english data, whereas the embedding space of rrs might diverge substantially from the original embedding space. as a result, it is unclear whether the rest of the model parameters can be adapted to the new embeddings during fine-tuning, especially on smaller datasets like taxi1500. we thus suggest the alteration of the embedding space through reinitialization with rrs as a likely factor as to why we do not achieve good performance. ## conclusion in this work, we introduce mosecrot, a novel and challenging task that is relevant for, in particular, low-resource languages for which static word embeddings are available but few resources exist. in addition, we propose for the first time a method that leverages relative representations for embedding space mapping and enables zero-shot transfer. specifically, we fine-tune a monolingual english language model using only english data, swap the embeddings with target language embeddings aligned using rrs, and apply zero-shot evaluation on the target language. we show that the proposed method is promising compared with mbert on unseen languages but only modest improvements are achieved. we provide several possible reasons and leave improvement possibilities for future research.