Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer (2025)

Abstract: Large Language Models (LLMs) are increasingly incorporating multilingual capabilities, fueling the demand to transfer them into target language-specific models. However, most approaches, which blend the source model’s embedding by replacing the source vocabulary with the target language-specific vocabulary, may constrain expressive capacity in the target language since the source model is predominantly trained on English data. In this paper, we propose Semantic Aware Linear Transfer (SALT), a novel cross-lingual transfer technique that recycles embeddings from target language Pre-trained Language Models (PLMs) to transmit the deep representational strengths of PLM-derived embedding to LLMs. SALT derives unique regression lines based on the similarity in the overlap of the source and target vocabularies to handle each non-overlapping token’s embedding space. Our extensive experiments show that SALT significantly outperforms other transfer methods, achieving lower loss and faster convergence during language adaptation. Notably, SALT achieves remarkable performance in cross-lingual understanding setups compared to other methods. Furthermore, we highlight the scalable use of PLMs to enhance the functionality of contemporary LLMs by conducting experiments with varying architectures.

## introduction as large language models (llms) continue to demonstrate remarkable performance across various knowledge-based benchmarks and sub-tasks, the demand for robust linguistic capabilities in multilingual or language-specific contexts has surged @xcite @xcite . however, most multilingual-considered llms show limited practicality in target languages due to their english- † corresponding author centric training and reliance on common vocabulary designed to incorporate a wide range of languages @xcite @xcite @xcite . this drawback also correlates with the large portion of their parameters, which are the embeddings of irrelevant language tokens to target language. to address this challenge, a promising line of studies has focused on cross-lingual transfer @xcite @xcite @xcite @xcite . these efforts include replacing a source vocabulary with a target language vocabulary, simply initializing embeddings with newly manipulated embeddings stem from source model @xcite @xcite . however, a newly initialized embedding that relies solely on the source model inevitably experiences limited expressiveness in the target language due to its immature learning process and less consideration of the target language. moreover, the majority of approaches adopt small encoder architectures such as bert-like models @xcite @xcite ) as a source model, leaving their applicability on contemporary decoder-based llms unexplored @xcite @xcite @xcite . in this work, we propose semantic aware linear transfer (salt), a novel cross-lingual transfer method that leverages the rich representational power of target language embeddings in conventional pre-trained language models (plms) 1 to convert english-centric llms into target language-specialized llms. to facilitate the transfer, we identify the semantically closest tokens for each non-shared token from a shared vocabulary space. based on these similar tokens, we fit a unique linear regression to transfer the embeddings from the plm's space to the llm's space. this procedure enables non-shared vocabulary to be transferred to the llm space while retaining the semantic representation embedded in the plm embedding. in our experiments, we focus on investigating the potential of transferred embeddings from plms in three aspects: (i) whether the transferred embeddings can be well-aligned with inherent knowledge in the source model, (ii) the influence on better initialization to the target language and convergence during continual pre-training, and (iii) the crosslingual understanding ability between target language and mainstream language. we empirically demonstrate that embeddings from target language plms can be more helpful for cross-lingual transfer in llms than existing methods. salt not only surpasses various strong baselines in downstream tasks but also provides a better initialization for target language adaptation with faster convergence during continual pretraining. notably, we find that salt preserves english capability more effectively while maintaining alignment between english and the target language in a cross-lingual setup. furthermore, by conducting additional studies on various plm architectures (encoder, decoder, and encoder-decoder), we discover that salt can be extended to plms with various architectures and may become a valid strategy for basic models such as bert @xcite . our contributions are as follows: ## related work cross-lingual transfer to a target language includes approaches that manipulate the vocabulary or initialize embedding for the new target vocabulary. in vocabulary manipulation, most research has considered adding target language relevant vocabulary @xcite @xcite @xcite @xcite . vocabulary expansion is widely adopted for developing language-specific models from a source model @xcite @xcite . this line of work requires access to a large amount of target language data. in light of this, @xcite delve into cross-lingual vocabulary expansion in lowresource settings across initialization approaches and training strategies. also, @xcite investigate training scales required for vocab extension and the influence of transfer on capabilities of language generation and following instructions. on the other hand, given that overall cost increase derived from extended vocabulary, numerous studies have pursued replacing the original vocabulary with a new target vocabulary @xcite @xcite @xcite . they focus on initializing new target embeddings by manipulating source embeddings according to semantic similarities between vocabularies. for instance, @xcite compresses the source model's vocabulary using an averaging-based technique to accommodate domain-specific terms. more recently, lines of work use source embedding and well-aligned external word embeddings to initialize new subword embeddings for a targetspecialized vocabulary @xcite @xcite . notably, focus @xcite computes a weighted mean from a multilingual language model, guided by token similarities from fast-text @xcite , to obtain new embeddings. similarly, ofa @xcite adopts a comparable strategy to build multilingual models and proposes a factorization-based dimension reduction embedding transfer method for efficiency. as a hybrid method, @xcite combines the source model's embedding under a statistical translation scheme across languages. however, initializing embedding relying on the se e(street) e(road) e(traffic) e(sidewalk) e(street) e(pedestrian) e(road) e(traffic) traffic street sidewalk unique shared similarity ... road traffic street ... road traffic street ... ... linear transfer ... e(sidewalk) sidewalk sidewalk road figure 1: summary of salt. by using the paired embeddings of semantically similar tokens for each non-shared token, we create a unique least square matrix x ti to transfer from plm to llm. k denotes the number of selected nearest tokens among overlapping tokens for linear transfer, and h t and h s refer to the hidden dimensions of the target and source models. ## salt our design is based on the assumption that embeddings from target language-specialized plms, mainly pre-trained on a target language corpus, may have richer semantic information than those from llms trained in a multilingual context with imbalanced language consideration. we explore the way of transferring embeddings from the plm to the source llm and aim to enhance generalization on various tasks within the target language. we show the summary of salt framework in figure 1 and describe the process step by step as follows: step 0: objective given source llm (vocabulary embedding e s ) with its vocabulary v s and target language plm (vocabulary embedding e t ) with its vocabulary v t , we want to make newly initialized target language embedding for v t to replace source llm's embedding while minimizing contextual misalignment and maintaining semantically rich information inherent in e t . step 1: subword embedding extraction for each v s and v t , we use external static embeddings from fasttext @xcite to extract auxiliary embeddings. we select fasttext trained in each target language, as it can provide embeddings of various words using combinations of multiple subwords. for rare tokens that do not exist in fast-text or cannot be formed via subword combinations, we initialize them from a normal distribution with mean and standard deviation from e s . ## results we present downstream task performance for knowledge-based benchmarks in section 5.1. in section 5.2, we show the training loss curves during language adaptive continual pre-training with each transfer method. we also report mrc performance in a monolingual and cross-lingual setup in section 5.3. additionally, we verify the extensibility of our approach by examining the performance when using various types of target language-specialized plms beyond the decoder type in section 5.4. ## conclusion we propose salt, a novel cross-lingual transfer method that recycles the embeddings of plms for transferring from english-centric llms to target language-specialized llms. by deriving a unique linear regression line that takes into account the semantic similarity of each non-overlapping vocabulary embedding, we map the rich semantic features from the plm's embedding space onto that of the llm. a series of experiments reveals that salt benefits from the rich expressiveness of plm's embedding in the target language. furthermore, salt promotes better initialization and faster convergence for language adaptation, enhancing generalization to the target language. notably, among existing initialization methods, salt preserves english capabilities more effectively and demonstrates greater superiority in cross-lingual setups. we also investigate expandability by experimenting with multiple types of target models, indicating that salt can be applied to various plm architectures. for future work, we plan to explore methods that facilitate the transfer of languagespecific knowledge embedded within model layers.