Measuring Cross-lingual Transfer in Bytes (2024)

Abstract: Multilingual pretraining has been a successful solution to the challenges posed by the lack of resources for languages. These models can transfer knowledge to target languages with minimal or no examples. Recent research suggests that monolingual models also have a similar capability, but the mechanisms behind this transfer remain unclear. Some studies have explored factors like language contamination and syntactic similarity. An emerging line of research suggests that the representations learned by language models contain two components: a language-specific and a language-agnostic component. The latter is responsible for transferring a more universal knowledge. However, there is a lack of comprehensive exploration of these properties across diverse target languages. To investigate this hypothesis, we conducted an experiment inspired by the work on the Scaling Laws for Transfer. We measured the amount of data transferred from a source language to a target language and found that models initialized from diverse languages perform similarly to a target language in a cross-lingual setting. This was surprising because the amount of data transferred to 10 diverse target languages, such as Spanish, Korean, and Finnish, was quite similar. We also found evidence that this transfer is not related to language contamination or language proximity, which strengthens the hypothesis that the model also relies on language-agnostic knowledge. Our experiments have opened up new possibilities for measuring how much data represents the language-agnostic representations learned during pretraining.

## introduction the emergence of self-supervised pretraining models such as bert has revealed a notable phenomenon of cross-lingual transfer even when these models are trained on multilingual corpora devoid of paired translation examples. for example, llama @xcite , which was trained self-supervisedly on an english-centric corpus, exhibits surprising multilingual capabilities @xcite . the underlying mechanisms driving this behavior remain unclear, with hypotheses ranging from the presence of shared "anchor" tokens @xcite to language contamination @xcite , yet no scientific consensus has been reached. research in this area often involves the use of pre-existing language models (lms), which are subsequently finetuned on supervised datasets in different languages @xcite . however, when evaluating multiple languages, conventional methodologies encounter two significant challenges: firstly, the dependence on supervised finetuning datasets, which often vary in size and quality, complicating cross-lingual comparisons; secondly, the use of subword tokenizers, which do not represent all languages equally. in this work, we avoid these problems by working with a byte-level tokenizer and by using auto-regressive language models trained in selfsupervised from scratch in one language and then finetuned on another. to measure the effect of transfer learning, we employ the concept of data transfer @xcite , which allows us to quantify how much each different source language contributes to the perplexity of the target language. our main contribution is providing a method that measures how much knowledge, in bytes, is transferred from one language to another. by applying it, our findings reveal a surprising trend: even when comparing linguistically distant languages, the data transfer metrics are of a comparable magnitude. this research contributes additional evidence supporting the language-agnostic hypothesis, which suggests that the internal representations developed by a model are not only influenced by the linguistic surface form but also by the cultural and semantic content of the training data. ## related work prior work attributed the success of multilingual models in cross-lingual transfer to "anchor" tokens @xcite . however, subsequent research demonstrated that models could perform well even without these tokens @xcite , highlighting the significance of shared parameters during training @xcite . competitive results were achieved by monolingual models with minimal or no adaptation @xcite . investigations by @xcite linked these findings to language contamination, where pretraining datasets contained target language data. additional factors contributing to cross-lingual transfer success include dataset statistics, language attributes @xcite , language structure @xcite @xcite , and token overlap between training and target languages @xcite . the role of language script @xcite and model tokenizer @xcite was also noted, prompting the use of a byte tokenizer to address these issues @xcite . recent research proposed a two-component model representation hypothesis-language agnostic and language specific @xcite @xcite . while promising, no study has measured how much of the language-agnostic component is used in settings with multiple source and target languages. additionally, existing research still applies the source language vocabulary to the target language, potentially compromising input representations and affecting results. to address these gaps, we draw on @xcite and employ a byte vocabulary in our experiments to overcome current literature limitations. ## methodology inspired by @xcite , our methodology focuses on quantifying the transferability of pretraining data across distributions, particularly between different languages. we select a target language and finetune models initialized from various source languages onto it. subsequently, we evaluate each model on a target language test set and compare their performance. we introduce the data transfer (d t ) metric to estimate knowledge transfer, explained in more depth in 3.1. this process is repeated across different target languages to observe effects across a broad linguistic spectrum. these experiments aim to quantify and compare cross-lingual knowledge transfer from different source languages. this analysis seeks to uncover the extent to which transferability depends on specific languages and the importance of languageagnostic components in learned representations. the following sections provide further details. ## experiments this section presents the languages, datasets, model architecture, and training details for our experiments. ## results results are compiled in given our methodology, where we vary the source language while keeping the target language constant to assess the impact of pretraining language in cross-lingual scenarios, it is essential to read the table vertically unless stated otherwise. each row represents the results obtained by finetuning the model from a specific source language to a given target language (indicated in the column). this vertical arrangement facilitates the comparison of model performance across different source languages for the same target language. furthermore, test sets vary significantly for each language due to the nature of the mc4 dataset. consequently, results across target languages are not comparable. throughout this section, we highlight findings from models finetuned on 6 million tokens (i.e., d @xmath0. 4 this extreme scenario tests models with minimal target language resources. ## discussion our study aims to measure how much knowledge, in bytes, is transferred from one language to another, enabling the investigation of the effectiveness of language-agnostic representations acquired during pretraining in cross-lingual scenarios. we hypothesize that these representations enable models to perform well on downstream tasks across diverse languages, which is observed in state-ofthe-art multilingual models. in our results, we consistently find that at least two source languages demonstrate very close d t values when evaluated against a target language, despite the diverse set of script systems and linguistic characteristics involved. this observation suggests that the data transferred from these languages to the target language is not primarily related to language-specific components but rather to language-agnostic ones. for instance, as illustrated in figure 3 , both english and russian, despite being known as distant languages, achieve nearly identical d t values when evaluated on korean, a language distinct from either. moreover, all three source languages are remarkably close when evaluated on the japanese test set. despite exposure to only a few tokens in the target language, our models demonstrate similar perplexity performance, indicating high adaptability and generalization across a broad range of languages. this reinforces the notion that the language-agnostic component plays a crucial, uniform role across source languages. notably, our results are not attributed to pretraining exposure to target languages, since there is a weak correlation of language contamination with the data transfer coefficient. additionally, the observed performance is not solely dependent on language proximity, as suggested in other works. while perplexity offers valuable insights, generalizable conclusions require evaluation in downstream tasks, especially in under-resourced languages. to explore this further, we conducted a small-scale experiment, detailed in appendix a, finetuning our pretrained models for a low-resource language inference task. surprisingly, we observe comparable accuracy scores between portuguese and russian, with good results also for english and chinese, suggesting that our initial findings with perplexity may extend broadly. however, a more detailed investigation is required to understand cross-lingual transfer mechanisms fully. the novelty of our approach is employing a byte-level tokenizer and adapting @xcite for a cross-lingual scenario. the bytelevel approach facilitates consistent model embeddings across diverse scripts, enabling effective cross-lingual knowledge transfer without languagespecific tokenization or preprocessing. this is supported by the strong performance of byt5 compared to mt5 in @xcite . in conclusion, our study suggests the presence of language-agnostic representations contributing to cross-lingual transferability, while also laying the foundation to measure it through the data transfer metric. the observed consistency in model performance across diverse languages, facilitated by the byte-level tokenizer, indicates the potential for more efficient and generalizable natural language understanding across linguistic boundaries in computational linguistics and nlp. ## limitations our study has certain limitations that merit consideration. firstly, our choice of initializing models with only three languages, while diverse, leaves room for improvement. including additional languages in the pretraining phase would enhance the robustness of our analysis by minimizing possible bias towards the selected languages while providing more samples for our correlation analysis. however, this expansion would necessitate a more substantial computational budget. secondly, our reliance on small models, specifically a 65 million parameter model, limits the scope of our findings as larger models may exhibit different behavior. additionally, the capacity of very large models for few-shot learning opens avenues for further exploration in the domain of transfer learning. lastly, the heterogeneity of the mc4 dataset across languages introduces a potential source of variability in the models' exposure to different knowledge. while the impact of this variation on data transfer remains unclear, conducting experiments with controlled datasets would offer valuable insights. moreover, employing a more comparable test set could help mitigate statistical variance, particularly in analyses such as the commutative property assessment. ## conclusion and future work our study delves into the transferability of knowledge in cross-lingual scenarios, leveraging a bytelevel tokenizer and an adapted methodology inspired by @xcite . by measuring the models' reliance on pretraining when executing tasks in diverse languages, our approach offers an understanding of the cross-lingual capabilities of language models. the results provide evidence that language-agnostic representations also play an important role in downstream tasks. this not only contributes to the current understanding of cross-lingual transferability but also serves as a catalyst for further exploration into the properties of language-agnostic knowledge transfer. for future research directions, we envision key investigations that can build upon the insights presented in this paper: