Multilingual Pre-training with Self-supervision from Global Co-occurrence Information (2023)

Abstract: Global co-occurrence information is the primary source of structural information on multilingual corpora, and we find that analogical/parallel compound words across languages have similar co-occurrence counts/frequencies (normalized) giving weak but stable self-supervision for cross-lingual transfer. Following the observation, we aim at associating contextualized representations with relevant (contextualized) representations across languages with the help of co-occurrence counts. The result is MLM-GC (MLM with Global Co-occurrence) pre-training that the model learns local bidirectional information from MLM and global co-occurrence information from a log-bilinear regression. Experiments show that MLM-GC pre-training substantially outperforms MLM pre-training for 4 downstream cross-lingual tasks and 1 additional monolingual task, showing the advantages of forming isomorphic spaces across languages.

## introduction empirical studies @xcite show multilinguality and crosslinguality emerge from mlm pre-training on multilingual corpora without any supervision. the model is trained/pre-trained as a generator that yields masked token probabilities over the vocabulary. to improve cross-lingual transfer, we present mlm-gc (mlm with global co-occurrence) with the combined objective of the generator and a global log-bilinear regression for multilingual pretraining. our starting point is from two observations on multilingual mlm pre-training. language's structural information is every property of an individual language that is invariant to the script of the language. @xcite ; @xcite ; @xcite ; @xcite show that structural similarities across languages can contribute to cross-lingual transfer. co-occurrence information or n-gram is the primary source of structural information available to all methods. some methods like span-based masking @xcite @xcite now exist to leverage this information for new masking schemes in monolingual mlm pre-training, aiming at improving context understanding. however, in multilingual mlm pre-training, the question still remains as to how meaning is generated from these statistics on multilingual corpora, how the structural similarities could be learned from that meaning across languages, and how cross-lingual transfer might be improved from that meaning. furthermore, glove @xcite ) prove that leveraging global co-occurrence information can search for relevant information on monolingual embedding space. inspired by glove, we assume that global co-occurrence information can also be used to search for relevant information across languages on multilingual corpora. this assumption underlies zipf's law @xcite ) that analogical words and compound words across languages might have similar frequencies/counts on the multilingual corpora. our empirical studies further justify our assumption that analogical/parallel compound words across languages have similar co-occurrence counts (normalized). meanwhile, in multilingual mlm pre-training, one of the ultimate goals is to form contextualized representations. then, global co-occurrence information might be used to regularize representation learning in multilingual mlm pre-training, which allows for better contextualized representations in cross-lingual transfer. in this work, we present mlm-gc to utilize global co-occurrence information. mlm-gc builds on mlm with an extra objective of global log-bilinear regression that minimizes the error between dot products of neighboring contextualized representations and the matrix of global co-occurrence counts. since mlm only needs to predict masked tokens, we only consider the contextualized representations of the masked tokens and their neighbors, factorizing relevant global cooccurrence counts from the matrix. the model is pre-trained to learn bidirectional information from mlm and the global co-occurrence information from the global log-bilinear regression. on multilingual corpora, mlm-gc pre-training can improve cross-lingual transfer because analogical/parallel compound words across languages might have similar co-occurrence counts allowing for cross-lingual transfer, which is justified in our empirical studies on translation pairs. we have three contributions. 1) we present mlm-gc pre-training for multilingual tasks. the model is additionally supervised by co-occurrence counts on multilingual corpora. 2) mlm-gc pre-training outperforms mlm pre-training on 4 multilingual/cross-lingual tasks. the objective of mlm-gc can be adapted to encoder-decoderbased mlm models, e.g., @xcite and encoder-based mlm models, e.g., xlm (lample and @xcite . mlm-gc pre-training can also work on monolingual corpora for language understanding tasks. 3) mlm-gc pre-training can help the model to form isomorphic embedding spaces across languages, which is potentially useful for cross-lingual and multilingual tasks. our empirical study shows that analogical compound words across languages have similar co-occurrence counts (normalized) contributing to structural similarities across languages for cross-lingual transfer. ## related work and comparison structural similarity and zipf's law zipf's law @xcite @xcite indicates that words or phrases appear with different frequencies, and one may suggest analogical words or phrases appear with relatively similar frequencies in other languages. in multilingual mlm pre-training, @xcite al. (2020c); karthikeyan et al. (2020); pires et al. (2019); karthikeyan et al. (2020); sinha et al. (2021) shed light on studying structural information and find that structural similarities across languages are essential for multilinguality, where in this paper, structural similarities mean similar counts as zipf's law indicated. we follow this line, consider structural similarities from co-occurrence counts, and provide an empirical study to observe how the model learns structural similarities from global co-occurrence counts on multilingual corpora. meanwhile, glove @xcite report that co-occurrence counts can provide regularities for embeddings to understand word analogies for monolingual tasks. we extend the scope of glove to contextualized representations and multilingual tasks, helping the model form isomorphic spaces across languages in multilingual mlm pre-training. n-gram, co-occurrence, and regularity in mlm pre-training studying co-occurrence or n-gram is not a novel idea in mlm pre-training. whole word masking @xcite , span-bert @xcite , and pmi-masking @xcite suggest n-gram spans across several sub-tokens for masking to improve context understanding in monolingual tasks because the model may only learn from easier multi-tokens instead of usefully hard context, where easier multitokens are in a subset of the context and result in sub-optimization. in contrast, we show that cooccurrence counts can refine contextualized representations for improving context understanding and allow for better cross-lingual transfer, suggesting a new objective for mlm pre-training instead of a new masking scheme to capture global cooccurrence information in multilingual pre-training. on the other hand, for cross-lingual transfer, the contextualized representations could be further regularized and refined by aligning cherry-picked pairs after mlm pre-training on multilingual corpora @xcite @xcite @xcite @xcite . compared to that, mlm-gc pre-training does not require dictionaries, translation tables, or statistical machine translation models. ## experiment all the links of datasets, libraries, scripts, and tools marked with ⋄ are listed in appendix e. a preview version of the code is submitted, and we will open the source code on github. ## conclusion in this work, we leverage the global co-occurrence information from multilingual corpora. the result is mlm-gc pre-training with a combined objective of mlm and global co-occurrence modeling. our experiments show that mlm-gc pre-training can substantially improve the performance of naive mlm pre-training for 4 multilingual tasks, and additional experiments show that it can work for monolingual tasks. the isomorphic space across languages benefits from co-occurrence information, which allows for cross-lingual transfer. meanwhile, the model is encouraged to distinguish relevant information from irrelevant information and to discriminate between the two relevant information across languages from co-occurrence counts (normalized) and refine contextualized representations accordingly. we believe that leveraging cooccurrence information for cross-lingual transfer is an interesting avenue in multilingual pre-training.