MTLS : Making Texts into Linguistic Symbols (2024)

Abstract: In linguistics, all languages can be considered as symbolic systems, with each language relying on symbolic processes to associate specific symbols with meanings. In the same language, there is a fixed correspondence between linguistic symbol and meaning. In different languages, universal meanings follow varying rules of symbolization in one-to-one correspondence with symbols. Most work overlooks the properties of languages as symbol systems. In this paper, we shift the focus to the symbolic properties and introduce MTLS: a pre-training method to improve the multilingual capability of models by Making Texts into Linguistic Symbols. Initially, we replace the vocabulary in pre-trained language models by mapping relations between linguistic symbols and semantics. Subsequently, universal semantics within the symbolic system serve as bridges, linking symbols from different languages to the embedding space of the model, thereby enabling the model to process linguistic symbols. To evaluate the effectiveness of MTLS, we conducted experiments on multilingual tasks using BERT and RoBERTa, respectively, as the backbone. The results indicate that despite having just over 12,000 pieces of English data in pre-training, the improvement that MTLS brings to multilingual capabilities is remarkably significant.

## introduction all languages can be considered as symbolic systems @xcite . this is a generally accepted linguistic concept. indeed, the meanings of words are prescribed by human convention, and all languages rely on symbolic processes to associate specific symbols with meanings. however, in natural language processing (nlp), languages are often treated as complex systems of word formation methods and syntactic rules, and their inherent properties as symbols are often overlooked. in this paper, we focus on the symbolic properties of languages and conduct a preliminary investigation. we propose mtls: a novel pre-training method to improve the multilingual capability of models by making texts into linguistic symbols foot_0 . remarkably, mtls does not rely on extensive multilingual corpora, massive computational resources, or complex a priori knowledge. it can significantly improve the multilingual performance of pre-trained language models (plms). by making texts into linguistic symbols, it is possible to obtain textual embeddings in any language without vocabulary. pre-trained language models @xcite @xcite , including multilingual models @xcite @xcite , assign ids to tokens via vocabulary and then obtain textual embeddings. language models typ-ically support a finite vocabulary of categorical inputs, e.g. characters, subwords or even words, and much effort has been devoted to vocabulary construction @xcite . as the number of languages the model can handle increases, the vocabulary needs to be extended or reconstructed, which is a burdensome task. taking into account the symbolic properties of languages, linguistic symbols always correspond to concrete abstract semantics and have unique compositions or structures from a visual perspective. in mtls, words are first converted into linguistic symbols and rendered as pixel images. then, by replacing the unique encoding in vocabularies with the linguistic symbols of the words, the problem of multilingual vocabulary construction and out-of-vocabulary (oov) can be avoided, as shown in figure 1(a) . universal meanings are represented by different linguistic symbols in different symbol systems. these universal meanings underlying all human natural languages are referred to as irreducible semantic cores @xcite . this semantic core can be used as a semantic bridge for transformations between different languages. based on this notion, some work (sherborne and lapata, 2022; @xcite has been done to construct multilingual universal representations by finding universal meanings behind natural languages. we transfer this idea to the linguistic symbolic perspective and call this semantic core the meta-symbol system (mss). the mss can be used as a bridge between linguistic symbols of different languages. therefore, we propose to apply the embedding space of the mss to represent the linguistic symbols of any language, and then to construct the mapping from the embedding space of the mss to the embedding space of the plm, as shown in figure 1 @xcite . this means that plms can handle linguistic symbols in any language and do not need to be pre-trained again. in this paper, we propose sss embedding in mtls for obtaining symbolic embeddings of linguistic symbols and then mapping them to the embedding space of the plm. sss embedding consists of three components: symbolic embedding, selective embedding, and spatial embedding. in mtls, the sss embedding and the plm embedding layers are first jointly pre-trained. then, the sss embedding replaces the plm embedding layers, and the multilingual capability of the plm can be improved. we evaluate the performance of mtls on syntactic and semantic pro-cessing tasks in multiple languages. the results show that mtls can significantly improve the multilingual capabilities of the model, but at the cost of some performance degradation in latin script languages. we release the code and models at https://github.com/wenlong1019/mtls . ## related work multilingual representation learning is studied for a variety of downstream multilingual tasks. some approaches rely on rich multilingual corpora to learn multilingual representations through extensive pre-training, such as mbert @xcite and xlm-r @xcite . such extensive pre-training can be extremely computationally intensive, so some work @xcite has relied on adapterbased methods to fine-tune only the adapter layer during training. other approaches utilize multilingual parallel corpora to learn generic representations of parallel sentences based on contrastive learning, such as infoxlm @xcite and hictl @xcite . however, obtaining a high quality parallel corpus is difficult, so some work has converted monolingual corpora into parallel corpora through translation @xcite , or computed semantic similarities in multilingual corpora and used them as supervised labels @xcite . all these approaches rely heavily on multilingual corpora, while mtls uses only monolingual data. a further theoretical comparison between mtls and these methods is given in appendix a. vision-based embedding differs from mainstream vocabulary-based embedding in that it generates embeddings based on the structure and construction of the text from a visual perspective. some work @xcite has used vision-based embedding to obtain some potential features of hieroglyphs on writing systems. there is also some work exploring the potential of vision-based embedding. @xcite proposed a fully vision-based plm and obtained results competitive with @xcite . @xcite used vision-based embedding and vocabulary-based embedding to construct a two-tower model to combine the advantages of the two embedding construction methods. the work in this paper is also an exploration of the potential of vision-based embedding. ## methodology figure 2 shows an overview of mtls. mtls uses the sss embedding and the frozen embedding layer of the plm for joint pre-training. then, by simply replacing the embedding layer in the plm with sss embedding, the multilingual capability of the plm can be improved. ## conclusion in this paper, we explore the properties of languages as symbolic systems and propose mtls: a pre-training method to improve the multilingual capability of models by making texts into linguistic symbols. we also propose sss embedding, which can obtain the symbolic embedding of any language and map the symbolic embedding to the embedding space of the plm. by replacing the plm embedding layer with sss embedding, the model can be made to process linguistic symbols in any language. it should be emphasized that the pre-training of mtls requires only a small amount of monolingual data and does not require multilingual corpora. in addition, mtls reuses most of the parameters in the plm, so pre-training costs few computational resources. our experimental results show that mtls can significantly improve the multilingual capability of plms, at the cost of some performance degradation in latin-script languages.