Lightweight Cross-Lingual Sentence Representation Learning (2021)

Abstract: Large-scale models for learning fixed-dimensional cross-lingual sentence representations like LASER (Artetxe and Schwenk, 2019b) lead to significant improvement in performance on downstream tasks. However, further increases and modifications based on such large-scale models are usually impractical due to memory limitations. In this work, we introduce a lightweight dual-transformer architecture with just 2 layers for generating memory-efficient cross-lingual sentence representations. We explore different training tasks and observe that current cross-lingual training tasks leave a lot to be desired for this shallow architecture. To ameliorate this, we propose a novel cross-lingual language model, which combines the existing single-word masked language model with the newly proposed cross-lingual token-level reconstruction task. We further augment the training task by the introduction of two computationally-lite sentence-level contrastive learning tasks to enhance the alignment of cross-lingual sentence representation space, which compensates for the learning bottleneck of the lightweight transformer for generative tasks. Our comparisons with competing models on cross-lingual sentence retrieval and multilingual document classification confirm the effectiveness of the newly proposed training tasks for a shallow model.

## introduction cross-lingual sentence representation models @xcite @xcite @xcite @xcite @xcite ; li 1 https://github.com/mao-ku/ lightweight-crosslingual-sent2vec and mak, 2020) learn language-agnostic representations facilitating tasks like cross-lingual sentence retrieval (xsr) and cross-lingual knowledge transfer on downstream tasks without the need for training a new monolingual representation model from scratch. thus, such models benefit from an increased amount of data during training and lead to improved performances for low-resource languages. the above-mentioned models can be categorized into two classes. on one hand, global fine-tuning methods like mbert @xcite and xlm @xcite require being fine-tuned globally which results in a significant overhead of its own. on the other hand, fixeddimensional methods like laser @xcite ) fix the sentence representations during the pre-training phase, and subsequently the fine-tuning for specific downstream tasks without back-propagating to the pre-trained model will be extremely computationally-lite. lightweight models have been sufficiently explored for the former group by either shrinking the model @xcite or training a student model @xcite @xcite . however, the lightweight models for the latter group have not been explored before, which may have a more promising future for deploying task-specific fine-tuning onto edge devices. in this work, we propose a variety of training tasks for a lightweight cross-lingual sentence model while retaining the robustness. to improve the computational efficiency, we utilize a lightweight dual-transformer architecture with just 2 layers, significantly decreasing the memory consumption and accelerating the training to further improve the efficiency. our model uses significantly less number of parameters compared to both global fine-tuning methods like mbert, and fixeddimensional representation methods like laser, method architecture d h d f c attn h enc. dec. params. given a fixed training-set and model architecture, the robustness of the sentence representation is dependent on the training task. it is much more difficult for a lightweight model to learn robust representations merely with existing generative tasks (see section 2 and section 4.5), which could be attributed to its smaller size. in order to ameliorate this problem, we redesign a crosslingual language model by combining the singleword masked language model (smlm) with crosslingual token-level reconstruction (xtr). furthermore, we introduce two contrastive learning methods as auxiliary tasks to compensate for the learning bottleneck of lightweight transformer for generative tasks. following the state-of-the-art fixeddimensional model laser, we proceed to learn cross-lingual sentence representations from parallel sentences, where we employ 2-layer dualtransformer encoders to shrink the model architecture. by introducing the above-stated training tasks, we establish a computationally-lite framework for training cross-lingual sentence models. we evaluate the learned sentence representations on cross-lingual tasks including multilingual document classification (mldoc) @xcite and xsr. our results confirm the ability of our lightweight model to yield robust sentence representations. we also do a systematic study on the performance of our model in an ablative manner. the contributions of this work can be summarized as follows: ## related work a majority of training tasks for learning fixeddimensional cross-lingual sentence representations can be ascribed to one of the following 2 categories: generative or contrastive. in this section, we revisit the previous work in these 2 categories, which is crucial for designing a cross-lingual representation model. generative tasks. generative tasks measure a generative probability between predicted tokens and real tokens by training a language model. bert-style mlm @xcite masks and predicts contextualized tokens within a given sentence. for the cross-lingual scenario, cross-lingual supervision is implemented by shared cognates and joint training @xcite , concatenating source sentences in multiple languages @xcite or explicitly predicting the translated token @xcite . the [cls] embedding or pooled embedding of all the tokens is introduced as the classifier embedding, which can be used as sentence embedding for sentence-level tasks @xcite . sequence to sequence methods @xcite @xcite ) autoregressively reconstruct the translation of the source sentence. the intermediate state between the encoder and the decoder are extracted as sentence representations. particularly, the cross-lingual sentence representation quality of laser @xcite benefits from a massively multilingual machine translation task covering 93 languages. in our work, we revisit the bert-style training tasks and introduce a novel generative loss enhanced by kl-divergence based token distribution prediction. our proposed generative task performs effectively for the lightweight dual-transformer framework while other generative tasks should be implemented via a large-capacity model. contrastive tasks. contrastive tasks measure (contrast) the similarities of sample pairs in the representation space. negative sampling, which is a typical feature of the contrastive methods is first introduced in the work of word representation learning @xcite . subsequently, contrastive tasks gradually emerged in many nlp tasks in various ways: negative sampling in knowledge graph embedding learning @xcite , next sentence prediction in bert @xcite , token-level discrimination in electra @xcite , sentence-level discrimination in declutr @xcite , and hierarchical contrastive learning in hictl @xcite . for the cross-lingual sentence representation training, typical ones include using correct and wrong translation pairs introduced by @xcite ; @xcite ; @xcite ; @xcite or utilizing similarities between sentence pairs by introducing a regularization term @xcite . as another advantage, contrastive methods have proven to be more efficient than generative methods @xcite . inspired by previous work, for our lightweight model, we propose a robust sentence-level contrastive task by leveraging similarity relationships arising from translation pairs. ## methodology we perform cross-lingual sentence representation learning by a lightweight dual-transformer framework. concerning the training tasks, we propose a novel cross-lingual language model, which combines smlm and xtr. moreover, we introduce two sentence-level self-supervised learning tasks (sentence alignment and sentence similarity losses) to leverage robust parallel level supervision to better conduct the cross-lingual sentence representation space alignment. ## experiments we evaluate our cross-lingual sentence representation models by cross-lingual document classification and bitext mining for these 2 main downstream tasks belong to 2 groups: unrelated and related to the training task. for the former, we select ml-doc @xcite to evaluate the classifier transfer ability of the cross-lingual model, while for the latter we conduct sentence retrieval on another parallel dataset europarl foot_5 to evaluate the performance of our models. ## conclusion in this paper, we presented a lightweight dualtransformer based cross-lingual sentence representation learning method. for the fixed 2-layer dualtransformer framework, we explored several generative and contrastive tasks to ensure the sentence representation quality and facilitate the improvement of the training efficiency. in spite of the lightweight model capacity, we reported substantial improvements on mldoc compared to fixeddimensional representation methods and we obtained comparable results on xsr.