LEALLA : Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation (2023)

Abstract: Large-scale language-agnostic sentence embedding models such as LaBSE (Feng et al., 2022) obtain state-of-the-art performance for parallel sentence alignment. However, these large-scale models can suffer from inference speed and computation overhead. This study systematically explores learning language-agnostic sentence embeddings with lightweight models. We demonstrate that a thin-deep encoder can construct robust low-dimensional sentence embeddings for 109 languages. With our proposed distillation methods, we achieve further improvements by incorporating knowledge from a teacher model. Empirical results on Tatoeba, United Nations, and BUCC show the effectiveness of our lightweight models. We release our lightweight language-agnostic sentence embedding models LEALLA on TensorFlow Hub.

## introduction language-agnostic sentence embedding models @xcite @xcite @xcite align multiple languages in a shared embedding space, facilitating parallel sentence alignment that extracts parallel sentences for training translation systems @xcite . among them, labse @xcite achieves the state-of-the-art parallel sentence alignment accuracy over 109 languages. however, 471m parameters of labse lead to the computationallyheavy inference. the 768-dimensional sentence embeddings of labse (labse embeddings) make it suffer from computation overhead of downstream tasks (e.g., knn search). this limits its application on resource-constrained devices. therefore, we explore training a lightweight model to generate low-dimensional sentence embeddings while retaining the performance of labse. we first investigate the performance of dimension-reduced labse embeddings and show that it performs comparably with labse. subsequently, we experiment with various architectures to see whether such effective low-dimensional embeddings can be obtained from a lightweight encoder. we observe that the thin-deep @xcite architecture is empirically superior for learning language-agnostic sentence embeddings. diverging from previous work, we show that lowdimensional embeddings based on a lightweight model are effective for parallel sentence alignment of 109 languages. labse benefits from multilingual language model pre-training, but no multilingual pre-trained models are available for the lightweight architectures. thus, we propose two knowledge distillation methods to further enhance the lightweight models by forcing the model to extract helpful information from labse. we present three lightweight models improved with distillation: lealla-small, lealla-base, and lealla-large, with 69m, 107m, and 147m parameters, respectively. fewer model parameters and their @xcite sentence embeddings are expected to accelerate downstream tasks, while the performance drop of merely up to 3.0, 1.3, and 0.3 p@1 (or f1) points is observed on three benchmarks of parallel sentence alignment. in addition, we show the effectiveness of each loss function through an ablation study. ## light language-agnostic embeddings to address the efficiency issue of labse, we probe the lightweight model for learning languageagnostic embeddings with the following experiments: (1) we directly reduce the dimension of labse embeddings to explore the optimal embedding dimension; (2) we shrink the model size with various ways to explore the optimal architecture. ## knowledge distillation from labse besides the large model capacity, multilingual language model pre-training benefits labse for parallel sentence alignment. as no multilingual pre-trained language models are available for lightweight models we investigated in section 3.3, we instead explore extracting helpful knowledge from labse. ## conclusion we presented lealla, a lightweight model for generating low-dimensional language-agnostic sentence embeddings. experimental results showed that lealla could yield solid performance for 109 languages after distilling knowledge from labse. future work can focus on reducing the vocabulary size of labse to shrink the model further and exploring the effectiveness of lightweight model pre-training for parallel sentence alignment.