EASE : Entity-Aware Contrastive Learning of Sentence Embedding (2022)

Abstract: We present EASE, a novel method for learning sentence embeddings via contrastive learning between sentences and their related entities. The advantage of using entity supervision is twofold: (1) entities have been shown to be a strong indicator of text semantics and thus should provide rich training signals for sentence embeddings; (2) entities are defined independently of languages and thus offer useful cross-lingual alignment supervision. We evaluate EASE against other unsupervised models both in monolingual and multilingual settings. We show that EASE exhibits competitive or better performance in English semantic textual similarity (STS) and short text clustering (STC) tasks and it significantly outperforms baseline methods in multilingual settings on a variety of tasks. Our source code, pre-trained models, and newly constructed multi-lingual STC dataset are available at https://github.com/studio-ousia/ease .

## introduction the current dominant approach to learning sentence embeddings is fine-tuning general-purpose pretrained language models, such as bert @xcite , with a particular training supervision. the type of supervision can be natural language inference data @xcite , adjacent sentences @xcite , or a parallel corpus for multilingual models @xcite . in this paper, we explore a type of supervision that has been under-explored in the literature: entity hyperlink annotations from wikipedia. their advantage is twofold: (1) entities have been shown to be a strong indicator of text semantics @xcite @xcite and thus should provide rich training signals for sentence embeddings; (2) entities are defined independently of languages and figure 1 : illustration of the main concept behind ease. using a contrastive framework, sentences are embedded in the neighborhood of their hyperlink entity embeddings and kept apart from irrelevant entities. here, we share the entity embeddings across languages for multilingual models to facilitate cross-lingual alignment of the representation. ease learns sentence embeddings with two types of objectives: (1) our novel entity contrastive learning (cl) loss between sentences and their related entities (figure 1 ); (2) the self-supervised cl loss with dropout noise. the entity cl objective pulls the embeddings of sentences and their related entities close while keeping unrelated entities apart. the objective is expected to arrange the sentence embeddings in accordance with semantics captured by the entities. to further exploit the knowledge in wikipedia and improve the learned embeddings, we also introduce a method for mining hard negatives based on the entity type. the second objective, the self-supervised cl objective with dropout noise @xcite , is combined with the first one to enable sentence embeddings to capture fine-grained text semantics. we evaluate our model against other state-of-the-art unsupervised sentence embedding models, and show that ease exhibits competitive or better performance on semantic textual similarity (sts) and short text clustering (stc) tasks. we also apply ease to multilingual settings. to facilitate the evaluation of the high-level semantics of multilingual sentence embeddings, we construct a multilingual text clustering dataset, mewsc-16 (multilingual short text clustering dataset for news in 16 languages). multilingual ease is trained using the entity embeddings shared across languages. we show that, given the cross-lingual alignment supervision from the shared entities, multilingual ease significantly outperforms the baselines in multilingual sts, stc, parallel sentence matching, and cross-lingual document classification tasks. we further demonstrate the effectiveness of the multilingual entity cl in a more realistic scenario for low-resource languages. using multilingual entity cl, we fine-tune a competitive multilingual sentence embedding model, labse @xcite , and show that the tuning improves the performance of parallel sentence matching for lowresource languages under-supported by the model. finally, we analyze the ease model by studying ablated models and the multilingual properties of the sentence embeddings to shed light on the source of the improvement in the model. ## model and training data in this section, we describe the components of our learning method for sentence embeddings, ease, which is trained using entity hyperlink annotations available in wikipedia. ## experiment: monolingual we first evaluate ease in monolingual settings. we fine-tune monolingual pre-trained language models using only english wikipedia data. ## experiment: multilingual to further explore the advantage of entity annotations as cross-lingual alignment supervision, we test ease in multilingual settings: we fine-tune multilingual pre-trained language models using wikipedia data in multiple languages. ## case study: fine-tuning supervised model with ease existing multilingual sentence representation models trained on a large parallel corpus do not always perform well, especially for languages that are not included in the training data. in contrast, ease requires only the wikipedia text corpus, which is available in more than 300 languages. 7 thus, one possible use case for ease would be to complement the performance of existing models in lowresource languages by exploiting the wikipedia data in those languages. to test this possibility, we fine-tune labse @xcite , which is trained on both monolingual and bilingual data in 109 languages, with our ease framework in five low-resource languages (kab, pam, cor, tr, mhr). these languages are not present in the original training corpus, so the model performed particularly poorly on these languages. we fine-tune the model using 5,000 pairs each from english and the corresponding language data. as shown in figure 2 , ease improves the performance of labse across all target languages, which is an intriguing result considering that labse has already been trained on about six billion parallel corpora. these results suggest the potential benefit of combining ease with other models using parallel corpora, especially for languages without or with only a few parallel corpora. ## discussion and conclusion our experiments have demonstrated that entity supervision in ease improves the quality of sentence embeddings both in the monolingual setting and, in particular, the multilingual setting. as recent studies have shown, entity annotations can be used as anchors to learn quality cross-lingual representations @xcite @xcite , and our work is another demonstration of their utility, particularly in sentence embeddings. one promising future direction is exploring how to better exploit the cross-lingual nature of entities.