D ef S ent: Sentence Embeddings using Definition Sentences (2021)

Abstract: Sentence embedding methods using natural language inference (NLI) datasets have been successfully applied to various tasks. However, these methods are only available for limited languages due to relying heavily on the large NLI datasets. In this paper, we propose DefSent, a sentence embedding method that uses definition sentences from a word dictionary, which performs comparably on unsupervised semantics textual similarity (STS) tasks and slightly better on SentEval tasks than conventional methods. Since dictionaries are available for many languages, DefSent is more broadly applicable than methods using NLI datasets without constructing additional datasets. We demonstrate that DefSent performs comparably on unsupervised semantics textual similarity (STS) tasks and slightly better on SentEval tasks to the methods using large NLI datasets. Our code is publicly available at https://github.com/hpprc/defsent .

## introduction sentence embeddings represent sentences as dense vectors in a low dimensional space. recently, sentence embedding methods using natural language inference (nli) datasets have been successfully applied to various tasks, including semantic textual similarity (sts) tasks. however, these methods are only available for limited languages due to relying heavily on the large nli datasets. in this paper, we propose defsent, a sentence embedding method that uses definition sentences from a word dictionary. since dictionaries are available for many languages, defsent is more broadly applicable than the methods using nli datasets without constructing additional datasets. defsent is similar to the model proposed by @xcite in that it generates sentence embeddings so that the embeddings of a definition sentence and the word it represents are similar. however, while @xcite 's model is based on recurrent neural network language models, def-sent is based on pre-trained language models such as bert @xcite and roberta @xcite , with a fine-tuning mechanism as well as sentence-bert @xcite . sentence-bert is one of the state-of-the-art sentence embedding models, which is based on pretrained language models that are fine-tuned on nli datasets. overviews of sentence-bert and def-sent are depicted on figure 1 . ## sentence embedding methods in this section, we introduce bert, roberta, and sentence-bert, followed by a description of def-sent, our proposed sentence embedding method. ## word prediction experiment to evaluate how well defsent can predict words from sentence embeddings, we conducted an experiment to predict a word from its definition. ## extrinsic evaluations next, to evaluate the general quality of the constructed sentence embedding, we conducted evaluations on semantic textual similarity (sts) tasks and senteval tasks @xcite . simplicity and intuitiveness. ## conclusion in this paper, we proposed defsent, a new sentence embedding method using a dictionary, and demonstrated its effectiveness through a series of experiments. its performance was comparable to or even slightly better than existing methods using large nli datasets. defsent is based on dictionaries developed for many languages, so it does not require new language resources when applied to other languages. since the model is trained with the same word prediction process as the masked language modeling, sentence embeddings derived by defsent are expected to be similar to contextualized word embeddings of a word when it appears with the same meaning as the definition. in future work, we will evaluate the performance of defsent when it is applied to languages other than english and when it is applied to a broader range of downstream tasks, such as document classification tasks. we will also analyze the relationship between the sentence embeddings by defsent and the contextualized word embeddings in the semantic vector space and investigate how model architecture and size influence the embeddings.