Contextualized and Generalized Sentence Representations by Contrastive Self-Supervised Learning: A Case Study on Discourse Relation Analysis (2021)

Abstract: We propose a method to learn contextualized and generalized sentence representations using contrastive self-supervised learning. In the proposed method, a model is given a text consisting of multiple sentences. One sentence is randomly selected as a target sentence. The model is trained to maximize the similarity between the representation of the target sentence with its context and that of the masked target sentence with the same context. Simultaneously, the model minimizes the similarity between the latter representation and the representation of a random sentence with the same context. We apply our method to discourse relation analysis in English and Japanese and show that it outperforms strong baseline methods based on BERT, XLNet, and RoBERTa.

## introduction understanding the meaning of a sentence is one of the main interests of natural language processing. in recent years, distributed representations are considered to be promising to capture the meaning of a sentence flexibly @xcite @xcite . one typical way to obtain distributed sentence representations is to learn a task that is somehow related to sentence meaning. for example, sentence representations trained to solve natural language inference @xcite are known to be helpful for many language understanding tasks such as sentiment analysis and semantic textual similarity @xcite @xcite . however, there is an arbitrariness in the choice of tasks used for training. furthermore, there is a size limitation on manually annotated data, which makes it hard to learn a wide range of language expressions. a solution to these problems is self-supervised learning, which has been used with great success @xcite @xcite . for example, inspired by skipgrams @xcite , @xcite proposed to train a sequence-to-sequence model to generate sentences before and after a sentence, and use the encoder to compute sentence representations. inspired by masked language modeling in bert, @xcite and @xcite presented methods to learn contextualized sentence representations through the task of restoring a masked sentence from its context. in self-supervised sentence representation learning, sentence generation is typically used as its objective. such an objective aims to learn a sentence representation specific enough to restore the sentence, including minor details. on the other hand, in case we would like to handle the meaning of a larger block such as paragraphs and documents (which is often called context analysis) and consider sentences as a basic unit, a more abstract and generalized sentence representation would be helpful. we propose a method to learn contextualized and generalized sentence representations by contrastive self-supervised learning @xcite . in the proposed method, a model is given a text consisting of multiple sentences and computes their contextualized sentence representations. during training, one sentence is randomly selected as a target sentence. the model is trained to maximize the similarity between the representation of the target sentence with its context, to which we refer as s pos , and the representation of the masked target sentence with the same context, to which we refer as s anc . simultaneously, the model is trained to minimize the similarity between the latter representation s anc and the representation of a random sentence with the same context as the target sentence, to which we refer as s neg . from the viewpoint of optimizing s anc , this can be seen as a task to capture a generalized meaning that contextually valid sentences commonly have, utilizing s pos and s neg as clues. from the viewpoint of optimizing s pos , this can be seen as a task to generalize the meaning of a sentence to the level of s anc . we show the effectiveness of the proposed method using discourse relation analysis as an example task of context analysis. our experiments on english and japanese datasets show that our method outperforms strong baseline methods based on bert @xcite , xlnet @xcite , and roberta @xcite . ## learning contextualized sentence representations figure 1 illustrates the overview of our method. the encoder takes an input text consisting of t (> 1) sentences and computes their contextualized sentence representations. the encoder is trained by contrastive self-supervised learning. ## discourse relation analysis we show the effectiveness of the proposed method using discourse relation analysis as a concrete example of context analysis. discourse relation analysis is a task to predict the logical relation between two arguments. an argument roughly corresponds to a sentence or a clause. we conduct experiments on english and japanese datasets. ## sentence retrieval to investigate what is learned by our contrastive objective, we did sentence retrieval based on the similarity between sentence representations. for targets, we randomly sampled 500,000 sentences with context from input texts used for training. for a query, we used a sentence with context in a wikipedia article. computing the sentence representations for the targets and query, we searched the closest sentences based on their cosine similarity. ## conclusion we proposed a method to learn contextualized and generalized sentence representations using contrastive self-supervised training. experiments showed that the proposed method improves the performance of discourse relation analysis both in english and japanese. we leave an in-depth analysis of the level of abstraction trained by the proposed method for future work.