Do sentence embeddings capture discourse properties of sentences from Scientific Abstracts ? (2020)

Abstract: We introduce four tasks designed to determine which sentence encoders best capture discourse properties of sentences from scientific abstracts, namely coherence and cohesion between clauses of a sentence, and discourse relations within sentences. We show that even if contextual encoders such as BERT or SciBERT encodes the coherence in discourse units, they do not help to predict three discourse relations commonly used in scientific abstracts. We discuss what these results underline, namely that these discourse relations are based on particular phrasing that allow non-contextual encoders to perform well.

## introduction this paper compares the ability of different sentence encoders at representing coherence of sentences from scientific abstracts, and more specifically discourse relations between clauses. our first hypothesis is that bert @xcite and scibert @xcite models enable to produce sentence representations that take coherence into account thanks to their training done on next sentence prediction. our second hypothesis is that this training should also enable to capture discourse relations between clauses. discourse relations (drs) represent the semantic and pragmatic links between discourse units (dus), that are either clauses, sentences or groups thereof, within a hierarchical structure that represents the whole text. in this work, we focus on sentences that are defined as textual sequences separated by a period. sentences may comprise one or more dus, and when they have at least two dus, the coherence links between them is represented through drs. these drs are either explicitly signaled, or left implicit. for example, the sentence "by wearing a mask, we can protect the others." conveys an enablement relation between the action "wearing a mask" and the event "protecting the others", which is here lexicalized by the connective "by". drs can be used to extract new knowledge, especially in scientific abstracts which are highly structured @xcite . sentence embeddings (se) represent the meaning of a sentence in a fixed-size vector space, and recent contextual approaches such as bert have shown promising results for downstream tasks such as semantic textual similarity (sts) or natural language inference (nli) @xcite . when further trained on downstream tasks, these models have shown promising results. however, their performance also rely on linguistic knowledge acquired at pre-training. for example, bert and scibert are trained on both masked language model and next sentence prediction tasks, in order to capture general linguistic properties that are then transferred to learn more specific representations. in this work, we want to understand if discourse properties are embedded in sentence representations that are built before further training on downstream tasks. we design probing tasks, that are classification tasks whose goals are to predict discourse properties of the sentences from their embedding. our goal is to highlight if discourse properties of sentences are captured by those vectors without finetuning. we rely on the corpus scidtb @xcite to build four datasets used to probe if embeddings capture some discourse properties of the sentences. the two first datasets probe the coherence of sentences, and the two others probe the presence of drs. we use four different sentence encoders to produce sentence embeddings, that we then use as input vectors for two classification models. if the classifier succeeds, it means that the vectors stores the discourse property that is probed. we evaluate the classifiers, thus high-lighting encoders that best encode the properties we probe. this paper is organized as follows. we first provide background to our work. second, we detail the tasks that we design to detect discourse properties. third, we present our experimental setup, including the different se that we evaluate and the choices that are specific to the corpus on which we rely. finally, we present our results and discuss some issues. ## background coherence and cohesion are two key notions in the perception of a text as a unified whole. coherence refers to logical and semantic relations between clauses and sentences in a text, while cohesion refers to grammatical or lexical devices such as pronouns, verb tense or connectives, that form external relations of a text @xcite . while the former may stand without the latter and vice versa, the connectivity model of renkema (2009) mixes coherence and cohesion cues. in this work we follow this approach, and will refer to as coherence the properties of both coherence and cohesion. drs may either explicitly signal coherence relations by discourse connectives, or left them implicit. in this latter case, the reader infers the relations based on coherence links between clauses. for example, the marker "to" is frequently used to links an action x and a way y to realize x, signaling that y is the manner-means to do x. however, even if some relations are explicitly signaled with discourse connectives such as "to" or "by", others are less salient. for example in "we propose a novel extension of this work using target context information.", the ellipsis of "by" make it harder to infer the relation. some of them (e.g enablement, manner-means or attribution) express a logical link between the content of the clauses they relate, while others (e.g elaboration or progression) only express a continuation or additive relation. in scientific abstracts especially, drs that convey a logical link are often lexically marked. several theories such as rhetorical structure theory (rst) @xcite or segmented discourse representation theory (sdrt) @xcite help to build text discourse structures, by providing both sets of drs that are defined based on the content of the dus and a framework for attaching the dus by means of drs. the rst defines a set of relations foot_0 that are either mononuclear (if for two dus, one is more salient than the other) or multinuclear (if two or more dus have the same importance). they serve to build the hierarchical discourse structure of the text, either as a constituency tree, or, more recently, with dependency trees @xcite , which are used to annotate scientific abstracts from scidtb @xcite . se have shown promising results for a wide variety of nlp tasks @xcite , but are not interpretable independently of the others, making it unclear what linguistic information they contain and what is the meaning that they represent exactly. previous works @xcite @xcite intended to clarify what linguistic information are contained in se by designing auxiliary tasks that take a single sentence vector as input and tries to predict a simple linguistic property of the sentence. they showed that some syntactic properties (e.g word order or syntactic tree depth) and some properties that they defined as semantic foot_1 (e.g the tense of the main verb or the number of the subject) are well captured by se. we then hypothesize that the latter may also capture discourse coherence and drs in sentences from scientific abstracts, thus being the first step to extracting information at a semantic-pragmatic level. ## probing tasks in this section, for each probing task, we explain how we build a dataset of sentences tagged with discourse properties. they are then encoded by various methods and used to train and evaluate classifiers. we introduce two tasks designed to check if coherence between clauses of a sentence is encoded (3.1, 3.2), and two tasks to check if drs in sentences are encoded (3.3, 3.4). ## results in this section, we comment the results obtained for each probing tasks. relation detection dr are almost as well encoded by bov than by bert and scibert, which by comparing two by two the results of bov, bert, and scibert encoders for binrel, we highlight that those models nearly give the same predictions (see table . 4). among the 290 sentences used as test, 94 are predicted differently by bert and bov, 85 by scibert and bov, and 67 by scibert and bert. among the sentences that are similarly predicted, less than 20% correspond to common mistakes, and the others to common good predictions. for bert and scibert, around 70% of the predictions are similar, showing that the models capture similar information for this task. for bert and scibert with bov respectively, more than 50% are common, but still they make a lot of different predictions, showing that they do not capture same aspects of the sentence. for semrel however, the models make very different predictions. among the 144 examples used as test, 47 are predicted differently by bert and bov, 78 by scibert and bov, and 71 by scibert and bert. the results of we show both precision and recall obtained for semrel in coherence detection representations build from bov fail at predicting if units have been swapped (51% foot_7 ), and perform badly for predicting if units have been scrambled (≈ 51.9%). google universal sentence encoder improves swapped only by 10% and scrambled by 4%. representations built from contextual embeddings improve accuracy for swapped by almost 26% for bert and 27.5% for scibert and for scrambled by almost 25% for bert and 27.8% for scibert. this shows that training on sts, on which google use is trained, is not enough to capture coherence links between clauses, and that the "next sentence prediction" (nsp) task on which both bert and scibert are trained seems indeed to help a lot. this task enables somehow to account for the order of the units in the sentence. even if bert and scibert are not trained at clause level, the corpus on which they are trained is large and therefore contains a large number of sentences that are made of a single clause. this enables the model to learn clause level contingency relations, which are in fact the task that we probe here. ## discussion in this section, we discuss potential biases in the probing tasks that we designed, how we control them, and the possible improvements that could be further done. relation prediction a bias that could affect dr detection is the length of the sentence. in particular, @xcite showed that bov obtain 66.6% accuracy for predicting the length of the sentence, and get up to 99% with other more elaborated encoders such as bilstm or gated convolutional networks. we took that into account when selecting the different sentence sets. by including sentences that have more than one du (sentences that only have relations with weak semantics) in the norel class , we somehow control the size of the sentences and ensure that the distributions of words per sentence are not specific to each class. the distributions are given in fig. 3 , and are close enough to guarantee that the sentence length does not bias our predictions for binrel. similarly, we ensured that the length of the sentence does not affect the detection of the relation semantics. the distributions which are given in fig. 4 are close and, so, cannot be the only parameter involved in the good quality of the prediction. however, good performances of bov for this task are influenced by the fact that most of them are linguistically signalled. even if this bias is in fact related to the specific genre of the corpus that we study, we discuss here the linguistic properties that play a role in the various encodings and predictions. verbs such as "we show" or "we demonstrate" typically occur in sentences which contain an attribution relation. conjunctions such as "to" or "in order to" express enablement relations. finally, manner-means are often signalled by verb forms such as "(by) +ing". we think that these signals play an important role in the good performances of bov especially for attribution and manner-means. ## conclusion we introduced four tasks for probing the discourse properties involved in sentences from scientific abstracts. we evaluated the ability of a classifier to predict a discourse property of a sentence from its embedding. the performance of the classifier highlights the extent to which discourse properties are encoded in those representations. we showed that coherence links are captured by vectors made from contextual models, as well as drs, but that those models in fact do not encode the semantics of those drs. we highlighted that bov embeddings perform nearly as good as contextual embeddings for both drs detection tasks, highlighting the fact that these relations are most of the time explicitly signalled in scientific abstracts. this confirms our hypothesis that bert and scibert training suffice to encode coherence to some extent. we think that the next sentence prediction task is the reason for this performance, as it allows the model to learn adjacency properties of sentences, and thus of clauses. the second hypothesis however is not confirmed, as we concluded from the experiments that semantics of drs are as well captured by bov, as bert or scibert. the present work opens various possibilities, for improving the tasks as well as the analysis of the results. we decided here to focus on the detection of three specific relations that occur within sentences. we adopted strategies to make this possible, for example by ignoring drs that are of continuation or additive and not logical. we could however go further by considering all relations involved in the sentences. a multi-label classification could be a solution to the problem of predicting all drs of a sentence from its embedding. a sequence-to-sequence model could be a solution to the problem of predicting the discourse structure (or sequence of relations) from its embedding. for coherence prediction, we also plan to determine to what extent the tasks we introduced (swapped and scrambled) depend on syntactic properties. a statistical analysis of the syntactic structures involved could help to clarify the possible biases coming from the syntax. we also plan to further check the datasets that we introduce for those tasks, as we have raised that our method created coherent sentences, labeled as incoherent in our training dataset. other corpora such as rst-dt @xcite or pdtb @xcite , which could be combined, could be used to produce other datasets for probing drs, enabling to evaluate how other drs (such as contrast or reason) are encoded in sentence embeddings.