Evaluating Discourse Cohesion in Pre-trained Language Models (2022)

Abstract: Large pre-trained neural models have achieved remarkable success in natural language process (NLP), inspiring a growing body of research analyzing their ability from different aspects. In this paper, we propose a test suite to evaluate the cohesive ability of pre-trained language models. The test suite contains multiple cohesion phenomena between adjacent and non-adjacent sentences. We try to compare different pre-trained language models on these phenomena and analyze the experimental results,hoping more attention can be given to discourse cohesion in the future. The built discourse cohesion test suite will be publicly available at https://github.com/probe2/discourse_cohesion .

## introduction pre-trained language models have achieved remarkable success in many downstream tasks, including question answering @xcite , reading comprehension @xcite , and machine translation @xcite , inspiring a growing body of research analyzing their ability from different aspects @xcite . however, to our best knowledge, there is no existing work to evaluate whether the abilities of these models to identify and generate discourse cohesion. cohesion is the foundation of an essay and an important form of showing style and character, and it is a semantic property of a document that represents the degree to which discourse entities are knit throughout the document @xcite . @xcite defined cohesion as "the set of possibilities that exist in the language for making text hang together". cohesion occurs where the interpretation of some element in the discourse is dependent on that of another. for example, an understanding of the reference of a pronoun @xcite requires to look back to something that has been said before. through this cohesion relation, two text clauses or sentences are linked together. therefore, cohesion plays an important role in discourse. however, to our best knowledge, existing available resources either only provide annotations for one cohesive phenomenon or mainly focus on lexical cohesion. for example, @xcite annotate verbal phrase ellipsis; @xcite annotate lexical cohesion for both german and english texts. however, neither single cohesion phenomena nor just lexical cohesion can fully interpret the ability of models from the perspective of cohesion. considering the above, this work has the following contributions: discourse cohesion modeling some discourse cohesion phenomena have been applied in various nlp tasks. a thorough survey of related work on this is far beyond the scope of this paper. to name just a few, voita et al. (2019) study repetition and ellipsis in machine translation; geva et al. (2019) tried to bring the connection between two sentences closer by combining rulebased methods with coreference and conjunction. similarly, there are also some works dedicated to cohesion phenomenon category example size repetition adj he decided to buy a pair of khakis. the pair he bought fit him perfectly. 200 non-adj jude was very excited about his college graduation ceremony. on the way to the arena, he got stuck in traffic. he only had an hour before the ceremony started. 73 synonyms adj jill became very scared. liam could tell jill was truly frightened. 200 non-adj she decided not to pursue the matter and just keep the service. it was after all only $12. but the issue kept bothering her. 64 ellipsis adj but we have an interest in hiring him; i just don't know when. 200 non-adj shawn felt that he could learn to make the website on his own. due to budget he could not pay a web designer. he took many web development classes to learn how. 50 substitution adj she wanted those cookies. she then decided to take one. 200 non-adj she began to drink a few beers. he had never been a drinker. she encouraged him to drink one. 61 reference adj at first he did not like the classes. however, over time he began to like them a lot. 200 non-adj once there jill marveled at all the beauty. it was dangerous, but exciting. she had a wonderful time on her trip to the amazon. 51 conjunction adj it was also cash only. therefore i had to turn around and go home. 200 non-adj the couple rented a yurt. it was very small. they did not like being so close. they left the yurt. they rented a hotel instead. ## the probability of generating the target word ## internal analysis of bert for cohesion phenomena for these 7 kinds of cohesion phenomena, we got some fine-grained information from the attention heatmap. the upper part of figure 1(a) indicates the attention between the words of sentence/clause one and the words of the second sentence/clause two, while the below of figure 1(a) demonstrates the attention between the words of sentence two and sentence one. we note that repetition and synonym have great attention in both directions, with almost equivalent attention. this explains why the models are better at identifying these two cohesion phenomena. what's more, the attention mainly gather on the deeper layers, which might reflect the deeper layers of bert capture more complex semantic features. in figure 1 (b), the upper part represents the attention between the first sentence and the conjunction word/discourse marker, whereas the below represents the attention between the second sentence and the conjunction word or discourse marker. the attention heatmap shows that much more attention can be seen between sentence two and the words, which means that the conjunction word or discourse marker is more closely related to the second sentence. however, it can be observed that the maximum attention of all head value for these two phenomena does not exceed 0.3, thus illustrating the poor performance of the pre-trained language models on these two phenomena is largely due to insufficient attention between the conjunction words or discourse markers and the sentences. ## conclusion we have created a benchmark test suite to evaluate the ability of pre-trained language models on seven discourse cohesion phenomena. and we consider the cohesion phenomena between adjacent sentences/clauses and non-adjacent sentences. moreover, we conduct analysis on the results of different pre-trained language models for six discourse cohesion phenomena. in the future, we would like to know the capability of language models in terms of global cohesion.