S ci P ara: A New Dataset for Investigating Paragraph Discourse Structure in Scientific Papers (2024)

Abstract: Good scientific writing makes use of specific sentence and paragraph structures, providing a rich platform for discourse analysis and developing tools to enhance text readability. In this vein, we introduce SciPara, a novel dataset consisting of 981 scientific paragraphs annotated by experts in terms of sentence discourse types and topic information. On this dataset, we explored two tasks: 1) discourse category classification, which is to predict the discourse category of a sentence by using its paragraph and surrounding paragraphs as context, and 2) discourse sentence generation, which is to generate a sentence of a certain discourse category by using various contexts as input. We found that Pre-trained Language Models (PLMs) can accurately identify Topic Sentences in SciPara, but have difficulty distinguishing Concluding, Transition, and Supporting Sentences. The quality of the sentences generated by all investigated PLMs improved with amount of context, regardless of discourse category. However, not all contexts were equally influential. Contrary to common assumptions about well-crafted scientific paragraphs, our analysis revealed that paradoxically, paragraphs with complete discourse structures were less readable.

## introduction writing a scientific paper that is understandable to readers is a challenging task. well-written scientific papers not only facilitate the comprehension of scientific discoveries but also reduce the risk of disseminating inaccuracies and misconceptions in research @xcite . as a rhetorical unit of writing, paragraphs contain valuable information regarding the logical and narrative connections among sentences @xcite . scientific papers with many well-written policy in china provides coverage ranging from rmb100k to 200k (about usd15,384 to 30768, using the exchange rate as of december 31, 2020) with a duration of 1 year.] topic [the policy can cover related hospitalization expenses and include a lump-sum benefit when the insured is diagnosed as being in a serious condition or in the event of death or disability.] supporting [the insurance policy takes effect the day after sign-up, but is subject to a 1-week waiting period to mitigate adverse selection.] transition para. topic attr. 1 attr. 2 attr. 3 attr. 4 attr. 5 para. topic para. topic paragraphs are easier to understand. in those paragraphs, related sentences are grouped and information is stitched in a thematically progressing manner @xcite . in recent years, significant efforts have been directed at utilizing nlp technologies to process and comprehend scientific texts. for instance, research has focused on automatic summarization @xcite ), text generation @xcite @xcite , as well as argument mining and discourse analysis @xcite @xcite , all in the context of scientific papers. however, few efforts have been devoted to identifying wellwritten scientific paragraphs from the perspective of discourse structure. in this work, we propose scientific paragraphs (scipara), a novel dataset specifically curated for studying the structure of scientific paragraphs. sci-para is a collection of scientific paragraphs that have been manually annotated by professional editors with strong biomedical backgrounds. the annotations include paragraph-level discourse com-pleteness, sentence-level discourse categories, and word-level occurrences of the paragraph topic. by training various language models on scipara, we address the following research questions (rqs): rq1 can language models distinguish sentences of different discourse categories? rq2 can topic, concluding, and transition sentences be generated from the rest of the corresponding paragraphs? rq3 are paragraphs with complete discourse structure more readable? our main contributions are as follows: 1) we propose a manually annotated dataset of scientific paragraphs, which is, to the best of our knowledge, the first dataset specifically designed for the study of the discourse structure of scientific paragraphs; 2) we fine-tune language models to perform sentence classification and generation tasks on our dataset; 3) we perform an in-depth analysis of the paragraph discourse structure with respect to our experimental results. ## methods in the following section, we detail the experimental methods applied to the scipara dataset to address our research questions. notably, our experiments primarily utilized the annotations corresponding to the sentence-level task, and the topic ontology annotations from the subsentence-level task. we plan to incorporate other annotation types, such as topic attributes and links, in future studies. ## related work previous works on automatic classification of discourse category of sentences from scientific papers are @xcite , @xcite al. (2019), gonçalves et al. (2020), dayrell et al. (2012), fisas et al. (2015) moreover, there is a scarcity of research on generating sentences across these discourse categories. @xcite and @xcite conducted related studies, with the former generating abstract "conclusions" and the latter generating topic-wordconstrained sentences. our approach, however, explores generating "topic sentences" and other categories from the remainder of the corresponding paragraph and varying additional contexts, such as preceding paragraphs, thus addressing a research gap. ## conclusion we introduced the scipara dataset which comprises scientific paragraphs with expert annotations of sentence discourse category and of topic information. leveraging pre-trained language models, we explored two tasks: discourse category classification and discourse sentence generation. while the models demonstrated high accuracy in identifying topic sentences, they encountered challenges in distinguishing concluding, transition, and supporting sentences, underscoring the inherent complexities in automating discourse category classification. we also examined the influence of contextual input on generating discourse sentences. our findings indicate that language models perform better with increased context, but that the context most useful depends on the sentence discourse category. for instance, topic ontology plays the most crucial role for topic sentence generation, whereas the next paragraph has the largest influence on transition sentence generation. we also assessed the readability of scipara paragraphs. surprisingly, our analysis reveals an intriguing paradox on the relationship between discourse structure and readability. scientific paragraphs containing at least one topic sentence and at least one concluding or transition sentence are commonly perceived as well-written. however, such paragraphs are more challenging to read.