Document-level Text Simplification with Coherence Evaluation (2023)

Abstract: We present a coherence-aware evaluation of document-level Text Simplification (TS), an approach that has not been considered in TS so far. We improve current TS sentence-based models to support a multi-sentence setting and the implementation of a state-of-the-art neural coherence model for simplification quality assessment. We enhanced English sentence simplification neural models for document-level simplification using 136,113 paragraph-level samples from both the general and medical domains to generate multiple sentences. Additionally, we use document-level simplification, readability and coherence metrics for evaluation. Our contributions include the introduction of coherence assessment into simplification evaluation with the automatic evaluation of 34,052 simplifications, a fine-tuned state-of-the-art model for document-level simplification, a coherence-based analysis of our results and a human evaluation of 300 samples that demonstrates the challenges encountered when moving towards document-level simplification.

## introduction text simplification (ts) is the process of transforming text into a simpler variant that is easier to understand for wider audiences @xcite @xcite @xcite . simplifications can vary depending on the audiences' needs and expertise. for example, people with disabilities, such as dyslexia, have a better understanding of content with shorter word lengths @xcite , however, this aspect is not necessarily relevant for non-native speakers @xcite . in the past decade, most of the research efforts in automatic ts have focused on simplification at the sentence level, without considering the impact of ts at a document level. when multiple sentences in text are simplified, the overall quality of the text is affected @xcite . incorrect simplifications impact the overall text meaning and create disruptions to its structure (e.g., a sentence split without using adequate sentence connectors). sentence simplification usually does not take into account the wider context to which sentences belong. nevertheless, most of the practical applications of ts are motivated by the target audience needing to understand complete documents rather than isolated sentences. in general, sentences are evaluated within the scope of the sentence that is being simplified without considering possible disruptions that can happen between the nearby context (e.g., sentences left unconnected or unrelated sentences). the generation and evaluation of document-level simplification foot_0 have been explored to a limited extent (section 2). meanwhile, the discourse features, such as cohesion, coherence and anaphora, have been widely considered in related fields @xcite . we choose coherence to measure the relatedness of sentences in document-level ts, because of the availability of annotated data. 2 coherence is a logical and structured relationship between co-located sentences. this relationship can be at a local level between nearby sentences. this is called local coherence. in contrast, this relation could be observed at a broader level, such as sentences in each section of a scientific paper, where they belong to a common topic. we refer to this scenario as global coherence @xcite . when a sentence is incoherent, the logical relationship between the events is disrupted, such as in example 1 @xcite . this example is readable, simple, and grammatically correct, but there is no logical sequence of events or discourse elements. hui went to a restaurant she ordered a pizza she read a menu and sat down @xcite in this paper, we contribute to the transition from sentence-level to document-level ts (docs), carrying out experiments at a paragraph level to understand the possible challenges of this setting. to achieve this, we enhance a state-of-the-art sentence simplification model to perform docs with paragraph-level data from the general and medical domains. in addition, we evaluate our system outputs using docs metrics, such as coherence, readability, and simplicity, to validate the suitability of simplifications when multiple sentences are present. we summarise our main contributions as follows: in the past, ts at the document level has scarcely been explored despite the known need for simplification methods and evaluation metrics beyond sentence-level @xcite . nevertheless, recently, new directions have been explored leveraging existing methods and resources from sentence-level domain @xcite @xcite @xcite @xcite . similarly for document-level corpora, there have been limited efforts to alleviate the lack of parallel texts at a document level @xcite . recently, datasets for the general @xcite and medical @xcite domains have been proposed, aligning existing corpora such as wikipedia and cochrane reviews. 3 these resources include complex and simpler variants of a text, which are leveraged for ts. the creation of new parallel corpora for document-level simplification is also increasing beyond english @xcite @xcite , which enhances opportunities for cross-lingual settings. in relation to the evaluation metrics, sentencelevel research has typically relied on the following automatic metrics: sari @xcite for simplicity, flesch-kincaid grade level (fkgl) @xcite for readability and bleu @xcite for grammaticality. however, bleu, typically used in ts and summarisation, has been discouraged due to its poor performance in simplification operations, such as sentence splitting, and its negative correlation with simplicity @xcite . similarly, there are also limitations considered for fkgl @xcite . however, we still use this metric in our work to compare with previous work. at a document level, @xcite proposed d-sari evaluation metric that considers additional document-level penalties for system outputs (e.g., simplifications that outnumber the gold standard in sentence count). ## methods we describe the adaptation of sentence-level ts methods into a document-level setting. we trained a sentence-level state-of-the-art ts model using paragraphs (section 3.1) for discourse generation (i.e., longer, well-structured and logically simplified texts). there is no limitation on the simplifications that can occur at the document level, which means that we can expect modifications at a lexical, syntactic or semantic level, inferred from the training data. after texts are generated, we evaluate our simplifications (section 3.2) through documentlevel metrics for simplicity, readability and coherence. we demonstrate our selected methods in figure 1 . ## experiments we adapted three sentence simplification models (section 4.2) for docs by training them with paragraph-level data (section 4.1). we evaluated our models with metrics that consider discourse factors to understand the impact on simplification when multiple sentences are involved (section 4.3). ## results we evaluated the results for simplification quality (d-sari) and readability (fkgl) in in terms of readability, the fgkl metric (lower is better) in the model mined+d-wikipedia+cochrane, showed the worst performance when evaluated using the cochrane test set, with a score of 12.69. this result is mainly because the cochrane articles are in the medical domain, where the vocabulary tends to be more complex and the sentences are longer. we also calculated the fkgl score of the gold reference corresponding to this simplification and we achieved a similar value of 12.43 for the cochrane test set. the mined+d-wikipedia model showed the best readability results. we selected the onestopenglish dataset as an external dataset for model generalisation. as shown in figure 2f in the appendix, almost all predictions are shorter than the gold standard simplification. therefore, all values for d add are low, due to the penalty of lp 1 . in terms of readability, all models evaluated with onestopenglish test set showed better performance compared to cochrane test set. we evaluated our selected measure of coherence in our models' predictions, including comparisons between the inputs (complex), predictions (simple) and gold-standard simplifications. as shown in also, we note that for professionally written samples (onestopenglish) the coherence is significantly high, especially for the complex texts, which are more elaborated. cochrane and on-estopenglish gold-standard have a higher coherence, which may be related to the fact that these are written by professional authors, rather than created by crowdsourcing community as the d-wikipedia dataset. overall, gold-standard values ## human evaluation due to the limitations of the coherence neural models, we further evaluate their performance against human criteria to better understand the existing gap with respect to automatic metrics. we performed a human evaluation of 300 samples of automatically simplified text, divided into 5 sets of 20 paragraphs; each set was annotated by 3 evaluators. for the evaluation, we recruited 15 annotators working within the nlp domain (staff and phd students from the university of manchester and manchester metropolitan university). we selected the mined+cochrane model evaluated in the cochrane dataset and the mined+d-wikipedia model evaluated in the d-wikipedia. we had a total of 50 unique texts for each model. we selected these models to measure coherence in both domains (medical and general) in their best setting (within their own test sets). 16 as a result of our human evaluation, we noticed that texts from the general domain were perceived as more coherent than those from the medical domain. while some of our annotators had experience with texts from the medical domain, these are still significantly technical and seem incoherent for some of them. however, most of the texts from both domains were rated as high coherence. we also correlated the automatic scores for each of 93 the evaluated texts. the correlation between automatic metrics and human evaluation was 0.029 and -0.085 for the general and medical domains. the correlation between the coherence estimation of human annotators and the trained model is clearly weak. 17 the main reason is that the model has to operate outside its original domain: it was trained on documents written by human authors but was evaluated on the machine-generated text of simplifications. designing architectures and training strategies for coherence assessment models that operate with good performance on substantially different data is a direction for future research. ## discussion we trained the sentence-level muss model using paragraph-level data, evaluated with ts metrics and coherence. in our results, we observed the generation of longer sentences, in comparison to the original model. in addition, we saw an improvement in readability for the mined+d-wikipedia model using the d-wikipedia test set compared to the other baselines. the mined+cochrane had the lowest performance, most likely since it belongs to the medical domain. the reliable evaluation of ts remains a challenge. we noticed that the use of d-sari evaluation is significantly affected by the penalties from differences in the number of words and sentences. this leaves other aspects of simplification unattended, mainly at a discourse level such as the generation of coherent, topically-related simplifications. when simplification is performed at the document level, there are more opportunities for elaboration @xcite , but also, for shortening the content when it is explained in simpler words. due to this, it is unlikely to find a strong correlation (i.e., equality in length) between the size of the predictions and the gold standard. this is one of the main weaknesses of traditional ts metrics (e.g., fkgl, d-sari), which rely mostly on length aspect. our analysis was done to demonstrate this limitation further and as a motivation for discourse-level evaluation metrics for ts. the evaluation of coherence has shown new directions that could be explored to address this need. when simplification is performed beyond the sentence level, it disrupts the flow of ideas in the text and leaves sentences in paragraphs unconnected. as shown in as we mentioned earlier, coherence itself is a challenging factor to assess. this applies not only to automated evaluation methods but also to humans, especially to non-trained experts @xcite when classifying average samples (e.g., medium level of coherence). however, there is value in classifying simplifications as an additional aspect to consider for document-level ts. by performing a comparison between our inputs, predictions and the gold standard we obtain a valuable notion of coherence in model evaluation. the evaluation of coherence is a first step, among the possible discourse elements that must be assessed during simplification, such as better readability @xcite and factuality @xcite . ## conclusion in this study, we demonstrated that with the models and resources available, implementing discourseaware simplification models becomes possible. we implemented a document-level model by extending a state-of-the-art sentence ts model and included different evaluations from a document-level perspective. the evaluation of docs based on coherence is necessary, but it remains a challenge due to the subjective nature of this task. nevertheless, the assessment of coherence represents a viable tool for detecting those simplifications that are unclear, inconsistent or lack a consistent narrative. in the future, we expect to explore additional directions towards discourse elements such as cohesion and anaphora to support coherence evaluation for ts. we will also consider the implementation of alternative coherence models to improve coherence assessment and its generalisation for other domains within ts. finally, we will consider baselines in which documents are simplified sentenceby-sentence to compare against our docs systems, which consider context in the generation step.