DECOR : Improving Coherence in L 2 E nglish Writing with a Novel Benchmark for Incoherence Detection, Reasoning, and Rewriting (2024)

Abstract: Coherence in writing, an aspect that L2 English learners often struggle with, is crucial in assessing L2 English writing. Existing automated writing evaluation systems primarily use basic surface linguistic features to detect coherence in writing. However, little effort has been made to correct the detected incoherence, which could significantly benefit L2 language learners seeking to improve their writing. To bridge this gap, we introduce DECOR, a novel benchmark that includes expert annotations for detecting incoherence in L2 English writing, identifying the underlying reasons, and rewriting the incoherent sentences. To our knowledge, DECOR is the first coherence assessment dataset specifically designed for improving L2 English writing, featuring pairs of original incoherent sentences alongside their expert-rewritten counterparts. Additionally, we fine-tuned models to automatically detect and rewrite incoherence in student essays. We find that incorporating specific reasons for incoherence during fine-tuning consistently improves the quality of the rewrites, achieving a level that is favored in both automatic and human evaluations.

## introduction automatic english writing tools have gained extensive popularity among second-language (l2) learners. these tools serve as a cost-effective supplement to traditional, expensive human tutoring, providing learners with timely and constructive feedback. much progress in this area includes automatic grammar correction systems @xcite @xcite and tools to improve the vocabulary usage of learners @xcite @xcite ). however, these tools primarily focus on the word and sentence-level issues that affect l2 writing rather than discourse-level issues. an aspect of l2 writing that could also benefit from automated tools is the overall textual coherence which is a requirement to efficiently convey one's ideas. to improve l2 writing skills, whether it is part of a course assessment or standardized test of english ability, learners are often required to carefully organize their thoughts in response to a predetermined writing prompt. previous research has identified coherence as a crucial feature to measure when assessing l2 writing proficiency, as it is an aspect that students often struggle with @xcite @xcite . current automated writing evaluation tools primarily provide learners with scores that indicate the level of coherence in their writing @xcite . they primarily detect coherence with simple surface linguistic features, such as syntax and parts of speech @xcite . however, merely detecting coherence in writing is insufficient to help l2 english writers enhance their writing. an automated system capable of detecting incoherence in l2 writing, identifying the underlying reasons, and correcting the incoherent sentences would be immensely valuable for both language learners and instructors. however, the absence of a benchmark dataset specifically designed for incoherence detection, reasoning, and rewriting in l2 english essays significantly impedes the development of such systems. hence, we introduce decor, a novel benchmark dataset that can be used to improve coherence in l2 english writing. to construct decor, we start by creating context-sentence pairs from the toefl-11 corpus @xcite , following the incremental annotation protocol suggested by @xcite . we then design a language-learning-oriented annotation scheme that guides expert annotators to detect incoherence in these pairs, identify specific reasons for incoherence, and rewrite the incoherent sentences. figure 1 demonstrates the overview of decor and the three tasks. to our knowledge, decor is the first benchmark to feature expert annotations for incoherence detection, reasoning, and rewriting, specifically tailored for l2 english writing. the resulting parallel corpus with pairs of original incoherent sentences and their expert-revised versions, provides a valuable resource for evaluating coherence in automated writing evaluation systems. moreover, while previous research demonstrated the effectiveness of using gpt-4 to assess writing coherence @xcite , challenges persist, particularly for users in developing countries @xcite . these include limited access to gpt-4, the high cost associated with its usage, and its tendency to produce overly invasive and non-essential revisions (as shown in figure 1 ). consequently, building smaller and more accessible models tailored specifically to our dataset could bring significant benefits. hence, we develop models to automatically perform the proposed three tasks on decor. the findings from our experiments indicate that our incoherence detection models deliver performance comparable to gpt-4 in zero-shot and few-shot scenarios, despite being significantly smaller and less costly. we also demonstrate that both automatic and human evaluations affirm that fine-tuning rewriting models with specific reasons for incoherence consistently enhances their ability to produce rewrites that match the quality of those generated by human annotators. overall our contributions are three-fold: these models achieve results comparable to gpt-4 in detecting incoherence and producing rewrites that match the quality of those generated by human experts. ## decor benchmark and annotation scheme in this section, we detail the data creation process for decor (section 3.1). we also outline the specific annotation schemes for each proposed task: incoherence detection (section 3.2), incoherence reasoning (section 3.3), and incoherent sentence rewriting (section 3.4). ## data annotation process and statistics considering the need for substantial experience in english essay grading, we recruited two annotators with extensive teaching experience in english and advanced degrees in applied linguistics, specializing in english language education. before annotating decor, we conducted a tutorial session to train the two annotators and familiarize them with our annotation scheme. subsequently, in accordance with our specified scheme, we tasked them with annotating five sample essays, which comprised 72 sentence-context pairs. we calculated the inter-annotator agreement for these pairs using cohen's kappa @xcite . the two annotators achieved a κ value of 0.83 for incoherence detection, indicating an almost perfect agreement. for incoherence reasoning, they reached an average κ = 0.90 across all reason types, also reflecting almost perfect agreement. the specific agreement scores for each reason type and more justifications for the annotation process are presented in appendix b. as for incoherent sentence rewriting, the leading authors validated whether the new sentences are acceptable. in particular, a new sentence s ′ is acceptable if it preserves the semantic meaning of the original sentence s and is coherent with the given context c. overall, the rewrites by the two annotators were deemed acceptable at rates of 88% and 89%, respectively. subsequently, the two annotators worked independently on the test set, with each annotating around 700 (c, s) pairs that are constructed from section 3. overall, among all 1, 352 (c, s) pairs, 906 sentences are coherent with their corresponding contexts, whereas 446 sentences are labeled as incoherent. we present the number of words per rewrite in figure 4 . note that we do not consider rewrites marked as delete, resulting in 213 rewrites that contain more than one word. in addition, we presented the distribution of the annotated reasons for incoherence in figure 2 . our analysis shows that the medium-level essays, randomly sampled from the toefl-11 corpus, generally maintain consistency and rarely contradict the context. moreover, we also find that the primary sources of incoherence in these essays are related to relevance and cohesion, with issues of tangential relevance and weak discourse relations being the most prevalent. ## incoherence detection, reasoning and rewriting we propose decor to benchmark the model's ability in incoherence detection, reasoning, and rewriting for english essays written by l2 language learners. in this section, we will outline each of the three tasks and describe their specific task formulations, evaluation metrics, data, baselines, and results and analysis. ## conclusion and future work we propose a novel benchmark decor aiming to assess and improve coherence in l2 english writing. specifically, decor contains three tasks: incoherence detection, reasoning, and rewriting. our annotation scheme allows us to produce a corpus comprising 1, 352 context-sentence pairs with coherence labels, as well as the first parallel corpus featuring 213 pairs of original incoherent sentences and their expert-rewritten counterparts. additionally, we fine-tuned various models with task-specific synthetic data, achieving results comparable to gpt-4 in coherence detection and generating rewrites favored by both automatic and human evaluations. in future work, we plan to enhance decor by expanding its size and quality, ensuring more balanced reasoning types and multiple edits for each incoherent context-sentence pair. this enhancement will create a more comprehensive evaluation set for incoherence detection and correction, specifically tailored to l2 writing.