Doolittle: Benchmarks and Corpora for Academic Writing Formalization (2023)

Abstract: Improving the quality of academic writing is a meaningful but challenging task. Conventional methods of language refinement focus on narrow, specific linguistic features within isolated sentences, such as grammatical errors and improper word use. We propose a more general task, Academic Writing Formalization (AWF), to improve the overall quality of formal academic writing at the paragraph level. We formulate this language refinement task as a formal text style transfer task which transfers informal-academic text to formal-academic and contribute a large-scale non-parallel dataset, Doolittle, for this purpose. Concurrently, we apply a method named metric-oriented reinforcement learning (MORL) to two large language models (LLM) where we incorporate different levels of automatic feedback into the training process. Our experiments reveal that existing text transfer models and grammatical error correction models address certain aspects of AWF but still have a significant performance gap compared to human performance. Meanwhile, language models fine-tuned with our MORL method exhibit considerably improved performance, rivaling the latest chatbot ChatGPT, but still have a non-negligible gap compared to the ground truth formal-academic texts in Doolittle.

## introduction writing in a second language often leads to characteristic errors. in english, such errors include subject-verb disagreement, noun-number disagreement, and determiner misuse @xcite . therefore, a language refinement system with the ability to suggest or automatically correct such errors is highly desirable @xcite . towards this end, research in grammatical error correction @xcite : we propose more sophisticated hierarchical model to include geographical informations. in light of this, we propose the novel task of academic writing formalization (awf) that aims to generalize the scope of gec for language refinement: given an informal-academic paragraph p, the objective of awf is to refine the language of p to make it grammatically correct, concise, and fluent, while preserving its semantics. different from gec, which solely concentrates on grammatical error correction for a single sentence, awf works on paragraph-level contexts and aims for refinements beyond grammar. this requires the model to comprehend and then rephrase the entire paragraph. specifically, awf considers three objectives in academic writing formalization to refine the language. 1) grammar correction: correcting grammatical errors in the paragraph, the objective of gec. 2) word refinement: replacing inaccurate words and phrases with more accurate and concise ones. evidence has shown that there are vital differences in vocabulary usage between native and non-native writers @xcite . for example, we replace "science" and "much" with "scientific" and "considerable" in although there exist several large-scale corpora for gec @xcite @xcite @xcite , they either focus on word/phrase/sentence level text refinement or do not target scientific texts, which makes none of them available for awf. we thus construct doolittle foot_1 , a large-scale, nonparallel dataset containing 55.6k formal-academic paragraphs and 13.0k informal-academic ones. doolittle is based on the semantic scholar open research corpus (s2orc; @xcite . for each paragraph, we ask human annotators to rate the academic formality of the language via crowdsourcing. expert annotators then refine the language for around 900 paragraphs to obtain a parallel corpus that serves as the development and test sets of doolittle. to investigate the performance of state-of-the-art models for awf, we adopt four baseline models of text style transfer, two baseline models from low-resource gec, the widely-available large language model (llm) chatgpt, and two pretrained language models (plm) fine-tuned with our pro-posed method: metric-oriented reinforcement learning (morl). we find that style transfer models are unsuccessful in discriminating the differences between formal and informal text, resulting in lower scores for academic formality, perplexity, and meaning preservation, while gec baselines perform relatively better across all metrics but only marginally modify the inputs. on the other hand, bartlarge @xcite and galactica-1.3b @xcite fine-tuned with our morl approach provide comparable gec results to chatgpt, despite having significantly fewer parameters. nonetheless, none could comprehensively outperform the ground truth formalacademic paragraphs. it is worth noting that metricoriented rl has been explored in the context of text generation, with some early studies @xcite using rl for neural machine translation optimized by bleu, but with limited success. to the best of our knowledge, we are the first to demonstrate that applying metricoriented rl to plms yields promising results, indicating that metric-based rl is well-suited for powerful backbone models. we summarize our contributions as follows: 1) we propose a novel setting for paragraph-level language refinement, formulating it as a text style transfer problem. 2) we construct doolittle, the first large-scale dataset for academic writing formalization. considering that awf is a common use case of llms such as chatgpt, we believe doolittle can serve as a good testbed for benchmarking llms. 3) we propose a method, metric-oriented reinforcement learning (morl), and show its effectiveness and cost-efficiency in tuning plms. 4) we conduct a comprehensive evaluation of neural approaches on our task and show that their performance still suffers from a sizable gap compared to formal-academic rewrites by humans. this highlights the need for the dataset and the awf task. ## related work language refinement. there are two tasks typical of language refinement, both focusing on enhancing the quality of sentences. post-editing @xcite @xcite is designed to rectify typical errors in machine translation, thereby augmenting the generation quality, as measured by bleu. the other task, grammatical error correction (gec), is formulated as a parallel translation task with phrase-based machine translation (pbmt) models @xcite , neural machine translation (nmt) models @xcite , and hybrid pbmt-nmt models @xcite . however, these methods require large amounts of parallel data which are expensive to annotate. to address this, lowresource gec @xcite builds models that do not rely on large parallel data. choe et al. (2019), grundkiewicz et al. (2019), and zhou et al. (2020) initially pretrain a transformer model with large synthetic parallel corpora which are generated from a realistic noising function and then fine-tune this model on a small in-domain parallel dataset. reinforcement learning from human feedback (rlhf). as a notable advancement within the realm of reinforcement learning (rl), reinforcement learning from human feedback (rlhf) integrates human feedback into the training process. this approach trains a model to align more closely with user intentions, thereby equipping the model with the ability to generate more reliable, authentic, and useful results @xcite @xcite . rlhf manifests its advancement and convincing capability in the recent state-of-the-art chatbot chatgpt (openai, 2022). the fundamental workflow of rlhf can be succinctly summarized in three steps below: step 1: train a policy model with supervised training on collected demonstration data. step 2: train a reward model on collected comparison data. step 3: optimize the policy against the reward model using reinforcement learning with proximal policy optimization (ppo) algorithm @xcite . ## dataset construction we present doolittle, a corpus of academic formality non-parallel texts from scientific paper sources ( § 3.1), where we manually annotate academic formality by crowdsourcing to obtain a large set of non-parallel training paragraphs in two styles ( § 3.2). we then conduct a second annotation task called formal-academic rewrite in order to create a small parallel dataset for evaluation ( § 3.3). ## method it could be speculated from our dataset analysis results ( § 4) that our doolittle task is advanced in terms of both quality and difficulty. to address our task with reduced cost and better performance, we proposed a method called metric-oriented reinforcement learning (morl). this methodology, inspired by reinforcement learning from human feedback (rlhf) (ziegler et al., 2019; ouyang et al., 2022), follows a similar three-step training process to rlhf but with crucial modifications: step 1: train a policy model (usually a plm) that can meet the requirements of a task. step 2: select some metrics that can accurately evaluate the quality of how the task has been performed. build a reward model that can score a given policy model's output with a scalar. step 3: optimize the policy against the reward model using reinforcement learning with the proximal policy optimization (ppo) algorithm @xcite . the key distinction between rlhf and morl lies in step 2 where rlhf trained a reward model with collected comparison data while morl utilizes any existing, machine learning-based or not, tuned or plug-and-play evaluation metrics to generate a reward model. through incorporating a variety of evaluation metrics into the reward model, the cost of implementing morl becomes flexible and the potential misalignment between human preference and a single metric can be alleviated. ## experiment 6.1 experimental settings we apply nine models -four from non-parallel text style transfer (controlledgen @xcite , deeplatentsequence @xcite , style-transformer @xcite , and deletean-dretrieve @xcite ), two from low-resource gec tasks @xcite and bart-gec (katsumata and komachi, 2020)), one from chatgpt, and two morl-based models -on doolittle to establish baseline performance. we use the official code released by the authors of our baseline (except chatgpt), and follow the recommended configuration settings from their corresponding papers. we report implementation details and hyper-parameter settings of the different models with their size and running speed in the appendix. ## conclusion and future work we propose a new setting for language refinement called academic writing formalization (awf), which bridges the gap between formal-academic and informal-academic writing. we contribute a new dataset, doolittle, and evaluate nine baseline models for awf using both automatic metrics and gpt-4 annotation, demonstrating that paragraph-level refinement is a promising task with significant room for improvement. to address awf, we propose a method called metric-oriented reinforcement learning (morl). leveraging morl, we successfully elevate the performance of bart-large and galactica-1.3b, yielding results comparable to those of gpt-3.5 turbo-based chatgpt, which possesses significantly more parameters than our morl-based models. even though such models do not outperform the formal-academic paragraphs due to the difficulty of academic writing formalization, this promising result has substantiated the advantages in terms of competence and cost-efficiency of morl. in the future, we plan to incorporate external knowledge and weakly supervised information into the text rewriting process. we also assert that the full potential of morl as applied to large language models remains to be tapped. it is foreseeable that through the inclusion of more relevant metrics and advanced modeling, morl can enhance its capability further and be adapted to a broader spectrum of nlp tasks.