G - SPEED : General SP arse Efficient Editing M o D el (2023)

Abstract: Large Language Models (LLMs) have demonstrated incredible capabilities in understanding, generating, and manipulating languages. Through human-model interactions, LLMs can automatically understand human-issued instructions and output the expected contents, which can significantly increase working efficiency. In various types of real-world demands, editing-oriented tasks account for a considerable proportion, which involves an interactive process that entails the continuous refinement of existing texts to meet specific criteria. Due to the need for multi-round human-model interaction and the generation of complicated editing tasks, there is an emergent need for efficient general editing models. In this paper, we propose G eneral SP arse E fficient E diting Mo D el ( G-SPEED ), which can fulfill diverse editing requirements through a single model while maintaining low computational costs. Specifically, we first propose a novel unsupervised text editing data clustering algorithm to deal with the data scarcity problem. Subsequently, we introduce a sparse editing model architecture to mitigate the inherently limited learning capabilities of small language models. The experimental outcomes indicate that G-SPEED, with its 508M parameters, can surpass LLMs equipped with 175B parameters. Our code and model checkpoints are available at https://github.com/Banner-Z/G-SPEED .

## introduction recently, the natural language processing (nlp) community has witnessed the rapid development of large language models (llms). with the help of instruction tuning, llms can effectively utilize the knowledge acquired during pre-training and have the ability to follow user instructions @xcite @xcite @xcite . therefore, due to the powerful ability of large models as assistants, users can conveniently accomplish a wide range of open-domain tasks by collaborating with llms. one particularly important applied scenario of the llm assistants is editing. editing is a crucial process in writing, requiring a diverse range of skills to refine texts from multiple perspectives. consequently, there is a long-term goal of using machines to assist in automating the editing process. however, the inherent difficulty of editing, coupled with the limited availability of annotated editing data, hinders the development of general editing models capable of meeting a wide range of editing requirements. therefore, before the era of llms, previous works mainly focused on developing specialized editing models tailored to specific tasks, such as grammatical error correction (gec) @xcite , style transfer @xcite , and sentence fusion @xcite . due to their specific capabilities, these specialized editing models have limited practical applications. besides, although recent work has explored the potential of llms as general editing models @xcite @xcite , the iterative nature of editing often requires multiple interactions with models to edit a single piece of text. therefore, relying solely on llms to meet diverse editing needs becomes impractical and cost-prohibitive. in this work, to develop a lightweight general edit model that caters to various editing requirements, we propose general sparse efficient editing model (g-speed). specifically, we introduce a novel unsupervised editing data clustering method to eliminate noise and annotate the intent, considering the limited quality of current edit intent annotations and the presence of noisy data in the revision history of wikipedia passages. besides, we propose a novel sparse editing model structure to enhance the learning capabilities of small models. experimental results demonstrate that g-speed achieves state-of-the-art performance on the editeval benchmark and exhibits a strong generalization ability to unseen tasks with limited data resources. in conclusion, our contributions are as follows: general text editing existing work on general text editing can broadly be divided into two categories: (1) instruction-based text editing @xcite @xcite ;(2) multi-intent editing @xcite . instruction-based text editing polishes up existing texts following user instructions, which is based on llms @xcite @xcite and needs a long time to generate and large-scale datasets for training. multi-intent editing uses intents to perform diverse editing actions. specifically, kim et al. ( 2022 ) improves text editing with intent span detection. however, edit intents are defined by humans, which have high annotation costs and may not cover all editing intents @xcite @xcite . in this work, we propose g-speed, which can handle general text editing without the dependence on llms and human-annotate editing intents. editing model editing models are efficient alternatives to seq2seq models @xcite when the source and target texts in the task have a large amount of overlap, e.g., grammatical error correction (gec) @xcite , style transfer @xcite , and sentence fusion @xcite . editing models learn to predict edit operations and directly leave the correct text unchanged, while seq2seq models generate target text from scratch. lasertagger @xcite is a general approach that predicts edit operations by sequence labeling and then generates inserted words with a fixed vocabulary. omelianchuk et al. (2020) proposes more complete tags (such as pluralization and capitalization) to make generation easier and improve the gec task. mallinson et al. (2020) and mallinson et al. (2022) perform arbitrary content insertion via non-autoregressive and semi-autoregressive methods, respectively. however, the current editing model is stuck on solving a single task. editing models do not perform well in multi-task problems @xcite . sparse language model sparse language models achieve promising results with larger model sizes while maintaining almost the same computational costs @xcite @xcite . there are various routing algorithms to determine where to send examples, such as hash routing @xcite , base layer @xcite , and multi-gate moe @xcite . @xcite , @xcite and @xcite design sparse feed-forward layers on bert @xcite and make a boost on glue @xcite . in this paper, we explore the use of sparse layers on editing models for general text editing tasks with a bert backbone. ## task formulation during text editing, the documents d undergo modifications based on user-selected intents i like fluency, clarity, simplification, and neutralization. each modification corresponds to a pair of documents (d t-1 , d t ) and an editing intent i t where t represents the number of modifications. text editing models are tailored to modify documents based on intent: ## unsupervised editing data clustering to facilitate text editing tasks, a dataset comprising editing intent and text pairs, both before and after editing, is required for pre-training purposes. it-erater @xcite ) collects text revisions from wikipedia, arxiv, and wikinews. it categorizes text intents into five categories: fluency, clarity, coherence, style, and meaning-changed. the intent is then predicted using a roberta-based model @xcite that takes both the original and revised texts as input. however, automatic annotation performs poorly on small categories due to the unbalanced distribution of categories, and certain categories, such as simplification, are not taken into account. formulating perfect intent categories and annotating large-scale data for pretraining pose significant challenges. similar to existing work @xcite , we first collect data from the revision histories in march 1 st , 2023 dump of english wikipedia. 1 each revision consists of an original text, a revised text, and a user comment that is associated with the editing intent. we extract revisions from the xml format dump and exclude revisions that do not have user comments. to ensure data clustering quality, we additionally perform data cleaning based on various factors, such as the blue value of the source and target, as well as the length of comments and text. we extract the source and target sentences using punkt sentence tokenizer foot_1 and difflib foot_2 , and then we eliminate wikipedia markup using wikiextractor foot_3 . to classify user comments into distinct intent categories, we utilize k-means clustering initialized with k-means++ algorithm @xcite . 5 with a collection of n comment embeddings represented as x 1 , x 2 , ...x n , k-means clustering aims to partition these embeddings into k sets {s 1 , s 2 , ...s k } so as to minimize the intracluster distance: in our study, we utilize sentence-bert @xcite to generate comment embeddings, which are then decomposed using singular value decomposition (svd). sentence-bert provides a more effective representation of text semantics compared to algorithms based on word frequency. to discern the purpose of each cluster, we also organize the designed prompts into clusters (such as fixing grammar errors and improving text cohesion). based on the prompts they contained, we then selected four clusters, namely fluency, readability, simplification, and neutralization, as illustrated in step i of figure 1 . ## experiments 6.1 datasets we conduct our evaluation using @xcite , which is an instruction-based text improvement benchmark. the study includes six datasets and six tasks; their details are provided below. fluency the primary objective of the fluency task is to rectify grammatical and spelling errors. the evaluation of this task utilizes iterater @xcite and jfleg @xcite . iterater is an edit-intention annotated corpus of iteratively revised text. we utilize the subset that focuses on fluency. in comparison to iterater, jfleg not only corrects grammatical errors but also makes the original text more native sounding. clarity and coherence both tasks use the corresponding subsets in the test dataset of iterater. the clarity task aims to enhance the formality, conciseness, readability, and comprehensibility of the text. the coherence task aims to enhance the cohesiveness, logical connectivity, and overall coherence of the text. ## conclusion in this work, we propose the general sparse efficient editing model (g-speed), a model designed to fulfill diverse editing needs while ensuring computational efficiency. specifically, we first propose a novel unsupervised clustering strategy to obtain a large amount of multi-intent editing data, which is collected from editing histories from wikipedia and can be used to conduct pre-training. subsequently, we propose a novel sparse editing model architecture to improve the learning abilities of small models. the experimental results on editeval show that, with the use of bert-base as the backbone model, g-speed can outperform llms while maintaining an efficient inference speed. ad-ditionally, we discuss different sparse structures and show the strong generalization capability of our method across downstream tasks.