Reducing Sequence Length by Predicting Edit Spans with Large Language Models (2023)

Abstract: Large Language Models (LLMs) have demonstrated remarkable performance in various tasks and gained significant attention. LLMs are also used for local sequence transduction tasks, including grammatical error correction (GEC) and formality style transfer, where most tokens in a source text are kept unchanged. However, the models that generate all target tokens in such tasks have a tendency to simply copy the input text as is, without making needed changes, because the difference between input and output texts is minimal in the training data. This is also inefficient because the computational cost grows quadratically with the target sequence length with Transformer. This paper proposes predicting edit spans for the source text for local sequence transduction tasks. Representing an edit span with a position of the source text and corrected tokens, we can reduce the length of the target sequence and the computational cost for inference. We apply instruction tuning for LLMs on the supervision data of edit spans. Experiments show that the proposed method achieves comparable performance to the baseline in four tasks, paraphrasing, formality style transfer, GEC, and text simplification, despite reducing the length of the target text by as small as 21%. Furthermore, we report that the task-specific fine-tuning with the proposed method achieved state-of-the-art performance in the four tasks.

## introduction large language models (llms), including chat-gpt foot_0 and bard foot_1 , have exhibited exceptional performance across a range of natural language processing (nlp) tasks and amassed a significant user base @xcite @xcite . as performance gains are brought from the increases in model size @xcite @xcite , llms are becoming larger and larger. however, the computational cost of inference is a severe bottleneck of many practical applications, especially when the number of parameters in an llm is massive @xcite . meanwhile, llms are also used for local sequence transduction tasks, such as paraphrasing, formality style transfer, grammatical error correction (gec), and simplification @xcite @xcite @xcite , where only a small portion of the source text is edited. most tokens in a source text are kept unchanged in these tasks. for example, the source text, "many years ago, the situation is different," and the target text, "many years ago, the situation was different," of the gec task mostly share the common tokens except for the underlined tokens (is and was). existing methods of downstream tasks do not make use of the characteristics of local sequence transduction @xcite @xcite , simply generating all target tokens. in this paper, we hypothesize that this treatment is disadvantageous in achieving high performance in terms of task accuracy and computational time. more specifically, it is inefficient to generate unchanged tokens (e.g. many, years, ago, the, situation, different) in the previous example because the model must copy many source tokens only to increase the length of the target sequence. this study proposes to predict a set of edit spans, which represent the changed parts of the target text relative to the source tokens. omitting unedited tokens that occupy most of the target text, we can reduce the length of the target text and the inference time for local sequence transduction tasks. figure 1 shows the process of creating a set of edit spans from source and target texts in gec. first, we align tokens in the source and target texts to extract the edit locations and tokens and convert them into a set of edit spans. in the example shown in figure 1 , the edit spans (1, 1, "the"), (8, 9, "have been"), (12, 13, "") are created from the source text "through thousands of years, most chinese scholars are greatly affected by the confucianism." and the target text "through the thousands of years, most chinese scholars have been greatly affected by confucianism.". llms are fine-tuned using pairs of source text and edit spans with the instructions. we conducted experiments on four local sequence transduction tasks: paraphrasing, formality style transfer, gec, and simplification. the proposed method achieved comparable performance to the baseline that directly outputs the target text. in these tasks, the proposed method could reduce the sequence length on the target side by 32% on average and by as small as 21% in gec. furthermore, the proposed method with task-specific fine-tuning achieved state-of-the-art (sota) performance in the four tasks. ## conclusion in this study, we proposed to predict a set of edit spans, which represent the changed parts of the target text relative to the source tokens. we showed our method omits unedited tokens that occupy most of the target text and reduces the length of the target text and the inference time for local sequence transduction tasks. moreover, we reported that instruction tuning with the proposed method achieves