Conciseness: An Overlooked Language Task (2022)

Abstract: We report on novel investigations into training models that make sentences concise. We define the task and show that it is different from related tasks such as summarization and simplification. For evaluation, we release two test sets, consisting of 2000 sentences each, that were annotated by two and five human annotators, respectively. We demonstrate that conciseness is a difficult task for which zero-shot setups with large neural language models often do not perform well. Given the limitations of these approaches, we propose a synthetic data generation method based on round-trip translations. Using this data to either train Transformers from scratch or fine-tune T5 models yields our strongest baselines that can be further improved by fine-tuning on an artificial conciseness dataset that we derived from multi-annotator machine translation test sets.

## introduction we evaluate a range of models on our newly collected conciseness test sets. our initial approach follows the recent paradigm of using massively pretrained neural models with either no or very little task-specific training data. inspired by @xcite we report on zero-shot experiments with the large language model lamda @xcite . we also fine-tune the large sequence model t5 @xcite on small conciseness data sets. we achieve our best results using an unsupervised synthetic data generation method based on round-trip translations, i.e. sentence pairs that were generated by translating an english sentence into another language (e.g. german) and back, a technique that was previously proposed for gec pre-training @xcite . we construct additional data sets by creating mappings from the longest to the shortest reference in multi-reference machine translation (mt) test sets. our experiments suggest that conciseness is a hard task for current nlp models. we conclude with a thorough investigation into the similarities and differences of our systems and map out the challenges ahead. ## the conciseness task in this work we define the conciseness task as applying the required edits to make a sentence less wordy without changing its meaning, intent or sentiment. we will shed more light on the limitations of this definition in sec. 6. we expect conciseness models to be useful mainly for native or advanced non-native writers who wish to improve their writing style. conciseness is related to several other nlp tasks, but we argue below that each of these tasks has a different focus and deserves an independent treatment. ## modeling conciseness the approaches in this section cover a wide range of nlp models to convey a better sense for the task. they are intended to serve as baselines to compare against, and as a starting point for future research. ## data sets openmt-based fine-tuning and development sets (multirefmt-*) we derive fine-tuning and development sets from existing publicly available mt test sets. it is common practice in several nlp areas to collect reference sentences from multiple annotators to increase the trustworthiness of automatic evaluation measures, for example in grammatical error correction @xcite @xcite , mt @xcite , and image caption generation @xcite . multi-reference mt test sets have been used in the past to evaluate paraphrasing or sentence compression systems @xcite . we make use of these multi-annotator test sets by selecting the longest reference sentence as the (wordy) source sentence and the shortest reference sentence as the golden (concise) target sentence (fig. hand-annotated test sets (concise-*) deriving conciseness test sets from multi-reference mt evaluation sets is viable as a first approximation given that all references have similar meaning, intent, and sentiment by design (apart from annotation errors). however, it does not allow us to determine how wordy the sentence is in the first place. if all mt references agreed, it would suggest that the original source sentence has a single obvious translation, not that the references are already concise. therefore, we collected two new data sets, consisting of 2000 sentences each, that were explicitly annotated for conciseness -concise-lite and concise-full. both data sets used the same set of source sentences drawn from wikipedia. sentences that a) were ungrammatical, b) contained fewer than 15 words or c) included mismatched quotation marks were not selected. while concise-lite annotators were asked to make minimal changes to the original sentence, concise-full annotators were given the flexibility to make larger changes to the original sentence. the exact annotator guidelines are listed in appendix b. we will make the test sets publicly available to establish a benchmark for researchers to evaluate conciseness models. ## results we use the gec evaluation toolkit errant @xcite to compute f 0.5 -scores on spacy foot_5 -tokenized text. like in gec, precision is weighted twice as high as recall using the f 0.5 -score, which matches our intuition that a conciseness system should act as a minimally intrusive writing assistant for which false positives are far worse than false negatives. ## limitations in terms of both information density (fig. 6 ) and number of unnecessary synonym replacements (fig. 7 ), the annotators are clearly separated from most of our automatic systems, illustrating the gap to human performance on this task. our experiments showed that the giant-lm (zero-shot) underperformed the other approaches. preliminary experiments using few-shot learning did not yield improvements over the zero-shot setting. we expect the performance of giant-lm to improve via systematic prompt engineering. another challenge lies in the intrinsic uncertainty @xcite of the conciseness task, i.e. the existence of multiple viable ways to make a sentence more concise. ## conclusion our work is an initial exploration of conciseness from an nlp point of view. we compared a variety of approaches to the problem using popular techniques based on synthetic data generation or giant pre-trained sequence models. round-trip translations provide a useful data source for training conciseness models but can introduce undesirable synonym substitutions. 11 our analyses show that our systems trade off the objectives in conciseness differently (e.g. reducing the sentence length vs. preserving semantics vs. improving readability vs. increasing information density). further experiments are necessary to understand how these tradeoffs would impact the user experience or potential downstream nlp tasks. we expect our study and our annotated test sets to provide impetus for researchers to explore this field further.