Learning to Paraphrase Sentences to Different Complexity Levels (2023)

Abstract: While sentence simplification is an active research topic in NLP, its adjacent tasks of sentence complexification and same-level paraphrasing are not. To train models on all three tasks, we present two new unsupervised datasets. We compare these datasets, one labeled by a weak classifier and the other by a rule-based approach, with a single supervised dataset. Using these three datasets for training, we perform extensive experiments on both multitasking and prompting strategies. Compared to other systems trained on unsupervised parallel data, models trained on our weak classifier labeled dataset achieve state-of-the-art performance on the ASSET simplification benchmark. Our models also outperform previous work on sentence-level targeting. Finally, we establish how a handful of Large Language Models perform on these tasks under a zero-shot setting.

## introduction paraphrasing a sentence to a targeted level of complexity is a natural language processing task that has not received much attention. most work focuses solely on sentence simplification: decreasing the syntactic and lexical complexity of a sentence in order to make it easier to understand while preserving its original meaning @xcite @xcite @xcite @xcite . this task has applications for second language (l2) learners and people with neural conditions that impede their reading comprehension abilities @xcite . there has been limited work on sentence complexification, which is the exact opposite of sentence * equal contribution. as far as we know, there has not been any work done on same-level paraphrasing, which we define as paraphrasing a given sentence without changing its complexity level. however, all three tasks have important potential applications in computer-assisted language learning. services like grammarly foot_0 and lingglewrite @xcite aim to correct grammatical and lexical writing errors, especially for l2 learners. others aim to generate example usage sentences for new words @xcite , as well as suggest potential paraphrases of learners' sentences in order to improve the diversity of their writing @xcite . in addition to suggesting general paraphrase rewrites, the online writing assistant wordtune foot_1 allows users to control both the length (correlated to complexity) and formality level of its paraphrase suggestions @xcite . despite the existence of these paraphrasing systems commercially, to the best of our knowledge, there has been no academic work on paraphrasing to different complexity levels. writing assistants and general language learning systems could benefit from this. a learner might want to see more concise ways of expressing their ideas (simplifications), more advanced or idiomatic ways of expressing them (complexifications), or suggestions that match their writing level (same-level paraphrases). we present models for all three tasks. for these tasks, we construct two automatically labeled (unsupervised) datasets and compare them to one human-labeled (supervised) dataset. our first automatic labeling method is rulebased according to flesch-kincaid grade level (fkgl). fkgl can be calculated automatically as a weighted score consisting of sentence length and syllable information @xcite . a lower score means simpler output, and the lowest possible score is -3.40. although this metric has been widely used for automatic evaluation of sentence simplification systems, it has been criticized for being easy to manipulate without increasing the simplification quality of the output @xcite . our second automatic labeling method is weak classification according to the six common european framework of reference for languages (cefr) levels. the cefr is used in standardized testing around the world to describe the language ability of l2 learners. 3 it contains six levels in the order of increasing complexity: a1, a2, b1, b2, c1, and c2. 4 unlike fkgl, the cefr is based on a holistic combination of lexical, syntactic, and conceptual features and requires professionals to determine scoring @xcite . we construct a new, weakly labeled cefr-annotated sentence and phrase dataset from the english profile and cambridge dictionary, which we call cefr-cep (cefr-cambridge-english-profile). we train a classifier to classify sentences and phrases into any of the six levels. from the paranmt dataset @xcite , we create both cefr-labeled and fkgl-labeled unsupervised sentence simplification, complexification, and same-level paraphrasing datasets. we also use a supervised dataset called newsela-auto @xcite . on all three datsets, we fine-tune t5 models. we conduct ablation studies on multitasking configurations, comparing performance of single-task, two-task, and three-task models. we also compare two prompting strategies: absolute prompting, where we prepend target complexity level to the input sentence, and relative prompting, where we prepend level direction to the input sentence. finally, we assess how large language models (llms) perform on these tasks in a zero-shot setting. our contributions are as follows: the source dataset is publicly available on evp, egp, and cambridge websites and can be obtained via their data request process. foot_5 ## cefr level classification in order to automatically label paraphrase data with complexity levels, we first train a sentencelevel classification model. in theory, any of the few english sentence-level readability datasets can be used for training. however, cefr-sp @xcite and newsela @xcite may contain data that we use for training and testing our later paraphrasing models, so we do not use either of those. the other option of onestopenglish (vajjala and lučić, 2018) has very few sentence pairs, and upon inspection, we find its simplest level to appear more complex than cefr a1. therefore, we create a new cefr-labeled corpus for our needs, cefr-cep. ## paraphrasing experiments we train models on the three tasks of sentence simplification, sentence complexification, and same-level paraphrasing. we train ablations for training dataset (newsela-auto, paranmt-cefr, paranmt-fkgl), multitasking configuration (1-3 tasks), and prompting strategy (relative/absolute). including our baselines, we train 42 models in total. see ## paraphrasing evaluation we perform both automatic and human evaluation. to compare all 40 experiment models, we only report automatic evaluation results. we perform human evaluation on just one model per task. ## conclusion in this paper, we provide a general investigation of the task of changing sentence complexity, with thorough fine-tuning experiments and brief experiments with llms. for sentence simplification, our models surpass or are comparable to state-of-the-art systems. for sentence complexification and same-level paraphrasing, we set new benchmarks. we show that weak classification is an effective way to create strong unsupervised datasets and that target level absolute prompting is more effective than level direction relative prompting. this research leaves opportunities for future work. for example, using a stronger level classifier to label paraphrase data might improve performance for the paraphrasing tasks. in the same vein, different filtering of paranmt or another paraphrasing dataset @xcite could potentially be used. a human-labeled same-level paraphrasing test dataset does not yet exist, and a modified sari metric that adequately penalizes repetitions is needed for sentence complexification. our methods focus on english data, but they can be easily applied to other languages if a different classifier is trained @xcite and a non-english paraphrasing dataset is used @xcite @xcite . finally, a thorough investigation on how well llms can change sentence complexity is necessary.