Split and Rephrase with Large Language Models (2024)

Abstract: The Split and Rephrase (SPRP) task, which consists in splitting complex sentences into a sequence of shorter grammatical sentences, while preserving the original meaning, can facilitate the processing of complex texts for humans and machines alike. It is also a valuable testbed to evaluate natural language processing models, as it requires modelling complex grammatical aspects. In this work, we evaluate large language models on the task, showing that they can provide large improvements over the state of the art on the main metrics, although still lagging in terms of splitting compliance. Results from two human evaluations further support the conclusions drawn from automated metric results. We provide a comprehensive study that includes prompting variants, domain shift, fine-tuned pretrained language models of varying parameter size and training data volumes, contrasted with both zero-shot and few-shot approaches on instruction-tuned language models. Although the latter were markedly outperformed by fine-tuned models, they may constitute a reasonable off-the-shelf alternative. Our results provide a fine-grained analysis of the potential and limitations of large language models for SPRP, with significant improvements achievable using relatively small amounts of training data and model parameters overall, and remaining limitations for all models on the task.

## introduction transforming complex sentences into a sequence of shorter sentences, while preserving the original meaning, can facilitate the processing of complex texts for humans and machines alike. it is, for instance, an important operation in text simplification @xcite @xcite , where simpler split sentences can provide content that is easier to read and comprehend. this task, commonly referred to as split * equal contribution. and rephrase (sprp) @xcite , also provides a relevant framework to evaluate natural language processing models, as complex linguistic properties need to be tackled to provide optimal splits and appropriate rephrasing. the sprp task is typically performed via dedicated transformation models, e.g. sequence-tosequence modelling with a copy mechanism @xcite or graph-based neural segmentation @xcite . the main limitation for this type of approach is the dependency on training datasets of paired complex and split sentences, which are scarce across languages and domains. as an alternative, @xcite exploit one-to-many alignments in parallel corpora and pivot machine translation (mt) to create an aligned corpus of complex and split sentences, which can in turn be exploited by sequence-to-sequence models. for this type of approach, the main bottleneck is the dependency on parallel corpora with sufficient many-to-one alignments and on quality pivot machine translation models. large language models (llms), such as the gpt models @xcite based on the transformer architecture @xcite , have demonstrated their strong potential on a large number of downstream tasks @xcite . in this work, we measure their performance on the sprp task, prompting the models to generate split and rephrase hypotheses from complex sentences. we evaluate different variants of the approach on datasets of varying complexity, contrasting zero shot, few shot in-context learning, and fine-tuning scenarios. we notably measure the impact of parameter and training data size, domain shift, and prompt optimisation. overall, our results demonstrate that llms outperform the previous state of the art by large margins across multiple metrics, with remaining relative deficiencies, notably in terms of split generation. results from two human evaluations, a comparative 3-way ranking task and a qualitative evaluation, further support our conclusions. our main contributions can be summarised as follows: the split and rephrase task was first proposed by @xcite , who created the web-split dataset from rdf semantic tuples and provided benchmarks with statistical mt models and lstms with attention. @xcite identified data representation in this dataset and provided enhancements along with a new benchmark augmenting the lstm with a copy mechanism @xcite . to address some limitations of websplit, notably in terms of vocabulary and naturality, @xcite presented wikisplit, a corpus created using wikipedia data that takes advantages of edits made on a page to split or fuse sentences. they showed that models trained on wikisplit improved those trained on websplit in terms of bleu @xcite . however, they also noted that wikisplit is a relatively noisy dataset, not suited for evaluation. among other approaches, @xcite leveraged dependency parsing information for the sprp task, whereas @xcite employed handcrafted rules, recursively applied to complex sentences to generate simpler ones. recent approaches have exploited neural modelling, in particular the transformer architecture. thus, @xcite proposed a tagging approach using a bert model @xcite as encoder and a single transformer decoder layer. @xcite also explored a tagging model, at the span level instead of the token level, achieving improvements with pointer networks @xcite for end of span prediction and a two-layer transformer decoder. berlanga neto and ruiz (2021) introduced a cross-lingual model based on part-of-speech tags, gru sequence-to-sequence modelling, and bert for next word prediction. recently, alajlouni and li (2023) reported improvements via transfer knowledge of data generated by the dissim rulebased system @xcite to fine-tune an encoder-decoder t5 model . in this work, we compare our llm-based variants with two recent approaches and associated corpora. first, we address bisect @xcite , a method and corpus based on extracting 1-2 alignments from parallel bilingual corpora and pivot machine translation. bisect achieved higher quality than wikisplit, and sequence-to-sequence models trained on the corpus improved on the sprp task. additionally, we compare our work to @xcite , who proposed the abcd framework where sprp is performed via neural graph segmentation, according to the number of predicates in the complex sentence, approaching or outperforming parsing-based and encoder-decoder models. they also provided two new datasets for the sprp task: minwiki, a reduced version of wikisplit where only pairs of complex and simple sentences with a matching number of predicates are kept, and desse, a corpus of students' opinion essays in social science consisting of complex sentences and corresponding splits; we used the latter in our experiments, in addition to bisect. finally, we used the wiki-bm corpus described in @xcite , who found that the standard websplit dataset contained easily identifiable syntactic cues, which even a simple rule-based approach could exploit to match the performance of state-of-the-art models. their benchmarks feature higher syntactic diversity to overcome this issue and are thus suited for our qualitative evaluation of llm-based sprp hypotheses. ## methodology the overall approach to sprp with llm is simple: probe a llm to generate sequences of meaningpreserving sentences from a single complex sentence. we explored two main variants for the task. we first fine-tuned pretrained llms on pairs of complex and split sentences. the objective in this scenario is twofold: (i) measure the performance of bare llms on the sprp task, and (ii) evaluate llm accuracy with varying number of parameters. for these models, we also evaluate different approaches to prompting, comparing soft and manually crafted prompts. additionally, we evaluate scenarios where we directly prompt llms that have been previously tuned on instruction datasets. the goal in this case is to measure the ability of this type of readilytrained models to perform the two tasks at hand, with or without positive examples. under this scenario, we explore both zero-shot and in-context few-shot task completion, by directly prompting the model for completion in the first case, and providing a limited number of examples within the prompt in the latter case. ## qualitative evaluation to gain further insights on the sprp capabilities of llms, we performed an additional human evaluation, this time centred on assessing the validity of individual sprp hypotheses on a set of predefined criteria. for this qualitative evaluation, we sampled 100 source sentences from wiki-bm @xcite , a manually crafted benchmark created from wikipedia data, and generated sprp hypotheses with the pythia-6.9b model fine-tuned on desse data. the evaluation involved 8 human evaluators, who had to answer 6 questions related to the quality of an sprp hypothesis considering a source sentence, addressing sencicality, grammaticality, deletion or introduction of new facts, correct splits and insufficient splits. 11 overall, the results, shown in although still featuring scores above 80%, the remaining aspects led to comparatively lower marks, with missed facts and not enough splits in 15.87% and 16.75% of the cases, respectively, and interrater agreement between 0.55 and 0.60. missed facts in particular were somewhat more difficult to judge, according to post-mortem feedback, as some rephrasing cases involved implicit rather than explicit logical connection between sentences, which could be rated more or less strictly. similarly, the need for additional splits can be viewed as more subjective than other categories, as evaluators could view some grammatically correct additional splits as cases of over-splitting. 12 it is worth noting that these results were obtained with a model fine-tuned on data from a different domain than wiki-bm, namely desse, showing that knowledge extracted from one dataset could be effectively transferred to other domains. ## conclusions in this work, we evaluated the performance of large language models on the split and rephrase task, improving over the state of the art by significant margins on all metrics over sprp datasets of varying complexity. the results of a manual 3-way ranking evaluation further confirmed human preference for llm-based hypotheses, and a qualitative evaluation on wiki-bm data further demonstrated the quality of the generated sprp output. among llm variants, models fine-tuned with lora achieved the best results overall. zero-short and few-shot instruction-tuned variants achieved reasonable performance, modulo a significant drop in splitting compliance for the latter, and might thus constitute a reasonable alternative in languages where training corpora are lacking for the task. increasing fine-tuning data size resulted in only gradual improvements, plateauing on most metrics with subsets of the available data. increased parameters led to only minor improvements, with models around 7b achieving the more balanced results. despite significant improvements, none of the examined variants achieved full task compliance, and core aspects of sprp remain a challenge.