Enhancing Sentence Simplification in P ortuguese: Leveraging Paraphrases, Context, and Linguistic Features (2024)

Abstract: Automatic text simplification focuses on transforming texts into a more comprehensible version without sacrificing their precision. However, automatic methods usually require (paired) datasets that can be rather scarce in languages other than English. This paper presents a new approach to automatic sentence simplification that leverages paraphrases, context, and linguistic attributes to overcome the absence of paired texts in Portuguese.We frame the simplification problem as a textual style transfer task and learn a style representation using the sentences around the target sentence in the document and its linguistic attributes. Moreover, unlike most unsupervised approaches that require style-labeled training data, we fine-tune strong pre-trained models using sentence-level paraphrases instead of annotated data. Our experiments show that our model achieves remarkable results, surpassing the current state-of-the-art (BART+ACCESS) while competitively matching a Large Language Model.

## introduction text simplification consists of making a text easier to read and understand by wider audiences while preserving most of its original meaning @xcite . simplification has a variety of critical social applications, for example, increasing accessibility for individuals with reading difficulties @xcite , cognitive disabilities such as aphasia @xcite , dyslexia @xcite , and autism @xcite , or for non-native speakers @xcite . moreover, expert-written texts, such as those in science, medical, financial, and legal fields, can exhibit a high level of complexity, making them difficult for the public to read and understand. this complexity stems from specialized jargon and technical language, which, while precise and necessary within the field, can pose significant reading challenges to the layperson @xcite . the plain language movement dates back to the 1940s, advocating for a straightforward writing style that avoids unnecessary jargon or complex vocabulary @xcite . however, legislation towards making legal and governmental texts more accessible to the public has become prominent only in this century foot_0 . in brazil, for example, while a few local governments have specific laws for simple language since 2010 @xcite , only in 2023 a national law has emerged foot_1 . given the amount of human knowledge still outside that movement, achieving the goals of plain language initiatives worldwide requires automatic approaches. additionally, the unique characteristics of cultural writing styles necessitate developing customized solutions for each. for instance, portuguese writers favor lengthy sentences, incorporate passive voice and complex verb conjugations, add implicit coreferences, and use extensively verbal phrases and other intricate structural elements. those practices usually make sentences more complex than they could be. however, most text simplification work has targeted english, with a relative abundance of aligned pairs for supervised training of automatic models. @xcite shows the amount of parallel simplification data available by language. parallel pairs are very scarce in languages other than english, french, and russian. this paper focuses on portuguese, although our model architecture is language-agnostic. the scarcity of both simplification models and datasets in portuguese highlights the need to make available more resources for the community, mainly because it is the language spoken by about 260 million people 3 . portuguese text simplification also inherits the challenges of natural language generation, namely the lack of accurate evaluation metrics @xcite . furthermore, we focus on models trained from non-aligned pairs to account for the lack of paired datasets. acknowledging that text simplification is highly audience-centric @xcite , recent work has focused on developing techniques to control the degree of simplicity of the output. controllable text simplification can be seen as a conditional language modeling task, where the source text x is rephrased as an output y that presents attributes v evaluated by a model p (y |x, v ) @xcite . @xcite provides an overview of the control token attributes introduced in prior work for text simplification. they range from high-level features, e.g., grade levels, to low-level syntactic, lexical, and semantic attributes. this work proposes a new controllable sentence simplification model trained using pairs of paraphrase data (s source , s target ) plus a sentence around the target paraphrase. in addition to conditioning the text generation on linguistic attributes of the target text, we also condition on its context. similar to @xcite , we take as context a sentence of the same document as the target sentence context, relying on the observation that style is a "slowmoving" feature, which tends to be uniform over spans of a document. we learn a representation from the low-level features of the target sentence, and another one from its context. we combine and feed them into a simple neural network to obtain our final style representation, which will guide the decoder. pt-t5 @xcite , a portuguese version of t5 @xcite , is our base neural sequenceto-sequence (seq2seq) architecture, given the successful results of this transformer-based model on several nlp tasks. the contributions of this paper are: 1. a novel few-shot approach to training simplification models with paraphrase data and the sentence adjacent to the target paraphrase. 2. to the best of our knowledge, this is the first study that uses both linguistic features and context to learn a representation to guide the simplification process. 3. in the experiments with three portuguese simplification datasets, our method outperformed strong baseline approaches, including sota and large language models. 4. we release pre-trained models, paraphrase data, a new dataset, and code for training foot_3 . ## conclusion this paper proposed a new method that leverages a pre-trained text-to-text model (t5), paraphrases, context, and linguistic features to address the controllable sentence simplification task. integrating linguistic features enhances the control and interpretability of the generated output. we experimented with three datasets, two meticulously curated by linguistics experts and one translated from english. the results demonstrated notable performance improvements on the sari metric, surpassing the current open-source state-of-the-art model and showing competitive results against an llm. our strategy is particularly advantageous due to its minimal requirement for exemplars during inference. future work could explore obtaining richer representations from the parsed dependency tree with graph neural networks and applying our models in other languages besides portuguese using multilingual models.