A Syntax-Aware Edit-based System for Text Simplification (2021)

Abstract: Edit-based text simplification systems have attained much attention in recent years due to their ability to produce simplification solutions that are interpretable, as well as requiring less training examples compared to traditional seq2seq systems. Edit-based systems learn edit operations at a word level, but it is well known that many of the operations performed when simplifying text are of a syntactic nature. In this paper we propose to add syntactic information into a well known edit-based system. We extend the system with a graph convolutional network module that mimics the dependency structure of the sentence, thus giving the model an explicit representation of syntax. We perform a series of experiments in English, Spanish and Italian, and report improvements of the state of the art in four out of five datasets. Further analysis shows that syntactic information is always beneficial, and suggest that syntax is more helpful in complex sentences.

## introduction automatic text simplification (ats) aims to reduce the linguistic complexity of a text for a certain target audience. ats is useful for people learning foreign languages, suffering cognitive disabilities, or with low literacy level. the area of application of ats is however not restricted to humans, as it has also been used to improve automatic nlp tasks such as parsing @xcite , summarization @xcite , semantic role labeling @xcite , information extraction @xcite and machine translation @xcite @xcite , among others. ats is often cast as a machine translation task, where the system receives complex sentences as input, and produces sentences that are simpler yet maintaining the original meaning. while early ats systems were rule-based, statistical or hybrid @xcite , in the last years neural network based ats approaches have also been proposed @xcite . in particular, sequence-tosequence (seq2seq) neural models @xcite @xcite have shown to obtain state-of-the-art results. such systems are trained on parallel corpora comprising pairs of complex/simple sentences, and implicitly learn the simplification rewrites needed to convert complex sentences into simpler ones. neural seq2seq systems are usually black boxes that are trained on an end-to-end fashion. as a consequence, built models are usually very difficult to interpret, and offer little control or hints that explain why a particular input word sequence has been rephrased. edit-based ats systems try to overcome this limitation by learning the transformations required to convert complex sentences into their simpler counterparts. the set of transformations is limited and known beforehand, and usually comprise edit operations such as delete, removal or lexical substitution @xcite @xcite . because the set of allowed operations is restricted, the search space is considerably reduced. as a consequence, edit-based models are usually sample efficient and require less training examples compared to traditional seq2seq systems @xcite . edit-based ats systems learn edit transformations at a word level, but often those operations are applied to whole phrases. besides, systems need to capture long range relations among words, such as syntactic and phrase structures. for example, in the english sentence presented in version sentence orig. (en) dry air wrapping around the southern periphery of the cyclone eroded most of the deep convection by early on september 12. aut. dry air wrapping around the southern periphery of the cyclone. ## related work sentence simplification (ss) aims to reduce a sentence's reading complexity while preserving its meaning. recently, systems based on neural networks are gaining attention in ats. for more information about previous works in ats the interested reader is referred to these works @xcite @xcite @xcite . in this section, we focus on work that are based on dependencies and on one of the recently developed techniques, which is based on editing to simplify the text. dependency based simplification has proven to be effective in rule based and statistical simplification to analyze the texts and induct rules in the rulebased and hybrid architectures of ats systems (indeed, the first module was a parser and many works have relied on dependency parsers) @xcite @xcite @xcite as well as to perform tree transformations together with statistical machine translation @xcite . however, in the last years neural approaches have gained interest in ats and ss research. the most popular framework has been the seq2seq models, which mainly rely on rnn and transformer based architectures @xcite . while the majority of ats systems are supervised, some works have obtained good results using unsupervised approaches @xcite . one of the main drawback of the aforementioned approaches is the lack of interpretability, i.e., the extend to which a system can explain in a meaningful way why certain actions have been performed. editbased systems @xcite @xcite offer a solution to the interpretability problem, as they directly learn a sequence of edit operations that convert complex sentences into simpler ones. @xcite present a machine translation system that predicts three explicit simplification operations (keep, replace and delete) in aligned pairs of complex-simple sentences. dong et al. (2019) propose editnts, a neural programmer/interpreter that learns to generate edit operations (add, keep, and delete) in a sequential fashion. @xcite design a scoring function that measures the quality of a candidate sentence based on the fluency, simplicity, and meaning preservation and generate the simplified candidate sentences by iteratively editing the given complex sentence. the operations they take into account are removal, extraction, reordering and substitution. syntactic information has been previously used in rule-based systems, for instance, as an indicator to identify the complexity of sentences @xcite . on deep learning systems, graph convolutional networks (gcn) over dependency trees is a usual method that leverages syntactic information into the models, and captures long-range syntactic relations among words. gcns generalize the convolution operation usually applied in images to arbitrary graphs, and allow to refine information associated to nodes according to the information in the neighbor nodes @xcite . they have been successfully used in nlp tasks such as semantic role labelling @xcite , information extraction @xcite and aspect-based sentiment analysis @xcite . contemporaneous to this work, zhe lin (2021) use semantic information in seq2seq systems by including in the graph of the source sentence the predicate-argument relations between content words in a sentence. ## a syntax-aware ats system our system is based on editnts, an edit-based system that has obtained state of the art results on many datasets. we start by briefly describing ed-itnts. then, we describe the graph convolutional network that leverages syntactic information derived from dependency trees. finally, we describe how to integrate the syntactic module into editnts. ## experiments in this section we describe the experiments performed within this work. we start by describing the datasets used for training and testing the system, followed by the experimental setting, which includes a description of the metrics used to evaluate the models. ## results it is worth noting that the results obtained by us when running editnts without syntax are different to those reported in @xcite , and that the gap is specially large in the wikilarge/turkcorpus dataset foot_2 . we attribute this difference to the fact that the reported results in the original paper are obtained using the model that performed best in the test split, whereas we performed model selection according to the development dataset (c.f section 4.2). there is also a slight difference in the newsela dataset, which we attribute to the use of different evaluation scripts. ## analysis in this section we perform an analysis of the results. we start by analyzing examples of the outputs, followed by an ablation study. finally, we analyze the effect of the sentence complexity in the simplification process. ## conclusion and future work in this paper we show that syntactic information is a valid source of information for edit-based text simplification systems. we have presented a system that extends a well known edit-based system with explicit syntactic information derived from dependency trees, by virtue of graph convolutional networks. the results show that the dependency information is useful, obtaining state of the art results on four out of five datasets in different languages. further analysis show that the syntactic information is always beneficial (sometimes by a large margin), and that the improvement is often correlated with the depth of the dependency tree. in the future we want to analyze the inclusion of dependency syntax information into transformer based seq2seq systems. in particular, we want to analyze whether explicitly modeling syntactic information is still a valid approach when the transformer based ats system is initialized with large language models such as bart @xcite .