Split-and-Rephrase in a Cross-Lingual Manner: A Complete Pipeline (2021)

Abstract: Split-and-rephrase is a challenging task that promotes the transformation of a given complex input sentence into multiple shorter sentences retaining equivalent meaning. This rewriting approach conceptualizes that shorter sentences benefit human readers and improve NLP downstream tasks attending as a preprocessing step. This work presents a complete pipeline capable of performing the split-and-rephrase method in a cross-lingual manner. We trained sequence-to-sequence neural models as from English corpora and applied them to predict the transformations in English and Brazilian Portuguese sentences jointly with BERT’s masked language modeling. Contrary to traditional approaches that seek training models with extensive vocabularies, we present a non-trivial way to construct symbolic ones generalized solely by grammatical classes (POS tags) and their respective recurrences, reducing the amount of necessary training data. This pipeline contribution showed competitive results encouraging the expansion of the method to languages other than English.

## introduction text simplification (ts) is the process of modifying natural language to reduce complexity and improve both readability and understandability @xcite . a simplified vocabulary or a simplified text structure can benefit people with limited language skills, such as those with low education levels, children, non-native speakers, and individuals with learning impairments (e.g., autism, dyslexia, or aphasia) @xcite . furthermore, when applied as a preprocessing step, ts may also improve the performance of several natural language processing (nlp) tasks, such as parsing, machine translation, semantic role labeling, text summarization, information extraction, among others @xcite this bottle was used until 2002 when it was dropped in favor of a traditional bottle . this bottle was used until 2002 . it was dropped in favor of a traditional bottle . he sought medical care in rome , but it was unsuccessful , and he died at the age of 42 . he sought medical care in rome . it was unsuccessful . he died at the age of 42 . 2019a; @xcite . most work on ts has concentrated on analyzing specific characteristics at the sentence level, fashioning the task of sentence simplification (ss) @xcite . ss applications aim to identify and solve two main aspects: lexical complexity, which refers to difficult words or expressions in the text (e.g., non-frequent words, specific terminologies, foreign words, etc.) @xcite ; and syntactic complexity, which refers to the length of the sentences and their grammatical complexities (e.g., number of subordinate or coordinate clauses, unusual sentence structures, depth of the syntactic tree, among others) @xcite . split-and-rephrase, proposed by narayan and co-authors @xcite , is a novel sentence simplification task that has attracted much research interest in the nlp field. its goal is to split and rephrase a complex input sentence into shorter sentences that retain equivalent meaning (see examples in figure 1 ). neither deletion nor lexical/phrasal simplification is intended. the core of this process is to properly make the syntactic transformations required by the split action (e.g., turn a relative clause into a main clause). this work innovates from previous split-andrephrase methods. we present a complete pipeline capable of performing the split-and-rephrase challenge by combining trained sequence-to-sequence neural models that rely on symbolic vocabularies accompanied by bert's masked language modeling. the main contribution is to construct a crosslingual solution that deals both with english and portuguese sentences. in addition, we enhanced a preliminary work @xcite promoting analysis against complete reference test sets and comparing results to similar models/pipelines. to the best of our knowledge, this is the first complete pipeline to address split-and-rephrase in a crosslingual manner, encouraging the expansion of the method to languages other than english. ## related work as discussed by narayan and colleagues @xcite , split-and-rephrase method must be distinguished from other sentence rewriting tasks, such as sentence compression, sentence fusion, and sentence paraphrasing. furthermore, in contrast to the conventional sentence simplification task, splitand-rephrase does not entail loss of information, thus targeting the meaning preservation despite the split behavior @xcite . in an observational study, gasperin and colleagues @xcite stated that sentence splitting was the most frequent syntactic simplification operation used by an annotator when creating simplified texts. among the techniques to perform the transformations required by sentence splitting, niklaus et al. @xcite segregates them into three classes: (a) syntax-driven rulebased approaches that use a set of hand-written rules to detect points where sentences may be split @xcite ; (b) semantic parsing based approaches that aim to decompose sentences into minimal semantic units that may be split into individual output sentences @xcite ; and (c) data-driven approaches where the splitting point and transformations are learned automatically from training in aligned corpora of complex-simple sentences @xcite . concerning split-and-rephrase previous works, narayan et al. @xcite recently presented data-driven baseline models to help with some insights about the task, together with the websplit benchmark corpus. after that, aharoni and goldberg @xcite established more robust baselines augmenting sequence-to-sequence neural models with copymechanism @xcite , and also released an updated version of websplit to reduce overlap in the data splits. given the small vocabulary and the unnatural linguistic expressions present in websplit corpora, botha et al. @xcite compiled the wikisplit corpus reuniting more than one million naturally occurring sentence rewrites obtained from mining english wikipedia's edit history. later, niklaus et al. @xcite constructed the minwikisplit corpus running dissim framework @xcite over the wikisplit data and applied a set of 35 handwritten transformation rules to decompose source sentences in more split simplified counterparts. as for the portuguese language's split-andrephrase task, based on the literature surveyed, we found no specific corpus built for this purpose. however, leal et al. @xcite made available the porsimplessent data set, a brazilian portuguese corpus to study sentence readability assessment, which we incorporated into this work to further test our pipeline. ## methodology and data we define the split-and-rephrase task as follows. given a complex sentence c, the goal is to produce a simplified text t consisting of a sequence of sentences t 1 , t 2 , . . . , t n , @xmath0, in such a way that t preserves the meaning of c. in this section we specify the details about the implementation of our proposed pipeline and all the above mentioned split-and-rephrase corpora employed in this work. ## experiments we assembled two different training setups concerning the sequence-to-sequence neural models attending different corpora as follows. training setup 1 from wikisplit @xcite training corpus, we selected aligned sentence pairs formed only by alphanumerical characters, commas, periods and whitespaces, eliminating any foreign/special characters as this corpus is slightly noisy as admitted by the authors foot_1 . this cut extracted 485,120 alignments, consolidating the training set for this first setup. we then executed our aforementioned custom implementation to construct the symbolic vocabulary and obtained 247 different items to train the first model; training setup 2 from minwikisplit @xcite ) corpus, we first established a limit to select aligned sentences with a maximum length of 100 tokens, due to the fact that this corpus has few long sentences that would lead to long padding. this first cut extracted 197,496 alignments. we then repeated the prior setup selecting aligned sentence pairs formed only by alphanumerical characters, commas, periods and whitespaces, finally consolidating a training set of 122,104 alignments. the symbolic vocabulary obtained by our custom implementation was composed of 230 different items to train the second model. ## discussion to achieve a detailed analysis, we manually inspected some of the predictions from the pipeline with the two built models bringing some examples to help explain the scores illustrated in tables 2 and 3 . these extracted examples are in in example 1, the same complex input sentence is transformed into different simplified outputs according to their training setups: output 1 performed a single split whereas output 2 performed two splits. this different behavior explains the higher numbers in the #s/c column and the lower numbers in the #t/s column from training setup 2. these two measures confirmed the hypothesis that models trained in minwikisplit might capture the tendency to split source sentences into multiple output ones. such multiple sentences may not be good for humans readers, but may benefit nlp downstream tasks. in example 2, even though both setups generated perfect outputs in terms of meaning preservation, only the output 2 achieved maximum bleu score since it is the unique that matched perfectly against one of the references. this brings the evidence that bleu requires high-quality data to produce more precise outcomes, ideally with multiple correct references @xcite . another limitation from bleu is the low correlation with simplicity when sentence splitting is performed, but it still holds the high correlation with human assessments of grammaticality and meaning preservation @xcite . in example 3, we note interesting contrasts produced by the models from the distinct training setups. while output 1 retained the same structure from the complex input sentence, output 2 promoted the reordering of the words preserving equivalent meaning and showing low conservatism. the only little mistake is observed by the repetition of word "was" in the output 2. @xcite 76.42 2.08 16.55 13.30 dissim @xcite 51.96 4.09 11.91 0.76 lastly, example 4 illustrates the pipeline working in a cross-lingual manner. output 1 produced a pronoun "ele" (he) instead of repeating "o projeto gemini" (the gemini project), as observed in output 2. it is exactly the same behavior seen in the english example 2 reflected for brazilian portuguese sentences. recent studies that analyze eye movements of human readers interestingly reveal that they quickly retrieve information upon finding pronouns when referred to a close syntactic antecedent @xcite . our detailed inspection together with the prediction logs confirmed that the pipeline could split complex input sentences into shorter simplified ones, often preserving equivalent meaning successfully. more than that, it showed ability to perform equivalent syntax transformations for different languages (english and portuguese). on the other hand, some of the predictions reveal common mistakes from sequence-to-sequence models, such as repetition or omission of tokens and "hallucination" of new unwanted information. another limiting factor is the noise from unsupported or missing statements observed in the referred test data sets. the low quality references eventually harmed the bleu scores in those cases. ## conclusion split-and-rephrase task conceptualizes that shorter sentences are generally better processed by hu-mans and by nlp downstream applications. we presented a complete pipeline for the split-andrephrase method that attends in a cross-lingual manner english and portuguese languages, by integrating sequence-to-sequence neural models and bert's masked language modeling. in contrast to conventional approaches, we train models making use of symbolic vocabularies defined by a custom implementation. this approach speeds up the training process and enables the models to acquire specific knowledge on how to split symbolic sequences, then demanding only a little step to convert them back to genuine texts in respective languages. furthermore, the pipeline is capable to foster new words to rewrite the complex input sentence, thanks to bert's mlm predictions. unlike most previous works on split-and-rephrase, we employed the four state-of-the-art corpora for the task and also a brazilian portuguese corpus, showing competitive results to equivalent approaches. as future work, we plan to exploit our pipeline in more languages. we should also inspect the effectiveness of the transformer architecture in replacement of the sequence-to-sequence models. moreover, we intend to promote an extrinsic evaluation of the benefits of the split-and-rephrase method in nlp downstream applications. example 1 (from wikisplit data set) input gavin confessed to the murder of george pollard and was held in the round house until he was hung on the 6th april 1844 , his body was buried south of the round house . ref. gavin confessed to the murder of george pollard and was held in the round house until he was hanged on 6 april 1844 . his body was buried south of the round house . output 1 gavin confessed to the murder of george pollard and was held in the round house until he was hung on the 6th april 1844 . • his body was buried south of the round house . output 2 gavin confessed to the murder of george pollard . • gavin was held in the round house until he was hung on the 6th april 1844 . • his body was buried south of the round house . example 2 (from websplit v0.1 data set) input a.s. livorno calcio are managed by christian panucci who is attached to the club genoa cfc . ref. 1 a.s. livorno calcio are managed by christian panucci . christian panucci is attached to the club genoa cfc . ref. 2 a.s. livorno calcio is managed by christian panucci . christian panucci played football for genoa c.f.c. ref. 3 a.s. livorno calcio are managed by christian panucci . christian panucci played football for genoa c.f.c. ref. 4 a.s. livorno calcio is managed by christian panucci . christian panucci is attached to the club genoa cfc . output 1 a.s. livorno calcio are managed by christian panucci . • he is attached to the club genoa cfc . output 2 a.s. livorno calcio are managed by christian panucci . • christian panucci is attached to the club genoa cfc . example 3 (from wikisplit data set) input born in huzhou , zhejiang , qian was trained in traditional chinese philology , and was a student of zhang binglin . ref. born in huzhou , zhejiang , qian was trained in traditional chinese philology . he was a student of zhang binglin . output 1 born in huzhou , zhejiang , qian was trained in traditional chinese philology . o projeto gemini é resultado de uma associac ¸ão de sete países. o projeto gemini envolve a construc ¸ão de dois telescópios com um espelho de oito metros de diâmetro. output 1 o projeto gemini é resultado de uma associac ¸ão de sete países .