An Unsupervised Method for Building Sentence Simplification Corpora in Multiple Languages (2021)

Abstract: The availability of parallel sentence simplification (SS) is scarce for neural SS modelings. We propose an unsupervised method to build SS corpora from large-scale bilingual translation corpora, alleviating the need for SS supervised corpora. Our method is motivated by the following two findings: neural machine translation model usually tends to generate more high-frequency tokens and the difference of text complexity levels exists between the source and target language of a translation corpus. By taking the pair of the source sentences of translation corpus and the translations of their references in a bridge language, we can construct large-scale pseudo parallel SS data. Then, we keep these sentence pairs with a higher complexity difference as SS sentence pairs. The building SS corpora with an unsupervised approach can satisfy the expectations that the aligned sentences preserve the same meanings and have difference in text complexity levels. Experimental results show that SS methods trained by our corpora achieve the state-of-the-art results and significantly outperform the results on English benchmark WikiLarge.

## introduction the task of sentence simplification (ss) is to rephrase a sentence into a form that is easier to read and understand while conveying the same meaning @xcite . ss is first used as a preprocessing task of machine translation, and then is used to increase accessibility for those with cognitive disabilities such as aphasia @xcite , dyslexia @xcite , and autism @xcite . most popular methods @xcite @xcite @xcite have addressed ss as a monolingual machine translation task that translating from complex sentences to simplified sentences, whose performance rely heavily on the quality of parallel ss corpus. however, much work @xcite @xcite pointed out that the public english ss benchmark (wiki-large @xcite ) which align sentences from english wikipedia and simple english wikipedia are deficient, because they contain a large proportion of inaccurate or inadequate simplifications, which lead to ss methods that generalize poorly. additionally, parallel ss corpus is difficult to obtain in all languages other than english. therefore, in the paper, we focus on how to build ss corpora in multiple languages using an unsupervised method. some work @xcite ; @xcite built pseudo parallel corpora by searching the nearest neighbor sentence for each sentence based on embedding model from a largescale text corpus. we can see that the built corpora are more like paraphrase corpora instead of ss corpora. it is because: (1) it only guarantees that the aligned sentences are highly similar, and cannot guarantee that the aligned sentences preserve the same meanings; and (2) each sentence pair does not distinguish between the simple sentence and the complex sentence. in this paper, we present an unsupervised method to build ss parallel corpora based on a large-scale bilingual translation corpus. many languages have large-scale training corpora, which lie at the core of the recent success of neural machine translation (nmt) models. if we plan to build a pseudo english ss corpus, the main idea of our method is demonstrated in figure 1 : we use a translator to translate the sentence in a bridge language (e.g., german) into english, and pair them as a pseudo ss sentence pair. the idea is motivated by the following two findings : each sentence pair in the pseudo ss corpus preserves the same meaning. we measure the difference of sentence complexity between the original sentence and the translated sentence using flesch reading-ease score @xcite , and keep pairs with a higher complexity difference as ss corpus. for each remaining sentence pair, the sentence with a higher score will be treated as the simple sentence and the other sentence as the complex. the contributions of our paper are as follows: researchers have attempted to design ss methods in other languages such as spanish @xcite , portuguese @xcite , japanese @xcite , french @xcite and italian @xcite . but, these approaches are limited by the availability of parallel ss corpora. in this paper, we propose a general framework that can be used to obtain large-scale ss data for these languages to train neural ss methods. ## method in this paper, we present our unsupervised method to build ss corpora in multiple languages, which is motivated by high-frequency tokens generated by nmt modeling and the difference of text complexity levels between bilingual translation corpora. the overall architecture of the proposed method for building the english ss corpus is illustrated in figure 2 . our method consists of two steps (paraphrase generation and selectors) to build ss corpora, to achieve the following two requirements: ## sentence simplification corpora our unsupervised method can be used to languages with large-scale bilingual translation corpora. according to this principle, we choose the three languages (english, french, and spanish) to build ss corpora, to train ss systems. ## experiments we design experiments to answer the following two questions: q1. effectiveness: is the english ss corpus built by our method a better dataset compared with the benchmark wikilarge and the dataset built by @xcite )? q2. universality: can our unsupervised method be used to build ss corpora for other languages with large-scale bilingual translation corpora? ## conclusions we propose an unsupervised method to build large parallel corpora for training sentence simplification (ss) models. our method consists of a highresource bilingual translation corpus and a translator. unsupervised ss models can be trained by pairing the source sentences in the bilingual translation corpus and the translated sentences of the bridge language generated by the translator. we conduct experiments and show that ss models trained on synthetic data generated by our approach significantly outperform the results on english benchmark wikilarge. in the future, we plan to investigate the influence of different text readability methods. source he was diagnosed with inoperable abdominal cancer in april 1999. reference he was diagnosed with abdominal cancer in april 1999. ours he was diagnosed with stomach cancer in april 1999. source heavy rain fell across portions of britain on october 5, causing localized accumulation of flood waters. reference heavy rain fell across britain on october 5, causing accumulation of flood waters. ours heavy rain fell on parts of the uk on october 5, causing localized flooding. source admission to tsinghua is extremely competitive. reference admission to tisinghua is competitive. ours admission to tsinghua is very competitive. source they are culturally akin to the coastal peoples of papua new guinea. reference they are similar to the coastal peoples of papua new guinea. ours they are similar in culture to the coastal peoples of papua new guinea.