Constructing Pseudo-parallel S wedish Sentence Corpora for Automatic Text Simplification (2023)

Abstract: Automatic text simplification (ATS) describes the automatic transformation of a text from a complex form to a less complex form. Many modern ATS techniques need large parallel corpora of standard and simplified text, but such data does not exist for many languages. One way to overcome this issue is to create pseudo-parallel corpora by dividing existing corpora into standard and simple parts. In this work, we explore the creation of Swedish pseudo-parallel monolingual corpora by the application of different feature representation methods, sentence alignment algorithms, and indexing approaches, on a large monolingual corpus. The different corpora are used to fine-tune a sentence simplification system based on BART, which is evaluated with standard evaluation metrics for automatic text simplification.

## introduction automatic text simplification (ats) is a sub-field of natural language processing mainly focusing on the automatic transformation a text from a complex form to a less complex form, and in that way make texts accessible for weaker readers. even though the modern ats techniques vary in scale and efficiency, there is one constant; the need for large parallel corpora of standard and simplified text, in order to train the simplification system. the acquirement of such corpora is however not an easy task. one theoretical option is to collect manually created simplifications, but that process is incredibly time consuming and often not feasible due to the enormous amount of text that is required by modern ats systems. a second option is to leverage already existing sources of parallel texts. one common example is the collection of articles from wikipedia alongside their simple wikipedia counterpart. however, @xcite identified numerous problems to the dual wikipedia approach, for example the fact the simple article most often is not a rewrite of the standard article. this can lead to a variation of the content in the articles that is large enough to make them unsuitable to be included in an aligned corpus. moreover, the simple english wikipedia presents a limitation in text simplification research due to its sole availability in the english language. one way to overcome this problem is to translate the english texts into another language. for instance, @xcite translated the wikilarge dataset @xcite into russian. another possibility would be to follow the approach of @xcite , where a monolingual sentence corpus is divided into a standard and simplified part, and aligned with the best sentence matches between the two corpora. the result is a "pseudo-parallel" monolingual corpus; a parallel monolingual corpus that has been aligned with an unsupervised alignment algorithm rather than been manually constructed or collected from an already divided source, circumventing the previously mentioned problems. the approach was proven to perform well for both english and japanese domains. the aim of the work presented in this paper was two-fold. first, we aimed to create swedish pseudo-parallel sentence simplification corpora 1 from a single monolingual swedish sentence corpus. second, we aimed to investigate how different methods and techniques used during the creation influence the performance of sentence simplification systems trained on the different cor-pora. the research question we explored was: in contrast to the previously mentioned corpora, which are based on alignment of sentences that are extracted from one source of standard sentences and another source of simplified sentences, the construction of a pseudo-parallel monolingual corpus includes the process of deciding if every given sentence should be considered as one of standard or less complexity. for this task, kajiwara and komachi (2018) calculated the, for english text widely used, flesch reading ease score (fres) for each sentence, and in that way determined its complexity. the swedish counterpart to fres is called läsbarhetsindex (lix) @xcite . since lix only measures the lengths of words, sentences, and ratios of long words, additional text complexity metrics have been developed for swedish texts, such as the scream @xcite and svit @xcite measures. with muss, @xcite implemented a method to align paraphrases based on their similarity measures. in order to train a simplifier to produce simplifications, as opposed to just paraphrases, the authors employed access @xcite . access enables controllable output of sequence-to-sequence models by including special control tokens, that-among other thingscan be used to limit the length of decoder output. ## data we used several different datasets for different tasks. the stockholm-umeå corpus (or suc) @xcite is a balanced corpus of swedish texts from the 1990s. the style of text is varied, and it is sometimes used as a baseline for standard use of the swedish language during the time period (see for example @xcite ). in total, the corpus consists of 1, 166, 593 tokens and 74, 245 sentences. the nyponvilja dataset consists of ocr scans of books from sweden's largest publisher of easyto-read books, nypon och vilja förlag, targeting children and youths. each book is graded by human experts with a readability level, where level 1 denotes a book that is the easiest to read and level 6 denotes books that provide the most challenge for the readers. the ccnet dataset is provided by common crawl foot_3 , a non-profit organisation that uses web crawlers to collect an enormous amount of text data from all around the web, and makes it freely available to the public. the organisation collects and publishes a new data snapshot approximately 10 times a year foot_4 , each snapshot in the size range of ≈ 100-300tb whereof 20-30 tb is raw text data. we used the swedish part of the cc-100 dataset, previously used to recreate the training of xlm-r @xcite , for the sentence alignment task. the dataset was created by researchers at statmt foot_5 , by applying the cc-net pipeline to extract datasets for 100 different languages from the common crawl snapshots created during the time period january-december 2018. the swedish dataset comprises 80gb uncompressed text, in the form of 580, 387, 314 paragraphs. from these paragraphs, 61, 959, 899 sentences were extracted for further pre-processing and annotation. the data was further prepared for alignment by roughly following the procedure in @xcite . however, an additional step was introduced to rearrange the data from paragraphs to sentences. this step was added since the task is to align sentences, not paragraphs. it was there-fore also necessary to annotate the dataset on the sentence level. we used the sapis @xcite pipeline to tokenise each sentence with efselab @xcite , and to annotate each sentence with a subset of the scream metrics previously identified by @xcite . pk18 @xcite is a corpus totalling 1,005 texts pairs. each pair consist of an original version of the given text, and a simplified version of the same text. the texts origin from four swedish organisations and municipal-, regional-and state departments; riksförbundet för utvecklingsstörda barn, ungdomar och vuxna (fub), linköpings kommun, region östergötland, and specialpedagogiska myndigheten (spsm). the simplified versions were written by experts, and were manually aligned with the corresponding original version of the texts. pk18 is currently the largest available corpus suitable for use as a gold standard for the evaluation of swedish ats systems. since this work focused on sentence-level simplification, only the pairs aligned in a 1-1 manner were used. the result was a dataset of 467 sentence pairs, with the purpose of being used as the test dataset for the fine-tuned text simplification system. ## implementation this section describes the creation of the pseudoparallel corpora and their usage in text simplification systems. the procedure can be outlined in four steps. first, the sentences were classified as being of either standard or simple complexity. second, the sentences were aligned. third, the different corpora were provided as training data to fine-tune multiple text simplification systems. finally, the performance of each of the systems was assessed with standard evaluation metrics. ## results for simplicity, the created corpora are referred to with the notation of [embedding type] [word alignment algorithm] [alignment threshold]. for example, the corpus in the second row of in all of the corpora performed better than the baseline sari. however, the best performance was shown by both word embedding-based corpora with a filtering threshold of 0.8, with a sari score of over 33. this is higher than the corpus aligned with the help of sentence embeddings, which had a sari of 30.43. of the two bestperforming word embedding-based corpora, the one aligned with the ma algorithm performed with a slightly higher sari score than the aa one. for the bleu score all of the corpora showed lower values than the baseline. the corpus based on word embeddings and with an alignment threshold at 0.9 did however show bleu scores fairly close to the baseline. the rest of the corpora performed significantly lower. it is clear that the number of sentence pairs is closely related to the alignment threshold. for all embedding type/word alignment algorithm combinations, the corpus with a higher threshold also consisted of fewer sentence pairs than their lower threshold counterparts. ## discussion in this section the results for the different conducted experiments will be discussed. ## conclusion the aim of the work presented in this paper was to create a swedish pseudo-parallel sentence simplification corpus from a single monolingual swedish sentence corpus, and to investigate how different methods and techniques used during the creation influence the performance of sentence simplification systems trained on the different corpora. from the results, it can be seen that the model fine-tuned on a corpus created with word-based embeddings, the maximum alignment algorithm, and an alignment threshold of 0.8 performed with the best sari and acceptable bleu scores. it is however unclear how much the different indexing methods impacted the performance of the alignment process, and exactly how the quality of the corpora was affected. both the investigated methods of creating pseudo-parallel corpora for sentence simplification show promising results. future studies should conduct a further investigation on different parameter choices, mainly when constructing the indices to help the alignment process, and explore how they impact the quality of the corpora. the resulting corpora should also be thoroughly evaluated with regard to different aspects of text simplification; with a combination of qualitative evaluations, additional evaluation metrics, and a larger test dataset.