Parallel Text Alignment and Monolingual Parallel Corpus Creation from Philosophical Texts for Text Simplification (2021)

Abstract: Text simplification is a growing field with many potential useful applications. Training text simplification algorithms generally requires a lot of annotated data, however there are not many corpora suitable for this task. We propose a new unsupervised method for aligning text based on Doc2Vec embeddings and a new alignment algorithm, capable of aligning texts at different levels. Initial evaluation shows promising results for the new approach. We used the newly developed approach to create a new monolingual parallel corpus composed of the works of English early modern philosophers and their corresponding simplified versions.

## introduction there has been a clear growth in research in the field of text simplification in recent years @xcite ). text simplification has many potential advantages, such as helping people who suffer from impairments like dyslexia @xcite . most recent approaches are data-driven and require learning text simplification transformations such as sentence splitting or word substitution from a parallel corpus. such a parallel corpus consists of a source document and a target document, which is the simplified version of it. the most widespread parallel corpora for text simplification are the parallel english simple wikipedia corpus @xcite and the more recent newsela corpus @xcite . a parallel corpus is obtained by aligning the units of text between the original-simplified pairs. the alignment can be done at different levels, however most research in the field is focused on sentence simplification @xcite , thus sentence level alignments is the gold standard. automated methods which can asses text similarity are highly desirable in order to produce such parallel corpora. there are few tools that can easily align text in an unsupervised way. massalign foot_0 by @xcite is a python library which can produce alignments at both paragraph and sentence level in an unsupervised manner using a tf-idf model. however, according to @xcite , a doc2vec model would yield results which would imitate human estimates closer than a tf-idf model when computing text similarity. the current work has a twofold contribution. firstly, we extend the existing massalign tool with a doc2vec language model to better capture text similarity and a new alignment algorithm to complement the language model. we manually label two pairs of original-simpliefied documents and use these pairs to evaluate the performance of the doc2vec-based method. we find some promising results, however more evaluation is needed in order to draw a strong conclusion. secondly, we create a novel monolingual parallel corpus from philosophical texts. the novelty lies in the type of texts that constitute the corpus, specifically original philosophical works written by early modern english philosophers and their simplified variants re-written by a group of editors, with the scope to make the texts more accessible while preserving the meaning. the newly developed parallel corpus is created using the improved text alignment tool and is intended to be used as training data for existing text simplification systems. we sample alignments at random to get an idea of the quality of the corpus. our initial findings show that the generated corpus seems to be of high quality. ## dataset the parallel dataset created is built from the works of four early modern philosophers, whose works were originally written in english: george berke-2 github.com/neosyon/simptextalign ley, david hume, john locke and john stuart mill. we obtained the original documents, which were in the public domain, from project gutenberg foot_1 . we obtained their simplified counter-parts of from early modern texts foot_2 . the simplified version of texts were re-written by a team of editors, with the specific goal of making the original document more accessible while keeping the original ideas intact. in order to be able to generate the parallel corpus, we cleaned-up and pre-processed the gathered data such that each document consists of a sentence per line, while empty lines represent the paragraph boundaries. the pre-processing pipeline consists of multiple steps. first, using regular expressions we remove unwanted characters from the texts such as hashtags or underscores, or in the case of the simplified versions, characters that mark omissions or that are used for formatting purposes, which were added by the editors. the next step was to remove the newline characters found in the middle of sentences. this was also done by means of regular expressions. at the end of this step, the documents were formatted such that each line of the document represents a paragraph. once this was achieved, a paragraph was split into sentences by using the punkt tokenizer provided in the nltk foot_3 python library. a list of common encountered abbreviations was supplied to the tokenizer such that sentences are not split midway. ## method we use the open-source python library, mas-salign, developed by @xcite as the base for our new alignment algorithm. we expand the tool with a doc2vec language model and a new alignment algorithm which can take advantage of the new language model. subsection 4.1 describes the language model, while subsection 4.2 describes the alignment algorithm. @xcite evaluated a number of language models for the task of computing document similarity and found that tf-idf embeddings are outperformed by doc2vec embeddings. this is in line with the intuition that a paragraph vector would capture meaning better than a simple bag of words approach since it makes better use of the context around words. therefore, we decided to extend the massalign tool with a doc2vec model. ## discussion the current work has a number of limitations. one of the biggest limitations is that the evaluation of the performance of the doc2vec model is done with limited data points. while, it shows some promising results, the limited evaluation is not enough to allow for a strong conclusion to be drawn. to overcome this, a more extensive intrinsic and extrinsic evaluation should be performed by testing with parallel corpora that have already been aligned, such as the simple wikipedia corpus or the newsela corpus and compare the number and quality of alignments obtained against already established methods. in addition to a better evaluation of the model, a method for determining the hyperparameters of the doc2vec model based on the characteristics of the texts to be aligned, such as number of sentences or number of words, would be highly beneficial and would improve the alignment process in terms of both quality and time investment. moreover, more recent, neural-network based language models, such as sentence-bert or universal sentence encoder, could be considered as an alternative to doc2vec. another limitation of the current work is the lack of evaluation of the produced parallel corpus. while the limited random sampling shows very promising results, this is not enough in order to draw a conclusion regarding the quality of the resulted dataset. a more structured approach to the random sampling method could give better insight into the quality of the dataset. another point of improvement is the preprocessing stage. ensuring that all text formatting elements, such as chapter numbers or titles are removed, would result in a more robust doc2vec model being trained on those documents. moreover, very short paragraphs or sentences are detrimental to the quality of the doc2vec embeddings and do not add a lot of value for the text simplification process, thus they should be filtered out. ## conclusion an approach to unsupervised text alignment was presented in this paper which makes use of doc2vec text embeddings in order to asses similarity between two pieces of texts. additionally, an alignment method derived from the vicinity-driven approach of @xcite has been presented. initial results have shown the current work has slightly better performance compared to the original approach when evaluated on a specific pair of texts, but it has worse results on a different pair of texts. however, due to the limited evaluation, the outcome cannot be readily generalized and more testing is required in order to draw a definitive conclusion. the massalign python library has been extended to include this new doc2vec model. a new monolingual parallel corpus has been created from documents consisting of works of english early modern philosophers and their simplified, corresponding, versions, which were redacted by a group of editors with the goal of making the original documents easier to follow and understand, while preserving meaning. the newly created parallel corpus, together with the extended version of massalign are available at: github.com/stefanpaun/massalign.