Zero-Shot Crosslingual Sentence Simplification (2020)

Abstract: Sentence simplification aims to make sentences easier to read and understand. Recent approaches have shown promising results with encoder-decoder models trained on large amounts of parallel data which often only exists in English. We propose a zero-shot modeling framework which transfers simplification knowledge from English to another language (for which no parallel simplification corpus exists) while generalizing across languages and tasks. A shared transformer encoder constructs language-agnostic representations, with a combination of task-specific encoder layers added on top (e.g., for translation and simplification). Empirical results using both human and automatic metrics show that our approach produces better simplifications than unsupervised and pivot-based methods.

## introduction sentence simplification aims to reduce the linguistic complexity of a text whilst retaining most of its meaning. it has been the subject of several modeling efforts in recent years due to its relevance to various applications @xcite . examples include the development of reading aids for individuals with autism @xcite , aphasia @xcite , dyslexia @xcite , and population groups with low-literacy skills @xcite , such as children and non-native speakers. modern approaches @xcite @xcite view the simplification task as monolingual text-to-text rewriting and employ the very successful encoder-decoder neural architecture @xcite . in contrast to traditional methods, which target individual aspects of the simplification task, such as sentence splitting @xcite or the substitution of complex words with simpler ones @xcite , neural models have no special purpose mechanisms for ensuring how to best simplify text. they rely on representation learning to implicitly capture simplification rewrites from data, i.e., examples of complex-simple sentence pairs. while large-scale parallel datasets exist for english @xcite and spanish @xcite , there is a limited amount of simplification data for other languages. for example, @xcite automatically aligned 7,000 complex-simple german sentences, 1 and brunato et al. (2015) released 1,000 complex-simple italian sentences. but datadriven approaches to simplification, in particular popular neural models, require significantly more training data to achieve good performance, making these datasets better suited for testing or development purposes. unsupervised approaches @xcite which forgo the use of parallel corpora are an appealing solution to overcoming the paucity of data. however, in this paper we argue that better simplification models can be obtained by taking advantage of existing complex-simple data in a high-resource language, and bilingual data in a low-resource language (i.e., a language for which no parallel simplification corpus exists). drawing inspiration from the success of machine translation @xcite @xcite , we propose a modeling framework which transfers simplification knowledge from english to another language while generalizing across language and task barriers during training. the backbone of our model is an encoderdecoder transformer @xcite trained using multi-task learning to either translate, autoencode, simplify, or language model in both high-and low-resource languages. regardless of the task or language, we employ the same base encoder on top of which task-specific transformer layers are added, while language-specific transformer decoders are used to generate the output sequence. since the same base encoder is used for all tasks and languages, the model learns task-and languageagnostic representations. a beneficial side-effect is that the proposed architecture can be trained using one language and tasked to simplify another. as simplifications for multiple languages can be produced within the same model, our approach is more scalable compared to pivot-based methods @xcite . the latter would first translate the complex sentence into a high-resource language, apply a monolingual simplification model, and then translate back the output to the original language. we avoid having to train multiple models and make multiple hops, where each hop can add noise and latency, and instead develop a one-hop crosslingual zeroshot approach. we evaluate our model using english as our high-resource language and german as our low-resource language on two test sets from different domains, and with different end-users in mind. these include textcomplexityde @xcite , a recently created corpus of german wikipedia sentences deemed complex by second language german learners. we also release a second dataset which contains manual simplifications of articles taken from geolino foot_1 , a popular children's magazine. empirical results using both human and automatic metrics show that our approach produces better simplifications than both unsupervised and pivot-based methods. our contributions in this paper are threefold: (1) a cross-lingual architecture which allows the transfer of simplification knowledge from high-to lowresource languages, alleviating the paucity of training data for monolingual simplification; (2) a comprehensive evaluation framework using automatic metrics and human judges; and (3) the release of a dataset in german which we hope will facilitate further research in automatic simplification. 3 2 related work simplification the majority of previous work has focused on english, using large-scale datasets like newsela and wikipedia @xcite . one of the first neural network approaches to simplification was presented in @xcite who use an encoder-decoder lstm, trained with reinforcement learning, to optimize for grammaticality, simplicity, and adequacy. dong et al. (2019) use a programmer-interpreter @xcite , which receives the source sentence as an input, and applies a sequence of edit operations (add, delete, keep). @xcite propose to rerank a diverse set of simplifications according to fluency, adequacy, and simplicity. martin et al. (2020a) translation data, in the form of paraphrases, has also been incorporated into simplification models leading to significant improvements. @xcite use multi-task learning to augment the limited amount of simplification training data. in addition to training on complex-simple sentence pairs, their model employs paraphrases, created automatically using machine translation. @xcite augment a transformer-based simplification model with lexical rules obtained from simple ppdb @xcite , a database of paraphrase rules, automatically annotated with simplicity scores. unlike previous approaches, we do not train models to create training data, either via backtranslation or extracting paraphrases. instead, our model is able to train directly on existing datasets, saving computation power and time. in the future, it would be interesting to explore whether additional datasets or tasks improve simplification performance. crosslingual generation cross-lingual transfer learning-based approaches have originated in machine translation. @xcite translate from one source language to multiple target languages (one-to-many) adding a separate decoder for each. follow-on work @xcite performs translation with multiple encoders and decoders (many-to-many). @xcite and @xcite train multilingual models where all languages share encoder and de-coder parameters, and language tags (prepended to the source sentence) are used to specify the target. multilingual models are also capable of translating between unpaired languages, thereby performing zero-shot translation @xcite @xcite . @xcite propose sharing all parameters but the attention mechanism, while @xcite develop a shared "interlingua layer" at the interface of language-specific encoders and decoders. advances in unsupervised machine translation @xcite have further spurred interest in modeling sequenceto-sequence problems without a parallel corpus. surya et al. (2019) learn from unpaired simple and complex english sentences using a shared encoder, two decoders, denoising, backtranslation and discrimination-based losses. @xcite propose a similar setup, they create a denoising objective by using simple ppdb, replacing simple phrases with complex phrases. reinforcement learning is further used to reward the fluency, adequacy and simplicity. while zero-shot approaches are effective for translating between unpaired languages, they do not consider the case where there exists no parallel data for a language. for simplification, we assume that there is no parallel corpus in the lowresource language (e.g., complex-simple german). furthermore, preliminary results showed that zeroshot translation approaches @xcite which prepend a tag in the source sentence -this tag would indicate the simplification task in our case -perform poorly, basically resulting in the source sentence being copied over with no changes made. we circumvent this by replacing tags with task-specific transformer encoder layers which are added on top of the base encoder. this proposed architecture allows us to transfer supervision signals across languages and is potentially useful for other generation tasks, including question generation @xcite and sentence compression @xcite . ## zero-shot simplification we first define a basic encoder-decoder transformer before adapting it for zero-shot crosslingual simplification with multi-task learning. ## experimental setup training set our training data is summarized in source target size wikilarge englishc englishs 300k wmt19 englishc germanc 6.0m geolino -germans 200k wikipedia -englishs 1.4m the low-resource language. simplification data in english is taken from wikilarge @xcite , a fairly large corpus which consists of a mixture of three automatically-collated wikipedia simplification datasets @xcite @xcite . english-german bilingual data is taken from the wmt19 news translation task. complex monolingual nonparallel data uses one side of the wmt19 translation data. simple english non-parallel data uses sentences extracted from simple wikipedia, a simplified version of wikipedia. simple german nonparallel data uses sentences scraped from geolino @xcite , a german general-interest magazine for children aged between 8-14. test set we evaluated our model on two german simplification datasets, each targeting different users. textcomplexityde @xcite consists of sentences from wikipedia, which were considered complex by second language german learners. these sentences were then simplified by a native german speaker. in addition, we created a test set from geolino. we extracted 20 articles foot_3 from three categories: nature, physics, and people. a trained german linguist then simplified the articles, sentence by sentence, to be understandable for children aged between 5-7 years. our simplifying instructions can be found in the appendix. geolino. we also observe a significant amount of sentence splitting in this dataset. textcomplex-ityde also has a significantly higher translation error rate (ter), however, geolino approximately matches the ter of the wikilarge test set (25.85). while both test sets use a large proportion of substitutions, textcomplexityde has a much large proportion of insertions, which could be explained by the greater amount of sentence splitting. model parameters during training, the base encoder stack consists of six transformer layers, the decoder stack six layers. the simplification stack consists of two weight tied transformer layers, we note that the simplicity level can be increased by applying the stack multiple times at test time. all other stacks consist of a single layer. each layer has a hidden dimension of size 512 and an inner dimension size of 2,048. word embeddings, size 512, were initialized randomly and shared between the encoder and both decoders. we used eight attentional heads. dropout was set to 0.1; source word dropout was also set to 0.1. the discriminator consists of a four layer feedforward network with dropout set to 0.1. the networks were optimized using adam (kingma and ba, 2014). multi-tasking was performed by alternating batches of different tasks. tasks varied in dataset sizes and had different difficulties. as we wished to do equally well with all tasks we select a minibatch from a task with a probability inversely proportional to the training loss of the task. one model was selected using the average fre-bleu score across both development sets. all text was preprocessed using the udpipe tok-enization script @xcite and truecasing was applied. sentencepiece was subsequently applied to the text to split words into subwords, with a sen-tencepiece vocabulary size of 50,000 and a sampling size of @xmath0.25 @xcite . evaluation as there is no single agreed-upon metric for simplification @xcite , we evaluate model output using a combination of four automaticallygenerated scores. 5 these metrics have been previously shown to correlate with human judgments of simplification quality @xcite and essentially quantify: a) whether the output is similar to the gold standard reference (target-based, t); b) whether the output is similar to the source (sourcebased, s); and c) whether the output is simple on its own, with no regard to preserving the meaning of the original sentence (readability-based, r). we indicate the type of each metric using superscripts. bleu t @xcite assesses the degree to which generated simplifications agree with the gold standard references. 6 i-bleu t,s (sun and zhou, 2012) combines self-bleu and bleu to reward systems with high overlap with the reference, and penalize those with high overlap to the source. self-bleu computes the bleu score between the output and the source. it allows us to examine whether the models are making trivial changes to the input. following @xcite , we set the parameter which balances the contribution of the two metrics to α = 0.9. @xcite is calculated using the average of three rewrite operation scores: addition, copying, and deletion. it rewards addition operations when the system's output is not in the input but occurs in the references. analogously, it rewards words deleted/retained if they are in both the system output and the references. 7 fre-bleu t,s,r is a modification of fkgl-bleu @xcite , which combines the difference in fkgl of the source and the output and the i-bleu score. fkgl is a shorthand for the flesch-kincaid grade level readability score which was originally developed for english but has not been ported to german. so instead we use the flesch ## results automatic evaluation previous work @xcite demonstrates how an unsupervised neural mt model can be trained by optimizing two objectives: (1) denoising, where a source sentence is noised and then the corresponding decoder is tasked with reconstructing the original sentence and (2) on-the-fly back-translation, which translates the sentence in inference mode; this translation is then encoded and the task is to reconstruct the original sentence. this model can be easily adapted for simplification by considering simple 8 calculated as @xmath1.5 • asw) where asl is the average sentence length and asw the average number of syllables per word. german and complex german to be different languages (u-nmt). @xcite extend this approach further (u-simp) by adding two losses, which they show result in better simplifications: c das ist nur etwa das doppelte [des weltenergiebedarfs] 4 , [was] 5 bedeutet, [dass] 5 [erdwärmenutzung] 6 [im] 2 großen stil immer auf eine lokale abkühlung des gesteins hinausläuft. r das ist nur etwa das doppelte [des energiebedarfs der welt] 4 . das bedeutet, [dass] 5 die [benutzung] 6 von erdwärme immer dazu führt, [dass] 5 an [sich] 2 diesen stellen das gestein abkühlt. p dabei handelt es sich nur um eine [verdoppelung] 6 [des weltweiten energiebedarfs] 5 , [was] 5 [bedeutet] 2 , [dass] 5 die großflächige [geothermische] 7 [nutzung] 6 immer einer lokalen [k ühlung] 6 [des gesteins] 4 entspricht. z das bedeutet, [dass] 5 erdwärme im großen stil immer auf eine lokale abkühlung [des] 2 gesteins hinausläuft. (a) textcomplexityde c von hier aus erhaltet ihr einen [eindrucksvollen] 1 rundum-blick über die ganze schlucht [hinweg] 2 bis hin zu ihren etwa [5000] 3 meter hohen kraterwänden. r von hier aus erhaltet ihr einen rundum-blick über die ganze schlucht. ihr seht hier bis hin zu ihren etwa [5000] 3 meter hohen kraterwänden. p von hier genießen sie einen [beeindruckenden] 1 rundumblick über die gesamte schlucht bis [zu] 2 den 500 m hohen kraterwänden. z von hier aus erhaltet ihr einen rundum-blick auf die ganze schlucht. (b) geolino the results in we further examined the impact different loss functions have on the performance of zest, and these results are presented in we observed that removing the autoencoding loss (-ae) led to sentences which strayed too far from the source sentence, thereby losing meaning; whereas removing the language modeling loss (-lm) led to sentences being too close to the source sentence, resulting in too little simplification. the inclusion of the adversarial loss (-adv) showed a small overall increase in fre-bleu and a small decrease in sari. human evaluation ## conclusions in this paper we developed a general approach for transferring generation data from high-to lowresource languages. experimental results on transferring simplification knowledge from english to german showed that our model was able to produce significantly better german simplifications than unsupervised and pivot-based approaches. in addition to zero-shot simplification, we showed that our model can generate german simplifications given english input, without any additional training. in the future, we plan to explore this approach with other language pairs and other generation tasks.