W iki L ingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization (2020)

Abstract: We introduce WikiLingua, a large-scale, multilingual dataset for the evaluation of cross-lingual abstractive summarization systems. We extract article and summary pairs in 18 languages from WikiHow, a high quality, collaborative resource of how-to guides on a diverse set of topics written by human authors. We create gold-standard article-summary alignments across languages by aligning the images that are used to describe each how-to step in an article. As a set of baselines for further studies, we evaluate the performance of existing cross-lingual abstractive summarization methods on our dataset. We further propose a method for direct cross-lingual summarization (i.e., without requiring translation at inference time) by leveraging synthetic data and Neural Machine Translation as a pre-training step. Our method significantly outperforms the baseline approaches, while being more cost efficient during inference.

## introduction although there has been a tremendous amount of progress in abstractive summarization in recent years, most research has focused on monolingual summarization because of the lack of high quality multilingual resources @xcite . while there have been a few studies to address the lack of resources for cross-lingual summarization @xcite @xcite , the datasets employed are very limited in size. scarcity in the availability of data for crosslingual abstractive summarization can largely be attributed to the difficulty of collecting high-quality, large-scale datasets via crowd-sourcing. it is a costly endeavor, since it requires humans to read, comprehend, condense, and paraphrase entire articles. moreover, subjectivity in content selection, i.e. identifying the salient points of a given article, only adds to the difficulty of crowd sourcing this task @xcite . to overcome the lack of a large-scale, high quality resource for cross-lingual summarization, we present a new benchmark dataset, wikilingua, foot_2 which consists of collaboratively written how-to guides with gold-standard summaries across 18 languages. each article and summary is written and edited by 23 people, and further reviewed by 16 people, on average, which ensures that the content is of a high-quality. the articles describe multiple methods with steps to complete a procedural task from a diverse set of topics, such as "how to make a creamy coffee", "how to exercise to ease back pain". each step contains a one sentence summary followed by a paragraph detailing the instruction, along with an image to illustrate the given instruction, as shown in figure 1 . since the ordering of steps may differ for the same article across languages, we align each step using the corresponding illustrative image, as shown in figure 2 , given that each image is specific to a particular step and shared across languages. 4 our final dataset consists of 141,457 unique english articles. each of the other 17 languages has, on average, 42,783 articles that align with an article in english. to the best of our knowledge, wik-ilingua is the largest dataset with parallel articles and summaries for cross-lingual abstractive summarization to date. this further opens up avenues figure 1 : an example method "watering, feeding and pruning orchids" from the guide for "how to care for orchids". this method consists of three steps where each step has an illustrative image, a one sentence summary (in blue), and a paragraph providing more details about this step (in red). we combine the paragraphs and summaries from all the steps in each method to create article-summary pairs. with the dataset in hand, we evaluate existing approaches for cross-lingual summarization as baselines. we then propose a method for direct crosslingual abstractive summarization, leveraging synthetic data and machine translation as a pre-training step. we show that our method outperforms existing baselines, without relying on translation at inference time. ## data collection and statistics wikihow is an online resource of how-to guides on a diverse set of topics, written and reviewed by human authors. to ensure high quality content, experts are involved in the writing and reviewing process of these guides. 5 each page includes multiple methods for completing a multi-step procedural task along with a one-sentence summary of each step. figure 1 shows an example method from the guide for "how to care for orchids". for this guide, the method "watering, feeding and pruning orchids" includes three steps. each step consists of a unique illustrative image, a one sentence summary and a paragraph providing more details. we combine the paragraphs and summaries from all the steps of each method to create article-summary pairs. thus, the summarization task is framed as follows: given an article detailing instruction on how to complete a procedural task, produce a summary consisting of a list of steps, in the correct order. this builds on prior work that collected data from wikihow for monolingual summarization in english @xcite . we note that, by design, the summaries do not incorporate any potential lead bias, which stands in contrast to single document news summarization, where position is an influential signal @xcite . a majority of the non-english guides on this platform are translated from the corresponding english versions by human writers, who are fluent in both english and the target language. once translated, they are further reviewed by wikihow's international translation team, before they can be published. each of the guides also links to parallel guides in other languages, if available. we collected the guides for all 18 languages available on wikihow, and aligned the steps for each method in each guide using the illustrative images. figure 2 shows an example step from the guide "how to care for orchids" and its aligned step in five selected languages (english, spanish, turkish, russian, and vietnamese). this approach ensures that the alignments of the steps are high-quality since the images are unique to each step and shared across all the languages. we merged the step summaries and paragraphs for each wikihow method as described above to obtain article-summary pairs for all the languages. ## existing multilingual abstractive summarization datasets there have been a few datasets created for multilingual abstractive summarization tasks in recent years, which we describe in this section. multiling'13 and '15. multiple versions of the multiling dataset have been collected by the organizers of multiling workshops @xcite @xcite . the multiling'13 dataset includes summaries of 30 wikipedia articles per language, describing a given topic. for multiling'15, an additional 30 documents were collected for evaluation purposes @xcite . we note that while this dataset contains article and summaries in several languages there are no parallel articles or summaries, which makes it difficult to use this dataset for cross-lingual evaluation. global voices. @xcite collected social network descriptions of news ar-ticles provided by global voices. 6 these descriptions, however, are not written with the purpose of summarizing the article content but rather to draw user clicks on social media; therefore, they have a lower coverage of the original article than a good summary would. to address this problem, the authors crowd-source a small set of summaries, in english, for 15 languages. we report statistics only on the crowd-sourced summaries, given the clickbait nature of the social media descriptions. note that unlike our dataset, this one contains summaries only in english, which makes it difficult to evaluate cross-lingual summarization into other languages. statistics for the datasets are provided in ## cross-lingual experiments following the prior work in cross-lingual abstractive summarization @xcite , we aim to generate english summaries from non-english articles, as an initial study. we experiment with five languages (i.e. english, spanish, russian, turkish, and vietnamese) covering three language families (i.e. indo-european, ural-altaic and austroasiatic). we split the data for each of the four non-english languages into train/dev/test splits. when splitting the english data, we ensure that all articles from the same topic as test articles in any of the four non-english languages, are included in the test set. this leaves us with ∼ 69k english articles that we randomly split into train and dev set (90/10 split). see appendix a.2 for more information. we use large, pre-trained language models as a starting point for our experiments, given their success on a variety of downstream natural language processing tasks @xcite , including state of the art results for text summarization @xcite . in particular, we use mbart @xcite , which is a multi-lingual language model that has been trained on large, monolingual corpora in 25 languages. the model uses a shared sub-word vocabulary, encoder, and decoder across all 25 languages, and is trained as a denoising auto-encoder during the pre-training step. @xcite showed that this pre-training method provides a good initialization for downstream machine translation tasks, particularly in lower resources settings, making this an ideal starting point for our cross-lingual summarization experiments. we also ran initial experiments with non-pretrained transformer models, but the results were significantly worse than those with the pre-trained models. we fine-tune mbart for both monolingual and cross-lingual summarization as a standard sequence-to-sequence model, where the input document is represented as a sequence of tokens (subword units), with a special separator token between each sentence, and a language indicator token at the end of the document. the output summary is represented in a similar manner, with a language indicator token at the beginning of the sequence, to prime the decoder for generation in the target language, as shown in figure 3 . we use fairseq @xcite for all our experiments, and we follow the hyper-parameter settings that were used by @xcite to fine-tune bart for monolingual summarization in english. see appendix a.1 for more details. ## results and analysis when comparing the performance of trans-sum vs. sum-trans, we find that performance depends on the amount of summarization data available in the source language. similar to previous work @xcite , we find that tran-sum works significantly better when the amount of data in the source language is limited. however, as source language training data size increases, we see that the gap in performance decreases, as in the case of spanish, which is similar in size to english, vs. turkish, which is the lowest resource language for summarization in our dataset. this suggests that when the source language data is comparable in size or larger than the target language data, sum-trans approach may be worthwhile to consider, as suggested by @xcite , since it is more cost effective (translating summaries instead of whole articles) and may avoid error propagation from translation systems. amongst the baseline methods, trans-sum-r works the best. it consistently does better than trans-sum baseline, suggesting that round-trip translation to create noisy data can be an effective way to make the model more robust to translation errors at inference time. since we have gold translations (trans-sum g) for each of the articles, we can measure the translation error in the trans-sum system. we see that on average, the round-trip translation method is able to recover about 22% of the performance loss due to translation errors. for direct cross-lingual summarization, we find that the performance of the base model (dc) is worse than the translate-then-summarize baselines for all languages except spanish, where it is better. topic: how to critique a speech: assessing the delivery. article: does the speaker talk in a way that makes you want to keep listening, or is it easy to tune out? ... the way the speaker holds him or herself should project confidence and charisma, making the audience feel engaged and included ... too many "ums", "likes" and "uhs" take away from a speaker's credibility ... a great speaker should have memorized the speech long in advance ... look for signs that the speaker is nervous so you can offer a critique that will help him or her improve next time ... reference: listen to the speaker's voice inflections. watch the speaker's body language. listen for filler words. see if the speech was memorized. assess how the speaker manages anxiety. trans-sum: keep the listener's attention. maintain good posture. memorize speech beforehand. identify signs of nervousness. trans-sum-r: recognize the listener's needs. pay attention to the posture of the speaker. memorize the speech. recognize the signs of nervousness. dc+synth+mt (ours): pay attention to the way the speaker is speaking. notice the way the speaker uses body language. keep track of the words they say. remember what they have to say. watch for signs of nervousness. this suggests that direct cross-lingual summarization is a difficult task and requires a larger amount of data, even with a pre-trained mbart model as a starting point. once we add some synthetic data (dc+synth), we see the performance improves significantly, especially for the lower resource languages (tr and vi), which are on par with the best baseline model. note that the dc+synth models would still be preferable, even over the best baseline, as they give similar performance while being much more cost effective for inference. finally, we see that fine-tuning the mbart model for document-level machine translation, before fine-tuning it for cross-lingual summarization, further improves the performance for all languages. this variant (dc+synth+mt) performs significantly better than all baseline systems for spanish, russian and vietnamese. for turkish, the performance of dc+synth+mt is statistically the same as trans-sum-r; we note, however, that our model is significantly better than the trans-sum baseline, while the trans-sum-r model is not. ## conclusion we present a benchmark dataset for cross-lingual and multilingual abstractive summarization. we then evaluate existing methods in cross-lingual abstractive summarization. we further propose an end-to-end method for direct cross-lingual summarization and show that it achieves significantly better performance than the baselines while being more cost effective for inference. our new benchmark dataset opens up interesting new directions for research in summarization. we would like to further explore multi-source crosslingual summarization architectures, i.e. models that can summarize from multiple source languages in to a target language. another interesting avenue would be to explore the feasibility of multilingual summarization, i.e. building models that summarize articles from any language to any other language for a given set of languages. topic: how to speak portuguese: studying basic portuguese vocabulary. article: practice saying ordinary phrases so you can carry on a casual conversation in any situation ... practice polite phrases like "please" and "thank you". learn a few ways to communicate your understanding or confusion ... if you will be traveling in portugal, learn some of the basic questions and statements you will need to get around ... learn how to ask for rooms, order off a menu, and talk to shopkeepers ... portuguese numbers sometimes reflect the gender of the word they are applied to. for instance, if you are talking about a woman, you would say "uma mulher", but for a man you would say "um homem.". reference: learn a few greetings. master some basic conversational phrases. learn key travel phrases. practice your shopping and restaurant vocabulary. learn how to count in portuguese. trans-sum: learn some basic phrases to communicate basic conversation. practice basic phrases to communicate understanding or confusion. learn some basic questions and statements to communicate when traveling. learn how to order a room or talk to merchants. learn how to talk about gender. trans-sum-r: learn basic conversational phrases. use simple phrases to communicate understanding or confusion. learn basic questions and statements when traveling to portugal. learn how to order a room, menu, or speak to merchants. learn how to say "um" or "um homem" if you're talking about a woman. dc+synth+mt (ours): learn some basic conversational phrases. learn some polite phrases learn some basic phrases when communicating in public. learn some basic phrases when communicating in public. learn how to communicate with people. learn how to communicate with numbers. topic: how to teach english as a second language to beginners: embracing best practices. article: one great way to facilitate learning is to encourage students to avoid speaking languages other than english in the classroom ... when explaining an activity or giving directions about homework, classwork, or a project, you should always give both verbal and written instructions ... this will aid in word association and in pronunciation ... no matter what type of lesson you are teaching or what activity your students are doing, you should monitor them constantly ... teaching english as a second language to beginners is a lot more effective when you use a variety of types of learning ... when teaching beginners or very young students, break the lesson into several pieces of about 10 minutes. topic: how to live an active life with copd: participating in exercise and activities with copd. article: with a serious lung disease like copd, you have to be exceptionally careful when you start physical activity. although exercise can help improve your copd, you still need to ease into activities slowly ... increasing your lifestyle activity is a great way to stay active without overdoing it. these are not cardio activities, but they also help keep your body moving and your lungs working ... when you're ready to progress to more structured exercise, you need to plan to include a warm-up. this is an essential component of safe exercise for those with copd ... unless cleared by your physician, you should only participate in aerobic activities that are low in intensity. this level is the most safe for patients with copd ... aerobic exercises are great to help improve the condition of your lungs and improve your cardiovascular system; however, strength training is an essential form of exercise as well.