S im CS um: Joint Learning of Simplification and Cross-lingual Summarization for Cross-lingual Science Journalism (2023)

Abstract: Cross-lingual science journalism is a recently introduced task that generates popular science summaries of scientific articles different from the source language for non-expert readers. A popular science summary must contain salient content of the input document while focusing on coherence and comprehensibility. Meanwhile, generating a cross-lingual summary from the scientific texts in a local language for the targeted audience is challenging. Existing research on cross-lingual science journalism investigates the task with a pipeline model to combine text simplification and cross-lingual summarization. We extend the research in cross-lingual science journalism by introducing a novel, multi-task learning architecture that combines the aforementioned NLP tasks. Our approach is to jointly train the two high-level NLP tasks in SimCSum for generating cross-lingual popular science summaries. We investigate the performance of SimCSum against the pipeline model and several other strong baselines with several evaluation metrics and human evaluation. Overall, SimCSum demonstrates statistically significant improvements over the state-of-the-art on two non-synthetic cross-lingual scientific datasets. Furthermore, we conduct an in-depth investigation into the linguistic properties of generated summaries and an error analysis.

## introduction cross-lingual science journalism is a recently introduced task that produces science summaries in a target language from scientific documents in a source language while emphasizing simplification @xcite . a real-world example of crosslingual science journalism is spektrum der wissenschaft foot_0 . it is the german version of scientific american and an acclaimed bridge between local readers and the latest scientific research in germany. spektrum's target audience are non-expert adults, so their journalists summarize complex scientific concepts in easy-to-understand terms in their local language. these scientific summaries are distinctive from other scientific texts/abstracts due to their length, which is more concise than regular scientific articles, containing non-complex and simplified terms, and are generated in a different language from their source. previous work on science journalism, including monolingual and cross-lingual, is quite limited. monolingual science journalism has been investigated as a downstream task of abstractive summarization @xcite with customized monolingual datasets @xcite . these datasets are, unfortunately, not suitable for cross-lingual science journalism. moreover, crosslingual science journalism has been investigated as a fusion of cross-lingual summarization and text simplification with a pipeline model @xcite with cross-lingual scientific datasets @xcite . in the dawn of cross-lingual summarization, various pipeline models @xcite @xcite with synthetic cross-lingual datasets have been introduced to explore the task. later, cross-lingual summarization models have been focused towards multi-task learning (mtl) @xcite @xcite and direct cross-lingual summarization with non-synthetic datasets @xcite . due to the limited prior work in cross-lingual science journalism, this task needs further investigation. @xcite introduce the crosslingual science journalism task with a pipelinebased model -select, simplify and rewrite (ssr). ssr consists of three components -an extractive summarizer as select, a reinforcement simplification model as simplify, and a crosslingual abstractive summarizer as rewrite. ssr is a plug-and-play model which shows promising results for cross-lingual science journalism. however, this model has a few limitations. the select component is not a trainable model and cannot be fine-tuned for the training data. the simplify and rewrite components require independent training and hyper-parameter settings. therefore, these models cannot learn the mutual representations for simplification and cross-lingual summarization. we find this as a research gap. to the best of our knowledge, there are no available models for crosslingual science journalism that can be trained at once on training data for simplification and crosslingual summarization. to fill this gap, we propose an mtl-based model -simcsum that jointly trains for simplification and cross-lingual summarization to generate crosslingual popular science summaries. simcsum consists of one shared encoder and two independent decoders for each task based on a transformer architecture, where we consider cross-lingual summarization as our main task and simplification as our auxiliary task. the proposed model also leads us to two important research questions. (rq1) can jointly trained models learn and perform better than pipeline models for cross-lingual science journalism? (rq2) which linguistic features can effectively measure conciseness and readability to compare cross-lingual science journalism models? to investigate these research questions, we empirically evaluate the performance of simcsum against ssr and several existing cross-lingual summarization models on two cross-lingual scientific datasets. we conduct a human evaluation to find the linguistic qualities of generated summaries. we further analyze the outputs for various lexical, readability and syntactic-based linguistic features. we also perform an error analysis to assess the quality of outputs. ## proposed model our model jointly trains for simplification and cross-lingual summarization (simcsum). we first define mtl and our tasks, and then discuss the architecture of our proposed model. ## readability analysis: spektrum the automatic evaluation of models show higher performance of simcsum for rouge and bertscore for wikipedia. however, it is not the case for fre. interestingly, for spektrum, simcsum shows better performance for fre. so we decide to further investigate readability with lexical diversity, readability scores and syntactic analysis to determine the quality of generated summaries. these types of analyses are well-known in nlp for textual analysis @xcite @xcite @xcite . the lexical diversity and readability scores are computed over all spektrum's reference summaries (gold) and outputs of mbart, ssr and simcsum. the gold summaries' score is a guideline for how similar the models' outputs are to gold summaries. ## error analysis to further explore the challenges of generating cross-lingual science summaries, we consider the base model -mbart and simcsum for analyzing the produced errors. we randomly select 25 × 2 (for each model) summaries from the simcsum and mbart outputs. we note three main categories of errors in the manual inspection. non-german words. this is the error type where the models either produce non-existent german words or partially english-german or words in another language. we find that mbart is more prone to produce such errors. we note that it is due to the imbalance between the pre-trained and finetuned dataset sizes. these models are pre-trained on many languages and usually fine-tuned on comparatively smaller data. simcsum tends to produce fewer errors due to data augmentation (simplification data) during the training. wrong named entities. this is the error type where the models produce wrong named entities, such as cities or country names and persons' first and last names. we find that both models tend to produce such errors, however, the percentage of such errors is quite low. we note that the models overestimate or underestimate the probability of word sequences present in data. unfaithful information. this is the error type where we find some (new) information in generated summaries that is not faithful to the source documents. we note that this error is caused by long inputs where the model tends to hallucinate and generates some content that cannot be verified from the source. we find that simcsum makes similar errors as mbart for this error type. ## conclusions in this paper, we investigate a recently introduced task -cross-lingual science journalism. we propose a novel multi-task model -simcsum-that combines two high-level nlp tasks, simplification and cross-lingual summarization. simcsum jointly trains for reducing linguistic complexity and crosslingual abstractive summarization. our empirical investigation shows the significantly superior performance of simcsum over the pipeline-based ssr model and other baselines on two non-synthetic cross-lingual scientific datasets. this is confirmed by human evaluation. furthermore, our in-depth linguistics analysis shows how multi-task learning in simcsum has lexical and syntactic impacts on the generated summaries. we perform error analysis to find what kind of errors has been produced by the model. in the future, we plan to add modules for linguistically informed simplification.