Abstractive Summarization for the U krainian Language: Multi-Task Learning with Hromadske.ua News Dataset (2023)

Abstract: Despite recent NLP developments, abstractive summarization remains a challenging task, especially in the case of low-resource languages like Ukrainian. The paper aims at improving the quality of summaries produced by mT5 for news in Ukrainian by fine-tuning the model with a mixture of summarization and text similarity tasks using summary-article and title-article training pairs, respectively. The proposed training set-up with small, base, and large mT5 models produce higher quality résumé. Besides, we present a new Ukrainian dataset for the abstractive summarization task that consists of circa 36.5K articles collected from Hromadske.ua until June 2021.

## introduction reading a large number of documents is a timeconsuming and frequently tedious process that requires a substantial investment of human resources. that is why creating pithy abstracts for financial articles, social media news, or even bug reports originated many real-life use cases for automatic summarization. in the meantime, the rapid development of ai methodology and the latest nlp progress with large transformer language models pushed the boundaries of text generation. producing a résumé for a document constitutes one of the applications for text generation that keeps attracting more attention of the academic community (see section 2.1) and practitioners. abstractive summarization is a generative task that foresees automatic creation of document summary by synthesizing an input while preserving its gist. observed limitations of language models (see section 2.1) frequently challenge this definition. recent papers discuss the problem of information distortion when it comes to solutions for the english language; however, for low-resource languages like ukrainian the differences between real and expected results might be even more significant. this paper tries to improve the ability of language models to capture a gist of text in order to generate summaries of better quality for news articles in ukrainian by finetuning the multilingual t5 transformer on the corpora that exploits training data for both summarization and text similarity tasks simultaneously and thus guiding the model to the essence of each article. the second objective is to construct and introduce the dataset of ukrainian news that can be further exploited for abstractive summarization. the next section presents problems of abstractive summarization and discusses mt5 architecture and training. section 3 focuses on training data, methodology and evaluation strategy. section 4 concludes with results and discussion.