V i T 5: Pretrained Text-to-Text Transformer for V ietnamese Language Generation (2022)

Abstract: We present ViT5, a pretrained Transformer-based encoder-decoder model for the Vietnamese language. With T5-style self-supervised pretraining, ViT5 is trained on a large corpus of high-quality and diverse Vietnamese texts. We benchmark ViT5 on two downstream text generation tasks, Abstractive Text Summarization and Named Entity Recognition. Although Abstractive Text Summarization has been widely studied for the English language thanks to its rich and large source of data, there has been minimal research into the same task in Vietnamese, a much lower resource language. In this work, we perform exhaustive experiments on both Vietnamese Abstractive Summarization and Named Entity Recognition, validating the performance of ViT5 against many other pretrained Transformer-based encoder-decoder models. Our experiments show that ViT5 significantly outperforms existing models and achieves state-of-the-art results on Vietnamese Text Summarization. On the task of Named Entity Recognition, ViT5 is competitive against previous best results from pretrained encoder-based Transformer models. Further analysis shows the importance of context length during the self-supervised pretraining on downstream performance across different settings.

## introduction in recent years, transformer-based architecture models and pretrained language models (lms) have played a crucial role in the development of natural language processing (nlp). large pretrained models such as elmo @xcite , @xcite , bert @xcite is trained on large corpora and have the ability to derive contextual representation of the language(s) in the training data. after pretraining is complete, these models achieved state-of-the-art results on a broad range of downstream tasks @xcite . these self-supervised learning methods make use of learning objectives such as masked language modeling (mlm) @xcite where random tokens in the input sequence are masked and the model attempts to predict the original tokens. the successes of pretrained models in english have inspired new research efforts to develop pretrained models in other languages such as vietnamese (i.e., phobert @xcite and @xcite ) and italian @xcite . there are also ongoing efforts to develop multilingual pretrained models (mt5 @xcite , mbart @xcite ), in order to improve performance across multiple languages by learning both general and languagespecific representations. a short time ago, bartpho @xcite , a large pretrained sequence-to-sequence model for vietnamese inheriting bart style @xcite , demonstrated the effectiveness of pretrained language models on vietnamese abstractive summarization. nevertheless, there are some past works that have shown that t5 architecture @xcite might outperform bart in some aspects (i.e., @xcite ). inspired by that, we propose vit5, trained on the vietnamese monolingual subset of cc100, following the architecture and training methodology in @xcite . we perform exhaustive comparisons on downstream performance to many different pretrained transformer-based models @xcite @xcite . specifically, we finetune the vit5 on two summarization datasets, wikilingua @xcite and vietnews @xcite , and one named entity recognition dataset @xcite ). text summarization is an important downstream task whose input is a free-form text paragraph or document(s), and the output sequence is expected to be a short summarization of the input. vit5 achieves state-of-the-art results on both two of the single-document summarization tasks. we also perform an analysis on the max-length hyperparameter for input and output sequences during self-supervised learning and showed that longer lengths that match the downstream document's length lead to better result. for ner, we reformulated the per-token classification task into a generation task, where the decoder reconstructs the original input sentence with inserted named entity tags following each token @xcite . this simple and straightforward formulation achieves competitive results in comparison to direct per-token classification done on encoder-only model @xcite ). ## related work there are lots of abstractive summarization studies in english. in an early example, @xcite employed a bottom-up content selector (bottomup) to determine which phrases in the source document should be part of the summary, and then a copy mechanism was applied only to pre-select phrases during decoding. their experiments obtained significant improvements on rouge for some canonical summarization datasets. in recent years, pretrained language models have been used to enhance performance on language generation tasks. @xcite developed a transformer-based encoder-decoder model so that pretrained language models like bert can be adopted for abstractive summarization. here, the authors proposed a novel document-level bert-based encoder (bertsum) and a general framework encompassing both extractive and abstractive summarization tasks. based on bertsum, @xcite introduced gsum that effectively used different types of guidance signals as input in order to generate more suitable words and more accurate summaries. this model accomplished state-of-the-art performance on four popular english summarization datasets. meanwhile, there are a small number of studies on vietnamese text summarization. most of these focus on inspecting extractive summarization. the researchers @xcite com-pared a wide range of extractive methods, including unsupervised ranking methods (e.g., lexrank, lsa, kl-divergence), supervised learning methods using tf-idf and classifiers (e.g., support vector machine, adaboost, learning-2-rank), and deep learning methods (e.g., convolutional neural network, long-short term memory). similarly, the authors @xcite also evaluated the extractive methods on their own dataset, which was released publicly as a benchmark for future studies. recent work @xcite investigated the combination of a pretrained bert model and an unsupervised k-means clustering algorithm on extractive text summarization. the authors utilized multilingual and monolingual bert models to encode sentence-level contextual information and then ranked this information using the k-means algorithm. their report showed that monolingual models achieved better results compared when to multilingual models performing the same extractive summarization tasks. however, due to the lack of studies on vietnamese abstractive summarization, we compare both multilingual and monolingual encoder-decoder models. ## vit5 in this section, we will explain our newly released vit5 models, the vocabulary generation steps, the pretraining data, and the training setup. ## discussion according to the results on both wikilingua and vietnews summarization tasks ( interestingly, a more general domain of pretraining texts can lead to a better domain-specific summarization performance. in section 4.4.1, our vit5 models while being trained on a more general corpus (cc100), outperform current models that are trained on news-related corpus. more technical domains such as laws, medicals, or engineering are not tested as we leave these domainspecific summarization tasks for future studies. the slightly better performance of vit5 base 1024-length compared to vit5 base 256-length suggests that longer document summarization (more than 512 tokens) need a comparatively longer context length during the pretraining stage. ## conclusion we introduce vit5, a pretrained sequence-tosequence transformer model for the vietnamese language. leveraging the t5 self-supervised pretraining formulation on massive and high-quality vietnamese corpora, we showed that finetuned vit5 models are performant on both generation and classification tasks. we exhaustively compare vit5 with other pretrained formulations on both multilingual and monolingual corpora. our experiments show that vit5 achieves state-of-theart results on summarization in both wikilingua and vietnews corpus, and competitive results in generating named entity recognition (ner) on the phoner covid19 dataset. we also analyze and discuss the importance of context length during the self-supervised pretraining stage, which strongly influences and positively leads to better downstream performance.