m L ong T 5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences (2023)

Abstract: We present our work on developing a multilingual, efficient text-to-text transformer that is suitable for handling long inputs. This model, called mLongT5, builds upon the architecture of LongT5, while leveraging the multilingual datasets used for pretraining mT5 and the pretraining tasks of UL2. We evaluate this model on a variety of multilingual summarization and question-answering tasks, and the results show stronger performance for mLongT5 when compared to existing multilingual models such as mBART or M-BERT.

## introduction in recent years, there has been development of making transformer-based models more efficient so that they can handle longer input sequences. many of the models though have been english-only, making them inapplicable to other languages. in this paper, we present our work in extending one of these models to be able to handle multilingual data. our model, called mlongt5, takes advantage of the efficient architecture of longt5 @xcite , and has been pretrained on the multilingual mc4 dataset @xcite to be able to work on multilingual tasks. we have applied mlongt5 to a variety of multilingual summarization and question-answering tasks, and results show that mlongt5 exhibits strong performance in these domains. the configurations 1 and checkpoints 2 have all been open-sourced. ## related work there are two areas of related work -efficient transformer models that can handle long inputs, and multilingual models. ## https://github.com/google-research/longt5 there has been much interest of late in making transformer models more efficient, such as to handle longer inputs. example of these include etc @xcite , big bird @xcite , longt5 @xcite , and longformer @xcite . these models were successful in taking various approaches to address the quadratic growth of the attention mechanism in transformers. unfortunately though, these models are trained on english datasets, limiting their use in multilingual domains. with respect to multilingual models, these would include mt5 @xcite , mbart @xcite , and the recent umt5 @xcite . these models re-used architectures used by english models but are pretrained on a larger, multilingual corpus, with mt5 and umt5 trained on 101 languages and mbart on 25. while these models showed strong performance on being able to handle a wide variety of languages, they suffered the same restrictions as their original english models on not being able to scale up to longer sequences. ## results as with the original longt5 paper, we look at two domains for evaluating our model: summarization and question answering. for all of these tasks, we use the default values as used for t5 finetuning, only explicitly setting the input and target lengths as described in the tasks below. ## conclusion we have presented our new model mlongt5. it has the benefits of the efficient architecture of longt5, with the ability to handle multingual inputs and outputs. as our report shows, the model is able to perform well on a variety of summarization and question-answering tasks.