BERTG en: Multi-task Generation through BERT (2021)

Abstract: We present BERTGen, a novel, generative, decoder-only model which extends BERT by fusing multimodal and multilingual pre-trained models VL-BERT and M-BERT, respectively. BERTGen is auto-regressively trained for language generation tasks, namely image captioning, machine translation and multimodal machine translation, under a multi-task setting. With a comprehensive set of evaluations, we show that BERTGen outperforms many strong baselines across the tasks explored. We also show BERTGen’s ability for zero-shot language generation, where it exhibits competitive performance to supervised counterparts. Finally, we conduct ablation studies which demonstrate that BERTGen substantially benefits from multi-tasking and effectively transfers relevant inductive biases from the pre-trained models.

## introduction recent work in unsupervised and self-supervised pre-training has revolutionised the field of natural language understanding (nlu), resulting in high performance ceilings across multiple tasks @xcite @xcite . the recent success of language model pre-training with masked language modelling (mlm) such as bert @xcite further paved the way for more complex approaches that combine language pre-training with images @xcite @xcite , video @xcite , and speech @xcite . most of these approaches follow a task-specific fine-tuning step after the model is pre-trained. however, there has been little work on exploiting pre-trained mlms for natural language generation (nlg) tasks. previous work argues that the mlm objective is ill-suited for generation tasks such as machine translation @xcite . recent work in this direction has predominantly investigated the use of pre-trained models to either initialise transformer-based encoderdecoder models @xcite @xcite or to distill knowledge for sequence generation tasks @xcite . in this work, we present bertgen, which extends bert in a generative setting ( § 2.1). this results in a single generator -without a separation between the encoder and the decoder -capable of consuming multiple input modalities and generating in multiple languages. the latter features are achieved by transferring knowledge from state-of-the-art pretrained models, namely vl-bert @xcite and multilingual bert (m-bert) @xcite . we train bertgen on various tasks, including image captioning, machine translation and multimodal machine translation, and datasets in four different languages ( § 2.2). based on a number of experiments, our findings ( § 3) show that bertgen (i) is surprisingly versatile as it is capable of describing images and performing translation in unimodal and multimodal settings, across all languages, (ii) generalises well across zero-shot image captioning, multimodal machine translation, and out-of-domain news translation tasks, and finally (iii) is parameter efficient when compared to state-of-the-art models for each of the tasks combined together. ## method in this section, we describe bertgen and the tasks we explore. we then detail the baselines and sota systems that we compare against. ## results and findings we train bertgen on lowercased sentences for 45 epochs, after which the overall performance on the tasks reached a plateau. we define one bertgen epoch as a single pass over all of the training data for the multi30k @xmath0. we use greedy search for all systems that we trained and merge back the word pieces before evaluation. we compute tokenised foot_5 bleu @xcite , meteor @xcite and cider @xcite using cococaption 6 . in what follows, we provide detailed quantitative and qualitative findings. ## conclusions in this paper, we presented bertgen, a novel generative, decoder-only model which extends bert by combining multimodal and multilingual pretrained models. our findings show that bertgen obtains strong performance on a variety of generative tasks and further generalises over unseen tasks. importantly, our model demonstrates the potential for general-purpose (instead of task-specific) generation that is above and beyond the traditional pre-training and fine-tuning practices. bertgen is also parameter efficient as it has 89.3m total parameters and is trained on thirteen tasks encompassing mt, multimodal mt and image captioning. on the other hand, each of the single-task fairseq nmt baselines has 31.5m parameters. our ablation studies show that bertgen is able to efficiently transfer relevant inductive biases from the pre-trained models and benefits from multi-task learning without suffering from catastrophic forgetting. we hope that these findings will motivate future research in exploiting more sophisticated pre-trained models in place of m-bert and vl-bert and others. bertgen mehrere ngos, darunter die mozilla -und greenpeace -stiftung, schätzen, dass diese neuen werkzeuge unfähig sind und zu spät kommen. wmt ref mehrere ngos, unter denen die mozilla -stiftung und greenpeace, schätzen, dass diese neuen tools unzureichend sind und zu spät kommen. bertgen immigration wird als ein großes problem für die ue betrachtet, für 45 prozent der deutschen und 40 prozent aller europäischen. wmt ref die einwanderung halten 45 prozent der deutschen und 40 prozent aller europäer für das größte problem der eu. bertgen das ist der grund, warum er in seinem buch die frage erforscht, ob es alternativen zu wahlen gibt. wmt ref deshalb geht er in seinem buch der frage nach, ob es alternativen zu wahlen gibt.