Extrapolating Multilingual Understanding Models as Multilingual Generators (2023)

Abstract: Multilingual understanding models (or encoder-based), pre-trained via masked language modeling, have achieved promising results on many language understanding tasks (e.g., mBERT). However, these models are not capable of generating high-quality text compared with decoder-based causal language models. Can we transform a pre-trained language understanding model into an effective language generation model? We propose a Semantic-Guided Alignment-then-Denoising (SGA) approach to adapt a multilingual encoder to a multilingual generator with a small number of additional parameters. Experiments show that the proposed approach is an effective adaption method, outperforming widely-used initialization-based methods with gains of 9.4 BLEU on machine translation, 8.1 Rouge-L on question generation, and 5.5 METEOR on story generation on XLM-R _{large} . On the other hand, we observe that XLM-R is still inferior to mBART in supervised settings despite better results on zero-shot settings, indicating that more exploration is required to make understanding models strong generators. Our code is available at https://github.com/chengzhipanpan/XLMR4MT.

## introduction multilingual encoder-based models (e.g., mbert @xcite , xlm-r @xcite ), pre-trained via masked language modeling, have demonstrated strong performance on a wide range of understanding tasks @xcite @xcite . existing multilingual pre-trained models can be classified into two settings: autoregressive models @xcite @xcite and nonautoregressive models @xcite figure 1 : an overview of semantic-guided generation using pre-trained understanding models. the encoding step is responsible for mapping the source input into a shared space that supervises the following generation. by taking the source input and a blank sentence (white noise) as input, the alignment stage generates target tokens simultaneously. then, we feed the source representations and the generated sequence into the denoising stage for nar denoising. the denoising step is performed iteratively until the generated text keeps unchanged or reaches the maximum loop. @xcite . typically, the ar framework, where a target sequence is generated from left to right, succeeds in multilingual generation tasks @xcite . as a comparison, encoder-based models are nar models that are usually limited to understanding tasks @xcite . despite superior understanding results over ar models, these nar models still struggle to handle a wide range of multilingual generation tasks. however, nar models still have obvious advantages in generation efficiency and decoding flexibility @xcite @xcite @xcite , which enables generating multiple tokens at one time in arbitrary order. considering these strengths, this paper aims to explore methods to make multilingual encoderbased models better generators with a small number of new parameters. there is limited research focusing on empowering understanding models with the generation ability. traditional methods usually use pre-trained encoders as initializers for ar models in various monolingual generation tasks @xcite . despite promising results, it does not satisfy our target that fixes pre-trained parameters to build a unified model for any language tasks. more recently, researchers have focused on learning-free approaches @xcite @xcite . one typical approach is iteratively choosing tokens to mask and sampling proposals using energy models @xcite , resulting in surprising high latency. furthermore, these learning-free methods are usually limited to controllable generation and are still inferior in handling complicated tasks like machine translation. unlike these monolingual studies, adapting multilingual understanding models to multilingual generation has its own challenges: semantic constraints under conditional generation where the generation process should follow the semantic constraints given source texts in any language, and parameter efficiency constraints where a single model can serve text generation in any language. we propose a semantic-guided approach to address these challenges, with a two-stage generation process: alignment-then-denoising. the two stages share the same pre-trained parameters and only add a small number of new prompt parameters for adaptation. given that masked language modeling (mlm) is also a denoising objective, existing multilingual pre-trained models can be naturally adapted to good denoisers. therefore, we introduce a denoising stage into our framework. the whole generation process is shown in figure 1 . the encoding part maps the source input into a shared space that supervises the following generation. by taking the source input and a blank sentence as input, the alignment stage generates target tokens simultaneously. we feed the source representations and the generated sequence into the denoising module for nar denoising. the denoising step is performed iteratively until the generated text keeps unchanged or the maximum loop is reached. experiments demonstrate that our model has achieved better results on various generation tasks than traditional fine-tuning-based approaches that directly use nar pre-trained models as initialization, with gains of 9.4 bleu on machine translation, 8.1 rouge-l on question generation, and 5.5 meteor on story generation on xlm-r large . more promisingly, our method has achieved im-pressive zero-shot cross-lingual ability in translation tasks, outperforming a multilingual ar adaptation model, mgpt + msp by a large margin. on the other hand, we also notice the gap between xlm-r and ar models. generally, xlm-r with fine-tuning is largely inferior to mbart fine-tuning. with our methods, the gap is largely reduced but still exists. in future work, we would like to explore pre-training methods to make multilingual understanding models better generators. our contributions can be summarized as follows: in this section, we review the related studies including parameter-efficient adaptation, adapting encoder models as generators, and non-autoregressive generation. parameter efficient adaptation pre-trained language models (plms) @xcite @xcite have achieved overwhelming performance in a variety of downstream tasks. parameter-efficient tuning is a hot research direction to adapt plms to downstream tasks with only training on a few parameters. adapter-based methods @xcite are one of the popular parameter-efficient approaches. recent studies @xcite proposed to use adapters on the top of an mbart @xcite ) model, enabling a flexible and well-performed method for plug-andplay translation. more recently, prefix tuning and other prompt-based methods @xcite @xcite have proved to be extremely helpful, and can easily support mixed-task inference as it does not require changing the architecture of plms. in this work, we follow this research thread for efficient adaptation and apply prompt-based approaches in our work. ## notation and background prompt tuning for efficient adaption, we follow mgpt+msp @xcite and use prompt tuning to adapt an existing pre-trained nar pretrained multilingual understanding models to generators. formally, we denote k l and v l as the key-value pairs in the l-th transformer layer. the introduced prompt-tuning parameters are (k, v ) pairs, which will be concatenated with current keyvalue pairs during training and inference. in prompt tuning, we denote the forward pass of a pre-trained lm as f lm (θ p , x), which accepts two inputs including prompt parameters θ p and the source sequence x. ## conclusion in this paper, we propose an effective approach to adapt existing pre-trained multilingual understanding models to multilingual generators. on translation tasks, experiments demonstrated that our proposed method achieves large performance improvements and notable acceleration with strong cross-lingual generation ability. on free-generation tasks including question generation and story generation, our method also achieves comparable performance with at-based method with impressive speedups. although still lagging behind pretrained multilingual at models (e.g., mbart) in supervised fine-tuning settings in translation, our proposed method show better zero-shot abilities and faster inference.