CULG : Commercial Universal Language Generation (2022)

Abstract: Pre-trained language models (PLMs) have dramatically improved performance for many natural language processing (NLP) tasks in domains such as finance and healthcare. However, the application of PLMs in the domain of commerce, especially marketing and advertising, remains less studied. In this work, we adapt pre-training methods to the domain of commerce, by proposing CULG, a large-scale commercial universal language generation model which is pre-trained on a corpus drawn from 10 markets across 7 languages. We propose 4 commercial generation tasks and a two-stage training strategy for pre-training, and demonstrate that the proposed strategy yields performance improvements on three generation tasks as compared to single-stage pre-training. Extensive experiments show that our model outperforms other models by a large margin on commercial generation tasks, and we conclude with a discussion on additional applications over other markets, languages, and tasks.

## introduction pre-trained language models (plms) have achieved impressive success in many nlp tasks across natural language understanding (nlu) and natural language generation (nlg) @xcite @xcite @xcite @xcite . these successes are usually achieved by pre-training models on large corpora in a taskindependent way, and fine-tuning them on a specific downstream task. researchers have also developed plms for specific domains or multiple languages by conducting either pre-training from scratch @xcite @xcite or a second phase of pre-training on the basis of existing checkpoints @xcite @xcite . however, plms in the domain of commerce, especially for marketing and advertising, remain less * equal contribution studied. on the one hand, adapting plms to the advertising domain is challenging because existing pre-training methods usually use open-domain corpora containing largely well-structured text such as books @xcite , news @xcite , stories @xcite , or web text @xcite to learn text representations. however, the input text for selecting advertisements is primarily web search queries, which are usually not complete, grammatical sentences. on the other hand, there is no publicly-available plm in the commercial domain. this paper introduces commercial universal language generation model (culg), which supports multi-lingual, multi-market, and multi-task ad generation. culg adopts a transformer-based @xcite encoder-decoder generative framework similar to prophetnet @xcite , which uses an n-stream self-attention mechanism and supports future n-gram prediction. to adapt to diverse markets, we use the multi-lingual version of prophetnet -prophetnet-x @xcite as our foundation model, and conduct a second phase of pre-training using a self-constructed large-scale commercial corpus. culg is trained auto-regressively on four sequence-to-sequence (seq2seq) generation tasks, including: (1) generate keywords with the same intent as the query (gks); (2) generate keywords that are relevant to a query (gkr); ( 3 ) generate an ad title based on a query (gat); and (4) generate an ad description based on a query (gad). the motivation of these tasks is to infer the user's intention based on the query as well as perform product matching and recommendation. all queries used in this research are real-life search queries that have been submitted to the bing 1 search engine, and the ground truth targets are created according to either the records of user's click behaviour or labels from hired human annotators. we collected more than ten million queries from 10 markets in 7 languages, and split them into three classes according to data quality. given the user query, the gold class is ads that were deemed as relevant to the query by human judges, the silver class is made up of ads clicked on by users, and the bronze class is all ads that been selected by search engine to show to users. splitting the data into different markets, tasks, and quality classes provides us with flexibility to compare the model's performance under different training setups. given that the collected data varies in quality, we split both the pre-training and fine-tuning into two stages, using low-quality data in the first stage and high-quality in the second stage. to demonstrate the effectiveness of this approach, we compare it with alternative combinations of pre-training and fine-tuning. we evaluate culg on three commercial generation tasks. experimental results show that splitting pre-training and fine-tuning into two stages not only outperforms the widely-used singlestage pre-train and fine-tune schema, but is also better than other combinations of pre-training and fine-tuning. we further compare culg with existing pre-trained multi-lingual models @xcite and show that it surpasses other models on commercial generation tasks. finally, we conduct transfer learning experiments on different markets, languages, and tasks by finetuning culg on a market, language, and task that has not been seen during pre-training. the results demonstrate that clug also generalizes well to unseen markets, languages, and tasks. ## related work pre-training for text generation pre-training has been widely used in nlp tasks to learn language representations @xcite @xcite @xcite . @xcite takes plain text as pre-training data to predict the next token in a left-to-right fashion. it performs well on story generation and creative writing. bart @xcite uses an encoder-decoder structure to regenerate the original text from a corrupted input using an arbitrary noising function. the denoising training strategy and encoder-decoder structure lead to impressive results on generation tasks. mass @xcite pre-trains a seq2seq model by masking continuous spans and learn to recover them. t5 @xcite investigates different pre-training objectives and model architectures, and pre-trains on a large-scale corpus containing 750gb of text data. prophetnet @xcite introduces a novel selfsupervised objective named future n-gram prediction, that explicitly encourages the model to plan for future tokens and prevent overfitting on strong local correlations. in this paper, we use the model structure of prophetnet, and the same n-gram objective function. multi-lingual model in nlp building multilingual models is becoming more common across nlp tasks. support for multi-lingual text is either implemented by aligning multi-lingual word embeddings in a universal space @xcite or by learning crosslingual models using a different corpus to exploit shared representations across languages. models such as mbert (and), mbart @xcite , xlm-r @xcite , mt5 @xcite , and prophetnet-x @xcite are multi-lingual variants of bert, bart, roberta, t5, and prophetnet, respectively. domain adaptive pre-training in this paper, we adapt the pre-trained prophetnet-x to a commercial domain by continuing to pre-train. similar work has been done by researchers in other domains. biobert @xcite is obtained by performing additional bert pre-training on a biomedical corpora, leading to improvements on a variety of biomedical text mining tasks. @xcite continues pre-training biobert on clinical data, and achieves performance gains on three clinical nlp tasks. ulmfit (howard and ruder, 2018) introduced task-specific fine-tuning, with the core idea being to continue pre-training language models on task/domain specific data. @xcite used the approach of ulmfit and continued training it on a reddit corpus, achieving state-of-the-art performance on four claim detection datasets in doing so. most recently, @xcite continued training roberta across 4 domains and 8 tasks, and showed that both domain adaptive pre-training and task adaptive pre-training lead to performance gains. ## conclusion in this paper, we propose culg: a large-scale commercial universal language generation model which supports multi-lingual, multi-market, and multitask ad generation. as part of this, we propose 4 ad generation tasks for culg pre-training. we then propose a two-stage pre-training and fine-tuning strategy, and demonstrate the effectiveness of the proposed strategy through extensive experiments. we further compare culg with other multi-lingual generation models, and show the superiority of culg on commercial generation tasks. finally, we demonstrate the transferability of culg in three different settings.