PLATO -Ad: A Unified Advertisement Text Generation Framework with Multi-Task Prompt Learning (2022)

Abstract: Online advertisement text generation aims at generating attractive and persuasive text ads to appeal to users clicking ads or purchasing products. While pretraining-based models have achieved remarkable success in generating high-quality text ads, some challenges still remain, such as ad generation in low-resource scenarios and training efficiency for multiple ad tasks. In this paper, we propose a novel unified text ad generation framework with multi-task prompt learning, called PLATO-Ad, totackle these problems. Specifically, we design a three-phase transfer learning mechanism to tackle the low-resource ad generation problem. Furthermore, we present a novel multi-task prompt learning mechanism to efficiently utilize a single lightweight model to solve multiple ad generation tasks without loss of performance compared to training a separate model for each task. Finally, we conduct offline and online evaluations and experiment results show that PLATO-Ad significantly outperforms the state-of-the-art on both offline and online metrics. PLATO-Ad has been deployed in a leading advertising platform with 3.5% CTR improvement on search ad descriptions and 10.4% CTR improvement on feed ad titles.

## introduction in recent years, online advertising has been regarded as one of the most popular ways of internet monetization. a captivating and persuasive ad can greatly improve the probability of users clicking the ads or purchasing recommended products @xcite . thus, advertisers usually spare no effort to improve the quality of displayed ads. traditionally, some advertisers may manually design ads for high quality. however, this approach suffers from low efficiency and high labor costs. some other works design pre-defined templates (fujita et al., 2010; thomaidou et al., 2013) * equal contribution. or use recurrent neural networks (rnn) @xcite to automatically create ads. however, these ads generated by templates are generally rigid and not unappealing enough to users. over 15% ads generated by rnn are reported to be non-sense or bad @xcite , which hinders the application of this method in real-world business scenarios with high-quality standards. recently, with the remarkable success of the pretraining plus fine-tuning paradigm, some works utilize pretraining-based natural language generation methods to generate content-rich, diverse and attractive ads based on large-scale corpora @xcite . although these methods achieve significant success in generating high-quality ads, they still face great challenges when applied in real-world commercial scenarios. firstly, the performance of these pretraining-based models relies heavily on large-scale high-quality training corpora, which may be yet difficult to be obtained in some low-resource foot_0 scenarios. secondly, these work usually train a separate model for each task respectively when applied in multiple ad tasks, e.g., ad title and description generation @xcite , copywriting generation @xcite , selling point generation @xcite . this is inefficient and expensive for multiple ad tasks that require multiple copies of the model's parameters, with also ignoring the associations between multiple ad generation tasks @xcite . to this end, we propose a novel unified text ad generation framework with multi-task prompt learning, called plato-ad, to tackle the aforementioned problems. concretely, the model architecture of plato-ad is a transformer-based pretrained language model with 12 transformer blocks. to effectively address the low-resource ad generation problem, we propose a three-phase transfer learning mechanism to train plato-ad. specifically, we first pre-train plato-ad on genericdomain text corpus to equip the model with the ability to generate fluent natural sentences. then, in the second phase, we consecutively post-pretrain plato-ad on the datasets of multiple resourcerich ad generation tasks where massive data is available and open-domain question answering (qa) datasets, which enables the model to learn to generate ad-domain and commonsense-enriched text. finally, we use the prompting mechanism @xcite to transfer the well-trained plato-ad in the second phase to low-resource ad generation tasks. in this way, we can generate high-quality ads for those low-resource ad generation scenarios that lack a large amount of high-quality humanwritten data. furthermore, to improve training efficiency and reduce application costs for multiple ad tasks, in the post-pretraining phase, we propose a novel multi-task prompt learning mechanism by introducing task prompts and multiple losses to better fuse multiple tasks into one single model without loss of performance. the main contributions are summarized as follows: ## related work previous works on text ad generation rely on designing pre-defined templates to construct readable ad sentences @xcite @xcite . however, these template-based ads are generally rigid and not diverse enough leading to unappealing to users. afterward, hughes et al. (2019) presents data-driven methods to learn to write text ads from existing examples. the model employs lstms and attention layers to encode product landing pages and decode text ads. moreover, reinforce with baseline (rennie et al., 2017) is leveraged to generate attractive ads that potentially have a larger click rate. this method achieves a certain degree of success in the automatic generation of text ads. however, over 15% text ads it generates are labeled as non-sense, broken, or bad, which fails to meet the high-quality standard for production @xcite . recently, some work utilize pretraining-based natural language generation methods to generate content-rich, diverse and attractive ads based on large-scale corpus @xcite @xcite . however, these methods require large-scale high-quality training corpus and low training efficiency when applied to multiple ad generation scenarios. in this paper, we propose a unified text ad generation framework with multitask prompt learning to tackle ad generation problems in low-resource settings and improve training efficiency for multiple resource-rich ad tasks.