Striking Gold in Advertising: Standardization and Exploration of Ad Text Generation (2024)

Abstract: In response to the limitations of manual ad creation, significant research has been conducted in the field of automatic ad text generation (ATG). However, the lack of comprehensive benchmarks and well-defined problem sets has made comparing different methods challenging. To tackle these challenges, we standardize the task of ATG and propose a first benchmark dataset, CAMERA, carefully designed and enabling the utilization of multi-modal information and facilitating industry-wise evaluations. Our extensive experiments with a variety of nine baselines, from classical methods to state-of-the-art models including large language models (LLMs), show the current state and the remaining challenges. We also explore how existing metrics in ATG and an LLM-based evaluator align with human evaluations.

## introduction the global online advertising market has witnessed significant growth and quadrupled over the last decade, particularly in the domain of search ads @xcite . search ads are designed to accompany search engine results and are tailored to be relevant to users' queries (search queries). these ads are displayed alongside a landing page (lp), providing further details about the advertised product or service. therefore, ad creators must create compelling ad texts that captivate users and encourage them to visit the lp. however, the increasing volume of search queries, which is growing at a rate of approximately 8% annually @xcite , poses challenges for manual ad creation. the growing demand in the industry has fueled research on the automatic generation of ad texts. researchers have explored various approaches, starting with template-based methods that generate ad text by inserting relevant keywords into predefined templates @xcite figure 1 : examples of our dataset, translated into english for visibility. the highlighted areas indicate the aspects of advertising appeals: speed , trend , and user-friendliness 2010; @xcite . recently, neural language generation (nlg) techniques based on encoder-decoder models, which are widely employed in machine translation and automatic summarization, have been applied to ad text generation (atg) @xcite @xcite . however, the automated evaluation of atg models presents significant challenges. previous research has been constrained to conducting individual experiments using proprietary datasets that are not publicly available @xcite . this limitation arises from the absence of a shared dataset (i.e., a benchmark) that can be universally applied across the field. moreover, the absence of benchmarks has resulted in a lack of consensus regarding task settings such as the models' input/output formats. while some studies use keywords as input @xcite , others employ existing advertisements @xcite or lps @xcite @xcite . this variation in the task setting indicates that the field as a whole has yet to establish a standardized problem setting, which hinders the generalization and comparability of atg techniques. this study aims to advance atg technology by standardizing the task setup, transforming it into a format accessible to potential players by providing a shared dataset, and exploring the current status and limitations. standardizing problem settings common to a variety of advertising applications as tasks allows for focused exploration of core issues in an academic context while maintaining the flexibility to be applied to a wide variety of applications ( §3). to engage a broader community of researchers beyond those who possess ad data, we construct the first publicly available benchmark, camera (cyberagent multimodal evaluation for ad text generation)(figure 1 ), which is meticulously developed a comprehensive dataset ( §4). 1 our dataset comprises actual data sourced from japanese search ads and incorporates annotations encompassing multi-modal information such as the lp images. to explore the current state and future challenges, we conducted extensive experiments using nine diverse baselines, including multimodal models and large language models (llms), as well as the dominant approaches in existing studies ( §5). furthermore, we also conducted a meta-evaluation of how well the existing metrics and llm-based evaluators reproduced human evaluations ( §6.1). our major contributions are: we observed the following: ## background various types of online advertising exist, including search ads, display ads foot_1 , and slogans foot_2 . however, since most existing studies are related to search ads @xcite , this study also focuses on search ads and provides an overview of atg research and its current limitations. ## standardization of ad text generation one of the goals of this study is to develop a task that is not specific to a particular platform or advertising medium but focuses on universal core problems common to these applications, to facilitate the generalization of atg technology. to meet these requirements, we standardize the atg task as follows: let x be a source document that describes advertised products or services, a a user signal reflecting the user's latent needs or interests, and y an ad text. atg aims is to model p(y|a, x). user signals, such as search keywords for search ads and user browsing and action history for display ads, can vary based on the application and domain. the specific data to be selected for each x, a, and y will be left to future dataset designers and providers. this standardization of atg allows a focused exploration of core issues in an academic context while maintaining flexibility for diverse applications in an industrial context. the requirements of ad text the purpose of advertising is to influence consumers' (users) attitudes and behaviors towards a particular product or service. therefore, the goal of atg is to create text that encourages users' purchasing behaviors. based on this, the following two requirements for ad text were defined: (1) the information provided by the ad text is consistent with the content of the source document; and (2) the information is carefully curated and filtered based on the users' potential needs, considering the specific details of the merchandise. requirement 1 relates to hallucinations, which is currently a highly prominent topic in the field of nlg @xcite @xcite . this requirement can be considered crucial for practical implementation since the inclusion of non-factual hallucination in ad texts can cause business damage to advertisers. regarding requirement 2, it is necessary to successfully convey the features and attractiveness of a product within a limited space and immediately capture the user's interest. therefore, ad text must selectively include information from inputs that can appeal to users. differences from existing tasks the atg task is closely related to the conventional document summarization task in that it performs information compression while maintaining consistency with the input document's content. particularly, queryfocused summarization (qfs) @xcite , a type of document summarization, is the closest in problem setting because it takes the user's query as the input; however, there are some differences. the task of qfs aims to create a summary from one or multiple document(s) that answers a specific query (explicit needs). in contrast, atg is required to extract not only surface information from user signals but also the latent needs behind them and then return a summary. for example, when a user's query is "used cars," the goal of qfs is to provide information about used cars. on the other hand, for users seeking higher-priced items like cars, factors such as quality become important even if they are used. therefore, the task of atg aims to present ads that include expressions appealing to high quality and reassurance, such as "all cars come with a free warranty!". another notable difference is that while summarization aims to deliver accurate text that fulfills task-specific requirements, atg surpasses mere accuracy and aims to influence user attitudes and behavior. consequently, unconventional and/or ungrammatical text may be intentionally used in ad-specific expressions to achieve this objective (refer to details in §4.2). therefore, qfs is a subset of atg (qfs ⊂ atg). one of the technical challenges unique to atg is capturing users' latent needs based on such user signals a and generating appealing sentences that lead to advertising effectiveness, which depends significantly on the psychological characteristics of the recipient users. therefore, realizing more advanced atg will also require a connection with advertising psychology @xcite based on cognitive and social psychology. the atg is an excellent research topic for advancing user-centered nlp technologies. ## benchmarking of atg models to clarify the current state and remaining challenges, we conduct benchmark experiments using the dataset constructed in §4 and various atg models. specifically, we investigate the following research questions: rq1 how do differences in the use of pre-trained language models (i.e., finetuning vs. few-shot) affect overall performance? rq2 is multimodal information useful for ad text generation? entity type input output time expression 2022年9月(september 2022) 2022年 (2022) katakana サイト (site) ホームページ (homepage) numerical expressions 6,800円 -8,000円 (6,800 yen -8,000 yen) 6,800円 (6,800 yen) named entity イシダ (ishida) 株式会社イシダ (ishida corporation) terms 求人情報 (job openings) 求人紹介 (job introductions) rq3 do trends in model performance vary by industry domain? rq4 what are the qualitative differences between generated ad text compared to humanproduced ad text? ## analysis 6.1 how well can automated evaluations replicate human evaluations? to clarify the limitations and possibilities of automatic evaluation, we performed a meta-evaluation by adding a gpt-4 based evaluator to the set of the metrics used in the experiment in §5. the gpt-4 based evaluator was constructed by giving the same instructions as those given to the human raters in the manual evaluation §5.2. ## discussion for reproducible research we want to situate our findings in the context of the broader nlp community, in line with our goal of discussion on increased transparency in the field. examples of data that are challenging to open include proprietary datasets primarily owned by companies, housing sensitive information for maintaining a competitive advantage (e.g., datasets managed by openai). ad data, the primary focus of this study, also exemplified this scenario. one of the reasons why ad data has not been shared with the community in the past is that ctrs and other performance data are confidential. also, measuring ctr is difficult except for a few companies in the advertising business. therefore, as an incentive mechanism to promote the creation of open research within the research community, there is a direction for the community to accept secondary information (e.g., pctr or human preference in §6.2) that is guaranteed to be consistent to some extent with sensitive primary data (e.g., ctr). ## conclusion we standardized atg as a cross-application task and developed the first benchmark dataset. through evaluation experiments using our dataset, we demonstrated the current status and remaining challenges. atg is a promising application of nlp and a critical and complex research area for advancing user-centric language technology. we hope that the research infrastructure we established will drive the progress and development of atg technology.