A d TEC : A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising (2025)

Abstract: As the fluency of ad texts automatically generated by natural language generation technologies continues to improve, there is an increasing demand to assess the quality of these creatives in real-world setting.We propose **AdTEC**, the first public benchmark to evaluate ad texts from multiple perspectives within practical advertising operations.Our contributions are as follows: (i) Defining five tasks for evaluating the quality of ad texts, as well as constructing a Japanese dataset based on the practical operational experiences of advertising agencies, which are typically maintained in-house. (ii) Validating the performance of existing pre-trained language models (PLMs) and human evaluators on this dataset. (iii) Analyzing the characteristics and providing challenges of the benchmark.Our results show that while PLMs have a practical level of performance in several tasks, humans continue to outperform them in certain domains, indicating that there remains significant potential for further improvement in this area.

## introduction online advertising, especially sponsored search advertising (figure 1 ), is a dominant sector for vendors to promote their products, and the market size is estimated to grow by billions of dollars over the next few years @xcite . to meet the increasing demands of advertising operations (adops), such as creating ad texts from product information (step 2 in figure 2 ), the remarkable success of natural language generation (nlg) by pre-trained language models (plms) @xcite @xcite has given a boost to practical applications @xcite @xcite , making advertising a huge industrial use case for natural language processing (nlp). in this paper, we focus on evaluating the quality of ad texts generated by such models. we refer to this process as ad text evaluation, as depicted in step 3 of figure 2 . ad text evaluation is crucial because low-quality ad texts , which may lack fluency, present inappropriate appeals, or convey misleading representations, can negatively impact advertisers. since it is costly and not scalable to have humans verify the quality of each text in high-volume domains, such as sponsored search advertising, there is a high demand for the development of automated quality estimators for ad texts. the quality has multiple dimensions, such as appropriate wording, effective appeals, consistency between ad text and product information, and high predicted performance. although these dimensions should be included when evaluating automatic quality estimators, no benchmark currently exists to assess them. consequently, the bottleneck lies in figure 2 : generalized adops workflow described in §2: (1) the advertiser creates an lp to promote a product. (2) based on the product information in the lp and target customers, text and graphics are designed by creators. (3) the creatives are evaluated based on fluency, attractiveness, regulations, legality, and other factors. (4) once the creatives pass the quality evaluation, they are submitted to a delivery platform. (5) customers respond to the displayed ads, such as page views, clicks, and purchases. (6) based on the customer engagement, ad performance is reported back to the advertiser, and steps 1-5 are repeated to improve the quality of the lp and ads. the primary challenge in constructing an ad text evaluation benchmark is the absence of a clear definition for tasks @xcite . the lack of domain knowledge in adops complicates the understanding of highquality ad text standards and the accurate definition of tasks. however, the adops workflow is complex, relies on various platforms and formats, and encompasses multiple methodologies and metrics. in addition, only a few companies possess the expertise to operate online advertising at scale. owing to legal and contractual obligations, advertising workflows and data are predominantly managed in-house, leading to a lack of publicly available datasets. this scarcity makes it challenging to systematically reproduce and validate diverse methodologies in academia. consequently, research activity is limited in the advertising field, which leaves potential issues unaddressed and delays the application and development of cutting-edge technology. to address these challenges, we propose adtec, the first public benchmark that defines and unifies tasks based on generalized adops workflows. our major contributions are as follows: the first public dataset on ad text evaluation. we organized real-world advertising workflows and carefully designed five tasks to evaluate the quality of ad texts. this was based on the practical operational experiences of advertising agencies, using japanese data as a case study. we are releasing this dataset foot_0 , marking it as the first publicly available dataset for ad text evaluation, as such data are typically maintained in-house and difficult to obtain. ## understanding sponsored search advertising and its workflow in sponsored search advertising, titles and descriptions of ads that are relevant to keywords entered into search engines by customers appear as part of the search results. as shown in figure 1 , when a customer clicks an ad's url, they are directed to a web page known as a landing page (lp). the lp contains texts and images related to the advertiser's products, prompting the customer to view the product or make purchase decisions. operating such ads requires high expertise, owing to the variety of delivery platforms, properties, and formats. to gain insights, we interviewed two types of experts familiar with adops: those overseeing adops departments at ad agencies and those directly involved in on-site operations. through these interviews, we explored the intricacies of adops, generalized the workflow, and delineated six steps, as illustrated in figure 2 . in this work, we focus mainly on step 3 of quality evaluation. ## constructing adtec our goal is to develop a benchmark that captures multi-dimensional aspects of quality that are relevant to practical scenarios. in addition, these aspects will be carefully curated to provide value for both real-world applications and various research purposes. ## experimental settings we employed two types of evaluators: the plms, including both fine-tuned settings with encoder models and zero-/few-shot settings for llms, as well as human evaluators. note that the detailed model description, hyperparameters, prompts, and additional information are provided in appendix c. fine-tuning setting with encoder models. we utilized publicly available encoder models as baselines, specifically tohoku bert, waseda roberta, and xlm-roberta @xcite , which are commonly employed for japanese nlp tasks. these models differ in pre-tokenizer, tokenization unit, and pre-training dataset. all the aforementioned models are of large size. zero-/few-shot setting with llms. we employed calm2 7b and elyza 7b as baselines for open llms. calm2 7b and elyza 7b are based on the llama 2 @xcite architecture but differ in training methods and data; calm2 7b was trained from scratch, whereas elyza 7b was continuously trained from the original llama 2. 3 human evaluators. we enlisted three human evaluators who are not involved in adops to evaluate all tasks except ad performance estimation. we followed the same procedure for instruction as described in §3.2. pilot evaluations were carried out twice on 100 randomly sampled instances from the training set for each task before the main run. in the final assessment, a majority vote per instance was conducted for the ad acceptability and ad consistency tasks, while the average scores of evaluator assessments were reported for other tasks. ## results and discussion fine-tuned setting with encoder models. xlm-roberta and tohoku bert achieved the highest or competitive scores in two or more tasks. in addition, the large model outperformed the base model in most tasks, suggesting that an increased parameter size plays a key role in understanding ad expressions. zero-/few-shot setting with llms. gpt-4 achieved high performance across all three tasks, while elyza 7b performed the best among the llms in ad consistency. a substantial difference was observed between the open llms and openai's llms in the ad similarity task. calm2 7b and elyza 7b scored close to 0, indicating no correlation, whereas openai's models, especially gpt-4, achieved competitive scores of 0.776/0.811. thus, the open llms used in this study struggle to handle semantic similarity or numerical answers, whereas gpt-4 performs considerably better. in the ad performance estimation task, all llms produced uncorrelated responses close to 0, indicating that accurately predicting ad performance remains a challenge for open llms. fine-tuned encoder models vs. zero-/fewshot llms. overall, the fine-tuned models outperformed the llms. the difference was substantial, ranging from 0.2 to 0.6, particularly for the ad acceptability, ad performance estimation, and a 3 recognition tasks. this can be attributed to the characteristics of the tasks; ad performance estimation involves predicting numbers in the range [0, 100], and a 3 recognition requires selecting all suitable labels from more than 20 labels, suggesting that the variety of data features and outputs could not be handled by few-shot alone. both ad acceptability and ad consistency are binary classification tasks, yet the performance gap between fine-tuned models and llms is more noticeable in the ad acceptability task, with a difference as large as 0.2 points, compared to the relatively smaller difference observed in the ad consistency task. the task with the smallest difference was ad similarity, with a difference of only 0.06 to 0.10. plms vs. humans. human evaluators outperform models in ad acceptability and ad consistency. in both tasks, models tends to have high accuracy and low f1-score, while humans exhibit the opposite trend. the evaluations by plms demonstrate high precision but low recall in both tasks, resulting in a low f1score. according to error analysis for ad acceptability and ad consistency tasks. we conducted a detailed analysis of two tasks: ad acceptability and ad consistency, where the model has not yet outperformed human evaluators. we analyzed three types of errors: human incorrect and model correct (ic-c), human correct and model incorrect (c-ic), and both incorrect (ic-ic), as shown in in both tasks, the model more frequently predicted false labels than true labels for both ic-c and c-ic errors. in contrast, humans provided more true labels in both correct and incorrect cases. this suggests that humans exhibit a relatively higher degree of leniency compared to models, which tend to be overly cautious when making decisions in the ad acceptability and ad consistency tasks. additionally, in the ad acceptability task, the model can easily identify elements that can be detected heuristically, such as "incorrect use of symbols" or grammatically incorrect or non-fluent expressions. however, it struggles with evaluating expressions that are fluent but semantically unnatural, such as "engineer's career / engineer's job" (redundancy in the same appeal) or "condominium sales / single-family home" (contradictory appeals). to address these challenges, strategies such as adding training data or providing the llm with more detailed judgment criteria could enhance accuracy. regarding the ad consistency task, the model generally performs well when expressions from the lp are included in the ad text. however, it tends to make errors when this is not the case. there are two potential causes for this scenario. one is the use of meta-expressions, such as the word "official." in ad text, "official" is often used to reassure customers, but it is frequently omitted in lps because there is no need to emphasize it. to address this issue, accuracy might be improved by further training on similar data or by incorporating a specific mechanism to infer whether the lp is "official." the second cause involves referring to non-verbal information, such as images, in the ad text. although this is beyond the scope of the current study, it might be addressed by leveraging multimodal models. case study on the model's behavior on ad similarity task. we used integrated gradients @xcite to visualize the attention of tohoku bert, aiming for a deeper understanding of the model's behavior, which can potentially enhance its performance. figure 5 shows examples of low and high errors between predictions and the ground truth in the ad similarity task. in the first example, the model fails to pay sufficient attention to "online seminar," which has a similar meaning to "webinar." however, high attention to the date expression "9/9 (thursday) at 13:00" allows the model to predict a score close to the correct answer. however, in the second example, the model did not pay enough attention to "english conversation coaching," which has a similar meaning to "gaining english skills," resulting in an incorrect prediction of dissimilarity. as we found many such cases, we believe that the model tends to prioritize surfacelevel information, especially for entities such as dates, times, and numbers. this may prevent the model from correctly identifying cases that are semantically similar, even when the surface-level information differs, and vice versa. another instance where we observed shortcomings in the models is in reasoning. for example, the models mistakenly interpret the phrases "half price" and "buy 2 get 1 free" as having the same meaning, whereas humans easily comprehend the difference. additionally, pairs of phrases like "available 24 hours a day, 7 days a week" and "closed every wednesday," as well as "property located near the station" and "20-minute walk from the station" often confuse the models in understanding. these types of paraphrasing are commonly used in the ad creation process; however, it is challenging to automatically capture them in a natural dataset. therefore, a specialized dataset focusing on these phenomena is necessary to enable models to capture them accurately. ## related work evaluating ad text quality @xcite proposed an ad text generation benchmark that aims to evaluate nlg models (step 2 in figure 2 ) with surface-and semantic-level overlap metrics, such as bleu @xcite , rouge @xcite , and bertscore @xcite , along with simple heuristics, including keyword insertion rate @xcite ) and text length. however, this approach is inadequate for real-world applications because (1) it requires reference texts to assess generated texts, making it difficult to evaluate new ads that have no references, and (2) simple heuristics cannot guarantee the quality of the ad text for delivery, as more specialized criteria are required in actual production, such as appropriate wording, effective appeals, consistency between ad text and product information, and high predicted performance. ad text quality is typically represented by metrics of user behavior obtained after ad delivery as illustrated in step 6 of figure 2 , and ctr is widely used to predict ad performance and make marketing strategies. the ctr prediction task was first studied by @xcite and has been actively researched by @xcite 2012 @xcite over the past decades @xcite . while there are many methods for ctr prediction, few studies have evaluated the quality of ad texts at a more granular level. coarse-level feedback may be insufficient to improve ad quality, as it is unclear where edits should be made or what changes are necessary. meanwhile, adtec provides more detailed evaluation results, such as acceptability and consistency, making it easier to integrate into actual adops workflows. ## conclusion we defined five tasks to verify the quality of ad texts and developed the adtec, a large, versatile, and comprehensive benchmark of japanese advertising data constructed for the nlp community, based on real-world adops workflows. we conducted evaluations with both plms and human evaluators on adtec to explore its characteristics, offering insights into practical workflow applications and identifying potential areas for future improvement and research. our findings suggest that sampling directly from real data generally benefits the model, highlighting the importance of tasks centered on natural language inference and semantic understanding. we hope that the combination of our defined tasks and datasets will advance research in ad text evaluation, bridging the fields of advertising and nlp and paving the way for new discoveries and applications.