Topic-driven Ensemble for Online Advertising Generation (2020)

Abstract: Online advertising is one of the most widespread ways to reach and increase a target audience for those selling products. Usually having a form of a banner, advertising engages users into visiting a corresponding webpage. Professional generation of banners requires creative and writing skills and a basic understanding of target products. The great variety of goods presented in the online market enforce professionals to spend more and more time creating new advertisements different from existing ones. In this paper, we propose a neural network-based approach for the automatic generation of online advertising using texts from given webpages as sources. The important part of the approach is training on open data available online, which allows avoiding costly procedures of manual labeling. Collected open data consist of multiple subdomains with high data heterogeneity. The subdomains belong to different topics and vary in used vocabularies, phrases, styles that lead to reduced quality in adverts generation. We try to solve the problem of identifying existed subdomains and proposing a new ensemble approach based on exploiting multiple instances of a seq2seq model. Our experimental study on a dataset in the Russian language shows that our approach can significantly improve the quality of adverts generation.

## introduction the task of abstractive summarization assumes getting short but still a semantically relevant representation of some text. the summarization helps advertisement to catch user attention, get him or her interested, and finally to engage in the full text reading. this is especially true for online advertising when it flashes in front of eyes on most website in the internet. crafting such short texts for online advertising requires a level of mastering and some degree of subject understanding. humans doing this job have to read at least the base text, which describes products or services and then to spend additional time thinking of suitable describing phrase. that may be very time-consuming and thus a good option for automation. modern natural language processing methods and techniques allow solving the task up to some degree of quality while still leaving space for improvement. the most advanced methods are based on neural network seq2seq models @xcite and consume large manually labeled datasets for supervised learning. due to being so popular and widespread, online advertising gives us an opportunity to obtain labeled data (in this particular case, a pair of the source text and banner text) using only data crawling techniques significantly decreasing human efforts needed to build a sufficient dataset for the task. at the same time, ollected open data may consist of multiple subdomains with high data heterogeneity. moreover, what data will come from the crawling depends on advertising providers and their strategies adopted to the end-user. the subdomains belong to different topics and may vary substantially in used vocabularies, phrases, text styles. this heterogeneity may be amplified by overrepresentation of some domains while underrepresentation of others. as it will be demonstrated in our experimental study, all mentioned above may negatively affect the quality of adverts generation especially for text belonging to some topics leading semantic inconsistencies or wrong endings. in this work, we try to smooth the consequences of such peculiarities in data by addressing arising issues with direct topic identification in texts and extracting groups of semantically similar texts in the dataset. it allows multiple models ensemble to be built with fine-tuned specifics to extract topic-driven groups. eventually, this approach demonstrates improvement in the overall quality of generation. contributions of the paper are as follows: (1) a case study of online advertising generation as an abstractive summarization problem; (2) a new topic-driven ensemble approach; (3) an experimental study on russian dataset of the new approach performed on online advertising generation task that demonstrates improved quality of the approach. ## related work there are multiple efforts in scientific community directed to automate building of advertising and presenting it in a right way @xcite @xcite . some works analyze impact of banners' visual appearance to user behavior and psychological reactions @xcite thus answering what banner should contain. others try to automate time-consuming job of banners generation @xcite including smart banners concept that creates personalized banners. these works assume different approaches such as constructing a banner using already prepared parts or generating several banners by substituting words in a template or initial banner with synonyms. our approach is different in two things: first, it works directly with the text from product page instead of altering some base banner or template text and thus it doesn't require additional efforts to prepare these materials; second, we use banner generation methods that differ from the ones from existing works -based on a modern neural architecture called transformer. in the field of natural language generation, one can find several major approaches based on encoderdecoder architectures: lstm-based solutions with or without attention layers @xcite , pointer-generator model @xcite and transformer-based solutions gained significant popularity recently @xcite @xcite . while pointer-generator specifically oriented on solving problems with the tendency of recurrent models to reproduce details incorrectly, caused by an unbalanced dictionary (a rare word is replaced by a more common one), transformer may give better quality and easily parallelize. work @xcite is dedicated specifically for generating short news headlines from articles and has similar conditions of text generating and methods being applied. however, online advertising, in contrary to news headlines, has a different style (related to getting the user to be engaged to sell him/her something) that should be reflected in the generated text. moreover, our approach goes further dealing with heterogeneity in data and utilize ensemble technique. concerning approaches working with dataset peculiarities and exploiting its structure, several directions may be found. transfer learning methods (usually in the form of pretraining and fine-tuning models) show promising results in many applications. however, they deal with various differences in data for datasets in various domains and not in the same dataset @xcite @xcite . within the dataset a hard negative mining @xcite @xcite tries to identify the most informative negative samples and to reduce the amount of other ("easy") negative samples in training data to bring more focus on hard ones. in (braden hancock and ré, 2019 accessed may 25 2020) authors discuss data slicing approach, which assumes the presence of subsets in data that have something in common, and that may influence the performance of the model. they train encoder using different heads (e.g. top layers that solve classification task) with only a corresponding subset of data. our approach is different in two points: we use topic modelling to identify semantically related subsets of data, we use pretraining/fine-tuning routine, which is more widespread in domain adaptation tasks. to the best of our knowledge, there is no neural network-based generation method proposed for the problem of online advertising (banners) creation using natural language at the present moment. more-over, no method considers dataset heterogeneity for this task. ## the topic-driven ensemble approach we will start by describing the collected dataset of online advertising followed by its cleaning procedure and will end with the proposed transformer-based approach. ## experimental study in this section, we present the results of our experimental investigation for the proposed ensemble approach. we start by describing the results of topic modeling and subdomains extraction than proceeding to the comparison of the performance of the proposed ensemble method with existing alternatives. we end with discussing of the ensemble method performance for individual subdomains and its impact on overall performance. for our experiments including topic modeling, we used a random sample of data (roughly half of it) to significant reduce training time. the final size of the dataset used for training and testing was 181798 pairs. ## conclusion and future work in this work, we presented an approach to generate banners for online advertising from texts of web pages. a high degree of data heterogeneity that may be found in a dataset and represented by various topic-based subdomains reduces the quality of generation. the proposed topic-driven ensemble approach helps to solve this issue and shows an increase in efficiency for up to 2% -5% on average for all subdomains and up to 10% -15% for individual subdomains. probably, the approach may be further improved by introducing a topic mixture directly into the generator model. in this case, it would play a role of extended context and thus provide additional features the generator can rely on. such an extension may improve quality even further due to eliminating of intermediate subdomains classifier and prevent errors that can happen on this level. also, there is a potential to integrate knowledge about user profiles into the generator to obtain varying banners according to user behavior to better reach the target audience. further efforts should be directed to the investigation of how different topic models and other schemes of a dataset dividing into separate domains influence the overall quality of generators.