A d BERT : An Effective Few Shot Learning Framework for Aligning Tweets to Superbowl Advertisements (2022)

Abstract: The tremendous increase in social media usage for sharing Television (TV) experiences has provided a unique opportunity in the Public Health and Marketing sectors to understand viewer engagement and attitudes through viewer-generated content on social media. However, this opportunity also comes with associated technical challenges. Specifically, given a televised event and related tweets about this event, we need methods to effectively align these tweets and the corresponding event. In this paper, we consider the specific ecosystem of the Superbowl 2020 and map viewer tweets to advertisements they are referring to. Our proposed model, AdBERT, is an effective few-shot learning framework that is able to handle the technical challenges of establishing ad-relatedness, class imbalance as well as the scarcity of labeled data. As part of this study, we have curated and developed two datasets that can prove to be useful for Social TV research: 1) dataset of ad-related tweets and 2) dataset of ad descriptions of Superbowl advertisements. Explaining connections to SentenceBERT, we describe the advantages of AdBERT that allow us to make the most out of a challenging and interesting dataset which we will open-source along with the models developed in this paper.

## introduction the joint consumption of television programming and social media participation has become increasingly popular, leading to the rise of social tv ecosystems @xcite @xcite . twitter has become an integral outlet for tv viewers, with a whopping 85% of users tweeting while watching television programming @xcite . marketers, television networks, and social media platforms have explored this rising potential of social tv ecosystems. for example, twitter and content providers on television networks collaborated recently @xcite to create social tv experiences, and companies such as nielson @xcite are investing in technologies to quantify and analyze social tv audiences. figure 1 : mapping the the tweets referring to advertisements telecast during the superbowl event. for e.g., the tweet mentioning #diversity and #inclusivity is mapped to the advertisement by olay featuring the hashtag #makespaceforwomen. social tv research is still in its infancy stage @xcite . however, a few studies @xcite have already explored the impact of television on social media word of mouth (wom) and "impactful" factors (celebrity presence) that influence the volume of social media wom. similarly, identifying "attentiongrabbing" moments in media (e.g., the performance of the speaker during the presidential debate or a funny ad during the superbowl), can help gauge the reaction of the viewers'. hence, it becomes necessary to build tools that capture these "attention-grabbing" moments and analyze the subsequent responses. these tools are not only necessary for program recall and re-contextualization @xcite , but also for the design of more personalized recommendations in the future. @xcite . with a large amount of social buzz generated online, analyzing the responses of the viewers towards televised events has now become much easier as opposed to earlier slow and costly methods that involved surveying the viewers. however, this also comes with its technical challenges. for instance, given a televised event and an associated set of social media posts, an approach that effectively maps the posts to the parts of the event they are referring to (figure 1 ) is necessary. this raises two follow-up questions: (a) discretely atomizing the event into segments and (b) identify if the tweet focuses on a specific event segment or the event as a whole. for ex-ample, tweets can be related to the commercials during the break or the game as a whole during the superbowl broadcast. therefore, a method that can align the tweets and their related televised events is an essential building block toward answering fundamental questions regarding the event's influence on the viewer's social tv activity. machine learning based methods towards this end have attempted event segmentation @xcite ; however, they analyze events and tweets independently. this is a big drawback as the event influences the viewer's response; hence there is a need to jointly model tweets ad televised content information. our research study considers the specific social tv advertising ecosystem during the high-stakes super bowl sporting event. in this event, since the ads telecast and audience responses are on different media channels (i.e. tv and twitter, respectively) over a fixed time, we view the problem as a closed system consisting of two interacting sets -the set of stimuli (advertisements) and the set of responses (tweets). our research focuses on modelling the function that maps these two sets to each other. tweet to ad mapping is non-trivial problem as the viewer's tweet could be about multiple aspects of the advertisement, such as its creative elements or the brand making the advertisement. for example, if we consider the tweet,"the pepsi ad was so amazing", this is a simplistic case as it is easy to map that the viewer is talking about the advertisement by pepsi. however, in the case of another tweet, "mc hammer is still making money with songs low key", it is not easy to understand that this tweet is even ad-related (mc hammer is a celebrity who featured in the cheetos advertisement). moreover, we are trying to capture ad-related tweets against the general noise of the superbowlrelated tweets. in this setting, wom is much less for ad-related tweets (limited representation) and the viewers are also likely to talk about some ads more than others (class imbalance). our mapping methodology adbert is an effective few-shot learning framework that establishes semantic relatedness between an advertisement and a tweet under the constraints of class imbalance and limited class representation. once this mapping is established, it can be used as an essential building block in an audience engineering pipeline that can help incorporate a feedback loop to an advertisement and aid in downstream tasks such as ad-engagement measurement and sentiment analysis. as a by-product of our experiments, we also developed a manually annotated rich dataset of ad-related tweets and a manually annotated dataset of superbowl ad descriptions, which can be used for further research in social tv literature. ## related work with the rise of social tv technologies, research has been done to examine how media multitasking affects viewers' response to advertisements and how advertisers can leverage this behaviour @xcite . (lewis and reiley, 2014) find a sudden increase in online searches for brands shown during superbowl commercials immediately after the ad is telecast. while the aligning of real-time social media responses to tv advertising has been explored in recent years @xcite , their methods to map the tweets to the advertisement is based on the underlying assumption that a person tweets about an advertisement as soon as they see it, which need not be true @xcite . our proposed method relies on content mapping, which would capture tweets about an advertisement irrespective of its time of airing. though @xcite consider the primacy effect, their topic-model based approach method cannot be applied to televised segments for which no transcripts are readily available. advertisements broadcast on tv are usually tiny time segments for which auto-generated transcripts are not meaningful, as they could be theme songs or even a catchphrase. however, even this kind of short tv content is impactful enough to generate significant wom. our approach to solving this correspondence problem with its unique challenges draws inspiration from some previous research works @xcite @xcite which use different encoders for pairwise sentence scoring tasks and @xcite which inspires the idea for joint learning. our approach aligns tweets with their corresponding tv advertisement through jointly learning from both the advertisement information and the tweets. ## main technical challenges a tweet could be a response to either the advertisement's creative elements (for example, a cute retriever in the weathertech ad) or the advertisement as a whole. therefore, detecting the ad-relatedness requires a holistic understanding of the advertisement's content. identifying ad-relatedness can be viewed through a semantic relatedness lens such that we try to establish a relationship between the tweet and the advertisement description. however, the short length of tweets and their characteristic lingo adds to the complexities of identifying semantic-relatedness. while the tweet is a short sentence, the ad description is a comma-separated list of key phrases or words. hence a semantic gap exists between the twitter lingo and the advertisement descriptions ("audience-annotator" gap.) identifying ad-relatedness can also be seen through the lens of multi-class classification, which involves scoring a set of candidate labels given an input context. the superbowl dataset shows the unique characteristic of class imbalance with 15 popular or controversial commercials having high representation in the dataset and we call these -"majority classes". for example, the hulu advertisement featured superbowl superstar tom brady and was a viral ad and hence, a "majority" class. our threshold for a majority class is that the number of samples for that class should be at least 40. 46 other commercials exist in our data with lesser than 40 samples for the model to train on and understand patterns in these cases. we call these classes -"minority classes". each class in our training data also suffered from limited representation, with the average number of samples in a majority class being 122 and in a minority class being 17. ## experiments from a semantic relatedness perspective, we can try to map the text and ad information into a common feature space wherein a dot product, cosine or (parameterized) non-linear function is typically used to measure their similarity. sentencebert (reimers and gurevych, 2019) is a bi-encoder model, which applies bert independently on the two inputs, followed by mean pooling on the output to create separate fixed-sized sentence embeddings. as the representations are separate, the bi-encoders is able to cache the encoded candidates and reuse these representations for each input resulting in faster prediction times than cross-encoders. however, the tweets and ad description information in our dataset are not in the same vector space because the tweet has a sentence structure, while the advertisement information is a set of key phrases describing the ad from the annotator's point of view. therefore we consider the multi-classification perspective where we can try to score a set of candidate ad descriptions given an input tweet. this kind of multiclass classification can be done via classical machine learning approaches @xcite such as logistic regression and multilayer perceptron (mlp) with tf-idf vectorization of features. in these approaches, words characteristic to an ad are given greater weight than words that frequently appear across all the ads. our implementation of the mlp has 12 hidden layers each with dimension of 6000. the model is trained for categorical entropy loss with a batch size of 20 and number of epochs as 50. deep learning based methods like bert @xcite uses a cross-encoder @xcite where a special sep token separates the input and label candidate and multi-head attention is applied over all input tokens. in our implementation of bert for multi-class classification, we fine-tune @xcite the pre-trained 'bert-base-uncased' model with 12 layers from transformers library @xcite to identify if a tweet can be identified as related to a super bowl commercial or not. if the tweet is "superbowl event related" and does not relate to any ad, it is categorized as a 'none' class. else, the tweet is classified as 'ad-related'. for all the tweets classified as ad-related, we compute the embeddings from bert and run a softmax on similarity scores to identify the ad class. the model is trained on 4656 tweets and 61 classes. we use a batch size of 32, a learning rate of 2e-5 and the number of epochs as 4. we also use the epsilon parameter eps with a value 1e-8 to prevent any division by zero in the implementation. in our implementation of sentencebert, we finetune the pre-trained "nq-distilbert-base-v1" model using the joint learning setup described in section 6 and using cosine similarity loss. we use number of epochs as 30, warmup_steps as 100 and evaluation_steps as 500. during test time, we compute the maximum cosine similarity of the input tweet against all of the ad descriptions to get the ad class assigned to the tweet, but with the embeddings obtained from the fine-tuned sentencebert model. ## adbert : proposed joint learning approach figure 2 : adbert architecture our adbert approach frames the multi-class classification problem of mapping a tweet to its respective advertisement as a binary classification and semantic relatedness task. as we faced a problem of a limited labeled dataset, we required a better training signal from our dataset. in order to solve this problem, we use an approach utilizing class verbalizers as seen in similar research works for few shot learning @xcite @xcite . in our case study, we propose learning from both the tweet as well as the textual descriptions of each ad class, which is a part of our ad information dataset ( the cross encoder in adbert takes as input to the network both the tweet and the ad description separated by a sep token and multi-head attention, is applied across all tokens of the inputs. compared to a bi-encoder, the cross-encoder offloads the similarity computation to the self-attention matrices and hence is able to better learn to identify ad-relatedness. this implies that both inputs are compared simultaneously and helps solve the ad-relatedness problem. the problem task reformulation we suggest, where we append the label information to the tweet and assist the cross encoder, also solves the limited representation problem, thus allowing our model to behave as an effective few-shot learning framework. ## adbert used in downstream tasks our model serves as an essential part of multiple audience engineering pipelines in a social tv setting. in the research by @xcite , the aim is to examine the influence of the viewer's temporary affective states during superbowl ad exposure. in order to compute the viewer's affective state, a key step is to be able to understand which advertisement impacted the user's affective state, thereby making them tweet in a specific way. this is done using adbert, which proves to be superior than time-based tweet-ad alignment. this is because the advertisements are typically very short (10 to 20s) and the user is more likely to tweet much later @xcite than during this brief time period. similarly, in the work by @xcite , ad-related tweets derived through adbert are analyzed for evidence of gender-targeted advertising during the super bowl. ## limitations and future work since our current adbert approach uses a fine-tuned bert-base-uncased model, using fine-tuned bertweet @xcite , which is a pre-trained language model for english tweets, seems like a suitable next step. adbert uses additional information about the ad classes for joint learning. three authors manually annotate this information in this research, but manually generated class verbalizers heavily depend on domain specific prior knowledge and finding appropriate label descriptions automatically is a challenging research problem that can be further explored. similar and multiple advertisement mentioning commercials pose a problem in ad-tweet mapping and can be further disambiguated by considering the tweet's timestamp in addition to the tweet content. the joint learning strategy described in adbert can also be extended to other social tv datasets. for example, in the social tv ecosystem of presidential debates telecast on television, tweets could be mapped to segments of the debates. this could have multiple downstream implications such as viewer stance detection, viewer engagement analysis etc. ## conclusion in this paper, we develop a model, adbert, that aligns tweets to the advertisements they refer to in the context of the social tv ecosystem of superbowl 2020. this problem is technically challenging because of the difficulties in establishing ad-relatedness of a tweet, class imbalance in the dataset and limited representation for each ad class. we find that framing this multi-class classification problem as a binary classification and semantic relatedness task results in superior f1 performance compared to our baseline models. in the joint learning setting, the model learns from both the input and label information together, leading to better classification even in lesser represented classes. thus our model generalizes well despite the class imbalance and limited labelling problems in the dataset. adbert makes no assumptions about the number of ad categories and is easily extensible. our model can be highly useful as a step toward incorporating feedback into advertisements and analyzing viewer engagement and attitudes. as a by-product of this research, we also developed a dataset of ad-related tweets and a dataset of ad descriptions of superbowl ads, which can be used to further social tv research.