What BERT s and GPT s know about your brand? Probing contextual language models for affect associations (2021)

Abstract: Investigating brand perception is fundamental to marketing strategies. In this regard, brand image, defined by a set of attributes (Aaker, 1997), is recognized as a key element in indicating how a brand is perceived by various stakeholders such as consumers and competitors. Traditional approaches (e.g., surveys) to monitor brand perceptions are time-consuming and inefficient. In the era of digital marketing, both brand managers and consumers engage with a vast amount of digital marketing content. The exponential growth of digital content has propelled the emergence of pre-trained language models such as BERT and GPT as essential tools in solving myriads of challenges with textual data. This paper seeks to investigate the extent of brand perceptions (i.e., brand and image attribute associations) these language models encode. We believe that any kind of bias for a brand and attribute pair may influence customer-centric downstream tasks such as recommender systems, sentiment analysis, and question-answering, e.g., suggesting a specific brand consistently when queried for innovative products. We use synthetic data and real-life data and report comparison results for five contextual LMs, viz. BERT, RoBERTa, DistilBERT, ALBERT and BART.

## introduction brands play a vital role in marketing strategies. they are essential to company positioning, marketing campaigns, customer relationships, and profits @xcite . a brand persona is broadly defined by a set of attributes or dimensions; for instance, 'mountain dew' may be recognized by attributes such as 'adventurous' and 'rugged'. while aaker's dimensions @xcite are widely used to define a brand persona, more fine-grained attributes are documented in @xcite . furthermore, evaluating a brand persona, i.e., how a brand is perceived by various stakeholders such as consumers, competitors, and market analysts has been an active area of research @xcite . following the widespread success of pre-trained word representations, alternatively called language models (lms), consumer-specific downstream tasks such as recommender systems, dialogues systems, and information retrieval engines look to make use of brand persona along with these representations to better fulfill consumer requirements. accordingly, we formulate our first research question (rq1) as do lms store implicit associations between brands and brand image attributes?. to answer this, we look specifically at brands and brand image defined as affect attributes. since lms are trained on real-world data; we believe that these representations may be useful in understanding correlations between a brand and its persona attributes. while numerous studies have investigated unintended biases in natural language processing systems @xcite @xcite @xcite , this is probably the first work that explores brand and affect attributes associations in pre-trained lms. these lms are trained in an unsupervised manner on large-scale corpora. the training corpora generally comprise a variety of textual data such as common web crawl, wikipedia dump, and book corpora. they are optimized to statistical properties of the training data from which they pick up and amplify real-world trends and associations along with biases such as gender and race @xcite . some of these biases may be beneficial for downstream applications (e.g., filtering out mature content for non-adult viewers) while some can be inappropriate (e.g., resume sorting system believing men are more qualified programmers than women @xcite . marketing applications such as recommender systems and sentiment analysis can also perpetuate and highlight unfair biases, such as consistently showing popular brands as recommendations and not considering uncommon brands with less positive sentiment. with this in mind, we formulate our second research question (rq2) as do the associations embedded in lms signify any bias? we also investigate whether these associations are consistent across all lms as rq3. brand personas are alternatively characterized as brand archetypes in bechter's work @xcite . brand archetypes are widely used as effective branding and marketing strategy. according to jung @xcite , archetypes are defined as inherent images within the collective human unconsciousness having universal meaning across cultures and generations. when successfully used, archetypal branding provides a narrative to connect with consumers. we formulate the following research questions: rq4 as do lms capture brand personality intended by a brand? and rq5 as do lms capture brand personality as perceived by consumers? we propose to use brand-attribute associations to understand brand archetypes perceived by lms. in this work, we probe five different lms ( bert @xcite , albert @xcite , roberta @xcite , distilbert @xcite and bart @xcite ) on affect associations by using masked language model (mlm) head. the choice of lms was guided by three factors: 1) availability of mlm head, 2) variety in model architectures and 3) type and size of training data used while pre-training. lm pre-training data architecture bert bookcorpus (800m words), english wikipedia (2,500m words) @xmath0, @xmath1, @xmath2, @xmath3, cc-news (63m articles), openwebtext (8m documents), stories @xmath4, @xmath5, @xmath6, @xmath7, english wikipedia (2,500m words) @xmath8, @xmath9, @xmath10, @xmath11, english wikipedia (2,500m words) @xmath12, @xmath13, @xmath14, @xmath15, cc-news (63m articles), openwebtext (8m documents), stories @xmath16, @xmath17, @xmath18: variants of lms. l-total layers, h-hidden size, a-self-attention heads, t-total parameters. we mention the architecture of the large version of all the lms. the success of pre-trained word embeddings in achieving state-of-the-art results has sparked widespread interest in investigating information captured in these representations. typically defined as 'probing task', a wide variety of analyses have been proposed. for instance, @xcite proposes a structural probe to test whether syntax trees are embedded in word representation space. experiments in @xcite are aimed to investigate the numerical reasoning capabilities of an lm. similarly, @xcite presents an in-depth analysis of relational knowledge present in pre-trained lms. penha and hauff (2020) probe the contextual lms (bert and roberta) for the conversational recommendation of books, movies, and music. our work seeks to apply the idea of probing to a relatively unexplored area of affect analysis. to the best of our knowledge, this is the first work that presents a multi-pronged investigation of brands and subjective knowledge like affect attributes represented in contextual representation. @xcite is the most relevant prior work in terms of affect analysis. they present an entity-centric affective analysis with the use of contextual representations, where they find that meaningful affect information is captured in contextualize word representations but these representations are heavily biased towards their training data. a significant effort has been seen in investigating the intrinsic bias in word embeddings. these representations are trained in an unsupervised manner using a large amount of training data typically consisting of common web crawls. as a result, all kinds of biases like gender, race, demography along with trends and preferences get encoded in lms. works in @xcite @xcite propose methodologies to measure and mitigate bias in word representations. our work is targeted at finding trends and preferences that certain entities have by using a combination of old and new such measures. ## dataset in this work, we evaluate affect information captured in the lms for different brands. accordingly, the selected brands should have large volumes of online data to get significant representation in the lms. we choose 697 major us national brands reported in @xcite . these brands are categorized into 16 different product categories. to analyze affect associations, we refer to surveys conducted by young and rubicam (y&r) @xcite to measure a broad array of perceptions and attributes for a large number of brands. we choose 40 affect attributes listed as a part of 'brand image' in @xcite . we also manually map (see ## experimental setup we outline our approach for exploring answers to the research questions stated above. ## discussion we present a battery of analyses aimed at finding how much knowledge do the off-the-shelf lms capture about brands and affect attributes. ## conclusion in this paper, we presented a series of exploration setups to address research questions pertaining to associations between brands and brand image attributes. our analyses were able to tease out varied responses even from the models having identical training data and pre-training learning objectives. we observed that there exists a definite association between brands and attribute affect words across all lms (rq1). this impression is observed across a range of abstraction i.e. from individual brands and broader categories to attributes and aaker's dimensions. in all our experiments, some categories such as 'cars' and 'technology product & stores' and brands such as 'disney' and 'intel' are found to have consistent associations across all lms (rq3). however, it is interesting to note that these biases do not concur with both consumer perceptions and intended perceptions of the brand (rq4 and rq5). lastly, it is seen that perturbations in sentence moderately influences the association between brands and affect words. improved performance in fine-tuning implies that affect associations are enhanced (rq2). since we do not have enough data, it remains to be seen how additional training data changes the landscape. this work documents an initial investigation of brand and attribute associations in different lms. with enough task-specific data, we plan to evaluate how the affect associations are enhanced. we also intend to use these observations in further defining brand-persona and brand-archetype definitions. these impressions can help understand perceptions about a brand. furthermore, this can be extended in investigating impressions about iconic entities such as sports teams, celebrities, and politicians.