Contextualized Embeddings for Enriching Linguistic Analyses on Politeness (2020)

Abstract: Linguistic analyses in natural language processing (NLP) have often been performed around the static notion of words where the context (surrounding words) is not considered. For example, previous analyses on politeness have focused on comparing the use of static words such as personal pronouns across (im)polite requests without taking the context of those words into account. Current word embeddings in NLP do capture context and thus can be leveraged to enrich linguistic analyses. In this work, we introduce a model which leverages the pre-trained BERT model to cluster contextualized representations of a word based on (1) the context in which the word appears and (2) the labels of items the word occurs in. Using politeness as case study, this model is able to automatically discover interpretable, fine-grained context patterns of words, some of which align with existing theories on politeness. Our model further discovers novel finer-grained patterns associated with (im)polite language. For example, the word please can occur in impolite contexts that are predictable from BERT clustering. The approach proposed here is validated by showing that features based on fine-grained patterns inferred from the clustering improve over politeness-word baselines.

## introduction linguistic analyses have often been computationally performed around the static notion of words or word categorization methods (e.g. liwc) where the context of words is not taken into account. previous work on politeness @xcite and gender @xcite have focused on comparing the use of non-contextual broad word categories such as personal pronouns across different categories/groups. for example, @xcite found that women use more pronoun words than men. @xcite found that requests which contain a hedge word (e.g. think) are more likely to be perceived as polite than impolite. however, words often occur in many different contexts and thus analyzing them statically hides cues which can potentially enrich our understanding of the phenomenon being studied. as opposed to static word embeddings which provide the same representation for a word regardless of its context (i.a. @xcite @xcite , the bert @xcite and elmo @xcite models provide methods for extracting pre-trained contextualized word representations. by leveraging contextualized representations, linguistic theories can be validated and enriched. given a dataset annotated for a downstream task, we build a model which automatically discovers fine-grained context patterns of words. discovering such patterns provides insight into the phenomenon the task attempts to model. we use pre-trained bert embeddings and exploit the fact that words which occur in similar contexts tend to have similar representations. our model takes as input contextualized representations for a given word and splits them into different clusters based on the context in which the word appeared and the labels of the items the word belongs to. by doing so, we are able to identify the different contexts in which the word appears and how they correlate with the categories of the task being studied. we use politeness as case study of a linguistic phenomenon. existing computational work on politeness developed feature-based @xcite and neural @xcite models which detect if a natural language request (e.g. can you please tell me how to do that?) is polite or impolite. @xcite developed a computational tool driven by existing theories in the literature on politeness @xcite . these theories highlighted linguistic constructions that speakers use to reduce the burden on the addressee by sounding indirect (e.g. could you please [...]). @xcite showed further that, for some words, their position in the request plays a role in whether the request will be perceived as polite or impolite. for example, even though please is considered polite, they show that requests starting with please (e.g. please explain this to me.) are more likely to be perceived as impolite while requests with please in the middle (e.g. could you please help me out?) are more likely to be perceived as polite. we take an exploratory approach that is driven to discover fine-grained word patterns that encode the surrounding context as opposed to simply the position of the word. the proposed model automatically discovers patterns that have already been discussed in the literature in addition to multiple novel ones. for example, the model discovers fine-grained context patterns of the word please one of which include occurrences of please in the middle of impolite requests (e.g. would you please stop?) signaling that our model does not simply encode the position of the word but also the surrounding context. we validate the proposed model by showing that features based on the fine-grained patterns that it discovers outperform current feature-based politeness models. uncovering contextual information on how words are used can enhance our understanding of the task/phenomena being studied. it can also help in enriching the linguistic theories associated with the task. striving for informativeness and interpretability can make models more useful in downstream applications. ## discovering fine-grained context patterns given a word w appearing in multiple labeled text items, we build a model to automatically discover clusters such that each cluster contains use cases of the word which (1) are contextually similar in the way the word w is used, and (2) belong to items which are for the most part of the same label. for example, the word think can appear in different contexts across items which express positive sentiment such as i think this movie is great! and great movie! do you think it will win the oscars?. the goal is to automate the process of splitting multiple use cases of a word into different clusters. this helps uncover information about how a word can be used and how its usage relates to the labels of the task at hand, thus enriching our understanding of the phenomenon being studied. the resulting clusters can also be used to extract fine-grained, contextualized unigram features for the target task. we now describe the process by which the clustering is performed. we not only want to encourage contextual similarity of the word's usage across elements of each cluster, but also encourage clusters to have high purity determined based on the labels of items the word appeared in. striving for such criteria simplifies the process of interpreting the resulting clusters. we use a decision tree @xcite )-like approach to obtain clusters with such criteria. we apply the following process on each word in a given vocabulary v. first, the pre-trained bert-base representations @xcite of the word in the training set of items are extracted. for every word, bert-base produces 12 representations that correspond to 12 layers. we average the representations of the last 6 layers for every word. given that bert embeddings are contextualized, the same word will have a different representation depending on the context in which it appears. we label each contextualized representation with the same label as the item it was extracted from. we denote the representations of the word in the training set as the parent cluster. elements (contextualized representations) of the parent cluster are recursively split into two child clusters. in order to determine the cluster membership of each element, we sample two centroid representations from the parent cluster and map each of its representations to the closer of the two centroids. the proximity measure that we use is the euclidean distance. we try all possible pairs of centroids and select the pair which results in the binary split which maximizes the information gain (ig). the information gain is calculated as in (1): ## application: politeness detection the task that we choose as case study for our model is the task of detecting politeness in natural language requests. in section 3.1, we describe the dataset used to train and evaluate our model. in section 3.2, we describe existing politeness features. in section 3.3, we describe the model instantiation and evaluate the model qualitatively. ## conclusion in this work, we proposed a model for obtaining insight on how words are used given the context of their usage and the labels of the items they belong to. when applied to requests annotated for politeness, our model not only re-discovers existing linguistic markers discussed in the literature but also enriches them by discovering novel finer-grained patterns of using words in (im)polite language. we quantitatively validated the model by showing that fine-grained features improve over politeness-word baselines. this shows that examining the context of using words helps enriching the linguistic analyses of the task/phenomenon being studied. in the future, we plan to extend our model to obtain insight on tasks based on annotated text pairs @xcite . we also plan to use our model to inform sociolinguistic analyses on gender in order to get a better understanding of the relationship between language use and gender.