Are “Undocumented Workers” the Same as “Illegal Aliens”? D isentangling Denotation and Connotation in Vector Spaces (2020)

Abstract: In politics, neologisms are frequently invented for partisan objectives. For example, “undocumented workers” and “illegal aliens” refer to the same group of people (i.e., they have the same denotation), but they carry clearly different connotations. Examples like these have traditionally posed a challenge to reference-based semantic theories and led to increasing acceptance of alternative theories (e.g., Two-Factor Semantics) among philosophers and cognitive scientists. In NLP, however, popular pretrained models encode both denotation and connotation as one entangled representation. In this study, we propose an adversarial nerual netowrk that decomposes a pretrained representation as independent denotation and connotation representations. For intrinsic interpretability, we show that words with the same denotation but different connotations (e.g., “immigrants” vs. “aliens”, “estate tax” vs. “death tax”) move closer to each other in denotation space while moving further apart in connotation space. For extrinsic application, we train an information retrieval system with our disentangled representations and show that the denotation vectors improve the viewpoint diversity of document rankings.

## introduction language carries information through both denotation and connotation. for example, a reporter writing an article about the leftmost wing of the democratic party can choose to refer to the group as "progressives" or as "radicals". the word choice does not change the individuals referred to, but it does communicate significantly different sentiments about the policy positions discussed. this type of linguistic nuance presents a significant challenge for natural language processing systems, most of which fundamentally assume words to have similar meanings if they are surrounded in similar word contexts. such assumption risks confusing differences in connotation for differences in denotation or vice versa. for example, using a common skip-gram model @xcite trained on a news corpus (described in §3.2), figure 1 shows nearest neighbors of "government-run healthcare" and "economic stimulus". the resulting t-sne clusters are influenced as much by policy denotation (shapes) as they are by partisan connotation (colors foot_0 ). using these entangled representations in applications such as information retrieval could have pernicious consequences such as reinforcing ideological echo chambers and political polarization. for example, a right-leaning query like "taxpayer-funded healthcare" could make one equally (if not more) likely to see articles about "totalitarian" and "horror stories" than about "affordable healthcare". to address this, we propose classifier probes that measure denotation and connotation information in a given pretrained representation, and we arrange the probe losses in an adversarial setup in order to decompose the entangled pretrained meaning into distinct denotation and connotation representations ( §4). we evaluate our model intrinsically and show that the decomposed representations effectively disentangle these two dimensions of semantics ( §5). we then apply the decomposed vectors to an information retrieval task and demonstrate that our method improves the viewpoint diversity of the retrieved documents ( §6). all data, code, preprocessing procedures, and hyperparameters are included in the appendix and our github repository. foot_1 ## philosophical motivation consider the following two sentences: "undocumented workers are undocumented workers" vs. "undocumented workers are illegal aliens". frege (1892) famously used sentence pairs like these, which have the same truth conditions but clearly different meanings, in order to argue that meaning is composed of two components: "reference", which is some set of entities or state of affairs, and "sense", which accounts for how the reference is presented, encompassing a large range of aspects such as speaker belief and social convention. in contemporary philosophy of language, the sense and reference argument has evolved into debates of semantic externalism vs. internalism and referential vs. conceptual role semantics. externalists and referentialists foot_2 continue the truthconditional tradition and emphasize meaning as some entity to which one is causally linked, invariant of one's psychological encoding of the referent @xcite . on the other hand, conceptual role semanticists emphasize meaning as what inferences one can draw from a lexical concept, deemphasizing the exact entities which the concept includes @xcite . naturally, a popular position takes the cartesian product of both schools of meaning @xcite . this view is known as two-factor semantics, and it forms the inspiration for our work. to avoid confusion with definitions from existing literature, we use the terms "denotation" and "connotation" rather than "reference" and "concept" when discussing our models in this paper. ## data we assume that it is possible to disentangle the two factors of semantics by grounding language to different components of the non-linguistic context. in particular, our approach assumes access to a set of training sentences, each of which grounds to a denotation d (which approximates reference) or a connotation c (which approximates conceptual inferences). we require at least one of d or c to be observed, but we do not require both (elaborated in §4.3). in this work, d and c are discrete symbols. however, our model could be extended to settings in which d and c are feature vectors. while we are interested in learning lexical-level denotation and connotation, we train on sentenceand document-level speaker and reference labels. we argue that this emulates a more realistic form of supervision. for example, we often have metadata about a politician (e.g., party and home state) when reading or listening to what they say, and we are able to aggregate this to make lexical-level judgements about denotation and connotation. we experiment on two corpora: the congressional record (cr) and the partisan news corpus (pn), which differ in linguistic style, partisanship distribution (figure 2 ), and the available labels for grounding denotation and connotation. figure 2 : vector spaces that result from training vanilla word2vec on the congressional record (left) and partisan news (right). we evaluate on both corpora, but note that the partisan news corpus better exemplifies the problem we target where words cluster strongly according to ideological stance. ## intrinsic evaluation we confirm that our decomposed denotation and connotation spaces reflect their intended purposes by measuring their structures with homogeneity metrics ( §5.1) on three sets of evaluation words ( §5.2) as well as inspecting their t-sne clusters. ## extrinsic evaluation ultimately, our work aims to be more than just a theoretical exercise, but also to enable greater control over how sensitive nlp systems are to denotation vs. connotation in downstream tasks. to this end, we construct an ad hoc information retrieval task. we compare a system built on top of v pretrained to systems built on top of v deno and v conno in terms of both the quality of the ranking and the ideological diversity represented among the top results. ## summary in this paper, we describe the problem of pretrained word embeddings conflating denotation and connotation. we address this issue by introducing an adversarial network that explicitly represents the two properties as two different vector spaces. we confirm that our decomposed spaces encode the desired structure of denotation or connotation by both quantitatively measuring their homogeneity and qualitatively evaluating their clusters and their representation of well-known political euphemisms. lastly, we show that our decomposed spaces are capable of improving the diversity of document rankings in an information retrieval task.