Dictionary-Assisted Supervised Contrastive Learning (2022)

Abstract: Text analysis in the social sciences often involves using specialized dictionaries to reason with abstract concepts, such as perceptions about the economy or abuse on social media. These dictionaries allow researchers to impart domain knowledge and note subtle usages of words relating to a concept(s) of interest. We introduce the dictionary-assisted supervised contrastive learning (DASCL) objective, allowing researchers to leverage specialized dictionaries when fine-tuning pretrained language models. The text is first keyword simplified: a common, fixed token replaces any word in the corpus that appears in the dictionary(ies) relevant to the concept of interest. During fine-tuning, a supervised contrastive objective draws closer the embeddings of the original and keyword-simplified texts of the same class while pushing further apart the embeddings of different classes. The keyword-simplified texts of the same class are more textually similar than their original text counterparts, which additionally draws the embeddings of the same class closer together. Combining DASCL and cross-entropy improves classification performance metrics in few-shot learning settings and social science applications compared to using cross-entropy alone and alternative contrastive and data augmentation methods.

## introduction we propose a supervised contrastive learning approach that allows researchers to incorporate dictionaries of words related to a concept of interest when fine-tuning pretrained language models. it is conceptually simple, requires low computational resources, and is usable with most pretrained language models. dictionaries contain words that hint at the sentiment, stance, or perception of a document (see, e.g., @xcite . social science experts often craft these dictionaries, making them useful when the underlying concept of interest is abstract (see, e.g., @xcite . dictionaries are also useful when specific words that are pivotal to determining the classification of a document may not exist in the training data. this is a particularly salient issue with small corpora, which is often the case in the social sciences. however, recent supervised machine learning approaches do not use these dictionaries. we propose a contrastive learning approach, dictionary-assisted supervised contrastive learning (dascl), that allows researchers to leverage these expert-crafted dictionaries when fine-tuning pretrained language models. we replace all the words in the corpus that belong to a specific lexicon with a fixed, common token. when using an appropriate dictionary, keyword simplification increases the textual similarity of documents in the same class. we then use a supervised contrastive objective to draw together text embeddings of the same class and push further apart the text embeddings of different classes @xcite . figure 1 visualizes the intuition of our proposed method. the contributions of this project are as follows. roberta are not state-of-the-art pretrained language models, dascl can augment the loss functions of state-of-the-art pretrained language models. ## related work use of pretrained language models in the social sciences. transformers-based pretrained language models have become the de facto approach when classifying text data (see, e.g., @xcite @xcite , and are seeing wider adoption in the social sciences. @xcite show that roberta and xlnet @xcite outperform bag-ofwords approaches for political science text classification tasks. ballard et al. (2022) use bertweet (nguyen et al., 2020) to classify tweets expressing polarizing rhetoric. lai et al. (2022) use bert to classify the political ideologies of youtube videos using text video metadata. dascl can be used with most pretrained language models, so it can potentially improve results across a range of social science research. ## method the approach consists of keyword simplification and the contrastive objective function. figure 2 shows an overview of the proposed framework. ## conclusion we propose a supervised contrastive learning approach that allows researchers to leverage specialized dictionaries when fine-tuning pretrained language models. we show that using dascl with cross-entropy improves classification performance on sst-2 in few-shot learning settings, on classifying perceptions about the economy expressed in new york times articles, and on identifying tweets containing abusive language when compared to using cross-entropy alone or alternative contrastive and data augmentation methods. in the future, we aim to extend our approach to other supervised contrastive learning frameworks, such as using this method to upweight difficult texts @xcite . we also plan to expand this approach to semi-supervised and self-supervised settings to better understand core concepts expressed in text.