Towards Few-Shot Identification of Morality Frames using In-Context Learning (2022)

Abstract: Data scarcity is a common problem in NLP, especially when the annotation pertains to nuanced socio-linguistic concepts that require specialized knowledge. As a result, few-shot identification of these concepts is desirable. Few-shot in-context learning using pre-trained Large Language Models (LLMs) has been recently applied successfully in many NLP tasks. In this paper, we study few-shot identification of a psycho-linguistic concept, Morality Frames (Roy et al., 2021), using LLMs. Morality frames are a representation framework that provides a holistic view of the moral sentiment expressed in text, identifying the relevant moral foundation (Haidt and Graham, 2007) and at a finer level of granularity, the moral sentiment expressed towards the entities mentioned in the text. Previous studies relied on human annotation to identify morality frames in text which is expensive. In this paper, we propose prompting based approaches using pretrained Large Language Models for identification of morality frames, relying only on few-shot exemplars. We compare our models’ performance with few-shot RoBERTa and found promising results.

## introduction while the nlp field has seen tremendous progress over the last decade, building models capable of identifying abstract concepts remain a highly challenging problem. this difficulty stems from two key reasons. first, these concepts can manifest in very different ways in text. for example, the concept of fairness, that we discuss at length in this paper, can be discussed in the context of the abortion debate (e.g., "right to privacy") or in the context of covid-19 vaccination (e.g., "everyone should have access to the vaccine"). learning to identify instances of this concept in previously unseen contexts remains a challenge. second, building nlp models using the supervised learning paradigm requires humans to annotate data, which for such tasks is a cognitively demanding process. in this paper, we investigate whether the recently introduced paradigm of zero/few shot learning using large language models @xcite is better equipped to deal with these challenges. we focus on a recently introduced framework for analyzing moral sentiment, called morality frames @xcite . this framework builds on, and extends, moral foundation theory @xcite , which identifies five moral values (i.e., foundations, each with a positive and a negative polarity) central to human moral sentiment which include care/harm, fairness/cheating, loyalty/-betrayal, authority/subversion, and purity/degradation. morality frames is a relational framework that identifies expressions of the moral foundations in text and associates moral roles with entities mentioned in it (see section 3 for details). unlike previous approaches to this task @xcite which use annotated data to train a relational classifier using drail @xcite , we define the task as a zero/few shot problem. we rely on incontext learning using large language models for the identification of morality frames. in in-context learning, a desired nlp task is framed as a text generation problem where the large language models are provided with zero/few shot input-output pairs and prompted to generate label for the test data point without updating parameters of the llms @xcite . in this paper, we introduce several prompting techniques for llms for the identification of morality frames in tweets that rely on only few-shot examples. we compare our models' performance with few-shot roberta-based @xcite classifiers. we found that prompting-based techniques underperform roberta in identification of subtle concepts like moral foundations, but in case of moral role identification, the prompting-based techniques outperforms roberta by a large mar-gin. note that moral roles are directed towards entities and are more evident than subtle moral foundations. our promising findings in this paper suggest that in-context learning approaches can be useful in many computational social science related tasks and we propose a few potential future directions of this work. ## related works there has been a lot of work towards exploiting existing knowledge in pretrained large language models (llms) and improving its few-shot abilities on various downstream tasks in nlp. some of these works have been influenced from areas related to instruction-based nlp @xcite . @xcite fine-tuned a 140m parameter bart @xcite model using instructions and few-shot examples for various nlp tasks such as text classification, question answering, and text modification. this work suggests that augmenting instructions in the fine-tuning process improves model performance on unseen tasks. on similar lines, through a large scale experiment with over 60 different datasets, @xcite showed that instruction tuning on a llm (≈137b parameters) improves zero and few-shot capabilities of these models. other notable works @xcite show that even a relatively smaller language model can achieve substantial improvement in a similar setting. furthermore, schick and schütze, 2020 use cloze-style phrases in a semi-supervised manner to help lm assign a sentiment label for the text classification task. another line of work focuses on improving lm on downstream tasks with no parameter updates. @xcite proposed to improve llm few-shot performance by conditioning on concatenation of training examples without any gradient updates. other works @xcite have further improved this work and have shown consistent gains in various nlp tasks. in addition, @xcite shows that sufficiently large lm can exploit its innate reasoning abilities to solve complex tasks when provided with a series of intermediate steps during prompting. however, having a generalized llm may have poor performance when the downstream task needs nuanced understanding of the text or is very different from language modeling in nature. while @xcite @xcite have studied sentiment classification task in fewshot settings, not many works are available towards utilizing llm without finetuning it to understand more nuanced concepts like political framing @xcite , moral foundations @xcite , among others. previous work @xcite ) has performed nuanced analysis of political framing by breaking the policy frames proposed by @xcite , into fine-grained sub-frames. it was observed that the sub-frames better captured political polarization by providing a structural breakdown of policy frames. a later work @xcite studied the moral foundation theory @xcite at entity level and proposed a knowledge representation framework for organizing moral attitudes directed at different entities. the structured framework is named morality frames @xcite . these nuanced structural frameworks, such as, frames, sub-frames, entity-centric moral sentiments (morality frames), are expensive to annotate as they largely depend on human knowledge. a few-shot automatic identification of such concepts is required to save manual human-effort and for performing these studies at scale. in this paper, we take the first step towards the analysis on how well llms can understand these psycho-linguistic concepts in few-shot settings. as our first study, we explore in-context learning of morality frames in this paper and leave the study of framing and sub-frames as a future work. ## dataset we conduct our study on the dataset proposed by @xcite . in this dataset, there are 1599 political tweets from us politicians that are annotated for moral foundations by @xcite . @xcite proposed morality frames and broke down the sentence level moral foundations into nuanced moral role dimensions that capture sentiment towards entities expressed in the text. the moral foundations and corresponding moral roles can be found in in this paper, our goal is to study the identification of morality frames when only few-shot train-moral foundations moral roles care/harm: care for others, generosity, compassion, ability to feel pain of others, sensitivity to suffering of others, prohibiting actions that harm others. ing examples are available. to build this setup, we randomly sampled 10 tweets from each of the 5 moral foundations, and used it as training set. we use large language models (llms) for incontext learning that are expensive and resource heavy even for inference only. so, we benchmark our approaches using a smaller test set containing randomly sampled 20 tweets per moral foundation. it resulted in 103 and 207 tweet-entity pairs in the training and the test set, respectively. ## task definition the identification of morality frame in a tweet involves the following two steps. ## experimental evaluation in this section first we discuss our experimental setting. secondly, we discuss our proposed models' performance in morality frame identification. ## summary and future works in this paper, we apply few-shot in-context learning for identification of one of the psycholinguistic knowledge representation framework named morality frames. we proposed different prompting methods to perform the task. we found that in-context learning using a comparatively smaller language model (gpt-j-6b) does not perform well in identification of moral foundations that are very subtle. but it excels in moral roles identification of entities that are more evident in text. we believe there is a lot of scope for improvement, and this study will encourage the application of in-context learning in more computational social science related tasks. below we list a few future directions of this work.