SOCIALITE - LLAMA : An Instruction-Tuned Model for Social Scientific Tasks (2024)

Abstract: Social science NLP tasks, such as emotion or humor detection, are required to capture the semantics along with the implicit pragmatics from text, often with limited amounts of training data. Instruction tuning has been shown to improve the many capabilities of large language models (LLMs) such as commonsense reasoning, reading comprehension, and computer programming. However, little is known about the effectiveness of instruction tuning on the social domain where implicit pragmatic cues are often needed to be captured. We explore the use of instruction tuning for social science NLP tasks and introduce Socialite-Llama — an open-source, instruction-tuned Llama. On a suite of 20 social science tasks, Socialite-Llama improves upon the performance of Llama as well as matches or improves upon the performance of a state-of-the-art, multi-task finetuned model on a majority of them. Further, Socialite-Llama also leads to improvement on 5 out of 6 related social tasks as compared to Llama, suggesting instruction tuning can lead to generalized social understanding. All resources including our code, model and dataset can be found through [bit.ly/socialitellama](https://bit.ly/socialitellama/).

## introduction instruction finetuned large language models (llms) have demonstrated impressive performance on many standard nlp tasks @xcite , but these models tuned on non-social tasks seem to have poor social pragmatics @xcite @xcite . the fact that instruction tuning has been successful with limited amounts of data @xcite and that instruction tuned models have the ability to generalize to new tasks in both few-and zero-shot settings @xcite , suggests that llms * equal contribution could become more socially capable by instruction tuning them on a wide variety of social nlp tasks. here, we introduce socialite-llama, a llama2 7b-based llm @xcite instruction tuned on a suite of social scientific classification tasks spanning 5 broad categories for which we hand-craft instructions. we evaluate its performance on both zero-and few-shot settings on seen and related social tasks, demonstrating that our model significantly outperforms prior open models. our results support prior research highlighting the effectiveness of instruction tuning when applying llms to a new domain. social and psychological factors have been shown important and beneficial to model in past interdisciplinary nlp studies @xcite @xcite . modeling human factors and social context can not only improve performance on primarily non-social nlp tasks @xcite @xcite , but can also prove to be beneficial for a number of psychological and social scientific tasks @xcite . however, a major limitation of prior models has been that they have been task-specific and do not generalize well to new tasks. many such models also are trained from scratch and typically need an extensive amount of human-annotated training data. with the development of llms that exhibit the capability to learn from instructions, we posit that this new capability can be leveraged to address these two major limitations. our contributions include: (1) we develop and systematically evaluate socialite-llama-an instruction-tuned language model for social science tasks -across 20 seen and 6 related social scientific tasks, (2) we show socialite-llama consistently improved over llama2 in all seen tasks and the improvement generalized to 5 of 6 related tasks; in fact, it matched the performance of a state-of-the-art multi-task tuned deberta on a majority of seen tasks, (3) we suggest that the benefits of few-shot examples (over zero-shot) become negligible on tasks seen during instruction tuning as opposed to related tasks where fewshot still provided a benefit, and (4) we release socialite-llama 1 as well as its instructions corpora, socialiteinstructions 2 , as open-source resources for the community. ## related work language is inherently social -the underlying meaning is constructed through social interactions @xcite @xcite . understanding communication requires reasoning about the social implications drawn from that message @xcite . prior work has sought to build language models for social scientific tasks, which we discuss next. lms for social scientific nlp social science nlp models are usually built by fine-tuning @xcite @xcite for specific tasks or pretraining language models on the the domain of language that captures the social factors. bertweet @xcite , the first public large-scale pre-trained language model on english tweets, was trained using the bert architecture with the roberta pre-training procedure @xcite for this purpose. @xcite built bernice, a multilingual model for social science nlp tasks using the roberta architecture and pretrained from scratch on 2.5 billion tweets. it outperforms a variety of models adapted to social media data as well as strong multilingual baselines. despite the strength of such existing task-specific models, there exists no base instruction-following llm tailored to the domain of social science tasks. instruction-tuning instruction tuning, in general, refers to the practice of finetuning pre-trained language models to better understand and respond to a wide variety of human requests that are expressed in natural language @xcite . the success of instruction tuning requires two key components: 1) a powerful pre-trained language model like llama2 @xcite , and 2) an instruction dataset that is diverse and representative enough to adapt the llm to potential downstream usage. however, 1 model available on huggingface models 2 dataset available on huggingface datasets existing instruction-tuning datasets @xcite @xcite are general-purpose and do not contain a significant amount of social science tasks. consequently, models built using this data have limited usability for social scientific tasks @xcite . to address this gap, we curate an instruction-tuning dataset aimed at modeling social scientific knowledge and use it to train our model. ## datasets and experiments we use a diverse collection of english social science nlp tasks to create socialiteinstruc-tions by (re-)framing each task into a binary or multi-class classification problem. for each task, we hand-craft instructions for the model to follow. these instructions were composed with task descriptive which included examples in some cases (e.g. biasedimplication from ## results we analyze the effect of instruction tuning llama2 on 20 seen and 6 related social scientific tasks and report macro f1 scores in instruction tuning improves social understanding. we show that socialite-llama outper-particularly, performance of our model is better for all tasks that are proximate to personal factors or how one feels, such as the affective dimensions, sentiment, emotion and empathy/distress. we also see consistent improvements over llama2 on offensiveness related tasks. despite the tremendous general-purpose capabilities of llama2, it appears to lack social understanding from text, and can be improved using instruction tuning on social scientific tasks. instruction tuning helps generalization. in related social science tasks, which are composed of new instructions and language samples, we find that socialite-llama is better on 4 out of 6 under few-shot setting and 5 out of 6 overall. these 6 related tasks were on a varying degree of relatedness to the ones in the seen tasks. socialite-llama shows significant improvement in performance over llama2 on the related social science tasks. through instruction tuning, our model generalizes social aspects of language to perform well on new related social tasks and task categories. to assess the generalization emerging from the commonalities present between the task, we applied the instructions from the seen tasks' set on a related task and compared its performance with the related tasks' instruction (which was unseen to socialite-llama). a non-author nlp expert suggested 4 seen tasks (offensive, sexist, in-tenttooffend and biasedimplication) that they thought were very similar to the hatespeech task. of those, instructions from offensive and intenttooffend gave improvement over zeroshot hatespeech performance, while the other two led to worse performance. this suggests it's possible to do better on zero-shot by using a related task prompt from instruction tuning but it can also hurt performance. ## conclusion we introduce socialite-llama, an instruction tuned llama2 7b for social science applications. socialite-llama is trained using socialite-instructions, a diverse collection of 20 social scientific tasks and tested on those plus an additional 6 related social tasks. it consistently performs better than llama2 on all the seen tasks and all but 1 related task, demonstrating its strong generalization abilities. instruction tuning on diverse social scientific tasks helps our model achieve state-of-the-art overall. our results provide valuable insights to improve the social understanding of llms.