Do LLM s Understand Social Knowledge? Evaluating the Sociability of Large Language Models with S oc KET Benchmark (2023)

Abstract: Large language models (LLMs) have been shown to perform well at a variety of syntactic, discourse, and reasoning tasks. While LLMs are increasingly deployed in many forms including conversational agents that interact with humans, we lack a grounded benchmark to measure how well LLMs understand social language. Here, we introduce a new theory-driven benchmark, SocKET, that contains 58 NLP tasks testing social knowledge which we group into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. In tests on the benchmark, we demonstrate that current models attain only moderate performance but reveal significant potential for task transfer among different types and categories of tasks, which were predicted from theory. Through zero-shot evaluations, we show that pretrained models already possess some innate but limited capabilities of social language understanding and training on one category of tasks can improve zero-shot testing on others. Our benchmark provides a systematic way to analyze model performance on an important dimension of language and points to clear room for improvement to build more socially-aware LLMs. The resources are released at https://github.com/minjechoi/SOCKET.

## introduction interpersonal communication is more than just what is said. understanding communication requires reasoning not only about the content of a message but also the social implications drawn from that message @xcite . as nlp systems, particularly large language models (llms), are increasingly used in interpersonal settings, these models' abilities to understand social knowledge become critical. however, despite the recognized need for social knowledge (hovy and yang, * equal contribution 2021), the nlp field has limited abilities to test it. here, we introduce socket, a new benchmark for evaluating social knowledge. evaluating nlp systems has remained a key component for benchmarking the field's progress. indeed, the rapid replacement of traditional models by llm-based approaches was strongly motivated by substantial gains by llms on a variety of comprehensive natural language understanding (nlu) benchmarks like superglue @xcite and natural questions @xcite . however, despite the fundamental social aspect of language, comprehensive benchmarks of social language remain absent. instead, existing computational studies of social language have built individual datasets and models for specific types of information like empathy @xcite , politeness @xcite , and humor @xcite . while beneficial, these semantic-level tasks omit broader social and narrative-level information @xcite and present only a narrow view of model performance. we introduce socket (social knowledge evaluation tests), a theory-grounded, systematic collection of 58 social language tasks. foot_0 socket covers five categories of social information: sentiment & emotion, trustworthiness, humor & sarcasm, offensiveness, and social factors, each motivated by specific theories. to examine models' generalizability, socket includes four task formats: classification, regression, pairwise comparison, and span identification. this construction aims at assessing not only nlp models' performances on individual tasks but their ability to perform multiple task types and to productively benefit from related tasks and task categories during learning. our study offers the following three contribu-tions to the research community. (1) we motivate a theoretically-grounded organization of social tasks ( §2) and subsequently introduce a new easy-to-use benchmark, socket, that systematically organizes 58 tasks ( §3). (2) we benchmark multiple current llm approaches to multitask nlu via standard supervised training and zero-shot llms ( §4). across all tests, our results show that baseline llms perform moderately, at best, but offer promising signs of being able to leverage task correlations. (3) we test the abilities of models to make use of cross-task transfer ( §5) showing multi-task training on strongly correlated tasks can maintain or even improve performance in specific tasks, but doing so on weakly correlated tasks can hurt the overall performance of llms ( §6). we release our framework code and prepackaged datasets at https://github.com/ minjechoi/socket and https://huggingface . ## social information in natural language processing language is inherently social, as meaning is constructed through social interactions @xcite . a substantial body of research in linguistic theory and communication studies have examined how social knowledge is communicated via language understanding. theories of language grounded in interaction and communication systems such as systemic functional linguistics (sfl) by @xcite assert that the function and appropriacy of language in a given context is the key to our understanding of language and its use @xcite @xcite . we use these insights to probe linguistic models for their ability to capture social information, which we define as information conveyed through text about broader metatextual function and contextual appropriacy of the utterances in conversation. ## the socket benchmark here, we describe the steps taken to curate socket as robust benchmark for identifying social information embedded in language in interpersonal communication contexts. ## benchmarks on the social knowledge capabilities of llms we first train and evaluate several commonly used multitask llms on our datasets to obtain benchmark results, which provide a first glimpse of how good llms are at learning social knowledge tasks. experiment details are described in appendix §b. ## do we see cross-task transfer of social knowledge? in this section, we examine the relations and dependencies between tasks using the predictions of llms trained on different tasks and test for dependencies between tasks that are predicted by theory. quantifying task dependency we quantify the dependency between two tasks as follows. we finetune a pretrained llm on task t a to obtain a model m a , which is used to make predictions on the test set of another task t b . the correlation between the predicted values from model m a and the true labels of the test set of t b is considered as the task dependency that t a has on t b . we report the absolute correlation value, as negatively correlated tasks are still informative. we describe how the correlations are obtained across different task types in the appendix ( §b.6). span identification tasks are omitted from this analysis, resulting in 55 × 55 scores. we also measure the pairwise correlation between models m a and m b as well as task dependency to gain an additional perspective of task similarity. details for the model correlation can be found in appendix §b.6 and figure 7 . the task dependencies for all task pairs, shown in figure 2 , reveal salient block structures within the category, foot_3 especially for the offensiveness, sentiment & emotion, and social factors categories, suggesting the existence of shared knowledge within our thematically grouped tasks. these correlations align with existing findings from interpersonal pragmatics on the relationships between social knowledge. for instance, increased selfdisclosure or pain-related interactions are known to promote both intimacy (questionintimacy) and empathy (empathy) @xcite , two elements within the social factors category, while the usage of emojis (tweet_emoji) as effective symbols are indicative of emotional states such as valence (emobank#_valence) and arousal (emobank#_arousal) @xcite , which belong to the sentiment & emotion category. the offensiveness category shows mixed results in comparison with @xcite , whose results show that hate speech datasets are often overfit and do not generalize well to other similar datasets . figures 2 & 6 , however, show that of the seven datasets included in socket, five of them included at least one task which showed comparable correlations when tested both within and out of domain. indeed, persondirectedabuse, a task labeled for offensive language specifically directed towards an individual, is actually predicted better by models fine-tuned on jigsaw# tasks than it was on its own. interestingly, correlations are scarce within the humor & sarcasm, and trustworthiness categories. this is consistent with findings from @xcite which show that models without exposure to linguistic forms lack the requisite social information to perform well on non-literal pragmatic phenomena such as humor and deceit. another notable individual task is humor_rating from the humor & sarcasm dataset, which performs well as both the fine-tuning and predicted task alongside a number of tasks from the emotion & sentiment category-particularly discretized emotion tasks, as well as hateoffensive in the offensiveness category-which labels comments as either "hateful," "offensive," or neither. while relationships between offensiveness and humor have been theorized as early as @xcite and sentiment recognition has been shown to bolster offensive language detection @xcite , relatively little has been said regarding connections between the three categories and thus, this result presents an opportunity for further research. we observe that politeness shows strong transfer with many of the offensive and hate speech detection tasks in the socket benchmark. in particular, those tasks with high correlation within the offensive category are highly correlated in predicting the politeness classification task. this finding is supported by literature showing that impoliteness can fall under the umbrella of offensive language (b ączkowska, 2021) and, although key differences exist in the pragmatics of the two, the constructs are closely related @xcite . interestingly, regression tasks (from the hahackathon, emobank, and empathy datasets) in general have strong correlations with several other tasks. this trend suggests that tasks labeled with continuous variables may have more expressive power compared to ordinal or nominal categorization, and thus have a higher potential for stronger task dependencies. however, the magnitude of the correlation may be influenced by the relative value distributions of different correlation methods. this finding calls for a need for more datasets with continuous labels, which requires more effort but allows models to capture more fine-grained concepts of social knowledge. ## can multi-task training improve social knowledge? our findings reveal significant task transfer, both within and across task categories, which hints at shared knowledge among tasks. linguistics studies of social language also note the interrelated perceptions of different dimensions such as humor and offensiveness @xcite @xcite . we now examine whether llms can learn a more robust sense of social knowledge by training on multiple tasks. ## conclusion people increasingly interact with llms in natural conversation. to what degree are these models able to pick up on the social cues? to help answer this question, we introduce socket, an nlp benchmark to evaluate how well models perform at learning and recognizing concepts of social knowledge. we provide benchmark results using several popular models and provide case studies of studying the inherent social capabilities of llms in a zero-shot setting. surprisingly, llms perform moderately at best, with even large llms (>10b parameters) varying widely in their abilities. additionally, we show that there exist significant task dependencies both within and across task categories, and that multi-task training on task categories can affect model performance. our work contributes to the broader nlp community by fostering future efforts toward building and evaluating more socially responsible and coherent llms.