Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Future (2024)

Abstract: As Natural Language Processing (NLP) systems become increasingly integrated into human social life, these technologies will need to increasingly rely on social intelligence. Although there are many valuable datasets that benchmark isolated dimensions of social intelligence, there does not yet exist any body of work to join these threads into a cohesive subfield in which researchers can quickly identify research gaps and future directions. Towards this goal, we build a Social AI Data Infrastructure, which consists of a comprehensive social AI taxonomy and a data library of 480 NLP datasets. Our infrastructure allows us to analyze existing dataset efforts, and also evaluate language models’ performance in different social intelligence aspects. Our analyses demonstrate its utility in enabling a thorough understanding of current data landscape and providing a holistic perspective on potential directions for future dataset development. We show there is a need for multifaceted datasets, increased diversity in language and culture, more long-tailed social situations, and more interactive data in future social intelligence data efforts.

## introduction as early as the 1920s, psychologists like @xcite and @xcite considered social intelligence to be a distinct branch of intelligence that underlies all successful human interpersonal relationships. many researchers now argue that social intelligence is a prerequisite of humanlike artificial intelligence @xcite @xcite @xcite . however, existing work still lacks a 1 project page figure 1 : our social intelligence data infrastructure gives a comprehensive overview and synthesis of social intelligence in nlp, with a theoretically grounded taxonomy and an nlp data library. researchers can use our infrastructure to build and organize tasks, evaluate language models and derive future insights. prior studies define social intelligence by its cognitive aspect, or the ability to understand others @xcite , while others emphasize the behavioral component by defining it as the ability to interact with other people @xcite . both dimensions are relevant but incomplete, as social intelligence is multifaceted @xcite . besides the narrow definitions, related empirical efforts, such as dataset collection and model development, also have isolated focuses and only address a single aspect of social intelligence @xcite . therefore, there exists a pressing need for a holistic and synthesized definition for social intelligence to create an organized space for existing datasets. without such organization, it is difficult to identify overar-ching research questions and emerging trends for future exploration. besides, although various social intelligence datasets have been proposed, the lack of data organization creates barriers for researchers to gain insights from previous work. in light of this, we establish social intelligence data infrastructure, which consists of a comprehensive taxonomy for social intelligence and an organized data library of 480 nlp datasets (see figure 1 ) to structure current data efforts and navigate future directions. the taxonomy ( §2) formally defines various aspects of social intelligence, to introduce standardization and comprehensiveness to the definition of social intelligence in ai systems. the data library ( §3.1) maps crawled datasets to different categories in our taxonomy, to provide structures for existing datasets. the taxonomy and data library can collectively aid researchers to identify existing dataset gaps and guide future dataset development for social intelligence. moreover, we demonstrate how the proposed social intelligence data infrastructure can be applied to gain insights into future dataset development with the following contributions: ## social ai taxonomy to introduce a standardized and comprehensive definition of social intelligence, we propose social ai taxonomy, to capture diverse dimensions identified in previous work. as shown in figure 2 , different from previous categorization which is thematic with a focus on social understanding @xcite , our taxonomy considers the social interaction component and is hierarchical with three distinct types of social intelligence based on past literature: (1) cognitive intelligence, (2) situational intelligence, and (3) behavioral intelligence. as identified in social cognitive theory (bandura, 2009), these three intelligence types mutually influence each other to shape human behaviour. prior work @xcite shows that these three intelligence types can comprehensively cover different factors and dimensions under social intelligence. now we detail each intelligence type below. ## model performance now we use our social ai data infrastructure to evaluate current llms' performance on social intelligence and gain insights on models' strengths and limitations, shedding light on aspects which future social intelligence datasets need to address. ## recommendations for the future from analyzing current data landscape ( §3.3) and evaluating llms' performance ( §4), we unveil the most challenging aspects of social intelligence that remain unaddressed by existing data resources or model capabilities. guided by insights from our results, we discuss possible future directions for dataset development below. ## conclusions we introduce social ai data infrastructure with a theoretically grounded taxonomy and a data library of 480 nlp datasets, which facilitates standardization of the social intelligence concept in ai systems and organization of previous nlp datasets. we also conduct comprehensive analysis on the data library and evaluate llms' performance, offering insights on the current data landscape and future dataset development to advance social intelligence in nlp systems. it enables curation of high-quality datasets and holistic development of social intelligence in the nlp field.