D e SIQ : Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding (2023)

Abstract: Social intelligence is essential for understanding and reasoning about human expressions, intents and interactions. One representative benchmark for its study is Social Intelligence Queries (Social-IQ), a dataset of multiple-choice questions on videos of complex social interactions. We define a comprehensive methodology to study the soundness of Social-IQ, as the soundness of such benchmark datasets is crucial to the investigation of the underlying research problem. We define a comprehensive methodology to study the soundness of Social-IQ, as the soundness of such benchmark datasets is crucial to the investigation of the underlying research problem. Our analysis reveals that Social-IQ contains substantial biases, which can be exploited by a moderately strong language model to learn spurious correlations to achieve perfect performance without being given the context or even the question. We introduce DeSIQ, a new challenging dataset, constructed by applying simple perturbations to Social-IQ. Our empirical analysis shows De-SIQ significantly reduces the biases in the original Social-IQ dataset. Furthermore, we examine and shed light on the effect of model size, model style, learning settings, commonsense knowledge, and multi-modality on the new benchmark performance. Our new dataset, observations and findings open up important research questions for the study of social intelligence.

## introduction social intelligence is a long-standing research area in social science and psychology @xcite @xcite . it is the capacity to understand and navigate complex social situations. social intelligence is more than the perception of objects and human actions, as it requires a deeper understanding of human intents and interactions behind these actions or words. the study of social intelligence is an emerging area in both the nlp and computer vision communities. one representative work, social-iq (social intelligence queries) @xcite , is a benchmark dataset measuring social intelligence of current ai systems. it is a multiple choice question answering dataset with multi-modal inputs, including questions, answer options, videos, etc; see an example in figure 1 . although social-iq contains rigorously human-annotated data, surprisingly, we find even small models like t5-small @xcite could easily achieve 100% answer option accuracy ( the perfect performance of such an underpowered model prompted us to conduct further investigation to identify its source. through employing different models and perturbation methods on the answer options, we identify significant biases in the social-iq dataset, in which the representations of correct and incorrect options are easily separable, regardless of the questions (figure 3 ). thus, the models are able to exploit such a shortcut @xcite to answer questions with a high accuracy, without necessarily understanding social intelligence. to debias the social-iq dataset, we propose a simple yet effective debiasing approach and present a new unbiased benchmark desiq, by substituting all the incorrect answer options with correct answer options from randomly selected other questions. we establish a performance baseline on desiq with t5-small and delphi @xcite , a language model pretrained with commonsense and social norms knowledge. given answer options only or question-answers, both t5-small and delphi obtain close to random accuracy. by making use of multi-modal inputs, both t5-small and delphi achieve an accuracy of up to 77%. these results demonstrate that desiq is unbiased and challenging. interestingly, both models also outperform gpt-3 and chatgpt, further indicating the challenging nature of the social intelligence understanding problem. our contributions are: ## desiq: debiased social-iq in this section, we first describe our approach to debias social-iq . we then study the effectiveness of our debiasing approach and the resultant desiq datasets, by comparing the performance of both lstm and t5-small on desiq in different settings. ## setting baseline performance on desiq for our more challenging desiq benchmark, we introduce a new baseline model to better handle multi-modal inputs. its architecture is shown in figure 4 . compared with the model in figure 2 , we add three more projection layers (three yellow mlps) to map the original feature representations into the same dimensions. we then concatenate all the resulting representations as the inputs to a backbone mpm. for desiq d -2.0 containing raw data, we employ vision transformer (vit) @xcite and wav2vec 2.0 @xcite to obtain the video and audio representations respectively. we note again that raw video and audio files are not available for v1, thus we develop the above architecture to uniformly handle both datasets, and leave how to best use multi-modal inputs in desiq-2.0 for future work. as social intelligence usually requires commonsense knowledge, we posit that injecting commonsense knowledge into the backbone language model in our architecture would improve the model's performance. therefore, inspired by @xcite , we distill commonsense social knowledge from the following datasets into t5small: social chemistry 101 @xcite , ethics @xcite and moral stories @xcite . specifically, we pretrain t5-small on these corpora and then finetune it on the downstream social-iq and desiq datasets. we call this variant t5-small delphi . ## conclusion social intelligence is an essential ingredient for effective human-computer communications. in this paper, we analyze social-iq, a multiple-choice question answering benchmark dataset for social intelligence. our empirical analysis reveal the severe biases present in social-iq, which can be easily exploited by modestly-sized language models such as t5-small to achieve perfect accuracy on its development set. we construct the desiq benchmark by applying simple perturbation-based techniques on social-iq and show that the desiq vastly reduce the biases in social-iq. moreover, we propose a new model architecture on desiq and set strong performance baselines for this challenging new dataset. finally, our comprehensive analyses open up a number of important research questions for further investigation.