N orm D ial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violation (2023)

Abstract: Social norms fundamentally shape interpersonal communication. We present NormDial, a high-quality dyadic dialogue dataset with turn-by-turn annotations of social norm adherences and violations for Chinese and American cultures. Introducing the task of social norm observance detection, our dataset is synthetically generated in both Chinese and English using a human-in-the-loop pipeline by prompting large language models with a small collection of expert-annotated social norms. We show that our generated dialogues are of high quality through human evaluation and further evaluate the performance of existing large language models on this task. Our findings point towards new directions for understanding the nuances of social norms as they manifest in conversational contexts that span across languages and cultures.

## introduction social norms-implicitly learned notions of acceptable behavior-both develop from and guide our everyday interactions @xcite . as with the value systems that underlie these notions, the acceptability and deemed typicality of behaviors varies across cultures @xcite . for example, due to a strong emphasis on individualism, open and direct expression of opinions and disagreement is often encouraged and valued in western cultures @xcite , while such acts may often be viewed as disruptive to social order in eastern asian cultures that value collectivism @xcite . understanding these cultural nuances is key to empower computational systems to reason across cultural contexts @xcite . we introduce normdial, a bilingual synthetically generated dyadic dialogue dataset of social norms as they appear within the context of conversational interactions. gathering realistic data at scale in this domain presents a challenging and potentially cost-prohibitive task, particularly in the 哪里哪里， 谢谢夸奖，其实也没什么，只是按照食 谱做的 (not at all, thanks for the compliment, it's nothing, just follow the recipe) 李叔叔， 这道菜真好吃，您的厨 艺真厉害 (uncle li, this dish is delicious and your cooking skills are amazing) family members uncle li 真的吗？我们都觉得这已经很好吃啦 (really? we all think this is already delicious) 我还有很多需要学 习的地方，希望下次能再 给大家带来更 好的菜品 (i still have a lot to learn, and i hope to bring you better dishes next time) chinese norm: when responding to each others' compliment, one can respond with euphemism to downplay their achievement. ## the normdial dataset normdial is a human-in-the-loop synthetically generated bilingual (chinese & english) dyadic dialogue dataset for studying social norms as they appear in different conversational contexts. dialogue turns are further labeled on whether they adhere to or violate a given social norm with textual explanations. our human-ai collaboration framework for creating normdial is shown in figure 2 . 2 ). the linguistic data consortium (ldc) (linguistic data consortium, 2022) taxonomizes 10 categorizations of social norms in its guidelines-apology, compliment, condolence, criticism, greeting, leave, persuasion, request, response to compliment, giving thanks-and provides a detailed set of associated norms (5 for each category) for chinese culture. examples of verbal evidence of adherence to a norm in a conversational context, alongside the relationship data of each hypothetical interlocutor, are provided as details for norm definitions. ## evaluation we conduct a human evaluation on the quality of dialogues and the correctness of model-predicted synthetic dialogue quality. to determine the quality of our chinese and american synthetic dialogues, we perform a human evaluation against two types of baselines: (1) specifically curated domainrelevant dialogues or pre-existing and naturally occurring dialogue datasets, and (2) human-written dialogues specifically for our task. for the former, we compare our chinese dialogues against a collection of 200 dialogues randomly sampled from the linguistic data consortium (ldc), which contains 413 chinese conversations with expert-annotated turn-by-turn chinese norm observations from video transcripts. these conversations contain dialogues and chitchat from sources such as tv shows, vlogs, and other types of videos from bilibili, a chinese video-sharing platform. for english dialogues, as no existing domain-comparable dialogue is available, we compare against a set of 200 realistic, human-written dialogues reflecting daily communication that covers various topics about daily life, dailydialog @xcite ; a comparison that has been used for previous evaluations of synthetic dialogue quality @xcite . for the latter, to evaluate our dialogues against human-written counterparts specific to our task, we asked three native speakers of chinese and english to creatively write a set of 20 dialogues for each language, based on a sample of 20 norms and situations, which we selected from each of our 10 norm categories by randomly selecting an adherence and violation (norm, situation) pair from each category. we ask sets of 3 annotators to rate each conversation, formatted consistently, on their (1) naturalness, or how natural each dialogue sounded, (2) nativeness, if they thought the dialogue came from a native speaker, (3) coherence, and (4) interestingness, each on a 5-point likert scale, with final scores for each aspect being the average of the scores received. on a separate task, we ask annotators to rate if synthetic dialogues were faithful and on-topic to their provided social norm, i.e. does the main topic of the dialogue match the provided social norm, yes/no?, of which final labels are obtained via majority-voting. a comparison of quality evaluation scores is shown in for baseline (1), annotators rate normdial dialogues higher in almost all dialogue quality aspects than their pre-existing curated and naturally occurring dialogue baseline counterparts, with only dailydialog outperforming normdial in interestingness and ldc outperforming our synthetic dialogues in nativeness; synthetic dialogues were rated higher to a statistically significant degree in coherence and interestingness for chinese and nativeness and coherence for the american side. as for benchmark (2), normdial dialogues were found to be of higher quality than their specificially tasked human written counterparts for english and lower for chinese, in line with language performance differences for chatgpt. despite this performance difference in chinese dialogues, it took the average annotator more than an hour to finish writing 20 dialogues; as this is a highly creative task, tasking annotators in our task can prove to be a challenge in scalability, given emerging evidence of annotator fatigue @xcite , especially for creative tasks. 2 on the other hand, taking the majority vote of dialogue on-topic labels from annotators showed that 92.5% and 86.5% of dialogues for chinese and english, respectively, were faithful to their prompted (norm, situation) pairs. automatic turn-level annotation. as our automatic dialogue-turn norm adherence/violation labeling via prompting is separate from dialogue generation, a natural question arises as to how well existing llms are able to perform this task, i.e., how well can llms accurately detect if a given social norm is adhered to or violated for a conversational round? here, collecting a sample of 200 dialogues for each language, two annotators manually validated the correctness of chatgpt labeled dialogue rounds to check for correctness, resolv-2 https://www.reddit.com/r/prolificac/comments/17btpjs/ ing disagreements via discussion to produce a set of final gold standard labels for 1281 chinese and 1503 english dialogue rounds. shown in ## conclusion we presented normdial, a synthetic, validated, high-quality dyadic dialogue dataset with turn-byturn annotations of social norm adherences and violations for chinese and american cultures in chinese and english. our evaluation of synthetic dialogue quality reveals that our dataset is comparable to and/or exceeds the quality of naturally occurring and domain-specific dialogue datasets. furthermore, our analysis of llm predictions of norm observance reveals areas for existing models to improve in this domain. our resource points towards new directions for understanding the nuances of social norms as they manifest in conversational contexts that span across languages and cultures.