E ti C or: Corpus for Analyzing LLM s for Etiquettes (2023)

Abstract: Etiquettes are an essential ingredient of day-to-day interactions among people. Moreover, etiquettes are region-specific, and etiquettes in one region might contradict those in other regions. In this paper, we propose EtiCor, an Etiquettes Corpus, having texts about social norms from five different regions across the globe. The corpus provides a test bed for evaluating LLMs for knowledge and understanding of region-specific etiquettes. Additionally, we propose the task of Etiquette Sensitivity. We experiment with state-of-the-art LLMs (Delphi, Falcon40B, and GPT-3.5). Initial results indicate that LLMs, mostly fail to understand etiquettes from regions from non-Western world.

## introduction etiquettes define a system of rules and conventions that regulate social and professional behavior. etiquettes are an integral part of every culture and society. consequently, etiquettes carry significant cultural and regional implications in every region of the world. while some of the social norms 1 are common across many cultures, every region has some society-specific norms that may be contradictory to norms in other societies. when a person is visiting another region/culture, they need to be aware of social norms in that region to avoid undesirable situations that may hamper the business and sometimes lead to conflicts. in recent times, generative models and large language models (llms) have become part of many digital technologies (such as web search and personal digital assistants (pda)). in the past, people have referred to various books written to teach and explain etiquettes for a region @xcite @xcite @xcite . however, given the advances in ## related work in recent years, there has been works in the nlp and ml communities to develop algorithms to understand human ethics and social commonsense @xcite @xcite . we describe some of the major works in here. to develop an ethically sound ml model @xcite have developed the delphi system. the model is trained to perform three different tasks: a) classify (binary: yes/no) if a given social situation is as per social norms, b) open text judgment: whether the model agrees with the action taken in the particular situation or not c) relative selection: given a context and two actions, the task is to select the action that is more appropriate. the authors create a new dataset, commonsense norm bank, that incorporates existing datasets such as social chemistry @xcite , scruples @xcite , ethics @xcite , commonsense morality @xcite , moral stories @xcite , and social bias inference corpus @xcite . delphi uses a pre-trained language model 3 eticor: corpus for etiquettes we create eticor (etiquettes corpus), a corpus of regions-specific etiquettes. in this work, we use a standard dictionary definition of etiquette and define it as a set of social norms/conventions or rules that tell how to behave in a particular social situation. etiquettes are region-specific, dictate social and professional behavior, and are subjective. in the following text, we sometimes refer to each etiquette as a social norm or norm for short. the corpus contains specific etiquettes (as english sentences) relevant to a particular region, chosen from relevant sources to ensure authenticity. in eticor, we cover five regions across the globe: east asia, india, the middle east and africa, north america and europe, and latin america. eticor creation: to avoid conflicting information, we created the etiquettes corpus by collecting texts from authentic and publicly available sources (e.g., government-aided websites). the sources of information included websites providing information about regional etiquettes, tour guide points and pamphlets, etiquette information channels, and tweets and magazines on etiquettes (details about sources on github repo). the collected (and scrapped) data was noisy and sometimes repetitive, so it was pre-processed. we manually removed texts that described the same norm; sentences were filtered if the length was less than 4 tokens, and large paragraphs were summarized. the pre-processed data were manually labeled. is an act that is frowned upon by people of that culture. about 20% of the norms were self-labeled as these started with words like "do" and "do not." the remaining norms were manually labeled after careful examination. app. ## etiquette sensitivity task definition: existing llms are trained on huge corpora of texts, which has been shown to result in an improved overall general understanding of language. however, we are also interested in testing if the llms understand region-specific societal etiquettes. eticor serves this purpose. we propose the task of etiquette sensitivity to test our hypothesis. given a statement about etiquette, the task is to predict whether the statement is appropriate for a region. experiments: we evaluated various llms in a zero-shot setting for the task. we experimented with delphi (a pre-trained language model with 11b parameters) @xcite , falcon-40b (an open-source llm with 40b parameters) (h t t p s : / / h u g g i n g f a c e . c o / b l o g / f a l c on) and gpt-3.5 turbo (a closed llm with 175b 2 https://tinyurl.com/7ayaxdj2 parameters) @xcite . we analyzed the overall and etiquette type-wise performance of each of the models. we measured the model performance using standard accuracy and the f1-score metric. we also calculated overall scores by taking a weighted sum of the scores of each of the regions, where the weight for each region is the normalized count of number of instances for that region (more details in app. b). each of the models was queried for etiquette appropriateness; a majority of times, a model replied in binary yes or no answer; in some instances, it provided an explanation that was manually mapped to labels +1 or -1. however, in some cases, the model did not provide an answer or gave an unclear response that could not be mapped to either of the labels. such samples were not included in the evaluation and are referred to as abstentions. for checking etiquette sensitivityof falcon-40b and gpt-3.5 models, we prompted each of the models to provide a single word (yes or no) answer. the prompt used was: "in a single word, say whether this is a good etiquette or socially acceptable norm @xmath0:<etiquette-statement>." results and analysis: fine-tuned models: we also experimented with fine-tuned models for the task of etiquette sensitivity. in particular, we tried the bert @xcite model. we fine-tuned bert model on eticor (train-val-test @xmath1:10:20 for each fold, adam @xmath2.001, @xmath3. a separate model was trained for each region. the model was tuned to predict whether a given input is considered an appropriate etiquette in a particular region. the class imbalance was overcome by generating negative values of positive labels @xcite and elimination of excess positive class by merging smaller sentences together. furthermore, we did 5-fold cross-validation for each region as initial experiments showed variance in bert's performance across different runs. we also ensured the proper distribution of each type of etiquette in all the folds so that any form of bias could be avoided easily. overall results are calculated by a weighted sum of the mean scores for each region. we also combined the data from all the regions and performed 5-fold cross validation using bert. the bert model has fairly high performance for all regions, pointing out that region-specific training should be done when dealing with subjective and culture-specific norms. ## conclusion in this paper, we presented eticor, a corpus of etiquettes covering major world regions. we further evaluated the performance of llms on the task of etiquette sensitivity and the results indicate a significant gap in knowledge and understanding of llms. in the future, we plan to develop regionspecific adapters and integrate them into an llm via a mixture of experts @xcite .