E ti C or++: Towards Understanding Etiquettical Bias in LLM s (2025)

Abstract: In recent years, researchers have started analyzing the cultural sensitivity of LLMs. In this respect, Etiquettes have been an active area of research. Etiquettes are region-specific and are an essential part of the culture of a region; hence, it is imperative to make LLMs sensitive to etiquettes. However, there needs to be more resources in evaluating LLMs for their understanding and bias with regard to etiquettes. In this resource paper, we introduce EtiCor++, a corpus of etiquettes worldwide. We introduce different tasks for evaluating LLMs for knowledge about etiquettes across various regions. Further, we introduce various metrics for measuring bias in LLMs. Extensive experimentation with LLMs shows inherent bias towards certain regions.

## introduction in recent times, large language models (llms) have shown drastic improvements across almost all nlp tasks involving language understanding and generation @xcite @xcite , resulting in wide-spread adoption in real life applications such as using llm as personal digital assistants where the llm is used for querying about various kinds of information including those related to cultural aspects of human societies. consequently, the nlp research community has recently started focusing on evaluating and improving cultural understanding (and possible biases) of llms @xcite @xcite . it has resulted in the need to develop new culture-centric tasks and datasets. culture is a multi-faceted topic and has been studied in the nlp community via various proxies @xcite . one aspect of culture is etiquettes. 1 etiquettes can be generic (common across the majority of societies/regions) as well as localized (specific to a society/region). llms have been trained on almost the entire internet's data @xcite and have very likely picked up information about etiquettes in various societies. however, it remains to be evaluated if llms are able to understand intricate and subtle differences in social norms across cultures and are possibly biased towards certain cultures. since llms are increasingly being used for seeking information, potential etiquettical biases can have detrimental consequences for the user. hence, there is a need for evaluation and understanding of inherent biases. in this resource paper, we attempt to achieve this goal. in a nutshell, we make the following contributions: though etiquettes vary from region to region, there are some commonalities. we develop an algorithm (algorithm 1) for measuring the correlation between etiquettes of different regions to check for similarities and differences. with the aim to deploy nlp technologies (e.g., llms) in human societies, recent research in the nlp community has focused on ethics and culturecentric techniques and models @xcite @xcite . various works have been proposed covering social reasoning @xcite , cross cultural understanding @xcite , social dimensions @xcite , culturellm @xcite , cultural corpora @xcite @xcite , llm alignment @xcite , and inter alia. due to space constraints we provide more details in app. a. ## tasks and bias metrics we use eticor++ to check llms for cultural bias. for this, we created various tasks and metrics for measuring bias, as outlined below. etiquette sensitivity (es) task: this task is similar to the one introduced in @xcite . given an etiquette, the task is to predict whether the etiquette is acceptable or unacceptable for a region. we evaluate llms via zero-shot setting (app. d provides the prompt). this task is useful as we need the models to be sensitive to different cultures and not discriminate against any of them. the models should not deem some cultural values acceptable while others are unacceptable. es is measured using the standard metric of accuracy and f1 score. region identification (ri) task: this newly introduced task aims to test if a model can correctly identify the region corresponding to an etiquette. the model is provided with an etiquette text and asked to identify the region from a list of regions (see app. d for the prompt). we created this task by keeping the following use cases in mind. let's say that a person asked the llm to suggest a gift for their friend's wedding. however, the llm is unaware of the friend's cultural belonging. suppose the model responded with "you can gift them white flowers, as it represents purity and peace" but then you respond with "in our culture, white is an ominous color for us. please suggest a different gift." now, the model can actually guess what culture is involved here (e.g., east asian) and respond accordingly. there are many other use cases, such as asking the model to assist you in writing a speech at somebody's funeral or promotion, which can involve etiquette regarding first and last names, etc. we devise three metrics to evaluate a model. 1. preference score (ps(r)) for a region @xmath0, in, mea, la, ne} calculates how often the model prefers to select the region across all etiquettes in the corpus, i.e., n where, {i @xmath1, n is total number of etiquettes in the corpus and ri(e i ) is the answer generated by the model for the ri task query. a higher value of ps(r) than expected is indicative of a model's bias towards a region. the expected value for each region is their share in the actual data distribution (ne -26.50%, in -8.73%, ea -21.22%, la -15.22%, mea -28.30%). to estimate this deviation, we calculate (ps(r)% -d(r)), where d(r) is the percentage share of the region r ′ s data in the whole dataset. we also calculate standard deviation (σ p s(r) ) for each model ( §e.1). using this score, we look at the defaulting behavior of the llms. we are trying to quantify that whenever a model is wrong about the region of an etiquette, what region does it prefer in those cases. a high bfs(r) score indicates model bias for a region. similar to ps(r), we calculate standard deviation (σ bf s(r) ) for each model ( §e.2). ## experiments and results we experimented with 5 llms (mix of closed and open-weights models): gpt-4o (openai, 2024), gemini-1.5-flash (gemini, 2024), llama-3.1-8b-instruct (meta, 2024), gemma-2-9b-it (google, 2024) and phi-3.5-mini-instruct (microsoft, 2024). except for gpt-4o and gemini, we conducted the quantitative experiments three times (@xmath2.3 and top-@xmath3.9) to account for output variability. due to high cost of experiments, for gpt-4o and gemini, we took 200 samples for each region (1000 samples in total) for each of the experiments. note we did not experiment with the models used in eticor since these are outdated/unavailable and have in general shown to have poorer performance than the more recent models used in this paper (details in §f.2). ## conclusion in this paper, we introduce eticor++ and propose new tasks for evaluating llms for etiquettes. we also develop new measures for quantifying bias in llms. our experiments show inherent biases in llms. in the future, we plan to develop methods for mitigating etiquettical biases in llms.