Should We Respect LLM s? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance (2024)

Abstract: We investigate the impact of politeness levels in prompts on the performance of large language models (LLMs). Polite language in human communications often garners more compliance and effectiveness, while rudeness can cause aversion, impacting response quality. We consider that LLMs mirror human communication traits, suggesting they align with human cultural norms. We assess the impact of politeness in prompts on LLMs across English, Chinese, and Japanese tasks. We observed that impolite prompts often result in poor performance, but overly polite language does not guarantee better outcomes. The best politeness level is different according to the language. This phenomenon suggests that LLMs not only reflect human behavior but are also influenced by language, particularly in different cultural contexts. Our findings highlight the need to factor in politeness for cross-cultural natural language processing and LLM usage.

## introduction in natural language processing, large language models (llms), such as openai's chatgpt 1 and meta's llama @xcite , have attracted widespread attention. these models have shown significant performance in many tasks, such as logical reasoning, classification, and question answering, playing a crucial role in many practical applications. the input to an llm, a prompt, is a vital starting point for the model to process information and generate appropriate responses. however, despite the continuous improvement of the capabilities of llms, their behavior and generations still need to be improved in many factors. this study explores one of the possible influencing factors: the politeness of the prompt. in human social interactions, politeness, which expresses respect to others, is basic etiquette, which is reflected 1 https://openai.com/product in our language and behavior. however, politeness and respect may have different definitions and manifestations in different cultures and languages. for example, the expression and degree of respect in english, chinese, and japanese may differ significantly. this difference may make the performance of llms vary with language on the same politeness level. we hypothesize that impolite prompts may lead to a deterioration in model performance, including generations containing mistakes, stronger biases, and omission of information. in addition, we also hypothesize that the best level of politeness for performance is different across languages, which is strongly related to their cultural background. to verify these hypotheses, we design eight prompts with politeness levels ranging from high to low for english, chinese, and japanese, respectively. our experiments are conducted on three tasks: summarization, language understanding benchmarks, and stereotypical bias detection. our contributions are two-fold as follows: llms reflect human desire we observed that impolite prompts often result in poor performance, but excessive flattery is not necessarily welcome, indicating that llms reflect the human desire to be respected to a certain extent. this finding reveals a deep connection between the behavior of llms and human social etiquette @xcite . jmmlu to evaluate llms' multitask language understanding capabilities in japanese, we create jmmlu, a japanese version of mmlu @xcite . ## jmmlu construction to build a practical llm benchmark in japanese and to use it for evaluation in this study, we constructed the japanese massive multitask language understanding benchmark (jmmlu). this involved translating mmlu and adding tasks related to japanese culture. from each of the 57 tasks of mmlu, since the mmlu questions are not ordered, we selected up to former 150 questions. then, ten translators from an english-japanese translation company machine-translated the selected questions into japanese and reviewed the translations to remove questions and tasks that were difficult to translate, irrelevant, or contradictory to japanese culture. finally, the translators revised the remaining questions to fluent japanese. meanwhile, additional tasks based on school subjects, such as civics and japanese history, were added to supplement the aspects that were not covered in the western culture-oriented mmlu @xcite . the questions in the additional tasks were manually created by japanese teachers from two cram schools in japan. jmmlu consists of 56 tasks. the list of the tasks and examples of removed questions are shown in appendix a. the number of questions per task ranges from 86 to 150, totaling 7,536 questions. ## experimental settings we conduct experiments on three highly concerning tasks to evaluate the performance of llms according to prompt politeness. ## conclusion our study finds that the politeness of prompts can significantly affect llm performance. this phenomenon is thought to reflect human social behavior. the study notes that using impolite prompts can result in the low performance of llms, which may lead to increased bias, incorrect answers, or refusal of answers. however, highly respectful prompts do not always lead to better results. in most conditions, moderate politeness is better, but the standard of moderation varies by languages and llms. in particular, models trained in a specific language are susceptible to the politeness of that language. this phenomenon suggests that cultural background should be considered during the development and corpus collection of llms.