CP op QA : Ranking Cultural Concept Popularity by LLM s (2024)

Abstract: Many recent studies examining the knowledge capacity of large language models (LLM) have focused on knowledge explicitly learned from the pretraining data or implicitly inferable from similar contexts. However, the extent to which an LLM effectively captures corpus-level statistical trends of concepts for reasoning, especially long-tail ones, is largely underexplored. In this study, we introduce a novel few-shot question-answering task (CPopQA) that examines LLMs’ statistical ranking abilities for long-tail cultural concepts (e.g., holidays), particularly focusing on these concepts’ popularity in the United States and the United Kingdom, respectively. We curate a dataset of 457 holidays across 58 countries, generating a total of 9,000 QA testing pairs. Experiments on four strong LLMs show that open-sourced LLMs still lag way behind close LLM API (e.g., GPT-3.5) in statistical ranking of cultural concepts. Notably, GPT-3.5 exhibited its potential to identify geo-cultural proximity across continents.

## introduction large language models (llms) have shown their potential to capture multiple facets of the world, benefiting a variety of downstream applications such as constructing knowledge bases with reduced reliance on human intervention @xcite . despite that, the capacity of knowledge embedded in llms is still an open question, causing uncertainty in practical model deployment. to address this concern, researchers have been actively examining llms on various knowledge-intensive tasks, from word-or sentencelevel linguistic phenomena @xcite ) to language's statistical tendencies @xcite @xcite , and from commonsense @xcite , factual knowledge @xcite to basic science @xcite @xcite . given the generative nature of llms, these tasks can be easily conducted in a question-answering (qa) format. early studies in this thread emphasize llms' memory capacity and discover that llms possess a remarkable ability to embed the aforementioned knowledge. recently, advances in prompting like chain-of-thought @xcite and self-reflection @xcite have enabled llms to elicit complex multihop reasoning under zero-and/or few-shot learning manners. beyond common knowledge, the latest studies further explore llms' ability to process long-tail knowledge @xcite , particularly centering geo-diverse knowledge in sociocultural contexts @xcite @xcite . despite insightful findings, existing examinations largely focus on the capacity of llms to grasp knowledge explicitly learned from the training data or implicitly inferable from similar contexts. there is scant research exploring the capacity of llms to capture the broad statistical patterns of concepts within extensive datasets for in-depth comparisons, especially long-tail concepts spanning significantly diverse sociocultural contexts. this alternative perspective focuses on models' potential to embed macro-level phenomena derived from widely scattered knowledge points in the training corpus, which can broaden llms' benefits as exploratory tools in support of corpus-centered computational analysis @xcite , such as helping digital humanists and social scientists to gain new insights into historical, cultural, and social problems @xcite . in this study, we attempt to explore the statistical ranking ability of llms, with a specific focus on a research question: can large language models compare cultural concepts, especially longtail ones, regarding their popularity? to examine this question, we design a ranking-based statistical qa task that compares cultural concept popularity across countries (called cpopqa). this new task expands the study of llms' cultural awareness beyond existing studies on llms' sensitivity to crosscultural disparities @xcite @xcite . instead, we delve into the models' ability to encode broader cross-cultural social visibility patterns. this exploration could illuminate the potential of llms in tracing the dissemination of cultural concepts across geopolitical regions throughout history. to support this study, we curate a benchmark dataset of 9,000 qa testing pairs, covering 457 holidays across 58 countries. note that, our dataset construction is flexible and scalable, allowing for the easy generation of diverse testing instances. experiments on four popular llms show that close llm api (e.g., gpt-3.5) outperforms open-sourced llms in ranking holidays based on their statistical tendency. both close and open-sourced models struggle with ranking country-specific holidays in low-population countries compared with those in high-population countries. notably, gpt-3.5 showed a potential to identify geo-cultural proximity across continents. ## cpopqa in this section, we introduce our cpopqa by describing the task formulation, the process of dataset construction, and a prompt-based llm approach. tasks considering the geo-association between holidays and countries, we propose two levels of ranking-based statistical qa tasks: (1) fine-level holiday ranking (see below) and (2) coarse-level country ranking (see appendix f). specifically, we referred to the population statistics and the number of holidays on each country's wiki page, and collected holidays from the top five and bottom five countries regarding population statistics in each continent. ## experimental setting llms and baselines we chose 4 popular llms for evaluation. the first llm is gpt-3.5 (i.e., text-davinci-003, 175b parameters). through finetuning gpt-3 by reinforcement learning from human preferences @xcite , gpt-3.5 shows a higher quality in handling complex instructions compared to prior gpt-based models. we next chose llama, with 7b (llama-7b) and 13b (llama-13b) parameters, pre-trained on the english-dominated corpora covering diverse domains @xcite . the final model is bloom-7b1, a multilingual llm with ∼7b parameters @xcite . we selected this bloom variant because of its comparable model size with llama-7b. we employed 3 baselines, including random guess and statistical simulation by google trends and wikipedia article length, respectively (see details in appendix e). metrics we used three evaluation metrics, including accuracy (acc.) measures the degree of the exact match; precision@1 (p@1) calculates the precision of the first ranked item; average difference (diff. = 1 n n @xmath0, where ρ j is the spearman correlation coefficient between the model prediction and the ground-truth ranking on the j-th qa example. ## results and analysis can llms elicit long-tail cultural statistics and rankings? what ranking-based factors challenge the prediction? looking into the ranking setting, we find that llms show a noticeable drop in p@1 (∼5%-10%)and acc (∼20%-30%) when adding ranked items (see in the us, appendix h on results in the uk), we find that holidays ordered at two ends are usually easier to be predicted than those in between. items ordered at the third and fourth positions are more prone to confuse llms than others. ## conclusion we introduce a novel qa task, cpopqa, to assess llms in ranking holiday-centered cultural concepts based on their popularity in the us and uk. our results show that llama and gpt-3.5 tend to capture implicit statistical tendencies of holiday popularity. comparatively, gpt-3.5 displays superior ranking abilities. the model predictions are highly sensitive to the number of ranked items, and they encounter more difficulties in capturing statistical trends for under-represented geo-cultural concepts than well-represented ones. further exploring the optimal llm (gpt-3.5), we observe its potential to show geo-cultural proximity across continents. by examining llms' statistical ranking ability on long-tail cultural knowledge, this preliminary work benefits incentivizing future work on sociocultural tendency exploration by llms.