L o FTI : Localization and Factuality Transfer to I ndian Locales (2025)

Abstract: Large language models (LLMs) encode vast amounts of world knowledge acquired via training on large web-scale datasets crawled from the internet. However, the datasets used to train the LLMs typically exhibit a geographical bias towards English-speaking Western countries. This results in LLMs producing biased or hallucinated responses to queries that require answers localized to other geographical regions. In this work, we introduce a new benchmark named LoFTI (Localization and Factuality Transfer to Indian Locales) that can be used to evaluate an LLM’s contextual localization and factual text transfer capabilities. LoFTI consists of factual statements about entities in source and target locations; the source locations are spread across the globe and the target locations are all within India with varying degrees of hyperlocality (country, states, cities). The entities span a wide variety of categories. We use LoFTI to evaluate Mixtral, Llama3.3-70B, GPT-4 and two other Mixtral-based approaches well-suited to the task of localized factual transfer. We demonstrate that LoFTI is a high-quality evaluation benchmark and all the models, including GPT-4, produce skewed results across varying levels of hyperlocality.

## introduction large language models (llms) are proficient in text generation and are also extensive repositories of world knowledge, owing to their pretraining and fine-tuning on vast and diverse internet data. this suggests that llms might be effective at associating and transferring factual knowledge across geographical locations. they can generate localized text in a given target location by transferring from a reference text in a source location. however, there is no existing benchmark that helps assess this specific form of localization and fact-driven transfer. benchmarks that measure llms' ability to understand cultural concepts and their transference across geographical regions are steadily emerging in recent work @xcite . we argue that it is also important to evaluate the ability of models to transfer factual knowledge across geographical regions. figure 1 illustrates this point by showing two use-cases: 1) generating a localized response given a common question asked across locations and, 2) accurate factuality transfer from one locale to another. in this work, we introduce a new evaluation benchmark called lofti (localization and factuality transfer to indian locales): ## methodology for dataset creation common question generation. in addition to the reference and target text pairs, lofti also contains questions that capture common aspects shared by t ref and t tar . these common questions are ones a user would ask the model from a particular location for which answers should be appropriately localized. given a pair of text (t ref , t tar ), we generate these questions by identifying shared properties or descriptions of the entities mentioned in the text pairs. we use few-shot prompting on mixtral-8x7b-instruct-v0.1.q4_k_m model for common question generation, using the prompt in appendix a.3. note that common question generation is used exclusively in creating our dataset and is not evaluated as part of our experiments. these generated questions undergo human annotation for quality control. the human-validated questions are further used to assess how well llms perform at answering questions within localized contexts. ## properties of lofti dataset figure 3 shows the distribution of entities across reference and target locations. 2 ## discussion errors. two types of errors were mainly observed in the llm generations. (i) hallucinations: the llm generates fictional entities or events that do not exist in order to create a localized sentence for the target location. this was mainly observed with mixtral models for hyperlocal scores 2 and 3. (ii) lack of fine-grained knowledge: due to insufficient knowledge about specific categories and entities, the model produces sentences involving entities from closely related categories or with lower hyperlocal scores. this pattern was observed with both mixtral and gpt-4 models for regions with hyperlocal scores 2 and 3. ## conclusion this work introduces a new evaluation benchmark lofti to test the localization and factual transfer capabilities of llms. we attempt to localize factual statements from across the globe to multiple target locations within india, spanning different levels of hyperlocality. we establish various baselines (mixtral, gpt-4, etc.) and multiple benchmark tasks for the different models. we find that gpt-4 struggles with localization at higher levels of hyperlocality (i.e., when localizing to indian cities), so much so that it cannot be reliably used as an automatic evaluator. we hope lofti helps the research community in designing improved localization and factual transfer techniques.