C hrono B ias: A Benchmark for Evaluating Time-conditional Group Bias in the Time-sensitive Knowledge of Large Language Models (2025)

Abstract: In this paper, we propose \texttt{ChronoBias} , a novel benchmark for evaluating \textit{time-conditional group bias} in the \textit{time-sensitive} knowledge of large language models (LLMs).Our benchmark is constructed via a template-based semi-automated generation method, balancing the quality-quantity trade-off in existing benchmark curation approaches.For knowledge that changes over time, \textit{time-conditional group bias} exhibits varying patterns across time intervals, evident in both the best- and worst-performing groups and in the bias metric itself.In addition to \textit{parametric knowledge bias} –which influences group bias across all time intervals–we identify \textit{time-sensitivity bias} as an additional factor after a model’s knowledge cutoff, accounting for much of the variation in \textit{time-conditional group bias} over time.Since both biases are irreducible, retrieval-augmented generation (RAG) can be a promising approach, as it can address post-cutoff knowledge and better leverage pretraining knowledge that is underrepresented in the model parameters.While RAG improves both overall performance and group bias, we observe that the disparate patterns of \textit{time-conditional group bias} still persist.Therefore, through extensive experiments with various model configurations, we illustrate how accurate and fair RAG-based LLMs should behave and provide actionable guidelines toward constructing such ideal models.

## introduction despite their success on a wide range of downstream tasks, large language models (llms) often show disparate performance across factors such as demographic groups, geographical regions, or levels of popularity, undermining their trustworthy and fair deployment for diverse users @xcite @xcite @xcite . while the benchmarks proposed by the authors effectively demonstrate the presence of such biases, they do not account for the time-sensitive nature of knowledge. we define the time-sensitivity of knowledge as the rate at which it evolves over time. for questions involving time-sensitive knowledge, llms may provide correct answers only during specific periods @xcite . existing group bias evaluations, however, effectively average over all time points, thereby overlooking such temporal variability. we refer to this type of evaluation as time-marginalized group bias. meanwhile, several benchmarks show that llms struggle to answer questions involving timesensitive knowledge @xcite . specifically, they categorize knowledge into three groups based on its degree of time-sensitivitynever-changing, slow-changing, and fast-changingand show that there is a negative correlation between the degree of time-sensitivity and model performance on downstream tasks. in other words, the more time-sensitive the knowledge is, the more difficult it is for llms to generate correct answers. however, the analyses lack the notion of group fairness: when the knowledge content is the same, its degree of time-sensitivity remains similar across groups, yet llm performance may still differ. although benchmarks exist that assess llm performance with respect to both group and timesensitivity, these factors have been considered independently rather than in a joint analysis @xcite . see in addition to the challenge of addressing group fairness in the context of the time-sensitivity of knowledge, llms must adapt to slow-changing and fast-changing knowledge that are updated after their knowledge cutoffs. the knowledge cutoff of an llm is defined as the most recent date of the knowledge used for its pretraining. a number of benchmarks have been proposed to evaluate whether it is possible to continually pretrain llms, before right before right after after 0.0 0.1 0.2 0.3 0.4 0.5 0.6 group bias knowledge cutoff time-conditional time-marginalized [t 1 , t 2 ] before right before right after after marginal g max brazil g min algeria conditional g [t 1 ,t 2 ) max brazil spain england spain g [t 1 ,t 2 ) min algeria algeria korea korea (a) gpt-4o-vanilla before right before right after after 0.0 0.1 0.2 0.3 0.4 0.5 0.6 group bias knowledge cutoff time-conditional time-marginalized [t 1 , t 2 ] before right before right after after marginal g max spain g min saudi conditional g [t 1 ,t 2 ) max spain algeria england england g [t 1 ,t 2 ) a process often referred to as continual knowledge learning @xcite @xcite . in these benchmarks, the categories of new, unchanged, and updated knowledge are defined based on two consecutive time steps, and the evaluation measures whether a continual pretraining method can adapt to the new or updated knowledge while preserving the unchanged information. despite their additional training costs, most continual pretraining methods face two critical challenges: they struggle to adapt to new or updated knowledge, and they often forget unchanged information already encoded in the model parameters. as a result, they exhibit suboptimal performance on downstream tasks. as an alternative, retrievalaugmented generation (rag), which augments a query with relevant context retrieved from an up-todate knowledge base, enables llms to process new information without requiring further pretraining or fine-tuning @xcite . moreover, rag can also supplement knowledge acquired before the knowledge cutoff that was not well retained in model parameters. in seeking to construct fair and accurate ragbased llms on time-sensitive knowledge across all time intervals, the problem reduces to two central challenges: (1) identifying the properties of such ideal rag-based llms, and (2) developing methods to construct them. to mitigate these challenges, we propose a novel benchmark, chronobias, designed to support these analyses and prototyping efforts, through which we derive actionable guidelines for building such models. chronobias consists of question-answer (qa) pairs in the sports domain across geographical regions, each aligned with a gold passage. in addition, each qa pair is annotated with labels for the group (defined as geographical region), the degree of time-sensitivity, and the date of knowledge update. by leveraging chronobias, we first observe that the vanilla performance of llms exhibits varying patterns of time-conditional group bias across time intervals, defined as the discrepancy between the maximum and minimum group performance on knowledge from a given time interval (fig. 1a ). in alignment with existing research on group bias, we demonstrate that parametric knowledge bias-i.e., bias in group-specific knowledge encoded in model parameters-affects time-conditional group bias across all time intervals. in contrast, we further show that time-sensitivity bias-i.e., bias in the degree of time-sensitivity across groups-serves as an amplification factor only after the knowledge cutoff, leading to discrepancies in time-conditional group bias across time intervals. for fig. 1b , most rag baselines improve both accuracy and group bias compared to vanilla performance, but timeconditional group bias still exhibits distinct patterns across time intervals. thus, we further identify how an ideal rag should behave based on results using gold passages, and provide practical guidelines for building continuously fair and accurate rag-based llms. additional details on related work can be found in the appendix a. as for technical contributions in terms of benchmark construction, chronobias is built upon a novel, template-based benchmark generation method--a semi-automated approach that well balances the quality and quantity tradeoff of existing benchmarks (fig. 2 ). once the qa template and data extraction method are defined, the dataset can be continuously expanded over time without additional labeling of the degree of time-sensitivity for each question. our contributions are summarized as follows: for each league, we consider three main types of data: league tables by round, top scorers at the end of season, and historical records. for league table questions, we collect data from https://www. worldfootball.net , a comprehensive football database that provides global competition statistics and match results. questions regarding top scorers are sourced from https://www.flashscore. com/ , a live sports results website offering realtime scores, statistics, and match information. the statistics of the dataset are provided in table a1 . taxonomy of questions @xmath0. in contrast to existing automated benchmarks @xcite ## conclusion in this paper, we propose a continuously expandable benchmark, chronobias, by means of a template-based semi-automatic generation method, which effectively addresses the trade-off between quality and quantity. with labels of group, timesensitivity, and the date of knowledge update, we demonstrate with chronobias that it is imperative to conduct time-conditional group bias evaluation on the time-sensitive knowledge of llms for holistic fairness evaluation. specifically, time-conditional group bias shows notable shifts on either side of the knowledge cutoff of a given llm. these shifts involve not only which groups are the best-and worst-performing, but also in the values of the group bias metrics themselves-patterns that cannot be captured by existing time-marginalized group bias analysis. while parametric knowledge bias is the primary cause of performance disparities across groups, we find that time-sensitivity bias is an additional source of group bias, which manifests only after the knowledge cutoff and thereby leads to variations in timeconditional group bias across different time. since parametric knowledge bias and timesensitivity bias are irreducible, rag is considered an alternative for accurate and fair language generation, as it can adapt both to new knowledge and to pre-cutoff knowledge that is attenuated in model parameters. by comparing rag baselines with oracle results using gold passages, we propose actionable guidelines for building ideal rag-based llms. in particular, group-wise retrieval performance should be adjusted to negatively correlate with vanilla model performance, while acknowledging that time-sensitivity bias makes the amount of partial information in incorrect retrieval vary across groups. although we define group in terms of geographical region in the sports domain, our analysis can be generalized to any categorization and topic subject to parametric knowledge bias and time-sensitivity bias. we hope our benchmark serves as a stepping stone for future work emphasizing that holistic evaluation and mitigation of group bias must take the temporal dimension into account.