Fine-Tuned LLM s are “Time Capsules” for Tracking Societal Bias Through Books (2025)

Abstract: Books, while often rich in cultural insights, can also mirror societal biases of their eras—biases that Large Language Models (LLMs) may learn and perpetuate during training. We introduce a novel method to trace and quantify these biases using fine-tuned LLMs. We develop BookPAGE, a corpus comprising 593 fictional books across seven decades (1950-2019), to track bias evolution. By fine-tuning LLMs on books from each decade and using targeted prompts, we examine shifts in biases related to gender, sexual orientation, race, and religion. Our findings indicate that LLMs trained on decade-specific books manifest biases reflective of their times, with both gradual trends and notable shifts. For example, model responses showed a progressive increase in the portrayal of women in leadership roles (from 8% to 22%) from the 1950s to 2010s, with a significant uptick in the 1990s (from 4% to 12%), possibly aligning with third-wave feminism. Same-sex relationship references increased markedly from the 1980s to 2000s (from 0% to 10%), mirroring growing LGBTQ+ visibility. Concerningly, negative portrayals of Islam rose sharply in the 2000s (26% to 38%), likely reflecting post-9/11 sentiments. Importantly, we demonstrate that these biases stem mainly from the books’ content and not the models’ architecture or initial training. Our study offers a new perspective on societal bias trends by bridging AI, literary studies, and social science research.

## introduction books are a rich source of knowledge, offering insights across diverse topics, historical contexts, and in-depth subject matter. this makes corpora derived from books, such as bookcorpus @xcite and project gutenberg foot_1 , incredibly valuable for training large language models (llms). pretrained llms show impressive abilities in various writing tasks @xcite . however, fine-tuning on book-based corpora can enhance performance in text summarization @xcite , creative writing @xcite , and genrespecific content generation @xcite . despite these benefits, pre-training and finetuning llms on books introduces a challenge: the perpetuation of societal biases. these biases, defined as "skews that produce harm" @xcite , often manifest in the model outputs and are deeply embedded in cultural artifacts like literature @xcite . since models' biases typically stem from their training data @xcite , using literary sources for their development has been shown to exacerbate these biases @xcite . while extensive research has been conducted on biases in diverse text corpora @xcite , gaps persist in understanding how these biases vary across different historical periods. traditional qualitative methods, like close §2 §3 §4 figure 2 : overview of methodology from corpus creation to bias analysis, with section references reading @xcite , offer depth but lack scalability, limiting their ability to track evolving biases across large corpora @xcite . similarly, standard quantitative methods, which analyze large corpora, often miss subtle biases due to their reliance on static statistical techniques like n-gram models or word embeddings @xcite . to address these limitations, our research leverages three key capabilities of llms: their adaptability to specific textual sources through finetuning, complex language understanding, and responsiveness to targeted prompts. we develop bookpage (book publications across generational eras)-a curated corpus of 593 fictional books from seven decades . by fine-tuning llms on decade-specific subsets of bookpage, we create models that capture eraspecific linguistic patterns and biases, functioning metaphorically as 'time capsules'. we then probe these models with structured prompts, analyzing biases across multiple demographic categories (such as gender, race, and religion) and historical eras. this approach reveals complex patterns of bias that conventional close reading or simple word association techniques might miss. figure 2 provides a general overview of our methodology. key contributions of our research include: taken together, these contributions aim to show how ai can serve as a powerful tool for hypothesis generation -in this case, surfacing new questions about society's evolution that we hope humanities and social science experts will find valuable to investigate further. ## book publications across generational eras (bookpage) bookpage is a corpus comprising seven decadespecific subsets of fictional books, spanning from 1950 to 2019. formally, the bookpage corpus, d, is defined as: @xmath0, s @xcite , . . . , s @xcite } each subset s [x,y) includes books b [x,y) popular between years x and y -1, with n representing the total number of books in each subset: s @xcite @@xmath1,1960) , . . . , b n [1950,1960) } each book b [x,y) is a tuple containing the title t as a label and its complete content c: b @xcite ← {t, c} for example, the 1950s subset s @xcite may include: 1 [1950,1960) = {"the catcher in the rye", [full text of the book]} b 2 [1950,1960) = {"from here to eternity", [full text of the book]} . . . ## conclusion we introduce a novel methodology for analyzing historical societal biases using fine-tuned llms on a decade-stratified book corpus. we developed bookpage, a corpus of 593 fictional books spanning seven decades, and used targeted probing techniques to reveal temporal variations in societal biases. our approach shows that llms, when finetuned on era-specific literature, can effectively capture and reflect the evolving societal attitudes of different time periods. this methodology provides a powerful tool for understanding the historical progression of societal biases, offering valuable insights for sociologists, historians, and ai ethicists alike. future work could extend this method to non-fiction literature, cross-cultural comparisons, or more granular temporal analyses, while also exploring strategies to mitigate the perpetuation of harmful biases in ai systems.