Disentangling language change: sparse autoencoders quantify the semantic evolution of indigeneity in F rench (2025)

Abstract: This study presents a novel approach to analyzing historical language change, focusing on the evolving semantics of the French term “indigène(s)” (“indigenous”) between 1825 and 1950. While existing approaches to measuring semantic change with contextual word embeddings (CWE) rely primarily on similarity measures or clustering, these methods may not be suitable for highly imbalanced datasets, and pose challenges for interpretation. For this reason, we propose an interpretable, feature-level approach to analyzing language change, which we use to trace the semantic evolution of “indigène(s)” over a 125-year period. Following recent work on sequence embeddings (O’Neill et al., 2024), we use k -sparse autoencoders ( k -SAE) (Makhzani and Frey, 2013) to interpret over 210,000 CWEs generated using sentences sourced from the French National Library. We demonstrate that k -SAEs can learn interpretable features from CWEs, as well as how differences in feature activations across time periods reveal highly specific aspects of language change. In addition, we show that diachronic change in feature activation frequency reflects the evolution of French colonial legal structures during the 19th and 20th centuries.

## introduction the concept of indigeneity, particularly its relationship to language and culture, is the subject of longstanding, interdisciplinary academic study. postcolonial researchers and writers, for example, have repeatedly emphasized the role of language as central to understanding colonial power @xcite @xcite . the effects of historical and contemporary structural inequalities have also been addressed in computational linguistics, as low-resource indigenous languages pose technical @xcite and ethical @xcite figure 1 : we use k-saes to identify interpretable features in over 200,000 contextual word embeddings corresponding to the word "indigène(s)". features identified by a k-sae are subsequently given natural language descriptions by gpt-4o and validated by gpt-4o-mini. just as indigenous peoples and languages are themselves not interchangeable and must be understood within their appropriate historical and linguistic contexts, the category of "indigenous" is likewise contingent on the specific context in which it is deployed. while the french cognate "indigène" has a similar contemporary definition to "indigenous", it also has its own historical, cultural, and linguistic connotations related to french colonial rule, which was often overtly violent and oppressive towards subjects deemed "indigène(s)" @xcite . to understand the historical and linguistic impact of french colonial policies related to indigeneity, we examine here how "indigène" changes in meaning over the 19th and early-20th centuries. due to the complexities and sensitive nature inherent to the study french colonialism, we strove to maximize the breadth, specificity, and interpretability of our analyses, while at the same time leveraging the power of contemporary methods and technologies. with this in mind, we present an interpretable method for analyzing historical language change through a new large-scale french-language diachronic dataset, which we collected. whereas contemporary approaches to measuring historical language change often compute word similarities or clusters directly from dense contextual word embeddings (cwe) extracted from large language models (llms; @xcite , we instead use sparse autoencoders (sae) to identify interpretable features from dense cwes corresponding to our target word, "indigène(s)". this method, directly adapted from recent work in interpretability research on llms @xcite and sequence embeddings @xcite , has two key advantages. first, it enables us to find and validate natural language descriptions of features learned from cwes without supervision. second, it allows for feature-level analysis of diachronic change, as opposed to discrete sense-level or wordlevel approaches. through our approach, we find that temporal changes in specific feature activations reflect known events and periods of french colonial history. our method and results represent relevant contributions to several domains. notably, we demonstrate how saes can be used for analyzing historical language change when similarity-or clusteringbased approaches are intractable or undesirable. in addition, our method provides additional insight into precisely what cwes can capture about wordlevel meaning, as well as the current capabilities of automated feature labeling techniques. ## motivation and background the word "indigène(s)" differs from its english cognate "indigenous" both grammatically and semantically. it can be used as either an adjective (as in "plantes indigènes", "indigenous plants") or as a substantive (as in "les indigènes", "the indigenous [people]", or "the natives") (trésor de la langue française informatisé, 2024). specific to "indigène(s)" are its highly charged historical connotations tied to french colonial rule, including its deployment as a legal category. early precedents include the sénatus-consulte of april 22, 1863, which legally defined indigenous land-holdings (archives nationales d'outre-mer, 1863), and the crémieux decree in 1870 @xcite , which selectively granted french citizenship to algerian colonial subjects on the basis of religion and ethnicity. those deemed "indigène(s)" were subsequently subjected to social segregation policies analogous to jim crow in the united states, under a set of laws referred to as the code de l'indigénat ("indigenous code"; @xcite , which took form between 1881 and 1946 @xcite . by the 1960s, it had been noted in algeria that "indigène(s)" had largely taken on a pejorative connotation and had therefore begun to disappear from common use @xcite . given these factors, we wanted to determine whether historical trends in the use of "indigène(s)" were visible in large-scale textual data and would reflect the evolution of these legal and social frameworks, such as ethnic and racial segregation. more specifically, we hypothesized that the botanical or zoological senses of "indigène(s)" would be more prominent in the early 19th century, whereas the uniquely human, specifically ethnic, legal, or military senses would be more visible from the late 19th century onward. while there has been longstanding interest in the relationship between semantics, history, and society in the humanities @xcite , there are also a variety of approaches to semantic shift detection (ssd) in the computational linguistics literature. prior work typically relies on contextual word embeddings (cwe), word-level representations extracted from pretrained masked language models like bert @xcite or roberta @xcite , which are subsequently averaged or clustered and then compared across time steps @xcite . other supervised or more specialized approaches also exist, such as in hoffman et al. (2020), which uses graph representations of social networks in addition to temporal labels to enrich language model representations with additional sociotemporal context. more generally, there exist diverse computational historical analysis techniques that have been developed for other modalities, such as for historical change in melody @xcite and tracing the circulation of archival images @xcite . at the same time, there is a growing body of work showing that using cwes generated by llms may come with significant drawbacks. these include anisotropy @xcite ; ethayarajh, 2019; @xcite , "rogue dimensions" @xcite , lack of noise robustness @xcite , and social bias @xcite . these studies suggest that even though cwes perform reasonably well on downstream tasks, comparing cwes directly (particularly with measures such as cosine similarity) may not be a reliable proxy for semantic similarity. within the mechanistic interpretability literature, however, there have been significant recent advancements using sparse autoencoders (sae) to interpret llm representations @xcite @xcite . individual neurons in an llm (or, in our case, dimensions in a cwe) may correspond to multiple human-interpretable features, a phenomenon referred to as "superposition" @xcite . saes, by learning to represent dense model representations as higher-dimensional sparse representations, encourage greater "monosemanticity", where individual neurons in the sae hidden state correspond to a single, human-interpretable feature @xcite . more recently, this technique has been applied to sequence embeddings, further demonstrating that saes can identify highly interpretable, monosemantic features even from pooled or sequence-level representations @xcite ). an additional benefit of these approaches is the relative simplicity of finding and validating natural language descriptions for each feature using another language model @xcite , as well as the ability to analyze the sae's learned feature representations (that is, columns in its decoder weight matrix; @xcite . we offer several original contributions that draw from each of these research areas. building on prior work by @xcite and @xcite , we demonstrate how saes can be used to identify interpretable features from cwes, even when these cwes correspond to only two distinct lexical items ("indigène" and "indigènes"). we go on to show that features with high levels of diachronic variation have activation frequencies that correspond to known events in french history which influenced the signification of the term. our method and findings highlight how interpretability methods can be applied productively to difficult, nuanced questions in historical language change. finally, we find support for our hypotheses regarding the semantic shift of "indigène(s)" in our dataset. ## dataset construction to our knowledge, there is no existing, historical french-language dataset suitable for diachronic analysis. for this reason, we cooperated with the french national library (bnf) to source monographs preprocessed with optical character recognition (ocr) along with metadata, including date of original publication. all of the monographs we requested were free of copyright restriction as of 2023. while our original request encompassed nearly all monographs in the bnf's digital collection, the present study is limited to the first batch retrieved from their servers, which includes approximately 71k volumes. additional information regarding this data is available in appendix a. from these complete texts, we extracted all sentences that included the word indigène or indigènes using a regular expression. we retain sentences longer than ten whitespace-delimited words in length in order to ensure that our cwes are generated with sufficient context. to avoid ambiguity in our diachronic analysis, we excluded sentences sourced from texts that lacked a precise publication date (either due to missing metadata or where publication date was recorded as a range of years). due to extreme data sparsity in very early or late time periods, we limited our analysis to sentences between 1825 and 1950. in total, we include 210,305 sentences in our sentence dataset. 1 in order to facilitate diachronic analyses over 125 years (though not necessarily to align with typical periodizations of french history), we assigned each of these sentences to one of five 25-year time periods based on date of publication. we note that these time periods are severely imbalanced: our smallest time period (1851-1875) comprises 4,942 sentences, while our largest includes 92,767 sentences. we address this particular challenge further in section 4.2. to generate cwes from these extracted sentences, we use a masked language model, camem-bert @xcite , a variant of roberta @xcite optimized for french-language applications. we extract the last hidden state activations of the pretrained camembert-large model that correspond to the tokens in "indigène(s)", then mean pool token-level representations to construct a single cwe for each sentence. ## discussion our method allows for unsupervised identification, description, and validation of features from cwes, as well as diachronic analysis of these features. we demonstrate the successful application of our method on a novel french sentence dataset, sourced from texts across 125 years of french and francophone history. through our analysis, we find that the meanings of "indigène(s)" not only evolved over time, but changed relative to one another in a manner consistent with our original hypotheses: the features with the greatest decrease in activation frequency reflect highly specific botanical contexts, whereas we observe an increase in features associated with colonial, legal, and administrative use beginning in the late 19th century. as a historical research tool, we intend to employ our k-sae and generated feature descriptions to address additional qualitative concerns related to the concept of indigeneity in french history. more broadly, our findings demonstrate the potential of saes for analyzing word-level language model representations, which can complement or potentially replace similarity-based methods. in line with recent work using llms to find and describe concepts in unstructured text datasets @xcite , our approach demonstrates how embeddings can be meaningfully described by generative models, adding interpretable insights to nuanced and difficult tasks in text analysis. ## ethics our sentence data is composed of historical source material related to french colonial treatment of indigenous populations, and as such includes highly offensive or inaccurate descriptions, characterizations, and attitudes related to race, religion, and ethnicity. we in no way endorse irresponsible, discriminatory, or defamatory use of this data or our analyses. ## limitations while we made reasonable efforts to avoid bias when extracting sentences for our dataset, we cannot guarantee that our sample is entirely representative of the french language as a whole. similarly, though our method attempts to account for our dataset's temporal imbalance, we cannot rule out the possibility that our analyses are still affected by the low number of samples in early time periods. because we rely on our predictor to generate measures to gauge feature interpretability, generated feature descriptions may not accurately reflect the content of every sentence for which a feature is activated. because of financial constraints, our predictor uses a limited number of randomly selected positive and negative examples. finally, though we are interested in the overall change in pejorative or racial connotation associated with "indigène(s)" over time, this may only be indirectly inferred from extracted features or combinations of features, and may also require additional inspection of source text. linda wiechetek, flammie a. pirinen, børre gaup, trond trosterud, maja lisa kappfjell, and sjur moshagen. 2024. the ethical question -use of indigenous corpora for large language models. in proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (lrec-coling 2024), pages 15922-15931, torino, italia. elra and iccl.