Extended Abstract for “Linguistic Universals”: Emergent Shared Features in Independent Monolingual Language Models via Sparse Autoencoders (2025)

Abstract: Do independently trained monolingual language models converge on shared linguistic principles? To explore this question, we propose to analyze a suite of models trained separately on single languages but with identical architectures and budgets. We train sparse autoencoders (SAEs) on model activations to obtain interpretable latent features, then align them across languages using activation correlations. We do pairwise analyses to see if feature spaces show non-trivial convergence, and we identify universal features that consistently emerge across diverse models. Positive results will provide evidence that certain high-level regularities in language are rediscovered independently in machine learning systems.

## introduction whether certain structural patterns are shared across all natural languages, despite surface-level differences, has long been a topic of debate in linguistics. in natural language processing, studies have shown that multilingual language models possess semantically aligned capabalities across languages even without explicit parallel supervision @xcite @xcite . this suggests that machine-learned representations capture crosslinguistic regularities, but it should be noted that this alignment is aided by shared vocabularies and parameters @xcite . a more fundamental question remains: can independently trained monolingual lms -which share no parameters nor vocabulary -nonetheless converge on analogous high-level features? if so, this would suggest that certain structural principles of language emerge robustly in machine learning, even when models are trained in isolation. goldfish @xcite provides us with a suite of monolingual gpt-style models covering 350 languages. these models have identical architectures and training budgets but were each trained with strictly monolingual corpora. they thus form a controlled testbed for crosslinguistic comparison. a key technical challenge is how to identify and compare high-level features across different models. to overcome this, we adopt sparse autoencoders (saes) as an analysis tool. recent work @xcite showed that training a single-layer sae on a model's activations yields a set of sparsely activating features that are far more interpretable and monosemantic than the original neuron basis. in essence, the sae "discovers" a dictionary of latent feature directions in activation space, each corresponding to a distinct concept or pattern in the data. @xcite demonstrated that saes trained on multilingual llms uncover both monolingual and multilingual features. notably, @xcite recently employed saes to compare features across different english llms. they hypothesized that the spaces spanned by sae features are similar, such that one sae space is similar to another sae space under rotation-invariant transformations, and found high similarities for sae feature spaces across various llms, providing evidence for feature space universality. we build on this approach in a novel crosslinguistic setting. our research questions are framed as follows: rq1: can sae features trained on independently trained monolingual lms be matched across languages? after matching, do they show nontrivial (above baseline) convergence (i.e., have higher alignment score)? rq2: at which model depths (layers) is feature alignment strongest across languages? rq3: does the degree of alignment correlate systematically with linguistic relatedness (e.g., typological or genealogical distance)? rq4: are there features that emerge universally across languages, and can they be interpreted (e.g., punctuation, numerals, structural delimiters)? how prevalent are such features? ## universal features after aligning features between each pair of languages, we ask if these features are universal across all languages. definition. a feature cluster is universal at layer h if it contains aligned features from at least p% of languages (we choose @xmath0, 75, 90}). construction. we build a graph whose nodes are (language, feature) and whose edges connect matched pairs from section 2.3 (@xmath1. connected components (or communities via louvain) define crosslinguistic clusters. for each cluster we report: coverage (fraction of languages present), mean within-cluster correlation, and stability across bootstrap resamples. preliminary expectations are that a non-trivial fraction of the learned features -especially those capturing very general patterns -will be universal. for instance, we anticipate discovering features related to punctuation, numerals, and structural delimiters that appear in every model. interpretability. for each universal cluster, we list top activating n-grams/tokens per language and show cross-language trigger sets (digits, punctuation, brackets, etc.). we include exemplar sentences and activation traces for qualitative validation. we hope that uncovering such crosslinguistic universal features will shed light on whether machine-learned representations mirror long-standing hypotheses in linguistic theory, and may even provide a complementary empirical perspective to the study of linguistic universals in human languages.