Likelihood Variance as Text Importance for Resampling Texts to Map Language Models (2025)

Abstract: We address the computational cost of constructing a model map, which embeds diverse language models into a common space for comparison via KL divergence. The map relies on log-likelihoods over a large text set, making the cost proportional to the number of texts. To reduce this cost, we propose a resampling method that selects important texts with weights proportional to the variance of log-likelihoods across models for each text. Our method significantly reduces the number of required texts while preserving the accuracy of KL divergence estimates. Experiments show that it achieves comparable performance to uniform sampling with about half as many texts, and also facilitates efficient incorporation of new models into an existing map. These results enable scalable and efficient construction of language model maps.

## introduction in recent years, an increasing number of studies have been conducted to systematically compare and organize language models @xcite @xcite @xcite @xcite . in this context, @xcite proposed a method to estimate kullback-leibler (kl) divergence between language models based on log-likelihood vectors. in this method, language models with different architectures are embedded into a common space, and visualizing this creates a map of language models (fig. 1 ). the model map is constructed using a text set d @xmath0, • • • , x n }. since d n is a sample from a broader text population d † = {x † 1 , . . . , x † n 0 }, it introduces sampling error relative to the true relationships between models in the population 1 . additionally, the computational cost, which is proportional to the number of texts n , is also an issue. 1 when the entire pile corpus is converted into 1,024-byte text chunks, the total number of texts is @xmath1,703,791. the primary motivation of this study is to reduce the computational costs associated with adding new models to the map. to address this issue, we propose a resampling procedure in which n texts are drawn with replacement from the dataset d n . the resulting set, denoted by d * d , consists of d unique texts (@xmath2, which is then used to efficiently reconstruct the model map with the newly added models. when we discuss the distance between models calculated using log-likelihood vectors and the reliability of the model map, it is crucial to focus on the sampling error with respect to the population. to estimate the resampling error of the distance between models measured based on d * d relative to the true distances in the population, two types of errors need to be considered. the first is the sampling error that the original data d n has with respect to the population d † , and the second is the resampling error incurred when selecting d * d from d n . since uniform resampling from data d n may waste many samples on less informative texts, we propose a method that preferentially resamples texts with large variance in log-likelihoods across models. this approach builds on the column sampling method of drineas and @xcite , originally proposed for approximating matrix products. through experiments, we demonstrate that our text resampling method achieves an estimation error comparable to that of uniform random sampling while requiring only about half as many texts. furthermore, it facilitates the efficient incorporation of new models into an existing model map while maintaining accuracy. ## conclusion we proposed text selection methods to reduce the computational cost of constructing model maps. our experiments showed that these methods achieve estimation accuracy comparable to uniform sampling while requiring only about half as many texts, and that they also enable efficient incorporation of new models into an existing map. these findings facilitate more efficient comparative analysis of large-scale language models.