E ntropy2 V ec: Crosslingual Language Modeling Entropy as End-to-End Learnable Language Representations (2025)

Abstract: We introduce Entropy2Vec, a novel framework for deriving cross-lingual language representations by leveraging the entropy of monolingual language models. Unlike traditional typological inventories that suffer from feature sparsity and static snapshots, Entropy2Vec uses the inherent uncertainty in language models to capture typological relationships between languages. By training a language model on a single language, we hypothesize that the entropy of its predictions reflects its structural similarity to other languages: Low entropy indicates high similarity, while high entropy suggests greater divergence. This approach yields dense, non-sparse language embeddings that are adaptable to different timeframes and free from missing values. Empirical evaluations demonstrate that Entropy2Vec embeddings align with established typological categories and achieved competitive performance in downstream multilingual NLP tasks, such as those addressed by the LinguAlchemy framework.

## introduction linguistic typology provides a framework for classifying languages based on shared structural features, offering insights into language universals and diversity. databases like the world atlas of language structures (wals) @xcite , au-totyp @xcite , uriel @xcite , and uriel + @xcite catalog these features, serving as valuable resources for researchers and practitioners in the field of computational linguistics and beyond. however, these inventories face significant limitations: they often cover only a subset of languages, leading to missing values, and they represent static snapshots of linguistic knowledge, neglecting the dynamic and evolutionary nature of languages. recent advancements in neural language modeling have enabled the extraction of continuous representations of languages through pre-trained models. these embeddings capture semantic and syntactic properties, facilitating cross-lingual transfer in various nlp tasks. nonetheless, existing methods primarily focus on monolingual or bilingual settings and do not explicitly model the typological relationships between languages. moreover, they often rely on manually curated features, which may not generalize well across languages or over time. to address these challenges, we propose en-tropy2vec, a framework that derives language representations based on the entropy of monolingual language models (lms). entropy, a measure of uncertainty in information theory, reflects the predictability of a language's structure. by training a language model on a single language and analyzing its entropy when applied to other languages, we can infer typological similarities and differences. this approach offers several advantages: it is datadriven, scalable, and inherently adaptable to new languages and evolving linguistic features. in this paper, we demonstrate that en-tropy2vec embeddings align with established typological categories, such as phonological, morphological, and syntactic features. we also show that these embeddings outperform traditional typological inventories in downstream multilingual nlp tasks, including language identification, typol-ogy prediction, and cross-lingual transfer. by integrating entropy2vec into the lingualchemy framework @xcite , we achieve competitive generalization across languages, especially those underrepresented in existing typological resources. language vector in nlp language vectors, or embeddings, have become foundational in modern nlp, enabling models to represent words, sentences, and even entire languages as continuous vectors in a high-dimensional space. techniques like word2vec, glove, and fasttext have demonstrated that such embeddings capture semantic and syntactic properties, facilitating tasks like word similarity, analogy reasoning, and machine translation. these embeddings are typically learned from large corpora and reflect the statistical patterns in language use. however, they often treat languages as isolated entities, without explicitly modeling the relationships between them. recent advancements, such as multilingual bert and xlm-r, have sought to address this by training models on multiple languages simultaneously, capturing shared structures and enabling cross-lingual transfer.entropy2vec contributes to this landscape by offering a novel perspective on language representation. instead of relying solely on largescale pre-training on vast corpora, entropy2vec leverages the entropy of monolingual lms to infer typological relationships between languages. this approach not only aligns with existing language representation models but also extends their capabilities by incorporating typological insights, thereby enhancing multilingual understanding and transfer learning ## entropy2vec and language typology to assess the validity of entropy2vec, we compare it against several established language vector and tree baselines: uriel @xcite and uriel + @xcite vectors, as well as the glottolog tree @xcite . for the first two, we derive a hierarchical clustering tree representing inter-language distances based on geographical and syntactic features. we then evaluate how well the trees induced from entropy2vec vectors replicate these known typological groupings, and whether they reveal novel or diverging relationships. ## entropy2vec as language vectors in the previous section, we demonstrate that en-tropy2vec is able to represent meaningful linguistic properties such as language family relation, syntax similarity, and geographical distance. in this section, we establish the applicability of en-tropy2vec and compare it to other existing language vectors such as uriel @xcite and uriel + @xcite . we compare the effectiveness of entropy2vec and other language vectors by measuring the lms performance when applying the vectors on downstream tasks. ## discussion as highlighted in ## conclusion entropy2vec represents a significant advancement in the field of nlp, offering a novel, minimal human-derived knowledge and intervention approach to language representation that captures linguistic characteristics and achieves competitive cross-lingual generalization compared to baselines. by leveraging existing language models, en-tropy2vec is able to derive features with dynamic inventory without having to restart manual baseline-like typology studies and is free from the missing values that plague traditional typological language inventories. this adaptability and completeness make entropy2vec a powerful tool for representing languages, as demonstrated by its ability to mirror patterns observed in linguistic studies and enhance downstream nlp applications. the effectiveness of entropy2vec in improving crosslingual generalization-both as its sole vector and when integrated with baselines-highlights its dynamic nature and compatibility with other representations. entropy2vec holds strong promise for advancing linguistic inclusion and supporting language documentation and preservation efforts, making it a valuable contribution to the field with significant implications for future research in language representation learning.