A Reproducibility Study on Quantifying Language Similarity: The Impact of Missing Values in the URIEL Knowledge Base (2024)

Abstract: In the pursuit of supporting more languages around the world, tools that characterize properties of languages play a key role in expanding the existing multilingual NLP research. In this study, we focus on a widely used typological knowledge base, URIEL, which aggregates linguistic information into numeric vectors. Specifically, we delve into the soundness and reproducibility of the approach taken by URIEL in quantifying language similarity. Our analysis reveals URIEL’s ambiguity in calculating language distances and in handling missing values. Moreover, we find that URIEL does not provide any information about typological features for 31% of the languages it represents, undermining the reliabilility of the database, particularly on low-resource languages. Our literature review suggests URIEL and lang2vec are used in papers on diverse NLP tasks, which motivates us to rigorously verify the database as the effectiveness of these works depends on the reliability of the information the tool provides.

## introduction categorizing and quantifying variations and similarities between languages is critical for applications such as building multilingual large language models @xcite , examining the effects of cross-lingual transfer @xcite , understanding code-switching between languages @xcite , selecting pivot languages when translating from one language to another @xcite , or sharing language tools @xcite . however, there is no consensus on how to measure the similarity between languages due to the difficulty and subjectivity involved in assessing various aspects of languages. this challenge becomes even more pronounced when dealing with low-resource languages @xcite , where limited linguistic knowledge is available to researchers. uriel is a knowledge base that aggregates linguistic information for 4,005 languages from various data sources (figure 1 ) and computes distances based on this information. the lang2vec tool provides an interface for querying uriel @xcite . in many of the 198 citations of uriel and lang2vec, the distance values and feature vectors provided by uriel have been used to quantify language similarity and categorize language features. in this study, we analyze the uriel database to assess its capacity as a resource for quantifying language similarity. we evaluate the reproducibility and validity of the methodology employed in calculating language similarity measurements. we also examine the language and feature coverage of uriel, which affects the meaningfulness of the vectors and distance values. in addition, we conducted a literature review of papers that cite uriel to gain a better understanding of the influence of uriel in these works. ## conclusion in conclusion, in our attempt to reproduce uriel's "language distances", we identified several areas for improvement: as demonstrated in our literature review, there are broad use cases for measuring language similarity. by understanding and addressing areas of improvement for uriel and lang2vec, we can contribute to the progress of research in multilingualism and language diversity, especially for low-resource languages that are not properly represented by these knowledge bases and tools.