Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysis (2025)

Abstract: Probing techniques for large language models (LLMs) have primarily focused on English, overlooking the vast majority of other world’s languages. In this paper, we extend these probing methods to a multilingual context, investigating how LLMs encode linguistic structures across diverse languages. We conduct experiments on several open-source LLM models, analyzing probing accuracy, trends across layers, and similarities between probing vectors for multiple languages. Our key findings reveal: (1) a consistent performance gap between high-resource and low-resource languages, with high-resource languages achieving significantly higher probing accuracy; (2) divergent layer-wise accuracy trends, where high-resource languages show substantial improvement in deeper layers similar to English; and (3) higher representational similarities among high-resource languages, with low-resource languages demonstrating lower similarities both among themselves and with high-resource languages. These results provide insights into how linguistic structures are represented differently across languages in LLMs and emphasize the need for improved structure modeling for low-resource languages.

## introduction large language models (llms), such as gpt-4 @xcite , claude 3.5 @xcite , llama 3 @xcite , have demonstrated remarkable progress across a wide range of natural language processing tasks. as these models continue to advance, there is a growing need to understand their internal mechanisms and how they represent linguistic structures. probing techniques have emerged as a valuable tool for investigating how llms encode and process structural information, offering insights into their decision-making processes and the nature of their learned representations @xcite @xcite . however, a significant gap exists in our understanding of how llms represent linguistic structures across different languages. while extensive probing research has been conducted on english language representations, there are approximately 7,000 languages spoken worldwide, many of which remain understudied in the context of llms. this lack of multilingual analysis limits our understanding of how llms encode structural information across diverse linguistic contexts, particularly for low-resource languages that are often underrepresented in model training data and evaluations. to address this research gap, we propose a multilingual probing approach to investigate the structural representation capabilities of llms across a diverse set of 16 languages, including both highresource and low-resource languages. our study extends probing techniques from english to a multilingual context, examining how llms encode linguistic structures in factual knowledge and sentiment classification tasks across different languages. our key findings reveal that: (1) high-resource languages consistently achieve higher probing accuracy compared to low-resource languages; (2) highresource languages exhibit similar trends to english across model layers, with accuracy improving significantly in deeper layers, while low-resource languages show relatively stable or only slightly improving accuracy; and (3) there are high similarities between probing vectors of high-resource languages, whereas low-resource languages demonstrate lower similarities both among themselves and with high-resource languages. ## experiment in this section, we conduct comprehensive probing experiments to investigate how language models process different languages. our analysis focuses on two key aspects: comparing accuracy across different model layers and examining correlations between probing vectors of various languages. through these experiments, we seek to answer three fundamental research questions (rqs): ## related work in this section, we review two lines of research that are most relevant to ours. ## conclusions and future work in this work, our multilingual probing experiments on llms reveal significant disparities in performance and representational qualities across languages, suggesting potential limitations in how these models learn linguistic concepts. specifically, high-resource languages consistently achieve higher probing accuracy and exhibit similar trends to english, with accuracy improving significantly in deeper layers. we also observe high similarities between probing vectors of high-resource languages, while low-resource languages demonstrate lower similarities both among themselves and with high-resource languages. these findings not only indicate the current limitations of llms in handling low-resource languages, but also suggest that these models may not be learning deeper linguistic concepts effectively across all languages. in future, we plan to conduct research to address these gaps by developing more equitable and effective multilingual language models that can better capture universal linguistic concepts. besides, we plan to extend this research to multimodal models that incorporate visual and textual information.