Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization (2023)

Abstract: Large language models (LLMs) have exhibited considerable cross-lingual generalization abilities, whereby they implicitly transfer knowledge across languages. However, the transfer is not equally successful for all languages, especially for low-resource ones, which poses an ongoing challenge. It is unclear whether we have reached the limits of implicit cross-lingual generalization and if explicit knowledge transfer is viable. In this paper, we investigate the potential for explicitly aligning conceptual correspondence between languages to enhance cross-lingual generalization. Using the syntactic aspect of language as a testbed, our analyses of 43 languages reveal a high degree of alignability among the spaces of structural concepts within each language for both encoder-only and decoder-only LLMs. We then propose a meta-learning-based method to learn to align conceptual spaces of different languages, which facilitates zero-shot and few-shot generalization in concept classification and also offers insights into the cross-lingual in-context learning phenomenon. Experiments on syntactic analysis tasks show that our approach achieves competitive results with state-of-the-art methods and narrows the performance gap between languages, particularly benefiting those with limited resources.

## introduction cross-lingual generalization entails repurposing the knowledge acquired in one language to another with little supervision, thereby mitigating the digital language divide. despite the vast variations across languages, it is possible to identify corresponding concepts among them, which provides a basis for cross-linguistic generalization @xcite @xcite . this has been instantiated by frameworks such as universal dependencies (ud) @xcite , where the structural concepts including word classes (e.g., "noun" and "verb") and grammatical relations (e.g., "subject" and "object") are defined in a cross-linguistically consistent way. while large language models (llms) have demonstrated their capacity to induce these concepts within individual languages @xcite @xcite , they encounter difficulties in generalizing the knowledge across languages @xcite @xcite . this raises questions about whether llms are able to capture the underlying conceptual correspondence and how to harness the knowledge for improved generalization. previous work has shown that cross-linguistic similarities are automatically captured in the representation space of llms, enabling zero-shot cross-lingual transfer @xcite @xcite @xcite . efforts have been made to further enhance their generalization by exploiting high-resource languages for parameter and information sharing @xcite @xcite or enforcing alignment between languages @xcite @xcite , whereas these approaches typically rely on structural and lexical similarities between languages and fall short when dealing with low-resource languages distant from high-resource ones @xcite . with the ever-increasing size of llms, recent work has explored methods to elicit the multilingual ability of llms via in-context learning @xcite , alleviating the cost of parameters updates, but the generalization performance lags behind and is highly sensitive to prompt design @xcite . overall, the generalization is predominantly realized implicitly and it remains unclear whether the commonalities shared across languages have been fully exploited. in this paper, we investigate the potential to explicitly leverage the conceptual correspondence between languages for cross-lingual generalization. we focus on the structural concepts outlined in ud, which are at the core of syntactic analyses across languages @xcite and have shown to be learned by llms, offering a valuable testbed for our analyses. for each language, we learn a linear transformation of an llm's representation space, which defines a conceptual space where samples can be classified by their distances to prototypes for each concept. concepts represented by the llm are then regarded as clusters discernible from others based on their prototypes. analyses across 43 typologically distinct languages reveal a high degree of alignability among their conceptual spaces, indicating that the conceptual correspondence is implicitly established in llms (section 2). we then present a meta-learning-based method that learns to explicitly align different languages with limited data available, facilitating zero-shot and few-shot generalization in concept classification. we demonstrate the effectiveness of our approach for both encoder-only (section 3) and decoder-only llms (section 4), achieving encouraging results especially for low-resource languages. in summary, our contributions are as follows: 1) we demonstrate that the conceptual correspondence between languages, in terms of the structural concepts defined in ud, is implicitly established in both encoder-only and decoder-only llms. ## aligning conceptual correspondence for cross-lingual generalization the previous section shows that the universal structural concepts are readily alignable in the llms. next, we investigate how to leverage the knowledge for cross-lingual generalization. we rely on meta-learning to learn to align conceptual correspondence between different languages in both zero-shot and few-shot scenarios, analyzing how different factors including the number of available examples and the languages used for meta-training may impact the generalization. ## aligning conceptual correspondence during in-context learning in this section, we further explore whether the correspondence between structural concepts can be harnessed for cross-lingual generalization in the in-context learning setting. specifically, we first show that the cross-lingual syntactic abilities of llms can be elicited through in-context learning. we then rely on our method to probe the underlying mechanisms and further enhance the cross-lingual generalization by aligning the structural concepts within different languages. ## conclusion we have demonstrated that multilingual lms are able to induce the correspondence between structural concepts within different languages without any explicit supervision. this knowledge is encoded in their geometry and can be exploited for generalization, whereby we rely on metalearning to learn to align different languages with minimal examples available. our approach can be used to evaluate the correspondence between different systems that has been acquired by llms, and explicitly leverage them for sample-efficient generalization, suggesting a new path toward measuring and manipulating the knowledge encoded in llms. future research may generalize it to other contexts (e.g., other modalities) to probe the commonalities and differences shared between systems and develop more sophisticated way for alignment and generalization.