L ingu A lchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization (2024)

Abstract: Pretrained language models (PLMs) have shown remarkable generalization toward multiple tasks and languages. Nonetheless, the generalization of PLMs towards unseen languages is poor, resulting in significantly worse language performance, or even generating nonsensical responses that are comparable to a random baseline. This limitation has been a longstanding problem of PLMs raising the problem of diversity and equal access to language modeling technology. In this work, we solve this limitation by introducing LinguAlchemy, a regularization technique that incorporates various aspects of languages covering typological, geographical, and phylogenetic constraining the resulting representation of PLMs to better characterize the corresponding linguistics constraints. LinguAlchemy significantly improves the accuracy performance of mBERT and XLM-R on unseen languages by ~18% and ~2%, respectively compared to fully finetuned models and displaying a high degree of unseen language generalization. We further introduce AlchemyScale and AlchemyTune, extension of LinguAlchemy which adjusts the linguistic regularization weights automatically, alleviating the need for hyperparameter search. LinguAlchemy enables better cross-lingual generalization to unseen languages which is vital for better inclusivity and accessibility of PLMs.

## introduction significant advancements in language processing technology have been achieved through the development of plms with their impressive capability in language comprehension and generation @xcite @xcite @xcite . the development has been further expanded to non-english languages @xcite @xcite @xcite . however, there is still a gap in these models' ability to generalize effectively to low-resource and unseen languages, although there have been a numerous work in the field @xcite @xcite @xcite . ## related work plms with their transformer-based architectures have been demonstrating exceptional capabilities in language comprehension and generation @xcite . @xcite have explored how these models learn intricate linguistic features, including syntax and semantics to enhance their performance across a wide range of language tasks. incorporating new unseen languages has been a longstanding problem in the multilingual re-search, mad-x @xcite employ a language adapter to learn new unseen languages using language adapters that mitigate the risk of forgetting pre-trained knowledge, which is known as the curse-of-multilinguality @xcite . nonetheless, this approach requires training for generalizing to new unseen languages, which makes it costly and difficult to scale to thousands of languages. mad-g @xcite and udapter @xcite further generalize this approach by utilizing a linguistic-driven contextual parameter generator (cpg) module to generate language-specific parameters, allowing the models to generalize to other languages with similar linguistic characteristics. recently, rathore et al. (2023) introduce zgul, which combines representations over multiple language adapters and adding linguistic vector information to generate the unseen language representation. despite the effectiveness, all these approaches rely on two assumptions: (1) strict categorization of languages and (2) knowing the language category of the query apriori-our definition of "a priori categorization" as incorporating language-specific information into the model. in parallel, the development of linguisticallydriven resources such as the uriel vector and the lang2vec utility @xcite has been notable in extending multilingual nlp research, particularly for less-resourced languages, by providing methods to represent and compare the structured lingustic features across different languages. complementing this, @xcite pointed out the underexplored typological features in existing approaches and the need for integrating datadriven methods of typological knowledge into language models. previous studies have focused on extending multilingual nlp using uriel vectors @xcite @xcite , but none of their approaches use uriel vectors for alignment during the finetuning process of language models. however, plms still face significant challenges in generalizing to unseen languages, particularly when adapting to low-resource and unseen languages. these challenges stem from the vast structural and semantic variation across languages @xcite , the scarcity of resources @xcite , and the limitations inherent in the models themselves @xcite . this situation highlights the complexity of generalizing these models effectively to a broader and scope of language. this situation highlights the complexity of generalizing these models effectively to a broader scope of languages, making it difficult for them to perform well on languages they haven't been specifically trained on. ## unseen languages adaptation with lingualchemy in this section, we provide an overview of how lingualchemy captures linguistic constraints and explain the intuition behind it. we also discuss in detail how we align model representations with the linguistic vectors. ## conclusion we introduced lingualchemy, a novel approach that demonstrates strong performance across 30+ unseen languages on intent classification and semantic relatedness tasks. our method hinges on the integration of linguistic knowledge through the uriel vectors, enhancing the language model's ability to generalize across a diverse set of languages. we also proposed alchemyscale and alchemytune, which employs a hyperparameter search for the uriel scaling factor. this is achieved by two key strategies: (1) weightaveraging classification and uriel loss, and (2) learning to balance the scale between classification and uriel loss. lingualchemy achieves a massive performance improvement on low-resource languages in multiple downstream tasks including intent classification (↑ 18.50), news classification (↑ 31.46), and semantic relatedness (↑ 0.32).