Clustering Pseudo Language Family in Multilingual Translation Models with Fisher Information Matrix (2023)

Abstract: In multilingual translation research, the comprehension and utilization of language families are of paramount importance. Nevertheless, clustering languages based solely on their ancestral families can yield suboptimal results due to variations in the datasets employed during the model’s training phase. To mitigate this challenge, we introduce an innovative method that leverages the fisher information matrix (FIM) to cluster language families, anchored on the multilingual translation model’s characteristics. We hypothesize that language pairs with similar effects on model parameters exhibit a considerable degree of linguistic congruence and should thus be grouped cohesively. This concept has led us to define pseudo language families. We provide an in-depth discussion regarding the inception and application of these pseudo language families. Empirical evaluations reveal that employing these pseudo language families enhances performance over conventional language families in adapting a multilingual translation model to unfamiliar language pairs. The proposed methodology may also be extended to scenarios requiring language similarity measurements. The source code and associated scripts can be accessed at https://github.com/ecoli-hit/PseudoFamily.

## introduction multilingual neural machine translation (mnmt) aims to construct a single model to translate multiple languages and has proven its effectiveness @xcite . the application of such models for low-resource languages has revealed that leveraging supplementary languages during the finetuning phase can yield results that surpass traditional methods @xcite . nevertheless, the practical challenge lies in the development of the optimal strategy to identify the most beneficial auxiliary languages that can bolster the translation of low-resource language pairs. * co-first and corresponding author current academic investigations in this domain generally fall into two distinct categories: firstly, the integration of diverse prior knowledge, which incorporates the utilization of various forms of existing knowledge and language family classifications, as evidenced in research such as @xcite ; @xcite . secondly, the exploration and computation of language similarity center on examining language representations and their comparative resemblances @xcite @xcite . conventional methodologies predominantly rely on supplemental resources or the alteration of model architecture to categorize language families. however, those approaches need more adaptability, especially when dealing with recently developed pre-trained large mnmt models, e.g., m2m100 @xcite , where the original training data and model architecture are inaccessible. when it comes to a new scenario this limitation significantly hampers the effectiveness of existing methodologies for language family clustering. to address this gap, we introduce a novel approach to pseudo language family clustering, which inherently depends on the mnmt model itself. our proposed methodology hypothesizes that language pairs exhibiting similarities in their impact on model parameters are likely to possess a high level of congruence and should, consequently, be grouped together. we employ the fim to quantify such similarities between language pairs. we demonstrate the efficacy of our approach by enhancing low-resource translation with pseudo language family clustering, yielding superior results compared to traditional language family clustering. our main contributions are as follows: the accurate representation of language pairs is foundational for computing their pairwise similarities and ensuring efficacious clustering. as such, devising profound and robust representations for these pairs becomes imperative to attain excellence in this arena. drawing inspiration from contemporary strides in parameter-efficient fine-tuning where innovative strategies have emerged to harness model parameters for representation @xcite @xcite we propose the deployment of the fim. our approach seeks to capitalize on the task-specific parameters embedded within the model. ## conclusion and future work in this study, we introduce a pioneering approach and three methods to quantify the similarities between language pairs and subsequently propose a pseudo family, based on this measurement, to enhance the selection of auxiliary languages for multilingual training in low-resource language scenarios. our strategies for assessing language similarity leverage fisher information to sieve through model parameters, subsequently identifying taskspecific ones and utilizing them to formulate an fim to gauge the similarity between language pairs. the experimental outcomes significantly improve bleu scores on low-resource languages, affirming our proposed method's feasibility and accuracy. future works include: 1) expanding the application of the proposed fim methodology to other tasks that require the calculation of language similarity and 2) integrating our findings into curriculum learning frameworks @xcite and consistency learning frameworks @xcite @xcite .