Theoretical Linguistics Rivals Embeddings in Language Clustering for Multilingual Named Entity Recognition (2023)

Abstract: While embedding-based methods have been dominant in language clustering for multilingual tasks, clustering based on linguistic features has not yet been explored much, as it remains baselines (Tan et al., 2019; Shaffer, 2021). This study investigates whether and how theoretical linguistics improves language clustering for multilingual named entity recognition (NER). We propose two types of language groupings: one based on morpho-syntactic features in a nominal domain and one based on a head parameter. Our NER experiments show that the proposed methods largely outperform a state-of-the-art embedding-based model, suggesting that theoretical linguistics plays a significant role in multilingual learning tasks.

## introduction language clustering has been used to facilitate an effective cross-lingual transfer for low-resource languages in various tasks, such as machine translation @xcite . while the majority of recent clustering approaches depend on embeddings from language models, linguistic knowledge has not yet been exploited enough. previous studies have merely used descriptive typological features @xcite and a coarse language family classification as baselines @xcite . we argue that there is large room for improvement in language clustering using linguistics knowledge. this study examines two language classifications based on theoretical linguistics and tests their effectiveness in multilingual ner. multilingual ner is selected because comparison models are available from @xcite , namely an embedding-based classification and a language family classification. although there are datasets available for ner in various languages @xcite @xcite , our study focuses on indo-european languages because there is a rich body of research in theoretical linguistics. our classification approaches draw on morphosyntactic parameters proposed primarily in theoretical syntax. the first classification is based on a language tree created by @xcite , which reflects various morpho-syntactic parameters in a nominal domain. the second classification uses the head parameter @xcite , which indicates the "head" of a phrase in relation to its complements. we select these parameters because ner is a task that identifies mentions and types of named entities that are mostly nouns. we show that clustering languages based on such parameters results in more effective language groupings beyond the state-of-the-art embeddingbased method. moreover, our clustering approaches demonstrate comparable or better performance than a model trained with all indo-european languages (hence regardless of a substantial difference in the data size). these results suggest that theoretical linguistics has a promising potential in multilingual nlp tasks. ## related work in the current age of globalization, collecting information using various languages is getting more important than ever. multilingual models have gained increasing attention for this purpose. recently, pre-trained large-scale multilingual models using neural networks, such as multilingual bert (mbert) @xcite and xlm-roberta @xcite , have provided competitive results. however, the amount of labeled data available for fine-tuning these multilingual models is highly skewed toward "major" languages. in fact, there are more than 2,000 low-resource languages with little or no labeled data @xcite . to alleviate the problem with low-resource languages, cross-lingual transfer learning has been proposed @xcite . the aim of this method is to adapt a language model trained with high-resource languages to low-resource languages. various transfer learning methods have been proposed. for example, @xcite proposed a technique using subword units (byte pair encoding @xcite ). @xcite investigated which conditions make crosslingual transfer learning possible by conducting artificial language experiments. language clustering is another kind of transfer learning method mainly used in machine translation. @xcite compared clustering by language family and by embeddings and reported that the embedding-based clustering better improved translation accuracy. @xcite proposed a language clustering method that integrates syntactic features of wals @xcite and embeddings from machine translation models. as for ner, shaffer (2021) compared clustering by language family and by embeddings and reported that the embedding-based clustering outperformed language family clustering. in sum, clustering by linguistic prior was used as baselines, and these baselines did not attain better results than the ones with embeddings. other than language clustering, linguistic knowledge has been widely used in various nlp tasks @xcite @xcite . for example, some approaches use typological or phylogenetic features in multilingual fine-tuning for cross-lingual transfer @xcite @xcite . likewise, language family information or typological features, such as word order, have been used in various kinds of multilingual tasks, such as machine translation @xcite , dependency parsing @xcite , and pre-training @xcite . crucially, however, the linguistic information used in all these studies is limited to the extent of language family and typological features which are directly observable. no studies using more profound linguistic knowledge have been conducted. therefore, it remains to be seen whether and to what extent linguistic knowledge other than linguistic family and typological features could help improve clustering for multilingual tasks. ## ner experiments we conduct experiments on ner using the two clustering methods described in section 3. ## conclusion we have proposed two language clustering methods based on the morpho-syntactic parameters proposed in theoretical linguistics. we showed that these clustering methods outperformed the embedding-based clustering in multilingual ner with indo-european languages. we have also compared the model using all the indo-european languages as the training data. despite the large difference in the data size, our approach outperformed this model as well. these results suggest that parameters in theoretical linguistics have a potential utility in multilingual nlp tasks and that this direction is worth exploring. future work will extend this approach to other language families as well as different multilingual tasks, such as machine translation. another direction would be to probe the clusters derived from the embedding-based method to explore features that might not have been captured by our approach or any approaches that make use of explicit linguistic features.