LIMIT : Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages (2023)

Abstract: Knowing the language of an input text/audio is a necessary first step for using almost every NLP tool such as taggers, parsers, or translation systems. Language identification is a well-studied problem, sometimes even considered solved; in reality, due to lack of data and computational challenges, current systems cannot accurately identify most of the world’s 7000 languages. To tackle this bottleneck, we first compile a corpus, MCS-350, of 50K multilingual and parallel children’s stories in 350+ languages. MCS-350 can serve as a benchmark for language identification of short texts and for 1400+ new translation directions in low-resource Indian and African languages. Second, we propose a novel misprediction-resolution hierarchical model, LIMIT, for language identification that reduces error by 55% (from 0.71 to 0.32) on our compiled children’s stories dataset and by 40% (from 0.23 to 0.14) on the FLORES-200 benchmark. Our method can expand language identification coverage into low-resource languages by relying solely on systemic misprediction patterns, bypassing the need to retrain large models from scratch.

## introduction building natural language processing (nlp) tools like machine translation, language identification, part of speech (pos) taggers, etc. increasingly requires more and more data and computational resources. to attain good performance on a large number of languages, model complexity and data quantity must be increased. however, for a majority of the world's 7000 languages, large amounts of data are often unavailable which creates a high barrier of entry @xcite @xcite . increasing model complexity for large-scale models also requires disproportion-1 data, code, and models are publicly available on github under permissive licenses. repository: https://github. com/magarw/limit 0 25 50 figure 1 : most languages in our dataset are from the indian subcontinent and sub-saharan africa, with significant minorities from europe (primarily in the role of the high-resource language parallel translation available for each story). color broadly indicates continent or region (north america, south america, africa, europe, asia, oceania) and size indicates number of languages per country in our dataset. a popular data collection approach is largescale web mining @xcite @xcite , where large parts of the internet are scoured to find training data for data-hungry nlp algorithms. when faced with a sentence or phrase, such algorithms must know how to reliably sort this text into the appropriate language bucket. since the web is replete with content in a variety of languages, a model needs to recognize text in a sufficiently large number of these languages with high accuracy. identifying parallel bitext is even more demanding as a machine translation system must also be available to correctly identify and align parallel data @xcite . this datacollection paradigm becomes inaccessible for lowresource languages because high-quality translation models usually require substantial amounts of parallel data for training, which is often unavailable. without high-quality language identification and translation system, it becomes practically impossible to mine the internet for relevant text during such collection efforts. additionally, mispredictions by language identification and data collection algorithms can increase inter-class noise, reducing the crawled data's quality, and harming performance in downstream tasks without strong quality evaluation metrics @xcite . how can we address these challenges and build high-quality identification and translation for lowresource languages? resource creation highlighting the need for resource creation in low-resource languages, we first share a new parallel children's stories dataset, mcs-350, created using two resources: african storybooks initiative foot_0 and indian non-profit publishing outfit pratham books' digital repository storyweaver foot_1 (available under permissive creative commons licenses). the combined dataset includes original and human-translated parallel stories in over 350 languages (visualized in figure 1 ) and we merge, preprocess, and structure it so it is easily utilizable by nlp researchers for training and benchmarking ( §2). machine translation armed with parallel stories in many low-resource african and indian languages, we tackle machine translation in resourceconstrained situations next. if we aim to collect parallel data in low-resource languages, language identification itself is insufficient and we need highquality translation models as well. we utilize a pretrained multilingual translation model @xcite and explore training with hierarchical language-level and language familylevel adapter units to translate children's stories at the page level ( §3). language identification finally, we take on the biggest bottleneck in low-resource language data collection efforts -language identification. we propose limit -a misidentification-based hierarchical modeling approach for language identification, that utilizes data and computational resources efficiently and shows cross-domain generalization. the proposed approach is exciting because unlike previously published language identification models like afrolid @xcite , cld3 @xcite ing large multilingual models for a new set of languages and still outperforms existing systems. large multilingual models often require thousands of sentences for training, ex. afrolid @xcite collects and trains on over 4000 sentences per language. on the other hand, for many low-resource languages in india and africa, we may not even be able to collect 1000 sentences at first 2. also, in contrast with other recent work in hierarchical language identification @xcite @xcite , our work stands out because it accounts for mispredictions made by existing trained models. unlike other work, it does not predict a group/language family first, but rather directly learns confusion relationships between language pairs (which may not be from the same language family). by leveraging hierarchically organized units on top of a root model, we avoid complete retraining, saving computational resources, while increasing coverage into many new and understudied languages and language pairs (especially those between two low-resource languages) ( §4). ## machine translation benchmark while it is true that resource creation in lowresource languages requires fine-grained and highquality language identification, collecting parallel data additionally requires high-quality mt ( §1). in this section, we explore phylogeny-based hierarchical adapter units to improve translation quality between two african languages, and between african languages and english/french. ## language (mis)identification benchmark language identification (lid) affects low-resource language resource creation efforts severely @xcite because to collect data, we need accurate language identifiers that themselves need high-quality data to train @xcite , creating a vicious cycle. low-quality systems often make mispredictions which increases inter-class noise and reduces the crawled data's quality @xcite both for the predicted language and the true language. to correct mispredictions and improve accuracy in supported languages with limited data, we propose a hierarchical modeling approach. hierarchical modeling is an extremely popular choice for a wide variety of algorithmic tasks and it has been explored for language identification as well @xcite @xcite . however, previous work has focused on predicting language group/family first, followed by finer-grained predictions with a smaller set of classes. our work departs from this paradigm in two ways -first, we bring focus onto expanding language identification coverage in pre-trained or off-the-shelf systems without retraining, and second, we predict a prior and posterior language based on confusion and misprediction patterns of the model directly (without predicting language family/group first). under our technique, we first choose a wellperforming root model with high-coverage that provides us with the base/prior prediction. such base predictions are obtained for a sample of mcs-350's training set, allowing us to identify systemic confusion patterns embedded within the model using a confusion matrix. based on the identified misprediction patterns (which may or may not be between languages in the same family), we train lightweight confusion-resolution subunits that can be attached onto the root model to make the posterior prediction. our results show that, with this architecture, a small sample of data is sufficient to investigate pretrained, off-the-shelf, or blackbox commercial models and identify systemic misprediction patterns across domains. ## conclusion in this work, we tackle the lack of resources for many of the world's languages and release a large, massively parallel children's stories dataset, mcs-350, covering languages from diverse language families, writing systems, and reading levels. since translation is crucial for parallel resource creation, we explore adapter-based networks fine-tuned on a phylogenetic architecture, and utilize mcs-350 to create new translation benchmarks for vulnerable and low-resource languages. we demonstrate large improvements in the children's story domain and cross-domain improvement for several language pairs (on the flores benchmark dataset). on the algorithmic front, we introduce limit, a hierarchical, misprediction-based approach to counter the inaccuracies of pre-trained language identification systems. our method increases language coverage and prediction accuracy bypassing complete retraining, and shows crossdomain generalization despite being trained on our mcs-350 dataset. in the future, we hope to further investigate misprediction-based hierarchical language identification across more datasets, with more configurations, and extensions such as probabilistic branching, automated constructions etc. as a natural next step, we will utilize limit in a web-crawl to find and collect more low-resource language data.