Datasets Creation and Empirical Evaluations of Cross-Lingual Learning on Extremely Low-Resource Languages: A Focus on Comorian Dialects (2024)

Abstract: In this era of extensive digitalization, there are a profusion of Intelligent Systems that attempt to understand how languages are structured for the aim of providing solutions in various tasks like Text Summarization, Sentiment Analysis, Speech Recognition, etc. But for multiple reasons going from lack of data to the nonexistence of initiatives, these applications are in an embryonic stage in certain languages and dialects, especially those spoken in the African continent, like Comorian dialects. Today, thanks to the improvement of Pre-trained Large Language Models, a spacious way is open to enable these kind of technologies on these languages. In this study, we are pioneering the representation of Comorian dialects in the field of Natural Language Processing (NLP) by constructing datasets (Lexicons, Speech Recognition and Raw Text datasets) that could be used on different tasks. We also measure the impact of using pre-trained models on languages closely related to Comorian dialects to enhance the state-of-the-art in NLP for these latter, compared to using pre-trained models on languages that may not necessarily be close to these dialects. We construct models covering the following use cases: Language Identification, Sentiment Analysis, Part-Of-Speech Tagging, and Speech Recognition. Ultimately, we hope that these solutions can catalyze the improvement of similar initiatives in Comorian dialects and in languages facing similar challenges.

## introduction the comoros are an archipelago composed of four islands in the indian ocean. approximately 850,000 people are living there (worldometers), speaking four dialects belonging to the bantu language family @xcite . these dialects are consequently impacted by geo-spatial features that progressively increase or eliminate similarities between them as shown in @xcite and according to the orelc foot_0 lexicon (see fig. 1 ) in which we can observe that in a dictionary of 7,386 entries, 15.38% of the words are shared by all the dialects, 6.47% by three and 16.10% by two dialects. indeed, these dialects can be divided into two groups: eastern group (shindzuani and shimaore) and western group (shingazidja and shimwali). moreover, a part of the experiments conducted in @xcite has shown through lexical distances calculation that these dialects could be classified into two other different groups, the first one composed of the shingazidja while the second one contains the other three dialects. the arrival of transformers @xcite was a real breakthrough in artificial intelligence (ai). this architecture allows us to better represent the context within texts which is a major spearhead in language understanding. pre-trained language models like bidirectional encoder representations from transformers (bert) @xcite have encapsulated this architecture, paving the way to better representation of multiple languages in all around the world @xcite through cross-lingual learning. in multilingual scenarios, this latter allows languages that suffer from data scarcity to learn from the others owing to a sort of transfer learning. this becomes more interesting when working on close languages as demonstrated in @xcite where a machine translation system was adapted to a language close to the source language used on training. the aim of this work is to contribute on the pioneering of natural language processing (nlp) on comorian dialects by (a) constructing datasets that could be used on different downstream tasks for future works and (b) experimenting the impact of using a cross-multilingual approach on close language to leverage nlp solutions on low-resource scenarios. the rest of this study is structured as follows: we first present in section 2 notable previous works in these dialects, then we describe in section 3 the data collection methodologies that we adopted to collect the different datasets, section 4 shows the experiments that we conducted to evaluate the constructed datasets while section 5 presents the future work that could result from this study following a final conclusion. ## related work one thing to know about these works is that only few of them use nlp approaches and the data used or resulted from them are not publicly available. this make it more interesting the data retrieving and the resorting to recent nlp solutions in order to contribute to the digital representation of these dialects, hoping that this could be helpful in different upcoming use cases by researchers, institutions, companies or particulars. ## datasets ensuring the quality of data has always been at the center of concerns when designing ai solutions, especially in nlp @xcite . this is more important in low-resource scenarios to the point that before trying to understand which model architecture could be more appropriate for a given task in a given language, ensuring quality and sufficiency of data is crucial. following the experiments conducted in @xcite , interesting propositions were advanced. in fact, the experiments consisted of measuring the impact of focusing on data processing in the basque language. they first estimated with native basque speaker the quality of three datasets (mc4, cc100 and euscrawl) then trained different models (topic classification, sentiment analysis, stance detection, named entity recognition and question answering) with the same parameters for each dataset. one of the main conclusions resulted from this study was that in language understanding on low-resource scenario, the quantity of data could be more helpful than its quality, even if this latter is a crucial feature to take into account when managing natural language. we consider two observations (the data quality and quantity importance) when constructing comorian datasets for the aim to manage different nlp tasks. for that we resort to different methodologies depending on the task, the nature of data and from each source the data was initially retrieved. we also investigate the effectiveness of using advance processing approaches like transfer learning from close languages and data augmentation in possible cases.