The KIND Dataset: A Social Collaboration Approach for Nuanced Dialect Data Collection (2024)

Abstract: Nuanced dialects are a linguistic variant that pose several challenges for NLP models and techniques. One of the main challenges is the limited amount of datasets to enable extensive research and experimentation. We propose an approach for efficiently collecting nuanced dialectal datasets that are not only of high quality, but are versatile enough to be multipurpose as well. To test our approach we collect the KIND corpus, which is a collection of fine-grained Arabic dialect data. The data is short texts, and unlike many nuanced dialectal datasets, it is curated manually through social collaboration efforts as opposed to being crawled from social media. The collaborative approach is incentivized through educational gamification and competitions for which the community itself benefits from the open source dataset. Our approach aims to achieve: (1) coverage of dialects from under-represented groups and fine-grained dialectal varieties, (2) provide aligned parallel corpora for translation between Modern Standard Arabic (MSA) and multiple dialects to enable translation and comparison studies, (3) promote innovative approaches for nuanced dialect data collection. We explain the steps for the competition as well as the resulting datasets and the competing data collection systems. The KIND dataset is shared with the research community.

## introduction the arabic language is one of the most spoken languages in the world with over 400 million speakers from more than 30 countries @xcite and has gained wide attention in natural language processing advancements recently. since most linguistic technologies rely on high quality training data, arabic data collection is, consequently, becoming the focus of an increasing number of studies. although a wide range of these studies propose effective approaches for arabic data collection, the need for large scale, high-quality datasets of nuanced dialect variations is constantly increasing with the demands of domain specific applications as well as large language models. a main challenge facing nlp technologies in arabic is the diversity of arabic dialects, with more than 30 modern dialects across the arab region, and over 20 documented dialects in saudi arabia, the largest country in the arabian peninsula @xcite . this challenge is most pronounced in downstream applications that involve machine translation from dialectal arabic to other languages. the multitude of nuanced or explicit varying synonyms and hyponyms in arabic dialects have high impact on the quality of translation models. several approaches were proposed to take advantage of msa as the universal formal arabic and ground other dialects to it through translation or similarity analyses. as part of these studies datasets of parallel dialectal and msa texts were proposed @xcite @xcite . although very useful, these datasets remain limited in size and dialectal coverage. to address this gap we propose, kind: king fahd university of petroleum and minerals (kfupm) in your dialect approach, a multiple-tasks competition for obtaining short texts of parallel corpora of fine-grain arabic dialectal data and question and answer pairs. the approach aims not only to promote innovative approaches for data collection, but also to raise awareness about the significance of representation in spoken language-based technologies. the competition is accompanied by social educational initiatives to raise awareness about linguistic technologies and to encourage the public to participate in linguistic data collection competitions. social engagement was solicited by demonstrating the effect of sharing their dialectal utterances on the quality of technologies that will result from their aggregate contributions. our approach is comprised of two general tasks. the first is a data collection task where participants compete to enter the largest amount of entries for (1) translation of an msa sentence to their dialect, and ( 2 ) answering an open-ended question in their dialect. the resulting data from this task is denoted the kind corpus. the second task is to solicit innovative systems that can compete with our approach for task 1. our approach follows a hackathon format for developing dialectal arabic data collection systems. in this paper we describe the approach for collecting the kind dataset and the quality requirements of the submissions. we demonstrate the effectiveness of our approach by describing the resulting corpora of high-quality training data. the dataset is suitable for training language models, machine translation tasks, as well as q&a tasks with the respective dialects labeled to a fine granularity level. we make this dataset publicly available to the community along with the labels for each dialect. we summarize the contributions of this paper as follows: the remainder of this paper is organized as follows. section 2 provides an overview of the related literature. section 3 describes the design and process of our collection approach. section 4 describes the results of the first task of the collection approach. section 5 describes the results of the second task of the approach. in section 6 we describe the resources resulting from this study. in section 7 we conclude. ## our approach in this section we describe our proposed approach for nuanced dialect data collection. ## data collection results in this section we describe the resulting datasets and methods. ## hackathon results in this section, we shed light on the top projects that received the highest scores from the hackathon judging committee per the evaluation criteria, which seek to balance technical knowledge with originality, creativity, and relevance to the hackathon's objectives. ## challenges and recommendations in this section we describe challenges to the collection approach and provide recommendations for organizing future hackathons. ## resources the dataset is released for the research community at: https://huggingface.co/kind-dataset . the repository holds both the data marathon submissions and the open-ended question dataset. ## conclusion in this paper we describe the design and process of collecting a multi-dialect arabic dataset as well as the resulting systems and data. similar in concept to acl shared tasks, the kind competition aims to encourage innovative contributions towards high-quality data collection. the competition resulted in a corpus of over 50k high-quality texts labeled with fine-grained arabic dialects. as well as over 20 approaches for arabic dialect crowdsourcing techniques. the resulting data is made public for the research community. as future work, the authors aim to propose new competitions for domain specific as well as nlp-task specific data collection for arabic dialects.