Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning (2024)

Abstract: Datasets are foundational to many breakthroughs in modern artificial intelligence. Many recent achievements in the space of natural language processing (NLP) can be attributed to the fine-tuning of pre-trained models on a diverse set of tasks that enables a large language model (LLM) to respond to instructions. Instruction fine-tuning (IFT) requires specifically constructed and annotated datasets. However, existing datasets are almost all in the English language. In this work, our primary goal is to bridge the language gap by building a human-curated instruction-following dataset spanning 65 languages. We worked with fluent speakers of languages from around the world to collect natural instances of instructions and completions. Furthermore, we create the most extensive multilingual collection to date, comprising 513 million instances through templating and augmenting existing datasets across 114 languages. In total, we contribute three key resources: we develop and open-source the Aya Dataset, the Aya Collection, and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case study in participatory research, involving collaborators from 119 countries. We see this as an important framework for future research collaborations that aim to bridge gaps in resources.

## introduction datasets are static representations of the world, far from the rich, ever-evolving environment we navigate as humans. yet, these frozen snapshots in time are the foundation upon which progress in ai has been built. many recent breakthroughs in language modeling can be attributed to fine-tuning pre-trained models on a diverse set of tasks that enable a large language model (llm) to follow instructions @xcite @xcite @xcite . instruction fine-tuning (ift) leverages the precept that natural language processing (nlp) tasks can be described via natural language instructions, such as "what were the reviews like for the barbie movie?". the factors underlying the construction of the datasets impact how models perform for users around the world. models perform better on the distribution they are trained to mimic @xcite . this often introduces known biases towards languages and dialects not included during training and introduces critical security flaws @xcite @xcite @xcite . in this work, our goal is to reduce this linguistic inequality. efforts that aim to improve multilingual performance have often focused on improving data coverage @xcite . however, most of the limited effort to date has focused on multilin- ## aya dataset the goal of the aya project is to facilitate annotations to a crowd-sourced dataset by individuals fluent in different languages. inputs from speakers of each language ensure that the dataset is more likely to be organic, articulate, and representative of the speakers' cultures. the aya project was initiated to provide annotations for the 101 languages available in the mt5 model @xcite . ultimately, some of these languages did not receive enough contributions to include them in the final dataset. conversely, we received substantial contributions from languages not initially part of the original list, like wolof, leading to their inclusion; the final aya dataset covers 65 languages. table e .4 provides details of these languages. ## aya collection we introduce the aya collection, a comprehensive, large corpus of datasets that can be used by researchers around the world to train multilingual models. our goal is only to include datasets with permissive licensing for manipulation and redistribution. 6 the aya collection consists of three different sources of data: 1 templated data: we collaborated with fluent speakers to create templates that allowed for the automatic expansion of existing datasets into various languages. 2 translated data: we translated a hand-selected subset of 19 datasets into 101 languages (114 dialects) using the nllb 3.3b parameter machine translation model @xcite . the full list of datasets translated is listed in table e .8. 3 aya dataset: we release the aya dataset described in section 2 as a subset of the overall collection. it is the only dataset in the collection that is humanannotated in its entirety. ## aya evaluation suite lastly, we release an evaluation suite tailored to multilingual models. this set aims to tackle the scarcity of multilingual data, a challenge that becomes even more apparent when considering evaluation sets. while there are several test sets available for evaluating multilingual models @xcite @xcite @xcite , they focus primarily on discriminative tasks or on regional subgroups of languages. to evaluate multilingual models' generations, the literature includes task-specific evaluation sets such as translation @xcite , summarization @xcite and question answering @xcite , as well as combinations thereof @xcite . however, there exists a gap in evaluating open-ended generation capabilities of llms within a multilingual context. we aim to address this gap by curating a multilingual evaluation set tailored for assessing the open-ended generation capabilities of llms, such as brainstorming, planning, and other unstructured, long-form responses. to strike a balance between language coverage and the quality that comes with human attention, we create an evaluation suite that includes (1) human-curated examples in a limited set of languages, (2) automatic translations of handpicked examples into a more extensive number of languages, and (3) human-post-edited translations into a small number of languages. we consider two primary sources of data: original annotations from aya dataset (comprising new examples culturally curated for different languages) and dolly prompts (high-quality, human-written examples carefully selected to have a universal reach). the subsets comprising the aya evaluation suite are: aya-human-annotated test set. we partitioned the aya dataset into train and test splits. the test set of the aya dataset contains 1,750 of the total instances (250 instances from 7 languages), randomly selected from original annotations. to guarantee enough data remains for training, we focused on languages with at least 2000 original annotations. to ensure linguistic diversity, we included languages that were varied in terms of resourcedness, script and language family. for these reasons, the test set consists of english (high-resource, latin script, indo-european), portuguese (highresource, latin script, indo-european), simplified chinese (high-resource, han, sino-tibetan), standard arabic (high-resource, arabic script, afro-asiatic), telugu (low-resource, telugu script, dravidian), turkish (high-resource, latin script, turkic), and yoruba (low-resource, latin script, atlantic-congo). ## conclusion open participatory research continues to be underresourced and undervalued, particularly when that work focuses on data creation @xcite . aya involved participants from many different countries, different ages, and different levels of familiarity with the field of natural language processing. we see continued opportunities for computational linguists and machine-learning engineers to collaborate with social scientists such as sociolinguists, anthropologists, sociologists, and media studies scholars. as new norms in open science emerge @xcite , collaborations like these can help ensure that projects in nlp are motivated by an understanding of what language means to the people who use it every day. with aya we hope to change the way data is created for multilingual nlp research. in line with this view, we release the aya dataset which is the first human-curated open-source, multilingual instruction-style dataset consisting of 204,114 prompt-completion pairs covering 65 languages. this dataset was built with the help of our openscience community of 2,997 collaborators from 119 countries over a period of eight months. we also release the aya collection , which consists of 44 instruction-style datasets. these were prepared by transforming existing nlp datasets into prompt-completion pairs that can be leveraged for instruction tuning. furthermore, we translate several high-quality datasets into 101 languages, thereby expanding coverage, particularly for many low-resource languages. this collection consists of 513m prompt and completion pairs covering 114 languages in total and is the largest multilingual instruction fine-tuning collection today. additionally, we release aya evaluation suite , consisting of human-curated examples in 13 languages and translation of carefully selected prompts in 101 languages.