Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024)

Abstract: Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages. What does it take to broaden access to breakthroughs beyond first-class citizen languages? Our work introduces Aya, a massively multilingual generative language model that follows instructions in 101 languages of which over 50% are considered as lower-resourced. Aya outperforms mT0 and BLOOMZ on the majority of tasks while covering double the number of languages. We introduce extensive new evaluation suites that broaden the state-of-art for multilingual eval across 99 languages —— including discriminative and generative tasks, human evaluation, and simulated win rates that cover both held-out tasks and in-distribution performance. Furthermore, we conduct detailed investigations on the optimal finetuning mixture composition, data pruning, as well as the toxicity, bias, and safety of our models.

## introduction a fundamental question in machine learning is how to effectively capture the nuances of the long tail. datasets have been the foundation of modern machine learning progress, but have coalesced around a few data-rich languages. which languages are favored is often a symptom of historical technological use and access to resources, rather than the languages most frequently spoken or written in the real world @xcite . recent breakthroughs in natural language processing (nlp) have been no different, with the instruction-following capabilities of existing opensource models, such as alpaca @xcite , dolly @xcite , and vicuna @xcite , mainly developed for english tasks. instruction finetuning (ift) involves curating pairs of prompts and completions, and has been shown to significantly improve the helpfulness and general instruction following capabilities of large language models (llms) @xcite @xcite @xcite . however, a sizable gap between the available amount of instruction prompts for english and all other languages exists -an astounding 73% of popular ift datasets are primarily english @xcite . bridging this widening language gap and conferring multilingual instruction-following capabilities is not a trivial problem. some multilingual abilities can be inherited by pretraining on diverse multilingual data (brown et al., 2020) -often described as surprising multilingual abilities noted in finetuned models @xcite which are not explicitly finetuned to be multilingual @xcite . however, this was not proven to be competitive with a second direction of both pretraining and instruction finetuning with a multilingual corpus. pursuing this second approach has been the subject of several recent works @xcite @xcite @xcite where the persistent struggle to secure comprehensive multilingual ift datasets remains a fundamental obstacle. in this work, we address several core limitations of recent multilingual ift models in order to reduce their linguistic inequality: we aim to create a model that performs well on downstream tasks when given prompts in any of the included languages, rather than requiring multilingual speakers to write prompts in english. we greatly expand the coverage of languages to 101, far beyond the current coverage of open-source massively multilingual models. to do so, we embark on an ambitious effort to expand the size of the training corpus as well as the breadth of evaluation. the core contribution of our work is an open-source multilingual instruction-finetuned llm with diverse linguistic representation: the aya model. our primary contributions can be enumerated as follows: by releasing the aya model, we hope to empower researchers and practitioners to advance multilingual models and applications. aya model is available with a fully open-source apache 2.0 license foot_2 here: https://hf.co/cohereforai/aya-101 . ## evaluation to measure our models' performance on various tasks and many languages, we create an extensive multilingual evaluation suite ( ## conclusion language representation is a consequence of the choices made and resources spent by the development community. we choose to tackle the widening gap both in who creates, and who is represented by modern language models. representing 101 languages, we more than double the languages covered in instruction finetuning, evaluation, and safety. we will release our model under fully permissive, open-source compliant licenses, to further our mission of multilingual technologies empowering a multilingual world. the aya model vastly improves over all massively multilingual, open-source models, across a battery of automatic and human evaluation settings. we expand the axes of evaluation to shed light on multilingual capabilities, both for aya, and for future development projects. we transparently characterize model biases, toxicity, and harm across languages to raise the bar of multilingual safety evaluations. we intend for this work to empower accessible future research, but also to set a new course in what constitutes ambitiously representative language model development.