Community OSCAR : A Community Effort for Multilingual Web Data (2024)

Abstract: The development of large language models (LLMs) relies heavily on extensive, high-quality datasets. Publicly available datasets focus predominantly on English, leaving other language communities behind. To address this issue, we introduce Community OSCAR, a multilingual dataset initiative designed to address the gap between English and non-English data availability. Through a collective effort, Community OSCAR covers over 150 languages with 45 billion documents, totaling over 345 TiB of data. Initial results indicate that Community OSCAR provides valuable raw data for training LLMs and enhancing the performance of multilingual models. This work aims to contribute to the ongoing advancements in multilingual NLP and to support a more inclusive AI ecosystem by making high-quality, multilingual data more accessible to those working with low-resource languages.

## introduction the success of large language models (llms) hinges on access to vast amounts of high-quality data. the exact composition, procurement, and curation of this data has been one of the more closely guarded secrets of commercial llms. recently, academic and open-source efforts have made significant strides in curating and refining large-scale corpora for english @xcite @xcite . these data-driven efforts are central to advancing open-source and transparent llm initiatives. nonetheless, a strong disparity remains between the availability of english-language datasets and those for other languages. we argue that access to high-quality data is imperative for ensuring linguistic diversity, academic and economic competitiveness, and ai sovereignty for non-english countries and speakers. however, clean, multilingual datasets like culturax @xcite , for example, can only provide 100b+ tokens for less than ten languages. to bridge this gap, we introduce community oscar, a publicly available multilingual dataset that covers over 150 languages and includes over four times as much data as previous corpora. the creation of community oscar is a collective, community-driven effort, highlighting the importance of collaboration in addressing the challenges of data scarcity for non-english languages 1 . by expanding the availability of non-english data, community oscar seeks to democratize access to resources essential for building inclusive, multilingual ai systems. our initial results indicate that community oscar provides valuable raw data for downstream llm training. ## community oscar as the name suggests, community oscar builds on prior work of the oscar corpus @xcite . we went ahead and extended these efforts. oscar. the oscar project (open super-large crawled aggregated corpus) aims to provide open-source, web-based multilingual resources. ## outlook the release of community oscar now enables further progress in multilingual language modeling. we are actively working on extending the dataset to at least all available cc dumps, curating a high-quality subset from the raw data, and training llms on that data. all three steps yield good initial results, which we will discuss in the following section. specifically, we conducted initial experiments with subsets of the data and plan to extend our insights to the rest of the dataset.