I ndic LLMS uite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for I ndian Languages (2024)

Abstract: Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. Our work aims to bridge this divide by introducing an expansive suite of resources specifically designed for the development of Indic LLMs, covering 22 languages, containing a total of 251B tokens and 74.8M instruction-response pairs. Recognizing the importance of both data quality and quantity, our approach combines highly curated manually verified data, unverified yet valuable data, and synthetic data. We build a clean, open-source pipeline for curating pre-training data from diverse sources, including websites, PDFs, and videos, incorporating best practices for crawling, cleaning, flagging, and deduplication. For instruction-fine tuning, we amalgamate existing Indic datasets, translate/transliterate English datasets into Indian languages, and utilize LLaMa2 and Mixtral models to create conversations grounded in articles from Indian Wikipedia and Wikihow. Additionally, we address toxicity alignment by generating toxic prompts for multiple scenarios and then generate non-toxic responses by feeding these toxic prompts to an aligned LLaMa2 model. We hope that the datasets, tools, and resources released as a part of this work will not only propel the research and development of Indic LLMs but also establish an open-source blueprint for extending such efforts to other languages.

## introduction building large language models (llms) is an inherently data-intensive process requiring a comprehensive set of resources for pre-training (raffel * equal contribution. † work done during an internship at ai4bharat ‡ corresponding author: miteshk@cse.iitm.ac.in @xcite @xcite @xcite and fine-tuning @xcite @xcite . the last year has seen remarkable progress in building english llms, thanks to open-source models @xcite @xcite developed using comprehensive datasets containing such resources. nonetheless, this progress has largely bypassed low and mid-resource languages due to the the lack of data resulting from the lack of open source pipelines for curating data for such languages from diverse sources such as websites (which require crawling and extraction), books (which require ocr) and videos (which require transcription). further, for instruction fine-tuning, english llms now rely on model-generated data such as sharegpt foot_0 , self-instruct @xcite , evol-instruct @xcite , ultra-chat @xcite , etc. however, for low and mid resource languages this option is not available due to lack of high quality llms, leading to a chicken and egg problem, further widening the gap between the haves and the have-nots. a case in point is that of languages from the indian sub-continent which collectively are spoken by over 1.4 billion people. we focus on the 22 languages recognised in the 8th schedule of the indian constitution. these languages, despite their significant number of speakers, receive minimal representation in the training datasets and tokenizers of current open-source llms @xcite @xcite leading to a notable exclusion of their rich cultural contexts and nuances. in this work, we address this disparity by making the following contributions: 1. sangraha: pretraining data containing 251b tokens foot_1 summed up over 22 languages extracted from curated urls, existing multilingual corpora, and large-scale translations. 2. setu: spark-based (zaharia et al., 2016) distributed pipeline customised for indian languages for extracting content from websites, pdfs and videos, with in-built stages for cleaning, filtering, toxicity removal and deduplication. 3. indicalign -instruct: a diverse collection of 74.7 million prompt-response pairs across 22 languages collected through four methods: aggregating existing instruction fine-tuning (ift) datasets, translating english datasets into 14 indian languages using an open-source translation model, creating context-grounded conversations from india-centric wikipedia articles using open source llms, and establishing a crowdsourcing platform called anudesh for prompt collection. 4. indicalign -toxic: 123k pairs of toxic prompt and non-toxic responses generated using open source english llms and translated to 14 indian languages for safety alignment of indic llms. we collectively refer to the above as indicllm-suite. we try to balance quality and quantity while acknowledging recent trends of using synthetic data for building powerful llms for english @xcite as well as low resource languages @xcite . to ensure quality, we take help from humans to verify websites to flag noisy or machine translated content and to create toxicity lists for indian languages. on the other hand, to ensure explicit representation of prompt-response pairs grounded in indian context we take the help of indian languages and has an average fertility of 1.3 to 2.79 across the 22 languages. we use this tokenizer for all the reported statistics unless mentioned otherwise. powerful open source llms to generate grounded conversations from india-centric wikipedia articles. we recognize the need to represent diverse knowledge and alignment information in indic languages for better performance of llms in indic languages. hence, we undertake large-scale machine translation of rich english resources like wikipedia as well as english finetuning datasets into indian languages using sota open-source mt models. we thus balance source original data with translated and llm-generated data to create the above collection. we believe that these choices can be replicated across other languages to create llmsuites. all the code, tools and datasets developed as a part of this work will be publicly released and hopefully advance the development of llms for indian languages. given that llm training is an expensive exercise, we plan to undertake community-effort to train llms, where multiple groups can pool together computing resources to build a high-quality indic language llm. ## related works we organise the related work into 3 sections in line with our main contributions. multilingual datasets. previous works like os-car @xcite , cc100 @xcite , and mc4 @xcite are curated from commoncrawl dumps through extensive cleaning stages. @xcite extends to 419 languages, incorporating human audits and iterative refinement, along with language family-specific filters. roots @xcite and @xcite combined existing datasets and used strict clean-ing procedure to create data in 59 and 167 languages respectively. in contrast, we curate a list of verified websites for indian languages and use a pipeline specifically built for indian languages to clean up data from some of the existing collections listed above. in addition to html documents from web crawls, we also include pdfs and speech transcripts from a variety of sources. data curation pipelines. for scraping websites we rely on popular tools like trafilatura (barbaresi, 2021a) and justext @xcite which are widely used in recent works @xcite . for language identification, in addition to popular tools such as, cld3 foot_2 , langdetect foot_3 , fasttext @xcite , sslid @xcite , we use a custom indic language lid tool @xcite . for ensuring high quality of the processed data recent works have proposed rule-based approaches @xcite @xcite , approaches which rely on kenlm perplexity @xcite @xcite and machine learning-based approaches @xcite . in this work, we largely rely on rule based approaches. similarly, for filtering toxic content we use word lists @xcite , blocklists of urls @xcite , and language specific heuristics @xcite . lastly, deduplication @xcite @xcite is important to reduce memorisation @xcite and improve llm performance @xcite , especially at scale @xcite . in this work, we perform deduplication based on urls, and fuzzy techniques like minhash @xcite . supervised fine-tuning datasets. existing ift datasets either contain human-human interactions such as dolly @xcite and @xcite , human-gpt interactions such as sharegpt 5 and wildchat @xcite or llm-generated instructions @xcite typically bootstrapped with a few prompts, such as alpaca @xcite , wizardlm @xcite , @xcite , baize @xcite amongst others. multilingual instruction tuning datasets are created from english ift datasets us- ## sangraha in this section we describe the composition and curation of sangraha spanning verified (64b), unverified (24b) and synthetic (162b) content for a total of 251b tokens.