I ndic V oices: Towards building an Inclusive Multilingual Speech Dataset for I ndian Languages (2024)

Abstract: We present INDICVOICES, a dataset of natural and spontaneous speech containing a total of 7348 hours of read (9%), extempore (74%) and conversational (17%) audio from 16237 speakers covering 145 Indian districts and 22 languages. Of these 7348 hours, 1639 hours have already been transcribed, with a median of 73 hours per language. Through this paper, we share our journey of capturing the cultural, linguistic and demographic diversity of India to create a one-of-its-kind inclusive and representative dataset. More specifically, we share an open-source blueprint for data collection at scale comprising of standardised protocols, centralised tools, a repository of engaging questions, prompts and conversation scenarios spanning multiple domains and topics of interest, quality control mechanisms, comprehensive transcription guidelines and transcription tools. We hope that this open source blueprint will serve as a comprehensive starter kit for data collection efforts in other multilingual regions of the world. Using INDICVOICES, we build IndicASR, the first ASR model to support all the 22 languages listed in the 8th schedule of the Constitution of India.

## introduction recent advancements in automatic speech recognition (asr) have achieved remarkable success in english @xcite @xcite @xcite , with single digit wers on multiple benchmarks @xcite @xcite @xcite . however, despite several massively multilingual efforts such as whisper @xcite , usm figure 1 : primary regions of india where each of the 22 languages is spoken. @xcite and @xcite , the progress on mid and low resource languages is not at par with english. the primary reason for this is the lack of sufficient, diverse and high quality training data in these languages. in this study, we address this problem and set out with an ambitious goal to collect spontaneous speech data for indian languages while respecting the linguistic, cultural and demographic diversity of india. the scale of the problem is highlighted by the fact that there are 22 languages listed in the 8th schedule of indian constitution, belonging to 4 different language families. these languages have a dataset #l #hours #sp #d type channel ub lb r e c wb nb fleurs 13 -163 --✓ ✗ ✗ ✓ ✗ (conneau et al., 2022) msr 3 -150 1286 1 ✗ ✗ ✓ ✗ ✓ (srivastava et al., 2018) openslr 3 -618 1513 -✓ ✗ ✗ ✓ ✗ (kjartansson et al., 2018) cms 6 -35 243 1 ✓ ✗ ✗ ✓ ✗ (he et al., 2020) mucs 3 -351 158 4 ✓ ✗ ✗ ✓ ✗ (diwan et al., 2021) kathbath 12 -1684 1218 3 ✓ ✗ ✗ ✓ ✗ (javed et al., 2023a) shrutilipi 12 -6457 --✓ ✗ ✗ ✓ ✗ (bhogale et al., 2023a) graamvaani 2 1 1000 108 --✗ ✗ ✓ ✗ ✓ iisc-mile 2 -500 1446 -✓ ✗ ✗ ✓ ✗ (a et al., 2022b,a) kdc 3 1 -1 43 -✗ ✗ ✓ ✗ ✓ vāksañcayah 1 -78 27 8 ✗ ✗ ✗ ✓ ✗ (adiga et al., 2021) iiith-isd 7 -11 35 1 ✓ ✗ ✗ ✓ ✗ (prahallad et al., 2012) iitb-msc 1 -109 36 1 ✓ ✗ ✗ ✓ ✗ (abraham et al., 2020) smc-msc 4 1 -2 75 4 ✗ ✗ ✗ ✗ ✗ iitm 5 3 -690 --✓ ✗ ✗ ✓ ✗ nptel 8 -857 -1 ✓ ✗ ✗ ✓ ✗ (bhogale et al., 2023b) indictts 13 -225 25 4 ✓ ✗ ✗ ✓ ✗ (201, 2016) svarah 1 -10 117 37 ✓ ✓ ✓ ✓ ✗ (javed et al., 2023b) spring-inx 10 -2005 7609 16 ✓ ✓ ✓ ✓ ✓ (r et al., 2023) spire-sies 1 171 23 1607 -✗ ✓ ✗ ✓ ✗ (singh et al., 2023) indicvoices 22 7348 1639 16237 52 ✓ ✓ ✓ ✓ ✓ to collect data which is inclusive and representative of this diversity of india, we ensure a comprehensive representation across various demographics such as gender, age, educational background, and geographic location, with specific quotas for each category. additionally, we aimed for diversity in the vocabulary and content, including a mix of read speech, voice commands, extempore discussions, and both wide and narrow-band recordings. we also aim to maintain a balanced representation of urban and rural speakers and recording devices and include a significant portion of data recorded in noisy environments representing everyday usage of asr systems. this is indeed a one-of-its-kind effort ensuring inclusion (see figure 2 ) while covering a large number of languages, districts, domains/topics with different recording channels and speech styles (see to achieve this, we created a clear framework for data collection which can be replicated across languages and diverse geographical locations. first, we created a frontend mobile application to enable data collection in a remote and distributed setup using a standardised procedure. second, to elicit meaningful responses from participants capturing local contexts and culture, we created a repository of 2.5k engaging questions, 46.6k prompts, 1.1k (dogri) to 4.1k (hindi) role-play scenarios covering 21 domains and 28 topics of interest, anchored in practical, day-to-day usage scenarios. to facilitate data collection, we set up a countrywide network consisting of agencies, local universities, local ngos, and social sector professionals who acted as regional influencers to engage a variety of participants. we also built an in-house quality control team who listened to every audio collected from the field and ensured adherence to a stringent acceptance criteria. lastly, we built a team of transcribers consisting of makers, checkers and supercheckers and created an elaborate two-level transcription guideline addressing the unique transcription challenges in indian languages. indicvoices is the result of this massive effort involving a total of 1893 personnel in a variety of roles such as language experts, local mobilisers, coordinators, quality control experts, transcribers, language leads and project managers. it contains a total of 7348 hours of read (9%), extempore (74%) and conversational (17%) audio data from 16237 speakers covering 145 districts and 22 languages, of which 1639 hours have already been transcribed (the rest is under progress). in addition to asr, the collected data can be used for several other purposes, such as speaker diarization, speaker identification, speaker verification, language identification, intent detection, entity extraction, query by example and audio denoising. we hope that all the data, tools, guidelines and other material created as a part of this work serve as an open source framework for data collection projects in other multilingual regions of the world. all artefacts developed in this work are open-source and can be accessed on github 6 . in the remainder of this paper, we describe the four stages (figure 3 ) in our process, viz., (i) preparation before data collection (ii) on-field data collection (iii) quality control and (iv) transcription. we then present statistics about the data and train/evaluate existing models on indicvoices. ## pre-collection data preparation simple prompts often fail to engage participants effectively, necessitating the creation of structured sentences, questions, and scenarios for participants to read, respond to, or act out, as described below. ## on-field data collection we now describe the process of collecting data. ## quality control we set up an in-house quality control team comprising 3-5 experts per language for verifying the meta-data, diversity criteria and content. verifying meta-data: in the early stages of data collection, discrepancies between participants' voices and their registered age or gender information were noted, sometimes due to intentional misinformation or errors in form completion. additionally, inconsistencies in voice across a participant's submissions suggested data contributions from multiple individuals under a single id or instances of a single individual posing as multiple participants. to address these authenticity concerns, we introduced a video recording micro-task for participants to verbally confirm their details, allowing our quality control (qc) team to verify age, gender, and voice consistency visually and auditorily. discrepancies led to data rejection. we added privacy safeguards ensuring videos were not downloadable, accessible only to female members of the qc team and promptly deleted post-verification. recognizing cultural sensitivities, especially among female participants hesitant to record videos, we facilitated alternative live verification through a whatsapp call with a female qc member. verifying diversity criteria: the qc team was responsible for checking that we meet all the diver-figure 7 : top: percentage of data that was rejected by our qc team for each language. the higher numbers are for languages where multiple pilots were done before we could find the right local partners. bottom: distribution of errors across the 23 categories. sity criteria outlined in figure 2 . in case of a gap, the local partners were asked to source participants from specific underrepresented categories. the qc team also ensured that there was enough diversity in the domains and topics across participants. to ensure a broad range of topics, the qc team addressed the tendency of participants to choose familiar subjects like "sports" and "entertainment" by advising local partners to encourage discussions on less represented domains and topics. lastly, the qc team would also monitor that easy conversation scenarios such as "teacher-student" or "customershopkeeper" were not repeated by disabling such options in the registration form. figure 8 shows the diversity in content across topics and domains summed up across the 22 languages. verifying content: during initial data collection, our qc team noted issues with audio quality that were not caught by the first level of quality control conducted by coordinators. problems included audio files with low volume and others with significant background noise, which compromised intelligibility. it was hypothesized that these issues slipped through due to coordinators' unfamiliarity with the context, allowing them to understand the content despite the poor audio quality. to address this, the qc team developed a detailed list of 23 error categories as shown in figure 7 (also see ## transcription we now discuss the process of transcription, focusing on the guidelines and the workflow used. transcription guidelines: in indian languages, there is often a significant divergence between spoken and written forms due to colloquialisms, accent variations, mispronunciations, and contractions in rapid everyday speech (see appendix h. this issue necessitated a two-level transcription approach. in level 1, we perform verbatim transcription, capturing spoken language as it is, despite deviations from standard written forms. in level 2, we standardized transcription, ensuring that all transcriptions conform to textbook representations, regardless of spoken variations, thereby improving usability in downstream applications. given the high correspondence between grapheme and phoneme sets for indian languages, level 1 transcription guidelines simply require the transcribers to listen to the sound and transcribe it using the corresponding grapheme. however, level 2 guidelines required more careful consideration, for which we collaborated with linguists to develop detailed rules for standardization. for each of the 22 languages, we crafted guidelines incorporating examples from the collected audio, undergoing several rounds of review to achieve consensus and finalize the guidelines, as detailed in appendix h. we release both level 1 and level 2 transcripts giving researchers the flexibility to use as per their needs. transcription workflow: to maintain transcription quality, we employed a three-tier review system: maker-checker-superchecker. this process involved an extensive team of remote and external transcribers alongside experienced in-house super-checkers. the workflow, supported by our in-house tool, facilitated task assignment and monitoring across all 22 languages. the initial transcriptions (level 1) were handled by the broader maker-checker team, while the super-checkers, as specialists, conducted secondary reviews and managed the transition to standardized level 2 transcriptions, ensuring high accuracy and consistency. ## indicvoices we collected a total of 7348 hours of data summed up across the 22 languages, covering 145 districts. of these, 1639 hours of data have already been transcribed, with the rest in the pipeline. figures 7 to 12 summarise different statistics of the collected data, which showcase its rich diversity. creating train-test splits: while creating traintest splits ( by example, crosslingual transfer in multilingual asr systems, multimodal speech-text embeddings. it can also be used for the evaluation of zero-shot transfer across districts with new accents. given the episodic nature of our data collection, it can also be used for training and evaluating continual learning methods for asr. this comprehensive dataset, thus, holds the potential to significantly advance a broad spectrum of speech and language processing tasks for multiple low-resource indian languages. ## conclusion we presented indicvoices, a comprehensive speech dataset adhering to an inclusive diversity wishlist with fair representation of demographics, domains, languages and applications. the materials developed as a part of this work will serve as a good starter kit for data collection for other languages. these materials include (i) wikipedia sentences from multiple domains (ii) prompts for digital interactions, (iii) questions from different domains and topics of interest, (iv) conversational role-play scenarios (v) elaborate transcription guidelines (vi) android application for on-field data collection and verification and (vii) a web based platform for transcription workflow management. these can be adapted with minimal effort to bootstrap large scale data collection efforts for other languages.