Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Tasks (2022)

Abstract: We present Bloom Library, a linguistically diverse set of multimodal and multilingual datasets for language modeling, image captioning, visual storytelling, and speech synthesis/recognition. These datasets represent either the most, or among the most, multilingual datasets for each of the included downstream tasks. In total, the initial release of the Bloom Library datasets covers 363 languages across 32 language families. We train downstream task models for various languages represented in the data, showing the viability of the data for future work in low-resource, multimodal NLP and establishing the first known baselines for these downstream tasks in certain languages (e.g., Bisu [bzi], with an estimated population of 700 users). Some of these first-of-their-kind baselines are comparable to state-of-the-art performance for higher-resourced languages. The Bloom Library datasets are released under Creative Commons licenses on the Hugging Face datasets hub to catalyze more linguistically diverse research in the included downstream tasks.

## introduction only a negligible fraction of the 7100+ living languages @xcite have sufficient, publicly available text, audio, and image data to train state-of-the-art language/speech models and/or models for downstream tasks like named entity recognition (ner) or image captioning. this data scarcity results in systematic inequalities in the performance of nlp tasks across the world's languages @xcite . indigenous language ecologies also represent profoundly different understandings of the nature and function of language @xcite , which might prioritize orality or translanguaging @xcite , for example, above a single, written mode of communication in all domains. the bloom library foot_0 is a web-based platform that is attempting to facilitate an increase in the amount of multimodal materials available to communities speaking non-dominant languages. the bloom library holds over 12,400 books in 545 languages (at the time this paper is published), covering subjects including agriculture, business, culture, math, science, religion, and health. many of these books include images aligned with text, and 1,600+ of the books have corresponding audio recordings (called "talking books"). language communities can create new books, create audio recordings, download existing books, and translate existing books using the open-source "bloom" software foot_1 . to boost language diversity and indigenous perspectives in the nlp research community, we present multimodal datasets post-processed out of the bloom library. we anticipate that more task-specific datasets will be created from the bloom library. however, as a starting point, we are presenting the following datasets: (1) bloom-lm for language modeling in 351 languages; (2) bloom-captioning for image-totext or text-to-image tasks in 351 languages; (3) bloom-vist for visual storytelling in 351 languages; and (4) bloom-speech for speech-totext and text-to-speech tasks in 56 languages. the languages in these datasets correspond to 32 language families, and many of the included languages are in extremely low-resource settings. further, to the authors' knowledge, bloom-vist represents the first (and certainly most) multilingual visual storytelling dataset, and bloom-speech includes more languages in the following language families than any other aligned speech dataset (number of languages in parenthesis): austronesian (8), mayan (6), niger-congo (7), sepik (2), tequistlatecan (2), and trans-new guinea (3). to assess the difficulty of language modeling, image captioning, and automatic speech recognition (asr) with the bloom library datasets, we trained baseline models on each of these tasks. for certain languages, the bloom library datasets facilitate the first known baselines with comparable to state-of-the-art performance for higher-resourced languages. we acheive a bleu score on image captioning of above 10.0 for 10 languages using only data from bloom-captioning. for asr, we demonstrate a word error rate (wer) below 0.5 for 18 languages and a character error rate (cer) below 0.2 for 21 languages. ## related work in terms of language coverage, various multilingual and single modality datasets have emerged recently. these include, by way of example, the jhu bible corpus @xcite , the cmu wilderness multilingual speech dataset @xcite , common voice 9 @xcite , multilingual babel (consortium, 2022), and mas-sive @xcite . the number of languages in these datasets is impressive. however, many are limited in domain (e.g., only including bible data), accessibility, licensing, or modality (e.g., only focusing on text or read speech). these datasets are also primarily rooted in content from large, dominant languages, like english, and are translated or adapted to other fairly large languages. bloom library data, in contrast, originates from local language communities, foot_2 which are producing bloom books to fit their own local language ecology and perspectives. as a result, the data presented here covers languages, language families, and topics that are not covered by any other aligned and prepared datasets. in terms of modality, the research community is presenting an increasing number of intriguing multimodal datasets. these include, by way of example, pano-avqa @xcite , which facilitates question answering regarding various objects, sounds, and their associations in videos, and @xcite , which facilitates sequential vision-to-language tasks. however, recent multimodal datasets are overwhelmingly monolin-gual. datasets representing both multiple modalities and many languages include multi30k, which is one of the few multimodal, multilingual datasets in existence, with ~30k images and corresponding text descriptions in several languages including english, german @xcite , french @xcite , and czech @xcite . one listing can be found in @xcite , which provides a helpful (and comprehensive) table of multilingual, multimodal resources, dividing them into two categories: (i) "translation" (with captions translated into another language); and (ii) "description" (with annotations independently created for each language). the table reveals that multi30k was, at the time, the largest translation dataset available in terms of image count, at approximately 31k images and 31k sentences covering 4 languages. the bloom library datasets fit into the "description" category of @xcite . however, with over 90k+ images and 110k+ captions covering 351 languages and additional speech data in 56 languages, bloom library represents a massive increase in language and modality coverage (up to two orders of magnitude wider than previous multilingual, multimodal datasets). further, the existing datasets referenced by @xcite focus on large languages in high-resource settings, with no representation of local languages in low resource settings. in contrast, our datasets include languages in extremely low resource and non-dominant settings like bisu @xcite and kagayanen @xcite , with estimated populations of 700 and 30,000 users, respectively. ## constructing the datasets the authors worked directly with the bloom library developers to gain access to and understand the raw data behind the bloom library website. we parse, clean, deduplicate, and publicly release this data for research use on the hugging face hub 45 in formats compatible with the hugging face datasets python package. 6 bloom-lm, bloom-captioning, and bloom-vist are created using one data pipeline starting with bloom-vist, because each of these datasets use some or all of the images and corresponding text within the bloom library. a separate data pipeline is used for bloom-speech to process only "talking books." ## dataset analysis the final statistics per language family for all the datasets are presented in our bloom-vist, bloom-lm, and bloom-captioning datasets include data from 351 of these languages across 31 language families. there is a mean of 32 stories and 319 image-caption pairs and median of 2 stories and 22 image-caption pairs per language. our bloom-speech dataset comprises 428 hours of total audio in 56 languages. there is a mean of 458 and median of 138 records per language. 18 language families are represented, with a mean of 159 minutes and median of 31 minutes of audio per language family. a full breakdown of the composition of the dataset is available in the appendix ( notably for bloom-speech, among some of the languages of wider use in the dataset (e.g. french, spanish, and english) the accent of the speaker is often not the same as in other public datasets. in common voice, a large majority of french data is labelled as coming from speakers in france. most of the french data in bloom-speech is from francophone africa. the spanish is primarily from central america, and the english includes many places where english is a language of commerce. ## baseline experiments to establish some of the first known baselines in languages included in bloom library and to assess the difficulty of language modeling, image captioning, and automatic speech recognition with the bloom library datasets, we trained baseline models for 44 included languages. the summarized results are included in ## conclusion we present 4 datasets processed out of the bloom library books. these datasets represent a massive increase in language and modality coverage for the tasks of language modeling, image captioning, visual storytelling, and speech recognition. in total, the datasets cover 363 languages across 32 language families. we demonstrated an ability to make use of the data as-is, and we foresee this dataset being used by researchers to fine-tune multilingual models (as recommended by, e.g., @xcite ), benchmark zero-shot performance for linguistically diverse languages, or train new models from scratch in combination with other datasets. in future work, we would like to continue to improve the size and quality of the dataset. we would also like to explore and understand baseline performance in the various tasks. some of the baseline performance numbers are quite low compared to numbers on established datasets, and it would be interesting to further understand the characteristics of the bloom data that make it challenging for various tasks (e.g., lexical diversity). finally, we would like to prepare aligned mutlilingual versions of the dataset that would be immediately useful for crosslingual and multilingual tasks that require parallel corpora.