BIG - C : a Multimodal Multi-Purpose Dataset for B emba (2023)

Abstract: We present BIG-C (Bemba Image Grounded Conversations), a large multimodal dataset for Bemba. While Bemba is the most populous language of Zambia, it exhibits a dearth of resources which render the development of language technologies or language processing research almost impossible. The dataset is comprised of multi-turn dialogues between Bemba speakers based on images, transcribed and translated into English. There are more than 92,000 utterances/sentences, amounting to more than 180 hours of audio data with corresponding transcriptions and English translations. We also provide baselines on speech recognition (ASR), machine translation (MT) and speech translation (ST) tasks, and sketch out other potential future multimodal uses of our dataset. We hope that by making the dataset available to the research community, this work will foster research and encourage collaboration across the language, speech, and vision communities especially for languages outside the “traditionally” used high-resourced ones. All data and code are publicly available: [ https://github.com/csikasote/bigc ]( https://github.com/csikasote/bigc ).

## introduction the bemba language, spoken by over 10 million people in zambia and other parts of africa, is a rich and vibrant language with a unique cultural heritage. however, despite its significance, bemba is a dramatically under-resourced language, lacking in high-quality language data and resources for natural language processing (nlp) experiments and for the development of language technologies. with this work, we address this issue by creating a new multimodal dataset for bemba. our goal is to improve the accuracy and effectiveness of nlp systems for speakers of bemba and support research in this under-served language. while most datasets are constructed with a specific task in mind and tailored to its characteris1 all data and code are publicly available: https:// github.com/csikasote/bigc . umwana alepeluka pa cimpelwa elo ba nyina pa mbali balemutamba a child is on a swing chair swinging while the mother watches on the side akamwana kali akasansamuka saana,kalelosha ba nyina nakasekelela. the child is very excited for swinging, looking at its mother with a smile. cipalile pali efyo balekeba ba nyina pantu bonse bali abasekelea it seems like she is telling the child something as they both have smiled. limbi kaleumfwafye bwino filyafine kalepeluka nabanyina balekasunka. it may be excited for swinging even as the mother swings it. awe cine abana balatemwa banyina, insansa muli uyu umwana shafulisha indeed children love their mothers, the child is just too excited. tics, we aim to provide a path towards building multi-purpose datasets. under a limited budget, we hypothesize that the ideal scenario is to create datasets that can be useful for developing multiple language technologies for both practical applications and also facilitate cutting-edge nlp research on many dimensions. our hope is that such datasets will aid in bridging the ever-widening language divide both in terms of data availability @xcite and nlp research @xcite , and make language technologies more accessible for speakers of bemba. in this work, we present our methodology and results of creating a new multimodal dataset for bemba, and demonstrate the potential of this dataset to develop nlp systems and support nlp research. our dataset will fill multiple roles: enable development of fundamental tools such as speech recognition, speech and text translation systems for bemba; serve as a benchmark for academic and in-images text audio dataset (#unique) (turns) (hours) languages(s) parallel task: image captioning mscoco @xcite 330k 1.5m -eng na flickr8k audio @xcite 8k 40k 65 eng na flickr30k @xcite 30k 158k -eng na pascal sentences @xcite 1k 10k -eng, jap partial iapr tc-12 @xcite 1k 10k -eng, deu, spa no multi30k @xcite @xcite dustry research even as nlp for low-resource and under-represented african languages gets developed; facilitate research in language grounding and multimodal model development, or building context-based dialogue agents, among other possible use cases. to our knowledge this is the first such dataset of its kind for any zambian and possibly african language. we hope that it will provide an example of how to create a multi-purpose dataset in an under-served language to facilitate its coverage by multiple technologies. the rest of the paper is structured as follows: in section 2, we briefly introduce the bemba language discussing any currently available resources. in section 3, we summarise work related to multimodal tasks and existing datasets. in section 4, we provide a description of the big-c dataset and the methodology used, and in section 5, we provide baseline experiments for some nlp tasks. ## the bemba language bemba, also known as icibemba or cibemba, is a bantu language native to luapula, muchinga and northern provinces of zambia. it is also spoken in other urban parts of the country like copperbelt, central and lusaka provinces. it is estimated that bemba is spoken by over 30% of the population of zambia as either the first or second language, making it the language with the most speakers in the country @xcite . a map of bemba usage in zambia is provided in appendix figure 3 . the bemba language has a number of dialects and the main varieties are: standard bemba also central bemba, aushi, bisa, chishinga, lamba, lala, luunda, ngumbo, swaka, tabwa and unga. these dialects show minor differences in phonology, morphology and vocabulary @xcite . in this work, we focus on the standard bemba dialect, i.e., the one spoken in urban centers around the country. datasets for bemba for asr, to the best of our knowledge, there is only a single dataset publicly available for bemba, bembaspeech (sikasote and anastasopoulos, 2022). it contains 24 hours of read-styled speech data recorded from text mainly sourced from various source but mainly literature books. the low resource nature of the be-mbaspeech @xcite dataset makes it difficult to build usable asr system for bemba. for machine translation (textto-text), there is not a single dedicated dataset for bemba. however, there exist some parallel text-to-text data in multilingual datasets such as jw300 (željko agic and vulic, 2020) and in evaluation benchmarks such as ntrex-128 @xcite and @xcite . the text in the jw300 (željko agic and vulic, 2020) is mostly religious as it is derived from the bible text. for speech translation (speechto-text; st), to our knowledge, no prior work or bemba dataset exists. this essentially renders it impossible to build a st system where bemba is a source or target language. the same is true for multimodal and dialogue datasets: there is no multimodal or dialogue-related dataset for any zambian language that would enable development of multimodal systems. our work aims to fill these gaps. ## related work in the recent years, nlp, speech processing (sp) and computer vision (cv) fields have rapidly advanced, with computational models' performance achieving new heights on a wide range of downstream tasks. this, to some degree, can be attributed to factors such as the emergence of pre-trained models leveraging self-supervised learning, the availability of large-scale datasets, and increased large-scale computational infrastructure @xcite . in nlp, language models like bert @xcite , t5 @xcite , @xcite and xlm-r @xcite , pretrained on massive text datasets such as c4 (raffel et al., 2020), mc4 @xcite and bookscorpus @xcite among others, have lead to significant performance improvements on several language understanding and generation downstream tasks. likewise, for speech processing, the unsupervised pretraining of models like wav2vec2.0 @xcite or xls-r @xcite -having been pretrained on publicly available speech datasets such as voxpopuli @xcite , mls (pratap et al., 2020), commonvoice @xcite , babel @xcite among others, have led to advances on speech downstream tasks like asr @xcite and st. in computer vision, deep learning models like deepcnn @xcite have become the de facto solution for standard vision problems like object recognition @xcite , image classification @xcite , or semantic segmentation @xcite . since these neural models are conceptually (and architecturally) quite similar they have also enabled the integration of multiple modalities, with models such as vilbert @xcite , uniter @xcite , unicoder-vl @xcite able to jointly model the relationship between text and image modalities resulting into breakthroughs across a myriad of tasks such as imagetext retrieval/search @xcite , image or video captioning @xcite , and vision-question answering (vqa; @xcite . a crucial necessary component for all of the above, of course, is the availability of relevant datasets. below we discuss works that go beyond the collection of raw datasets that are used for self-supervised learning. dialogue in the recent past, a lot of work has been focused on dialogue datasets. on one hand there exist goal-oriented dialogue datasets, such as the case of the ubuntu dialogue corpus @xcite , the largest corpus of dialogues (almost 1 million mainly 3-turn dialogues in english) for the specific topic of troubleshooting ubuntu problems. on the other hand, open ended conversations, such as those on the call-home/callfriend @xcite or fisher corpora @xcite , often leads to uninteresting conversations. grounding the dialogue to event-centric images and potentially a specific scenario constrains the topic of conversation to event-rich and contentful utterances. multimodality multimodal works combining visual and language information typically focus on image captioning and visual question answering @xcite . for example, the iapr tc-12 dataset @xcite provides images with titles and descriptions (mostly in english, german, and spanish), as do commonly used datasets like mscoco @xcite and @xcite . flickr8k audio @xcite extended a subset of the flickr images with audio, by crowdsourcing readings of the english captions, while multi30k @xcite further extended flickr30k with german translations and annotations. wikipedia-based image text (wit) dataset @xcite provided large multilingual coverage (108 languages) based on 11.5m images and captions from wikipedia. more recent, hausa visual genome (havg; abdulmumin et al., 2022) provided over 30k parallel descriptions in english and hausa of images from the hindi visual genome (hvg; @xcite . the dataset was created by automatically translating the english descriptions of the images in the hvg to hausa using google translate foot_0 and postedited by crowd-sourced hausa volunteers. similarly, @xcite provides over 40k human-annotated parallel english-bangla image description pairs based on 8,091 images from flickr8k @xcite . lastly, the bloom library @xcite provides a set of multilingual datasets for language modeling, image captioning and visual-story telling tasks containing more than 110k image captions for over 90k images in 351 languages. it also provides a speech dataset with 428 hours of speech data for speech synthesis/recognition tasks covering 56 languages. beyond captioning tasks, the dialog component was first explored by @xcite , who extended the vqa scenario collecting sequential questions grounded on images. @xcite went beyond goal-oriented dialogue to collect image-grounded conversations (contrasting this to open-ended dialogue research). more recently, the image-chat dataset @xcite collected open-ended conversations grounded in images with a focus on engagement, by assigning desired style traits to the speaker. discussion there are notable limitations with most publicly available multimodal datasets. to make comparisons easy, we outline most relevant works in ## dataset description description the dataset consists of a parallel corpus of speech and transcriptions of image-grounded dialogues between bemba speakers and their corresponding english translations. it contains 92,117 spoken utterances (complete and incomplete dialogues), amounting to 187 hours of speech data grounded on 16,229 unique images. there are 16,697 complete 5-turn unique dialogues grounded on 14,551 unique images. of the total 16,697 complete dialogues, 2,146 are unique dialogues grounded on duplicated images, each recorded by unique pairs of speakers. a second set of dialogues is comprised of 2,314 incomplete dialogues missing one or more utterances as a result of the preprocessing step that involved removing all audio files that are silent and corrupted. the sum of utterances that make up the incomplete dialogues is 8,632 of the total 92,117 utterances. all audio files are encoded in waveform audio file format (wave) with a single track (mono) and sample rate of 16khz. in ## baseline experiments in this section, we detail some baseline experiments carried out to demonstrate the potential of the dataset. we provide unimodal baselines using the train-validation-test splits in ## conclusion in this paper, we presented a multimodal corpus comprised of multi-turn dialogues between speakers of the zambian language, bemba, grounded on images, transcribed and translated into english. it contains over 92,000 utterances/sentences, 180 hours of speech grounded over 16,000 images. the dataset aims to fill multiple roles: enable development of fundamental tools like speech recognition, machine translation and speech-to-text translation systems between bemba and english; serve as a benchmark for academic and industry research; and to facilitate research in language grounding and multimodal model development towards building context-based dialogue agents, among other potential use cases. we have also provided baseline for asr, mt and st task. in future work, we plan to conduct multimodal baseline experiments, as well as attempt to mitigate the image diversity limitation by collecting an addendum to big-c using images taken locally in zambia. in addition, we plan to further expand to other zambian languages such as tonga, tumbuka, chewa, or lozi, by translating the existing dataset (creating an n-way parallel corpus for zambian languages) and by direct data collection. further down the roan we plan to study the dialectal varieties of bemba and the other languages, by collecting contrastive datasets from different regions of the country. 7 note that zambia is a land-locked country.