H a VQA : A Dataset for Visual Question Answering and Multimodal Research in H ausa Language (2023)

Abstract: This paper presents “HaVQA”, the first multimodal dataset for visual question answering (VQA) tasks in the Hausa language. The dataset was created by manually translating 6,022 English question-answer pairs, which are associated with 1,555 unique images from the Visual Genome dataset. As a result, the dataset provides 12,044 gold standard English-Hausa parallel sentences that were translated in a fashion that guarantees their semantic match with the corresponding visual information. We conducted several baseline experiments on the dataset, including visual question answering, visual question elicitation, text-only and multimodal machine translation.

## introduction in recent years, multidisciplinary research, including computer vision (cv) and natural language processing (nlp), has attracted many researchers to the tasks of image captioning, cross-modal retrieval, visual common-sense reasoning, and visual question answering (vqa, @xcite . in the vqa task, given an image and a natural language question related to the image, the objective is to produce a correct natural language answer as output @xcite . vqa is one of the challenging tasks in nlp that requires a fine-grained semantic processing of both the image and the question, together with visual reasoning for an accurate answer prediction @xcite . the general approaches followed by existing vqa models include i) extracting features from the questions and the images, and ii) utilizing the features to understand the image content to infer the answers. recently, research on improving visual questionanswering systems using multimodal architectures and sentence embeddings @xcite @xcite has seen tremendous growth. however, most of the vqa datasets used in the vqa research are limited to the english language @xcite . although the accuracy of the vqa systems for english improved significantly with the advent of transformer-based models (e.g., @xcite , the lack of vqa datasets has restricted the development of such systems in most languages, especially the low-resource ones @xcite . the availability of original datasets for state-ofthe-art natural language processing tasks has since been appreciated, especially on the african continent. while some of the efforts to create such datasets for african languages are supported by funding such as facebook's translation support for african languages, the lacuna fund foot_0 , and many others, including the dataset in this work are driven by the enthusiasm for developing quality nlp solutions for african languages that are useful to the local communities. contributions: the main contribution of this work is building a multimodal dataset (havqa) for the hausa language, consisting of question-answer pairs along with the associated images and is suitable for many nlp tasks. as per our knowledge, havqa is the first vqa dataset for hausa language, and will enrich hausa natural language processing (nlp) resources, allowing researchers to conduct vqa and multimodal research in hausa. muhammad et al. (2022) classified about 30,000 tweets in each of the four major nigerian languages as either positive, negative, neutral, mixed or indeterminate for sentiment analysis task. subsequently, the dataset was expanded to include 14 african languages, resulting in the largest sentiment dataset for african languages @xcite . @xcite collected about 4,500 partially code-switched tweets for detecting hate against the fulani herdsmen in nigeria. @xcite created masakhaner 2.0, the most extensive corpus for named-entity recognition tasks. @xcite created the multipurpose kencorpus, a speech and text corpora for machine translation, text-to-speech, question answering, and part-of-speech tagging tasks for three kenyan languages. kenpos @xcite corpus was created for part-of-speech tagging for kenyan languages. kenspeech @xcite is a transcription of swahili speech created for text-to-speech tasks. ## the havqa dataset this section provides a detailed description and analysis of havqa questions and answers. ## sample applications of havqa we tested the havqa data by experimenting with the following nlp tasks: i) questions, answers, and images for visual question answering, multimodal machine translation, ii) questions, and images for visual question elicitation, and iii) questions and answers for text-only machine translation. ## results and discussion this section presents the results obtained after implementing the experiments described in section 5. ## conclusion and future work we present havqa, a multimodal dataset suitable for many nlp tasks for the hausa language, including visual question answering, visual question elicitation, text and multimodal machine translation, and other multimodal research. the dataset is freely available for research and non-commercial usage under a creative commons attribution-noncommercial-sharealike 4.0 license at: http://hdl.handle.net/11234/ 1-5146 . we released our experimental code through github. 14 our planned future work includes: i) extending the dataset with more images and qa pairs, ii) providing ground truth for all images for image captioning experiments, and iii) organizing a shared task using havqa.