Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for T igrinya (2023)

Abstract: Question-Answering (QA) has seen significant advances recently, achieving near human-level performance over some benchmarks. However, these advances focus on high-resourced languages such as English, while the task remains unexplored for most other languages, mainly due to the lack of annotated datasets. This work presents a native QA dataset for an East African language, Tigrinya. The dataset contains 10.6K question-answer pairs spanning 572 paragraphs extracted from 290 news articles on various topics. The dataset construction method is discussed, which is applicable to constructing similar resources for related languages. We present comprehensive experiments and analyses of several resource-efficient approaches to QA, including monolingual, cross-lingual, and multilingual setups, along with comparisons against machine-translated silver data. Our strong baseline models reach 76% in the F1 score, while the estimated human performance is 92%, indicating that the benchmark presents a good challenge for future work. We make the dataset, models, and leaderboard publicly available.

## introduction question answering (qa) and machine reading comprehension (mrc) have seen significant advances in recent years, achieving human-level performance on large-scale benchmarks @xcite . the main factors driving the progress are the adaption of large pre-trained large language models and the proliferation of qa datasets @xcite . however, most studies focus on high-resourced languages, while the task remains unexplored for most of the world's diverse languages. the primary challenge for non-english qa is the lack of native annotated datasets. in particular, there is little to no study done on scarcely resourced languages such as tigrinya that article: ◊ym §m" @xcite paragraph: ◊ym §m" kô ¿bµm bƒr §'e°mb" zõw'u §mr³t 'a †m¹ ≈ †t b» §m", ðˆm §m", ◊ym §m" ¼m∫'wn ó'e× §m" …ï ~oe¤£l sω d≈ ¿bµm ƒb gì k m² bbzj z•bi bf‰y d≈ t•yádz√m ‚"t"n z° §h ‡ oerë≈t z√• °ƒåsñn bzhbç ◊ym mb" mff• ynú ‡ [the red sea is one of the four seas in the world that are named after common colors: the yellow sea, the black sea, the red sea and the white sea. the origin of its name is attributed to the red color given by the poisonous bacteria, especially the trichodesmium erythraeum, which breed in large numbers during the summer season.] question 1: ƒb 'a †m knôy bmb" zõw'u §mr³t ƒ †ã? [how many seas in the world are named after common colors?] answer: ƒr §'e° @xcite question 2: ƒb 'a †m bmb" zõw'u §@xmath0: b» §m", ðˆm §m", ◊ym §m" ¼m∫'wn ó'e× §m" [yellow sea, the black sea, the red sea and the white sea] question 3: t•yádz√m ‚"t"n ƒ¤î¹y âe² y•bi? [in which season do the trichodesmium erythraeum reproduce?] answer: k m² @xcite question 4: t•yádz√m ‚"t"n …n³y mb" yh¥? [what is the color of trichodesmium erythraeum?] answer: ◊ym @xcite figure 1 : example entry from tiquad: a paragraph as context and the corresponding annotated questionanswer pairs. some context was redacted for brevity. this work presents tiquad, the first publicly available question-answering dataset for tigrinya; see figure 1 for an example entry. we collaborate with native tigrinya speakers to collect documents and annotate the dataset, yielding a total of 10.6k question-answer pairs with 6.5k unique questions over 572 paragraphs gathered from 290 news articles. we assess the quality of annotations and explore strong baselines by fine-tuning tiroberta and tielectra @xcite as monolingual models of tigrinya and xlm-r @xcite and afriberta @xcite as representative multilingual models. in addition to the monolingual qa setup, we perform three scenarios of cross-lingual and multilingual experiments. first, we translate squad1.1 to tigrinya and evaluate the performance in conjunction with the native tiquad. second, we assess a zero-shot crosslingual transfer learning approach @xcite ) by evaluating on the new dataset. third, we explore the performance of a multilingual setup by jointly fine-tuning the models on english and tigrinya datasets. the experimental settings are depicted in figure 2 . the bestperforming baseline model achieves up to 76% in f1 score in the multilingual setup, while the estimated human performance is 92%. considering the challenges of constructing annotated datasets for under-represented languages, we believe this work could serve as a reference case for similar languages. in particular, the tiquad benchmark is an important milestone in the advancement of question-answering for the tigrinya language. the contributions of this work are summarized as follows: (1) we build the first questionanswering dataset for tigrinya and make it publicly available. (2) we present an in-depth analysis of the challenges of question answering in tigrinya based on the dataset. (3) we apply transformer-based language models to the question-answering task in tigrinya and compare it with datasets of other languages. (4) we investigate various resourceefficient cross-lingual and multilingual approaches to qa and assess the utility of the native dataset. ## dataset annotation tiquad is designed following the task formulation of squad @xcite , where each entry in the dataset is a triple consisting of a paragraph, a question, and the corresponding answer. the answer is a contiguous span of text in the paragraph, a typical setup of extractive questionanswering. the dataset was constructed in four stages: first, a diverse set of articles are collected from which we extract paragraphs that will serve as contexts. second, the initial question and answer pairs are annotated for all the extracted paragraphs. third, additional answers are annotated for all the questions in the development and test sets. fourth, we post-process the annotations for quality control and remove noisy examples. the final dataset contains over 10.6k question-answer pairs across 572 paragraphs. while the size is on the smaller end compared to the english datasets, it reflects a realistic amount of data that researchers of low-resourced languages can acquire with a limited annotation budget. 2 the dataset characteristics are presented in ## dataset analysis to assess the quality and diversity of the dataset, we perform various analyses of the annotations. ## results and discussions in this section, we present and discuss the results of the proposed experimental setups. in this setup, we train all models on the native and translated tigrinya datasets then evaluate on the tiquad development and test sets. the experimental results are presented in ## conclusion in this work, we presented the tigrinya question answering dataset (tiquad). the context paragraphs were collected from high-quality news articles of diverse genres, and we collaborated with native speakers to annotate over 6.5k unique questions and 10.6k answers. the development and test sets were further enriched with additional answers to enable a robust evaluation. we conducted comprehensive experiments in monolingual, crosslingual, and multilingual settings. the estimated human performance on the test set is 81.3% em and 92.1% f1 score, while the top performing model achieves 68.06% em and 76.58% f1, leaving a room for future improvements.