P ara S hoot: A H ebrew Question Answering Dataset (2021)

Abstract: NLP research in Hebrew has largely focused on morphology and syntax, where rich annotated datasets in the spirit of Universal Dependencies are available. Semantic datasets, however, are in short supply, hindering crucial advances in the development of NLP technology in Hebrew. In this work, we present ParaShoot, the first question answering dataset in modern Hebrew. The dataset follows the format and crowdsourcing methodology of SQuAD, and contains approximately 3000 annotated examples, similar to other question-answering datasets in low-resource languages. We provide the first baseline results using recently-released BERT-style models for Hebrew, showing that there is significant room for improvement on this task.

## introduction natural language processing has seen a surge in the pretraining paradigm in recent years with the appearance of pretrained models in a plethora of languages, including hebrew @xcite . while such models have shown to perform remarkably well on a variety of tasks, most of the evaluation of the hebrew models, however, has been focused on morphology and syntax tasks in the spirit of universal dependencies @xcite , while end-user-focused evaluation has been limited to sentiment analysis @xcite and named entity recognition @xcite . in this paper, we try to remedy the scarcity of semantic datasets by presenting parashoot, foot_0 the first question answering dataset in hebrew, in the style of squad @xcite . we follow similar work in constructing non-english question answering datasets @xcite @xcite , and turn to hebrew-speaking crowdsource workers, asking them to write questions given paragraphs sampled at random from hebrew wikipedia. through this process, we collect approximately 3000 annotated (paragraph, question, answer) triplets, in a setting that may be suitable for few-shot learning, simulating the amount of data a startup or academic group can quickly collect with a limited annotation budget or a short deadline. statistical analysis of parashoot shows that the dataset is diverse in question types and complexity, and that the annotations are of decent quality. we provide baseline results based on two recentlyreleased bert-style models in hebrew, showing that there is much potential in devising better pretraining and fine-tuning schemes to improve the performance of hebrew language models on this dataset. we hope that this new dataset will pave the way for practitioners and researchers to advance natural language understanding in hebrew. 2 ## dataset we present parashoot, a question answering dataset in hebrew, in a format that closely follows that of squad @xcite . each example in the dataset is a triplet consisting of a paragraph, a question, and a span from the paragraph text constituting the answer to the question. we scrape paragraphs from random hebrew wikipedia articles, and crowdsource questions and answers for each one, resulting in 3038 annotated examples. while larger datasets may facilitate betterperforming models, recent work has advocated for research on smaller labeled datasets @xcite , which more accurately reflect the amount of data a startup or academic lab can collect in a short amount of time and resources. figure 1 : the annotation user interface, containing the article's title, the paragraph, a slot for entering a question, and an additional slot for entering the answer. dragging the mouse over a span in the paragraph automatically fills the question slot, allowing for quick and accurate annotation of answer spans. ## analysis we analyze the dataset in various ways to assess its quality and limitations as a benchmark. ## baselines we establish baseline results for parashoot using bert-style models. results indicate the task is challenging, leaving much room for future work in hebrew nlp to advance the state of the art. ## conclusion in this paper, we present parashoot, the first question answering dataset in modern hebrew, in a style and data collection methodology similar to that of squad. baseline results demonstrate the