H e Q : a Large and Diverse H ebrew Reading Comprehension Benchmark (2023)

Abstract: Current benchmarks for Hebrew Natural Language Processing (NLP) focus mainly on morpho-syntactic tasks, neglecting the semantic dimension of language understanding. To bridge this gap, we set out to deliver a Hebrew Machine Reading Comprehension (MRC) dataset, where MRC is to be realized as extractive Question Answering. The morphologically-rich nature of Hebrew poses a challenge to this endeavor: the indeterminacy and non-transparency of span boundaries in morphologically complex forms lead to annotation inconsistencies, disagreements, and flaws of standard evaluation metrics. To remedy this, we devise a novel set of guidelines, a controlled crowdsourcing protocol, and revised evaluation metrics, that are suitable for the morphologically rich nature of the language. Our resulting benchmark, HeQ (Hebrew QA), features 30,147 diverse question-answer pairs derived from both Hebrew Wikipedia articles and Israeli tech news. Our empirical investigation reveals that standard evaluation metrics such as F1 Scores and Exact Match (EM) are not appropriate for Hebrew (and other MRLs), and we propose a relevant enhancement. In addition, our experiments show low correlation between models’ performance on morpho-syntactic tasks and on MRC, which suggests that models that are designed for the former might underperform on semantic-heavy tasks. The development and exploration of HeQ illustrate some of the challenges MRLs pose in natural language understanding (NLU), fostering progression towards more and better NLU models for Hebrew and other MRLs.

## introduction machine reading comprehension (mrc), a critical skill of nlp systems, is often assessed by means of testing the ability of models to answer questions about a given passage, a setup known in nlp also by the name question answering (qa). qa systems find utility in a large spectrum of applications, from chatbots @xcite and search engines @xcite to the evaluation and assessment of pre-trained models' @xcite @xcite comprehension and reasoning skills. despite these developments, the landscape of mrc is primarily dominated by datasets and models for english, where low-to medium-resource languages such as hebrew remain largely underrepresented. although recent research in nlp shows increased interest in the hebrew language @xcite , the focus remains largely on morpho-syntactic tasks. with the exception of parashoot @xcite , the absence of natural language understanding (nlu) benchmarks for hebrew leads to the design of models primarily tailored for morpho-syntactic tasks. however, it remains an open question whether the performance on existing morpho-syntactic benchmarks correlates with actual natural language understanding, and indeed, as we demonstrate in §5, this is not necessarily so. a central challenge in the development of qa systems for hebrew, a morphologically rich language (mrl), compared with morphologyimpoverished ones, is the identification of precise answer spans in morphologically rich internally complex surface forms. this is exacerbated by multiple processes of affixation in hebrew (prefixes, suffixes, clitics) that may obscure the boundaries between linguistic units, making it harder to pinpoint the correct span of answers. for instance, given the sentence ‫בבית.‬ ‫הייתי‬ ‫אתמול‬ (literally: "yesterday i-was at-the-house"), valid responses to the question "where was i?" can be either ‫בבית‬ (at-the-house), or ‫בית‬ (house). current annotation guidelines, crowdsourcing protocols and evaluation metrics, however, assume the annotated spans respect space-delimited token boundaries, and fail to accommodate such morphological patterns, conse-quently penalizing boundary alterations and causing annotation disagreements. to counter this, we propose a revised set of guidelines and new evaluation metric, token-level normalized levenshtein similarity (tlnls), which manifests lower sensitivity to affixation changes. we thus present heq, a new benchmark for mrc in hebrew, modeled around the format of the widely-used stanford question answering dataset (squad) @xcite . squad is a benchmark dataset for mrc that was collected via crowd-sourcing and consists of questions and answers based on wikipedia articles. however, squad has been shown to contain biases and spurious regularities that allow models to exploit heuristics and shallow cues without fully understanding the text @xcite . furthermore, squad covers only a limited range of topics and domains @xcite . hence, in this work, we introduce several novelties. first, we work with a fixed set of human annotators and train each annotator individually based on their previous annotations to increase the quality and diversity of their annotations. we also add a data source from a news domain (geektime), besides wikipedia, to increase heq genre diversity. finally, we change the unanswerable questions generation methodology so they become more challenging for the model. in our experiments, we observe that a multilingual model, mbert outperforms all other hebrew-trained models despite being exposed to the least amount of hebrew data during training, and having the smallest hebrew token vocabulary. this suggests that pretrained models in hebrew can greatly benefit from pretraining on other languages. furthermore, we find that models trained on our new dataset, heq, perform significantly better than those trained on the earlier parashoot dataset, which emphasizes that the quality of the collected data is greatly influential on model performance (in fact, a model trained on a subset of heq with the same number of questions as in the parashoot dataset, outperforms a model trained on parashoot, also on the parashoot test set). lastly, models based on the geektime section outperform the wikipedia ones and demonstrate better domain transferability, likely due to the former's varied text structure. in conclusion, despite the complexity of hebrew, multilingual pretraining proves beneficial for mrc tasks. the quality and diversity of the data appear to be as important as the size, underscoring the importance of dataset design and the value of our new hebrew mrc benchmark, heq. note that all data, tagging platform, user manual, and the training and evaluation scripts are publically online. foot_0 ## the challenge: extractive question answering in morphologically rich languages extractive question answering (extractive-qa) is a common and widely used task where a model is tasked with locating and reproducing an answer to a question posed directly with regard to a source text. in morphologically-rich languages, this task becomes more complex due to the linguistic features that these languages manifest. morphologicallyrich languages, such as hebrew, turkish, or arabic, exhibit high morphological complexity, including extensive inflection, agglutination, and derivational morphology. this characteristic makes extractive-qa more challenging because the model needs to process and understand complex word forms and grammatical constructions to correctly pinpoint the answer spans. in these languages, a single word can carry information typically spread across several words in languages with more impoverished morphology, such as english. this means that word-and sentenceboundaries may not always align with semantic units or logical chunks of information. consequently, extractive-qa models must be able to handle this morphological richness to accurately locate and extract the correct answer. due to the high inflectional nature of these languages, the correct answer may appear in different forms within the text, requiring the model to understand and match these varied forms. this aspect necessitates an understanding of the language's morphological rules and a richly annotated dataset to provide the model with enough examples to learn from. on top of that, existing evaluation metrics, primarily designed for morphologically simpler languages, often fall short when applied to languages with higher morphological richness. a major contributor to this is affixations, which are a prevalent feature in mrls. these languages often merge what would be considered separate words in languages like english into a single unit. this means that an answer span might not align with the tradi-tional notion of word boundaries. current evaluation metrics do not take this into account, leading to potential harsh penalties for minor boundary deviations. metrics like exact match (em) or f1 score, which function well in morphologically simpler languages, may not accurately reflect a model's performance in morphologically rich languages. for instance, slight changes in word endings might result in a dramatically different em score, even if the overall answer meaning remains intact. similarly, an f1 score may fail to capture the nuances in word formation and changes that are inherent to these languages. ## collection process this section provides a comprehensive account of the data preparation, annotation, and validation procedures employed in this study. specifically, we describe the methods used to preprocess the raw data and prepare it for annotation, the rigorous annotation process carried out to ensure high-quality data, and the validation steps taken to evaluate the reliability and accuracy of the resulting dataset. ## mrc evaluation for mrls evaluating the performance of mrc models is a fundamental aspect of their development and refinement. however, this evaluation process isn't as straightforward as it might seem -especially when it comes to mrls. the traditional evaluation metric of mrc is f 1 over answer span words (described in appendix b.1) or a variation thereof. the f 1 score is sensitive to small changes in the answer span boundary. for example, if the gold span is "in the house" while the extracted span is "house" the f 1 score will be 2 3 . english somewhat alleviates this problem by removing the words "a", "an", and "the" for the evaluation, which in the former example will increase the f 1 to 0.5. unfortunately, the boundary issues in mrls, and in hebrew specifically, are much more common and result in biased scores. for example, if we consider the same example, but in hebrew, the phrase "in the house" is translated to the single word ‫,בבית‬ while the word ‫בית‬ corresponds to "house". so while the english phrase will get an f 1 score of 0.5, the hebrew span using a token-based or word-based evaluation will be scored 0, leading to an underestimation of model performance. this is the result of compounding, combining two or more words into a single word in hebrew. compounding is very common in hebrew and happens in many situations like possessive, pluralization, definite article, prepositions, and other affixations. while some of these affixations also appear in english (like pluralization), they are much more common in the hebrew setting. one way to avoid the issue is to do a complete morphological decomposition of each word in the predicted span and the gold span and remove affixations from the evaluation. unfortunately, the morphologically rich forms are ambiguous, and hebrew morphological disambiguation is contextdependent, requiring a dedicated model to perform. we seek an evaluation method that is not dependent on the existence of a disambiguation model and which will not be influenced by mistakes of different disambiguation models. rather want a method that is simple, unbiased, and easy to run on multiple mrls. we thus present a novel evaluation metric for mrc tailor-made for mrls. ## experiments we experiment with heq in several scenarios. first, we create a baseline based on several known models, then we evaluate the improvement of heq trained model vs parashoot trained model. we continue by analyzing the impact of the data size of the train data on the models' performance. finally, we assess the effect of the different domains in the dataset on the overall performance and the cross-domain performance on heq. ## conclusion in this work, we presented the challenges of mrc for mrls. we introduced heq, a hebrew mrc benchmark designed to enable further research into hebrew nlu. experimental insights highlighted the surprising efficacy of mbert, trained on diverse languages, over hebrew-monolingual models, suggesting the value of multilingual pretraining. however, data quality was seen to be as important as size, underlining the need for high-quality, diverse hebrew datasets. the complexity of evaluating models in mrls was also recognized, leading to the proposal of the tlnls metric, better suited to handle morphological variations. overall, this work advances the understanding of mrc in morphologically rich languages, offering avenues for improved benchmarks, datasets, and evaluation metrics. it invites further exploration to broaden knowledge and enhance mrc performance in underrepresented languages. we recognize two main limitations of the dataset: dataset size. although heq is of high quality, it consists of a relatively smaller number of instances compared to well-established english mrc datasets like squad. the limited dataset size might impact the model's ability to generalize and capture the full range of linguistic and contextual variations in hebrew. however, efforts have been made to ensure the dataset's quality and diversity compensates for its smaller scale. lack of natural questions. one of the limitations faced in constructing heq is the unavailability of a dedicated resource containing natural questions in hebrew. natural questions, which reflect authentic and unconstrained human language use, provide a realistic and diverse set of challenges for mrc models. without access to such a resource, the dataset may not fully capture the breadth and complexity of real-world questions, potentially limiting the evaluation of mrc models' performance in natural language understanding. despite these limitations, the introduction of a new dataset and evaluation metric provided in this paper are valuable contributions to the field and the progression of hebrew nlp research.