A linguistically-motivated evaluation methodology for unraveling model’s abilities in reading comprehension tasks (2024)

Abstract: We introduce an evaluation methodology for reading comprehension tasks based on the intuition that certain examples, by the virtue of their linguistic complexity, consistently yield lower scores regardless of model size or architecture. We capitalize on semantic frame annotation for characterizing this complexity, and study seven complexity factors that may account for model’s difficulty. We first deploy this methodology on a carefully annotated French reading comprehension benchmark showing that two of those complexity factors are indeed good predictors of models’ failure, while others are less so. We further deploy our methodology on a well studied English benchmark by using chatGPT as a proxy for semantic annotation.Our study reveals that fine-grained linguistically-motivated automatic evaluation of a reading comprehension task is not only possible, but helps understand models’ abilities to handle specific linguistic characteristics of input examples. It also shows that current state-of-the-art models fail with some for those characteristics which suggests that adequately handling them requires more than merely increasing model size.

## introduction generative language models, and very large ones in particular, define the current state-of-the-art in a number of natural language processing tasks. yet, despite the impressive quantity of scientific studies dedicated to them, the capabilities, limitations, and risks of these models remain largely unknown. in this work, we argue that black-box evaluations across various tasks, datasets, and languages @xcite is not enough to portrait current models abilities and instead propose in section 2 a linguistically fine-grained evaluation methodology that capitalizes on semantic frame annotation @xcite to character-ize examples thanks to a small number of complexity factors we describe in section 3. question answering (qa) from documents has been extensively studied since the advent of deep neural network-based models, facilitated by large evaluation corpora such as squad @xcite and multirc @xcite , part of the superglue benchmark @xcite . transformer-based models consistently top leaderboards foot_0 , outperforming humans. for a nuanced view, see the position paper by @xcite , but we acknowledge this belief as highlighting the challenge of evaluating qa, due to the subjective nature of answer generation and models capturing training data biases @xcite . thus, qa offers an interesting playground of our evaluation method that we consider here. as a proof of concept, we apply our methodology to a publicly available reading comprehension benchmark calor @xcite , which includes french question-answer pairs with detailed semantic annotations on the relation linking questions and answers. we demonstrate that certain complexity factors can effectively predict model limitations, regardless of size or architecture. in section 5, we extended our methodology to the naturalqa @xcite benchmark, using chatgpt to compute complexity factors. our results show that models of various sizes and architectures struggle with certain examples, suggesting that addressing these challenges requires more than just scaling up model size. by presenting a method to automatically select these challenging examples, we provide a means for monitoring further progress in reading comprehension. the data used and collected in this study is available on the following link: https://gitlab.lis-lab. fr/calor-public/complexity-calor . ## method our goal is twofold: first, to partition an evaluation corpus into several subsets, each with a distinct (linguistic) level of complexity; and second, to identify linguistically motivated factors that explain the variations in complexity across these subsets. we partition examples based on the analysis of systems' output inspired by the rover method @xcite . to ensure independence from any single model when doing so, we propose using a set of models @xmath0, m 2 , . . . , m n adapted to perform the task and bin examples according to the number of models that agree in their answer with the majority vote. thus, examples are partitioned into n bins (from total disagreement to full agreement); partition 1 grouping examples where all n systems' outputs differ, while partition n gather examples where all systems agree. to explain why some subsets are more complex than others, we confront linguistic assumptions formulated as complexity factors to examples in each bin, proceeding as follows: as stated in the introduction, we applied our method to a reading comprehension task, which involves a qa process based on documents. the complexity factors we evaluate in this study were defined through a controlled experiment on the calor evaluation corpus, which was manually annotated with semantic frames and enriched with qa based on these frames. this process is described in the next section. ## controlled experiment we compare seven pre-trained language models: one is a classification model based on a bert architecture @xcite developed for the french language, camembert @xcite ; three models are multilingual generative models based on t5 (t5-large, flan-t5-large @xcite , mt5-large @xcite ), and three models are current large language models (llms): llama2 @xcite and chatgpt-3.5 foot_2 . all these pre-trained models, except chat-gpt3.5 and mixtral 8x7b, have been adapted to our qa task using the french corpus fquad @xcite . this corpus, constructed similarly to squad @xcite , contains questions based on french wikipedia documents. we used fine-tuning (on fquad) for camem-bert and the t5 models with 2 epochs, and the low-rank adaptation method (lora) @xcite on the llama2 model. for gpt-3.5 and mixtral 8x7b, respectively, a one-and two-shot prompting approach was used, which involved specifying to the model the requirement for an extraction of the original document with one example of input/output in the expected format. ## experiments with naturalqa to evaluate the transferability of our two main complexity factors (f nb f es and f entropy ) to other qa datasets and languages, we used a subset of natu-ralqa @xcite and the predictions of 48 models provided by helm @xcite on their natural_qa_openbook_longans foot_8 scenario. this subset consists of 1,000 examples from the naturalqa evaluation distribution, each comprising a question, a "short" answer, and the context, which in this case is the corresponding "long" answer from naturalqa (typically equivalent to a paragraph). for brevity, we present the results of 8 of the 48 models in applying f entropy to naturalqa. for this dataset, lacking an automatic frame analysis, we used a proxy method: we compiled all potential triggers from berkeley framenet frames and checked their exact presence in the questions. each question provided a list of triggers and their corresponding frames. using a custom prompt 12 , we employed gpt-3.5 to determine the most appropriate pair for each question. for example with the question : how long did the democrats control the house and senate? we can extract the following list of 11 triggers and their potentials frames : [('dura-tion_description', 'long'), ('buildings', 'house'), ('desiring', 'long'), ('dimension', 'long'), ('firefighting', 'control'), ('controller_object', 'control'), ('measurable_attributes', 'long'), ('containing', 'house'), ('experimentation', 'control'), ('being_in_control', 'control'), ('control', 'control')]. the chosen pair in this case being : ('being_in_control', 'control'). we decided to use a proxy via chatgpt rather than automatic analysis in a semantic framework for several reasons. first, this approach offers simplicity in implementation and scalability to other languages, requiring only hypothesis extraction via keyword search and api calls. second, our analysis is focused on questions, not paragraphs of text, unlike the typical training data for most semantic frame models, and we had reservations about the models' performance in this context. we performed a manual evaluation on 50 sentences, where two annotators assessed chatgpt's frame predictions as fully correct, partially correct, or erroneous. the results, shown in ## related work our work situates itself within the domain of model evaluation. our approach contrasts with broadscale evaluations that span multiple tasks, corpora, and languages @xcite @xcite @xcite . it relates to focused studies addressing specific linguistic phenomena such as negations @xcite @xcite , ambiguity in inference tasks @xcite , and open information extraction @xcite , that utilizes small, meticulously curated datasets to precisely evaluate the capabilities of models for the task. our study 13 appendix a.5 echoes the latter, exploring focused linguistic evaluations. this study aligns with other efforts evaluating 'closed' llms like chatgpt, accessible only through apis, on benchmarks such as knowledgebased question answering (kbqa) @xcite . these studies highlight chatgpt's robust performance across diverse nlp tasks @xcite , yet also note its potential to lag behind task-specific models. overall, this study pushes the idea that we need a more precise evaluation framework and can be related to other studies such as @xcite ) that identify critical failures in both commercial and state-of-the-art models by proposing a model and task-agnostic testing methodology or @xcite insisting on the fact that to compare models we need more "careful annotation process [...] to characterize their output quality and distinguish between them". ## conclusions this paper presents a methodology for identifying intrinsic complexity factors in nlp tasks. our results reveal that some examples consistently produce lower scores due to their inherent linguistic complexity. through an empirical study on a qa task, we identified and validated several factors of semantic complexity, with results directly linked to human evaluations of model predictions. we have also validated these factors on another dataset in another language, confirming their robustness. in addition, we have developed corpora of increasing semantic complexity, suggesting that taking these complexities into account requires more than simply improving the model's parameters.