MOCHA : A Dataset for Training and Evaluating Generative Reading Comprehension Metrics (2020)

Abstract: Posing reading comprehension as a generation problem provides a great deal of flexibility, allowing for open-ended questions with few restrictions on possible answers. However, progress is impeded by existing generation metrics, which rely on token overlap and are agnostic to the nuances of reading comprehension. To address this, we introduce a benchmark for training and evaluating generative reading comprehension metrics: MOdeling Correctness with Human Annotations. MOCHA contains 40K human judgement scores on model outputs from 6 diverse question answering datasets and an additional set of minimal pairs for evaluation. Using MOCHA, we train a Learned Evaluation metric for Reading Comprehension, LERC, to mimic human judgement scores. LERC outperforms baseline metrics by 10 to 36 absolute Pearson points on held-out annotations. When we evaluate robustness on minimal pairs, LERC achieves 80% accuracy, outperforming baselines by 14 to 26 absolute percentage points while leaving significant room for improvement. MOCHA presents a challenging problem for developing accurate and robust generative reading comprehension metrics.

## introduction reading comprehension (rc) has seen significant progress in the last few years, with a number of question answering (qa) datasets being created @xcite @xcite . however, a majority of datasets are presented using a span-selection or multiple-choice (mc) format. both formats are easy to evaluate, passage: . . . behind one door is a lady whom the king has deemed an appropriate match for the accused; behind the other is a fierce, hungry tiger. both doors are heavily soundproofed to prevent the accused from hearing what is behind each one. . . question: what feature do the doors have? reference: soundproofed candidate: they are heavily soundproofed to prevent the accused from hearing what's behind each one. figure 1: generative reading comprehension example. properly scoring the candidate requires access to the passage. current metrics, such as bleu, rouge and meteor, are agnostic to the end-task while lerc is trained with the passage and question as input. as a result, lerc assigns a score that better reflects human judgement. but in return, have restrictions placed on the questions that can be asked or the answers that can be returned. furthermore, both formats hinge on distractor spans/choices for learning to be effective. ensuring high quality distractors is a challenging task in and of itself, which can lead to models that exploit spurious correlations @xcite @xcite . posing rc as a generation task addresses the aforementioned issues. generative rc does not require distractors, circumventing biases that could be introduced by them, and allows arbitrary questions and answers. unfortunately, existing metrics for evaluating text generation come with significant shortcomings. many metrics score n-gram overlap, and it is well established that using token overlap as a measure of similarity has drawbacks @xcite @xcite . cur-rent metrics also only consider the reference and are agnostic to the end-task being evaluated. fig. 1 demonstrates that this is problematic for generative rc because scoring a candidate may require a metric to also consider the passage and the question. without cheap and reliable evaluation, progress in generative reading comprehension has been extremely slow. to address the need for better evaluation metrics tailored to reading comprehension, we present a dataset called mocha, aimed at developing learned metrics that model the correctness of candidates using human annotation scores. mocha contains human judgement scores on 40k candidates, an order of magnitude larger than prior work @xcite . the candidates come from six diverse qa datasets which test a wide range of rc phenomena such as commonsense reasoning and understanding narrative over movie scripts. after collecting all annotations, we follow work on creating more robust evaluation sets @xcite and augment the test set of mocha by manually writing a small set of minimal pairs ( using mocha, we train a learned metric for reading comprehension which we abbreviate as lerc. we compare lerc against two sets of baselines: (1) existing metrics such as meteor @xcite and bertscore @xcite ; and (2) a sentence similarity model trained on @xcite . to ensure fair comparison, we evaluate lerc in an out-of-dataset setting: lerc is trained on all datasets except the one it is being evaluated on. on the test set, lerc outperforms baselines by as much as 36 pearson correlation points and on the minimal pairs set, by as much as 26 accuracy points. error analysis and minimal pair results indicate that there is substantial room to improve the robustness of lerc and its sensitivity to different linguistic phenomena. we hope that mocha and lerc enables a continual cycle of generative rc model and dataset developments that will enable easier collection of more diverse and useful candidates, allowing better learned metrics to be trained. instance score passage: with the aid of his daughter, abigail, barabas recovers his former assets. barabas then uses his daughter's beauty to embitter lodowick and mathias against each other. q: why did lodowick and mathias fight? ref: over the affection of abigail cand: they fight over barabas's daughter. 5 passage: miss moppet ties a duster about her head and sits before the fire. the mouse thinks she looks very ill and comes down the bell-pull. q: what does the mouse think when she sees the duster on miss moppet's head? ref: that miss moppet is ill cand: miss moppet thinks it is ill and is trying to sniff him. 2 passage: robin took a very long time to clean the windows of her house. q: how would you describe robin? ref: a neat freak cand: a clean person 5 passage: the strangest thing that has happened was when they were singing the chinese national anthem she was standing in front of the tv swaying and singing. q: what is probably true about this story? ref: they are watching the olympics cand: the olympics are watching 2 ## a description of mocha reading comprehension is the task of probing how well systems can understand passages of text. framing reading comprehension as a generation problem provides a great deal of flexibility, but introduces the challenging problem of evaluation. these challenges are further amplified when applied to generative reading comprehension, where the introduction of a passage and a question can add to the complexity of evaluation ( in this section, we present mocha, a dataset that pairs reading comprehension instances, which consists of a passage, question, and reference, with candidates and human judgement scores. we describe the process of gathering candidates, collect-instructions 1. read the passage. 2. read the question, correct answer, and predicted answer. 3. select the score that best reflects how closely a predicted answer captures the same information as the correct answer. passage: ...i got all of the ingredients i would need together to make the coffee and brought them to the company coffee machine… question: how was the coffee made? correct answer: with a coffee machine predicted answer: with a personal coffee machine ing human judgement scores, and creating minimal pairs for evaluation. ## a learned metric we provide details on lerc, our learned metric. lerc is initialized using bert-base @xcite we define as input a tuple consisting of a passage, p, a question, q, a reference answer, a, and a candidate answer, â. the input to bert is ## experiments training lerc: we use the pytorch (paszke et al., 2019), huggingface transformers @xcite , and allennlp @xcite libraries to implement lerc. we pre-train lerc before fine-tuning on mocha. we evaluate lerc in two settings, an out-of-dataset (ood) setting and an all-datasets (ad) setting. in the ood setting, we train and tune lerc on all datasets in mocha except the dataset we are evaluating on. this reflects the use case where we want to apply lerc to evaluate a new dataset where we do not have human judgement scores. in the ad setting, we train on all datasets in mocha and evaluate on all datasets. all results reported for lerc are the average of three runs using the best set of hyperparameters found on the validation set of mocha. baselines: we compare lerc against bleu-1 @xcite , rouge-l @xcite , me-teor @xcite , and bertscore @xcite . we also compare lerc against a bert-base model fine-tuned on the sentence similarity task, @xcite . results for bert sts-b are the average of three runs using the best set of hyperparameters found on the validation set of sts-b. all baselines are agnostic to the passage and the question. ## related work there has been a long history of developing evaluation metrics, which have generally fallen into one of three categories. the first consists of metrics that use some variant of n-gram matching @xcite @xcite . they are easy to implement, but lack flexibility by focusing only on token overlap. the second cateogry of metrics eschew some of the aforementioned issues by calculating a softer similarity score using embeddings of tokens @xcite . however, it is unclear how to tailor them to question answering, where the passage and question should be assimilated. the final category consists of metrics learned end-to-end from human judgements @xcite . these metrics are flexible in that they can be tuned to the specific evaluation setting but depend on a large corpus of human judgement scores to train on. we hope that the release of mocha pushes the development of qa metrics that fall into this category. mocha is directly inspired by the annual wmt metrics shared task @xcite @xcite @xcite . participants submit automatic translations and human judgement scores are collected for the submitted translations. the annotations collected as part of the wmt metrics shared task have made it easy to evaluate and create new translation metrics @xcite @xcite . in a similar vein, summeval is a recently released dataset that evaluates a number of evaluation metrics for summarization @xcite . ## conclusion we present mocha, a dataset of human judgement scores for training and evaluating generative reading comprehension metrics. using mocha, we train a learned metric, lerc, that outperforms all existing metrics and is much more robust when evaluated on a set of minimal pairs. while we have demonstrated that lerc is a better metric for evaluating generative reading comprehension than any existing metric, considerable work remains. error analysis reveals that there exist gaps in lerc's ability to handle certain phenomena, such as correctly leveraging the passage. future work involves collecting data to addresses weaknesses of lerc. we also anticipate a con-tinual cycle of generative rc model and dataset developments that will enable easier collection of more diverse and useful candidates. this in turn will allow better learned metrics, which can be used to evaluate ever more complex models.