Generative Interpretation: Toward Human-Like Evaluation for Educational Question-Answer Pair Generation (2024)

Abstract: Educational question-answer generation has been extensively researched owing to its practical applicability. However, we have identified a persistent challenge concerning the evaluation of such systems. Existing evaluation methods often fail to produce objective results and instead exhibit a bias towards favoring high similarity to the ground-truth question-answer pairs. In this study, we demonstrate that these evaluation methods yield low human alignment and propose an alternative approach called Generative Interpretation (GI) to achieve more objective evaluations. Through experimental analysis, we reveal that GI outperforms existing evaluation methods in terms of human alignment, and even shows comparable performance with GPT3.5, only with BART-large.

## introduction asking questions about the passage enhances children's literacy development @xcite . in the context of children's learning, educational question-answer generation (qag) has gained considerable attention due to its practical utility @xcite @xcite . qag frameworks aim to generate relevant question-answer (qa) pairs based on a given story passage. with the significant research focus on qag, numerous frameworks have been proposed to generate diverse and accurate qa pairs @xcite @xcite while the generation capability of qag has witnessed significant advancements, precise automatic evaluation remains a challenge. current automatic evaluation metrics for qag primarily rely on assessing textual similarity, such as rouge @xcite , and bertscore @xcite , with respect to the ground-truth(gt) qa pairs @xcite . however, we have observed that gt similarity seldom poses high score for the high relevancy to the given passage, but only prefer gt similar qa pair, which follows misalignment with human assessment @xcite . we consider evaluation to be a crucial factor in education of qag, as inaccurate assessments can result in improper guidance @xcite . considering the role of qag in the educational field, automatic evaluation methods serve as substitutes for human judgment that discriminate the most appropriate qa pair for the given passage. in such setting, an improper evaluation approach may restrict creative responses @xcite and skew the purpose of the education towards mimicking answers from the gt qa dataset. in an effort to mitigate such limitations, we propose a more objective and precise evaluation method, generative interpretation (gi). gi employs a generative qag model trained with gt qa pairs and selectively measures teacher-forced logits that are highly relevant in evaluating qa pairs. by evaluating each qa pair in a referencefree manner, gi enables even objective assessment that cannot be figured out via comparison between gt qa pairs. we figure out that gi can yield even higher human correlation, compared with the existing evaluation method. in particular, we demonstrate that only with utilizing the bart-large model structure @xcite , gi can offer comparable performance to the chat-gpt (gpt3.5) evaluation @xcite . ## preliminary the evaluation on the qag framework is performed by measuring the quality of the candidate qa set @xmath0, a c j )} nc @xmath1, generated by the qag framework given a passage p . existing methods measure the textual similarity between the c and the gt qa set @xmath2, a r i )} nr @xmath3. we denote the textual similarity metric as metric, where existing studies primarily adopt two measures, rouge and bertscore. considering multi-reference and multi-candidate setting, we can find two strategies in evaluating c. ## generative interpretation (gi) gi estimates the adequacy of the generated qa pair, which encompasses relevancy to the passage and the connectivity between the qa. similar with bartscore @xcite , we adopt qa generation model and take teacher-forced logits of the qa generation. in particular, we train the qa generation model θ to return concatenated sequence of the qa pair, by feeding passage and the question start tokens, with l ce shown in equation ( 3 ). the number of question start tokens are priorly set by a hyper-parameter n s . we feed start tokens of each question as a part of input sequence, to alleviate the question type bias foot_1 . in utilizing θ, we can estimate gi as follows: ## conclusion in this study, we focus on challenges in existing evaluation methods of educational qag that measuring quality based on the similarity with gt qa pairs. we find out that existing automatic evaluation methods show inferior human alignment especially in measuring relevancy to the passage and question-answer pair. as alternatives, we propose more objective evaluation methodology, gi, that can relieve several challenges in existing metrics. we shows that gi demonstrates even higher human alignment than gpt3.5, only with bart-large. we plan to extend gi to more general metric that can cover more generalized question generation tasks.