E - KAR : A Benchmark for Rationalizing Natural Language Analogical Reasoning (2022)

Abstract: The ability to recognize analogies is fundamental to human cognition. Existing benchmarks to test word analogy do not reveal the underneath process of analogical reasoning of neural models. Holding the belief that models capable of reasoning should be right for the right reasons, we propose a first-of-its-kind Explainable Knowledge-intensive Analogical Reasoning benchmark (E-KAR). Our benchmark consists of 1,655 (in Chinese) and 1,251 (in English) problems sourced from the Civil Service Exams, which require intensive background knowledge to solve. More importantly, we design a free-text explanation scheme to explain whether an analogy should be drawn, and manually annotate them for each and every question and candidate answer. Empirical results suggest that this benchmark is very challenging for some state-of-the-art models for both explanation generation and analogical question answering tasks, which invites further research in this area.

## introduction analogy holds a vital place in human cognition, driving the discovery of new insights and the justification of everyday reasoning @xcite @xcite . due to their unique value in many fields such as creativity @xcite and education @xcite , analogy and analogical reasoning have become a focus in ai research. the grand question is, are artificial neural networks also capable of recognizing analogies? relatively little attention has been paid in nlp to answer this question. the problem of recognizing analogies is mainly benchmarked in the form after "textbooks" are printed in the "printing factory", they are sold in a "bookstore". but the terms order is inconsistent with the query. the "bookshelf" is in the "reading room". both "school" and "enterprise" are organizations. after "talents" are educated in "school", they are transported into "enterprise". of (a:b::c:d) @xcite @xcite and targeted for testing the ability of pre-trained word embeddings. given a tuple of terms as query (e.g., tea:teapot:teacup) and a list of candidate answers as in figure 1 , a model needs to find the most analogous candidate to the query, which is c in the example since it matches the relations inherent in the query better than others. most methods @xcite @xcite hold a connectionist assumption @xcite of linear analogy @xcite , that the relation between two words can be estimated by vector arithmetic of word embeddings. for example, kingman + @xmath0. however, current benchmarks focus on the recognition of binary analogies such as syntactic, morphological and direct semantic (e.g., is_a and syn-onym_of ) relations. and the analogical reasoning procedure behind them is far beyond the scope of this line of research. in addition, how to explain and rationalize analogical reasoning remains to be the major challenge. psychological literature @xcite @xcite suggests that analogical reasoning follows the structure-mapping process. that is, a target (the domain where a problem must be solved, i.e., candidates) and a source (the domain where the analogy is drawn, i.e., the query) are matched, and the relevant features of the source have to be mapped onto the target. in figure 1 , source structures are drawn (or abduced) from the query and mapped onto candidates, and candidates a, b, d all fail at certain structures. we argue that such a process can be verbalized into natural language to explain analogical reasoning. moving from simply recognizing analogies to exploring human-like reasoning for neural models, we emphasize the importance of a new kind of analogical reasoning benchmark. to fill in this blank, we propose a first-of-its-kind benchmark for explainable knowledge-intensive analogical reasoning (e-kar). we collect 1,655 analogical reasoning problems sourced from the publicly available civil service examinations (cse) of china. these cse problems are challenging multiplechoice problems designed by human experts, thus solving them requires the intensive involvement of linguistic, commonsense, encyclopedic, and cultural (e.g., idiom and historical) knowledge. to justify the reasoning process, we follow the aforementioned guidelines from psychological theories and manually annotate free-text explanations for each query and candidate answers in e-kar. since the annotation requires intensive involvement of knowledge and reasoning, we carefully design a double-check procedure for quality control. we also translate this dataset into an english version, resulting in 1,251 problems after discarding language and cultural specific cases. in summary, our contributions include: ## related work word analogy recognition in nlp benchmarks for word analogy recognition @xcite @xcite examine mostly linear relations between words @xcite . such analogies can often be effectively solved by vector arithmetic for neural word embeddings, such as word2vec @xcite and glove @xcite . recent studies @xcite also test such ability of pre-trained language models (plms) @xcite @xcite on these benchmarks. an exceptional benchmark is @xcite , where they build a knowledgeenhanced analogy benchmark that leverages word sense definitions in a commonsense knowledge base @xcite . however, these benchmarks are mainly set up for evaluating learned representations, and few of them ever investigated the analogical reasoning skills for neural models. thus, the goal of this work largely differs from this line of research, as we aim to build a knowledge-intensive benchmark to teach neural models analogical reasoning for correct thinking. ## explainable analogical reasoning in this work, we consider a classic setting of analogical reasoning within nlp: recognizing word/term analogies. 1 this task can be formulated as multiplechoice question-answering. given a query tuple q with k (two or three) terms, and m candidate answer tuples @xmath1, the goal is to find the most analogous one in the candidates to the query. we advocate that reasoning is about giving reasons explaining a prediction. in order to teach machines to analogize as humans do, we draw inspiration from theories in cognitive psychology to design the forms of explanations. ## methods in this section, we describe the baseline methods in both qa and eg tasks in easy and hard modes. we mainly evaluate some of the state-of-the-art language models for solving tasks in e-kar. some implementation details are reported in appendix a. ## results and analysis in the experiments, we wish to answer two questions: q1) can models do knowledge-intensive analogical qa? q2) can models generate rational reasons for analogical thinking? categorization of problems we first manually categorize the relational types of problems in e-kar according to a pre-defined schema. unlike free text, we are unable to induce a comprehensive set of relations that covers all candidates due to the complexity of cse problems. as a result, we carefully assign at least one relation to each query. to facilitate analysis, we also try to assign relations to each candidate and query in the development and test set, ending up covering 76% of the candidates and 100% of the queries. we refer to several sources of word analogy definitions and textbooks for analogy tests (listed in appendix b), and categorize the relations into five meta-relations (as well as their coverage in the test set) and several accompanying sub-relations: complete sub-relations are presented in appendix b, as well as their definitions and examples. how do humans solve analogical reasoning problems? in contrast to machines, humans achieve in e-kar 77.8% accuracy in hard mode and 83.3% in easy mode, indicating the challenge of this task as well as showing that current sota language models still fall far behind human performance. we also find the trend of human performance is generally aligned with machines, with accuracy boost (also ∼5 points) when prompted with query explanations. error analysis for qa we further conduct an error analysis based on the results in e-kar (zh) predicted by a fine-tuned roberta (large). the erroneous ones are classified based on the manually annotated meta-relations and sub-relations of queries, which is a fine-grained tool for analyzing a model's predictions. figure 2(a) shows that the model performs poorly on nearly all meta-relations, with r2 (extension) being the most error-prone one (only 40.3% accuracy, normalized) and r3 (intension) being the least one (56.8% accuracy). one of the most prominent reasons is that r2 and r3 rely heavily on commonsense and encyclopedic knowledge and reasoning skills such as commonsense and world knowledge, at which current models easily fail. figure 2(b) shows the error rate of sub-relations with more than 10 samples. consistent with figure 2 (a), the three most error-prone sub-relations (is_a, part_of and juxtaposition_of ) all belong to r2 (extension). besides, the model seems to do well in linguistic knowledge, with verb-object achieving only 33.3% error rate. these findings may shed light on future directions for knowledge-intensive reasoning with language models. 6.2 can models rationalize analogical thinking? we report the automatic evaluation results of generated explanations in ## conclusion and discussion in this work, we propose a first-of-its-kind benchmark e-kar (in both chinese and english) for explainable analogical reasoning, which sets a concrete playground and evaluation benchmark to boost the development of human-like analogical reasoning algorithms. the e-kar benchmark is featured by its rich coverage in knowledge and welldesigned free-text explanations to rationalize the analogical reasoning process. preliminary experiments show that this benchmark provides a rather difficult challenge for prevailing language models. however, there are still many open questions to be addressed. for example, humans solve the analogy problems in a trial-and-error manner, i.e., adjusting the abduced source structure and trying to find the most suited one for all candidate answers. however, the explanation annotation process in e-kar (not the eg task) is mostly post-hoc and reflects only the result of reasoning. such explanations cannot offer supervision for intermediate reasoning, though it is an interesting question whether an intelligent model should be deeply supervised at every step @xcite . furthermore, e-kar only presents one feasible explanation for each problem, whereas there may be several. this benchmark also invites reasoning models that can effectively interact with extra knowledge. it remains to be a great challenge to generate and evaluate factually correct explanation text. especially, how to generate negated facts is relatively under-explored in the research community but of much importance. finally, whether the analogical qa system can correctly exploit explanations and background knowledge is also worth investigating, which may intersect with research on debiasing @xcite . we hope this work to be a valuable supplement to future research on natural language reasoning, especially for research on analogical reasoning and explainable nlp.