A Pretraining Numerical Reasoning Model for Ordinal Constrained Question Answering on Knowledge Base (2021)

Abstract: Knowledge Base Question Answering (KBQA) is to answer natural language questions posed over knowledge bases (KBs). This paper targets at empowering the IR-based KBQA models with the ability of numerical reasoning for answering ordinal constrained questions. A major challenge is the lack of explicit annotations about numerical properties. To address this challenge, we propose a pretraining numerical reasoning model consisting of NumGNN and NumTransformer, guided by explicit self-supervision signals. The two modules are pretrained to encode the magnitude and ordinal properties of numbers respectively and can serve as model-agnostic plugins for any IR-based KBQA model to enhance its numerical reasoning ability. Extensive experiments on two KBQA benchmarks verify the effectiveness of our method to enhance the numerical reasoning ability for IR-based KBQA models.

## introduction knowledge base question answering (kbqa) aims at finding answers from the existing knowledge bases (kbs) such as freebase @xcite and dbpedia @xcite for the given questions expressed in natural language. kbqa has emerged as an important research topic in the last few years @xcite @xcite , as the logically organized entities and relations in kbs can explicitly facilitate the qa process. two mainstream methods including the semantic parsing based (sp-based) models @xcite @xcite ) and the information retrieval based (ir-based) models @xcite @xcite are commonly studied to solve kbqa task. the sp-based models heavily rely on the intermediate logic query parsed from the natural language question, which turns out to be the bottleneck of performance improvement @xcite . on the contrary, the ir-based models directly represent and rank the entities in a questionaware subgraph based on their relevance to the question. such an end-to-end paradigm is easier to train and more fault-tolerant. however, most of the ir-based models focus on the single-or multihop relation tasks. to answer the example question "which is the largest city in china?" in figure 1 , the answer "beijing" is supposed to encode not only the magnitude of its area but also the ordinal relationship with "largest"-the ordinal determiner in the question. existing ir-based models are not explicitly aware of the magnitude and ordinal properties of entities, making the entity representations fall short in the ability to support such numerical reasoning. in view of the issue, this paper targets at empowering the ir-based kbqa models with the ability of numerical reasoning to address the ordinal constrained questions. ordinal constraint is summarized as one of the most important constraints via web query analysis @xcite and ordinal is also defined as the second fundamental measurement to capture data in the forms of surveys foot_0 . some efforts have been made on numerical reasoning for machine reading comprehension (mrc) @xcite @xcite . for example, given a question and a passage from which the answer can be inferred, numnet @xcite is an end-to-end model to learn the number embeddings and the nonnumerical word embeddings together, which are encoded by graph neural network (gnn) @xcite and bert @xcite respectively. qdgat @xcite further wires the numbers and the words in a same graph and encode them together by gnn. however, most of them implicitly infer number embeddings based on the qa pairs without the explicit annotation of the magnitude and ordinal relationships of numbers. such weak supervision signals bring difficulties to infer accurate number embeddings, which becomes more prominent when the ordinal supervision signals are rarely available in existing kbqa datasets. in fact, the three well-known kbqa benchmarks, metaqa @xcite , we-bquestionsp (webqsp) @xcite and complexwebqeustions (cwq) @xcite only contain 0, 101 and 1821 ordinal constrained questions respectively. to tackle the above challenge, we propose a pretraining method with additional self-supervision signals to capture two critical ingredients for ordinal constrained kbqa: number embeddings which satisfy the above two ingredients are capable of numerical reasoning for ordinal constrained questions. to obtain such number embeddings, we propose two pretraining modules numgnn and numtransformer. the former one pretrains a gnn upon the constructed number graphs by a number-aware triplet loss function to preserve the relative magnitude, and the latter one pretrains a transformer upon the constructed question-aware number graphs by a number prediction loss function to capture ordinal relationships. compared with the weak supervision signals from qa pairs, such self-supervision signals explicitly denote the numerical properties. after pretraining, numgnn and numtransformer can be attached as model-agnostic plugins into any ir-based kbqa model to infer number embeddings. by fusing the number embeddings into the entity embeddings learned by the basic model, the numerical reasoning ability of the basic model is enhanced. finally, we evaluate our method on two benchmarks of kbqa: webqsp and cwq. experimental results demonstrate that numgnn plus num-transformer, serving as plugins of alternative irbased kbqa models, can achieve substantial and consistent improvement (+2.4 -14.8% in terms of accuracy) on the ordinal constrained questions. ir-based methods @xcite @xcite @xcite @xcite ) directly retrieve answer candidates from the kbs and represent them to encode the semantic relationships with the questions. these methods are more faulttolerant, but are unable to deal with ordinal constrained questions. this paper aims to enhance the ir-based models for numerical reasoning. numerical reasoning. numerical reasoning has been studied for various tasks such as word embedding @xcite , arithmetic word problems (awp) @xcite @xcite @xcite . word embedding and awp are a little far from our task. similar to kbqa, mrc also aims to answer questions, but infers the answers from passages instead of kbs. to enable numerical reasoning, numnet @xcite adopts a numerically-aware gnn to encode numbers and qdgat @xcite further extends the number graph with additional words. however, they are all end-to-end models weakly supervised by the final answers. this paper studies the explicit supervision signals about numerical figure 1 : the whole reasoning process includes basic reasoning on the relation subgraph g q r and numerical reasoning on the attribute subgraph g q a . for numerical reasoning, we first perform the pretrained numgnn and numtransformer to infer value embeddings and then attach them into the entity embeddings learned by the basic reasoning. the final prediction is based on the entity embeddings. ## method in this section, we first introduce the ordinal constrained kbqa. then the framework of our model is provided, followed by detailed descriptions of its components. ## conclusion the paper proposes a pretraining numerical reasoning model for ordinal constrained kbqa. via pretraining by explicit supervision signals, numgnn and numtransformer are capable of capturing the magnitude and ordinal properties of numbers. by attaching them as plugins into any ir-based kbqa model, the numerical reasoning ability of the model 1860 20011030 20031202 20050000 20050610 20061017 20130711 20131023 0.1265 0.1137 0.0979 -0.0458 -0.2542 -0.2150 0.1432 < < < < < < < < < < < original numbers reduced 1-dimensional scores of the number embeddgins can be enhanced. the experimental results on two benchmarks verify the effectiveness of our model. other types of constraints, such as multiple topic entities, type and aggregation constraints, are to be explored in the future.