R e T ra C k: A Flexible and Efficient Framework for Knowledge Base Question Answering (2021)

Abstract: We present Retriever-Transducer-Checker (ReTraCk), a neural semantic parsing framework for large scale knowledge base question answering (KBQA). ReTraCk is designed as a modular framework to maintain high flexibility. It includes a retriever to retrieve relevant KB items efficiently, a transducer to generate logical form with syntax correctness guarantees and a checker to improve transduction procedure. ReTraCk is ranked at top1 overall performance on the GrailQA leaderboard and obtains highly competitive performance on the typical WebQuestionsSP benchmark. Our system can interact with users timely, demonstrating the efficiency of the proposed framework.

## introduction knowledge base question answering (kbqa) is an important task in natural language processing that aims to satisfy users' information needs based on factual information stored in knowledge bases. over the years, it has attracted a great deal of research attention from academia and industry. early kbqa systems are generally rule-based. they rely on predefined rules or templates to parse questions into logical forms @xcite , suffering from coverage and scalability problems. recently, researchers usually focus more on neural semantic parsing approaches. these data-driven parsing methods @xcite @xcite @xcite significantly improve the state-of-the-art (sota) performance on kbqa tasks. although various neural semantic parsing methods have been proposed for kbqa, there are few works investigating how to leverage the advantages of sota models to build a comprehensive system, and how to fit the system with practical application purpose (e.g., balancing effectiveness and efficiency). to investigate, we identify two key issues hindering the development of kbqa systems. on the one hand, there is a lack of a generic and extensible framework for kbqa. for example, the popular sempre foot_0 toolkit @xcite provides infrastructures to develop statistical semantic parsers for kbqa with rich features, but its performance and scalability are inferior to recent neural semantic parsing methods. the tranx toolkit 4 (yin and neubig, 2018) employs a transition-based neural semantic parser to model the logical form generation procedure as a sequence of tree-constructing actions under grammar specification. however, tranx does not include the essential retriever components used in grounding, and thus does not support kbqa by now. on the other hand, recent neural semantic parsing methods mostly emphasize performance on benchmark datasets while neglecting the efficiency (speed) dimension. this limits the understanding of how designed approaches fit into real applications. for example, the popular query graph generation methods generate and rank a set of query graphs @xcite @xcite . since all query graph candidates keep in line with the knowledge base (kb) structure, these methods take full advantage of the kb. however, they suffer from poor efficiency due to the large number of candidates and heavily querying on kb. to verify that, we performed a preliminary study on available sota models 5,6,7,8,9,10 . according to our study, these models either have difficulties in supporting interactive online services, or limit the candidate space for specific datasets, which makes them difficult to apply in practice. to this end, we present retrack, a practical framework for large scale kbqa. we hope re-track can help standardize the kbqa model design process and lower the barrier of entry for new practitioners. retrack is designed with the following principles in mind: ## retrack framework given an input question q, retrack parses the question into a logical form which can be deterministically converted into a sparql query to retrieve answers from the knowledge base k. generally k consists of two parts: an ontology o ⊆ t ×r×t , which defines the schema structure, and the fact triples f ⊆ e × r × (e ∪ t ∪ l). here, t is the set of types, r is the set of relations, e is the set of entities, and l is the set of literals. as shown in fig. 1 , retrack consists of three components: retriever, transducer and checker. the retriever consists of an entity linker, which links explicit entity mentions to corresponding entities, and a schema retriever, which retrieves relevant schema items (types and relations) mentioned either explicitly or implicitly in the question. given the retrieved kb items (entities, types, and relations), the transducer employs a grammar-based decoder to generate the logical form with syntax correctness guarantees. meanwhile, the transducer interacts with the checker to discourage generating programs that are semantically inconsistent with kb. to make retrack more accessible and interpretable for end users, we build a user interface. as shown in fig. 2 , users can type a question in the text box. the interface then displays retrieved kb items, a graph visualization of predicted logical forms, generated sparql query and predicted answer (s). the schema items selected by our transducer are shaded. besides, users can refer to more information of any kb item by clicking on the subsequent "detail". next, we will introduce each component in detail. ## conclusion we present retrack, a semantic parsing framework for kbqa. retrack is flexible and efficient, achieving strong results on two distinct kbqa datasets. we hope that retrack will be beneficial for future research efforts towards developing better kbqa systems. ledell wu, fabio petroni, martin josifoski, sebastian riedel, and luke zettlemoyer. 2020. scalable zeroshot entity linking with dense entity retrieval. in proceedings of the 2020 conference on empirical methods in natural language processing, emnlp 2020, online, november 16-20, 2020, pages 6397-6407. association for computational linguistics. wen-tau yih, ming-wei chang, xiaodong he, and jianfeng gao. 2015. semantic parsing via staged query graph generation: question answering with knowledge base. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing of the asian federation of natural language processing, acl 2015, july 26-31, 2015, beijing, china, volume 1: long papers, pages 1321-1331. the association for computer linguistics. wen-tau yih, matthew richardson, christopher meek, ming-wei chang, and jina suh. 2016. the value of semantic parse labeling for knowledge base question answering. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers), pages 201-206. pengcheng yin and graham neubig. 2018. tranx: a transition-based neural abstract syntax parser for semantic parsing and code generation. in proceedings of the 2018 conference on empirical methods in natural language processing, emnlp 2018: system demonstrations, brussels, belgium, october 31 -november 4, 2018, pages 7-12. association for computational linguistics.