A System for Answering Simple Questions in Multiple Languages (2023)

Abstract: Our research focuses on the most prevalent type of queries— simple questions —exemplified by questions like “What is the capital of France?”. These questions reference an entity such as “France”, which is directly connected (one hop) to the answer entity “Paris” in the underlying knowledge graph (KG). We propose a multilingual Knowledge Graph Question Answering (KGQA) technique that orders potential responses based on the distance between the question’s text embeddings and the answer’s graph embeddings. A system incorporating this novel method is also described in our work. Through comprehensive experimentation using various English and multilingual datasets and two KGs — Freebase and Wikidata — we illustrate the comparative advantage of the proposed method across diverse KG embeddings and languages. This edge is apparent even against robust baseline systems, including seq2seq QA models, search-based solutions and intricate rule-based pipelines. Interestingly, our research underscores that even advanced AI systems like ChatGPT encounter difficulties when tasked with answering simple questions. This finding emphasizes the relevance and effectiveness of our approach, which consistently outperforms such systems. We are making the source code and trained models from our study publicly accessible to promote further advancements in multilingual KGQA.

## introduction a knowledge graph (kg) is a collection of subjectpredicate-object triples, for example ⟨paris, capi-tal_of, france⟩. large kgs are valuable resources for many tasks, including question answering (qa) @xcite . knowledge graph question answering (kgqa) is an active research area, as well as a popular application. even though all major web search engines implement kgqa capabilities -kg results can be easily * the first two authors contributed equally. recognized in their 'smart answers' -there are few operational kgqa research prototypes available online. a rare example is qanswer @xcite , a rule-based kgqa system over wikidata. there are also only a few free kgqa codebases available @xcite @xcite . in this work, we focus on simple questions such as "what is the capital of france?". there exists an opinion that the task of answering such questions is nearly solved @xcite , but openly available systems are scarce and do not support multiple languages. besides, their performance, as will be observed from our work, is still far from perfect even for models based on deep neural networks specifically pre-trained on qa data. in our work, our aim is to address these limitations of the prior art. we developed a kgqa method m3m (multilingual triple match) based on text-to-graph embedding search. the key idea illustrated in figure 1 is to combine a pre-trained multilingual language model for question representation and pre-trained graph embeddings that represent kg nodes and edges as dense vectors. in the training phase, we learn separate projections of the question text embeddings to the subject, predicate, and object of the kg triple corresponding to the question-answer pair. in the test phase, we first fetch a set of candidate kg triples based on the question's word n-grams and extract named entities to make the process more computationally efficient. then, we rank candidate triples according to the sum of three cosine similarities -between the embeddings of the triple's components and respective projections of the question's embeddings. finally, the object of the top-ranked triple is returned as an answer. our approach build upon @xcite expanding the method beyond a single kg and a single monolingual dataset. we experimented with the de facto standard english kgqa dataset simplequestions, which is based on the now deprecated freebase, to allow comparison with previous art. moreover, we conducted experiments with several wikidata-based datasets: simplequestionswd (a wikidata mapping from the original benchmark), russian/english rubq 2.0 dataset, as well as a recent mintaka dataset covering nine languages. our experiments demonstrate the applicability of the proposed method in different kgs and languages. our online demo (see figure 2 ) implements two kgqa methods: (1) a t5 model fine-tuned on qa data and (2) our approach dubbed m3m based on embedding search. we believe that a combination of an online demo, publicly available code, as well as evaluation results on several datasets will contribute to future developments in the field of mul-tilingual kgqa. to summarize, our contributions are the following: we also release pre-trained models and provide an api making seamless integration into nlp applications possible. foot_0 ## related work there are two main directions in kgqa research: semantic-parsing-and retrieval-based approaches. semantic parsing seeks to map a natural language question to a meaning representation (e.g. sparql query language) that can be directly executed against a kg. @xcite provide a comprehensive survey of semantic parsing methods in kgqa. they also point out that kg schema richness (diversity of entity and relation types), irregularities and incompleteness of kg, as well as its large size make the application of semantic parsing methods to kgqa especially challenging. the latter line of research follows information retrieval's notion of proximity between the query and sought entity. early methods relied on the lexical match of the question and the kg elements (entity labels, descriptions, and relation names) @xcite . in their pioneering work @xcite proposed to embed question words and kg components in a shared space, with question representations close to the answers' ones. this approach was further developed in a series of studies @xcite @xcite . notably, more recent approaches leverage existing knowledge graph embeddings that have been used for a variety of kg tasks such as link prediction and entity classification @xcite and pre-trained language models such as bert applicable to a multitude of nlp tasks @xcite . the embedding-based approach can be extended to complex kgqa @xcite . a comprehensive overview of kgqa methods can be found in recent surveys @xcite and a monograph @xcite . large language models (llms) accumulate not only linguistic, but also factual knowledge, and thus can be considered as alternatives to kgs. @xcite showed that bert (a rather small model by today's standards) possessed some relational knowledge without fine-tuning. @xcite experimented with the t5 model @xcite as an open-domain qa system. in particular, after fine-tuning on question-answer pairs, at inference time, the model was given a question and returned an answer without accessing any external knowledge source. the results were quite encouraging; the model with "salient span masking" (ssm) pre-training proposed by @xcite (2020) performed best. @xcite found that llms memorise knowledge about common entities but fall short in the case of less common ones. they proposed to complement llm-based qa with zero-shot and few-shot prompting with adaptive evidence retrieval. @xcite evaluate qa capabilities of @xcite and several gpt versions (gpt3, gpt3.5, and chatgpt) in eight kgqa datasets. the results show that llms are yet quite far beyond sota; chatgpt outperforms its contenders. multilingual kgqa has been relatively rarely addressed in the literature. proposed solutions include question translation @xcite , language-agnostic translation of syntactic parses into meaning representations @xcite , and unsupervised induction of bilingual lexicons for word-level translation of training examples @xcite . we hope that the advent of larger multilingual datasets with wikidata annotations @xcite will stimulate interest in this timely topic. ## m3m: multilingual triple match qa in this section, we describe the core of our contribution: a novel method for one-hop question answering over a knowledge graph. the proposed method dubbed m3m for multilingual triple match relies on matching independently kgs subjects, objects, and predicates with the questions' representation. overall, the approach can be considered as an improved version of the method by @xcite with which we perform extensive comparisons in the remainder of the paper among other baselines. ## experiments in this section, we present the benchmarking of the presented system on several one-hop kgqa datasets for freebase and wikidata knowledge graphs. the method is compared to strong baselines. both the datasets and the baselines are described below. ## conclusion in this study, we introduced m3m, a multilingual model, along with an open implementation, devised for one-hop knowledge base question answering. our approach leverages the use of a multilingual text encoder and pre-trained kg embeddings, which are aligned using a triple projection method of a question to subject/relation/object of kg triple to facilitate efficient answer search in the embedding space. for simple questions, our system not only outperforms previous strong alternatives, including rule-based approaches, embeddings-based similarity search, and pre-trained sequence-to-sequence neural models, but also excels when compared to advanced ai models like chatgpt. these comparative results were drawn from a comprehensive battery of one-hop qa datasets, including both monolingual and multilingual data.