An empirical analysis of existing systems and datasets toward general simple question answering (2020)

Abstract: In this paper, we evaluate the progress of our field toward solving simple factoid questions over a knowledge base, a practically important problem in natural language interface to database. As in other natural language understanding tasks, a common practice for this task is to train and evaluate a model on a single dataset, and recent studies suggest that SimpleQuestions, the most popular and largest dataset, is nearly solved under this setting. However, this common setting does not evaluate the robustness of the systems outside of the distribution of the used training data. We rigorously evaluate such robustness of existing systems using different datasets. Our analysis, including shifting of training and test datasets and training on a union of the datasets, suggests that our progress in solving SimpleQuestions dataset does not indicate the success of more general simple question answering. We discuss a possible future direction toward this goal.

## introduction simple factoid question answering over a knowledge base is an important task in natural language understanding. although it only deals with factoid questions about a single entity and a predicate, they cover much of the real user queries @xcite , and also, accurate mapping of these is a critical subproblem in semantic parsing-based complex query generation @xcite @xcite . simplequestions @xcite is the largest and most popular dataset on this task. it was recently argued that this task, given abundant training data, is nearly solved with standard techniques in machine learning @xcite . in this paper, we present a thorough empirical analysis to assess whether the success of one particular dataset indicates the success of the task itself in general. to this end, we evaluate the behaviors of four existing qa systems targeting simplequestions @xcite @xcite , across four different datasets @xcite @xcite , under different conditions. one of our research goals is to evaluate the robustness of a model trained on a single dataset against questions that are outside of the distribution of the training data. such robustness evaluation is recently actively studied in other language understanding tasks @xcite @xcite while little effort has been made on question answering over a knowledge base, though, in practice, it would be critical because a practical system has to be robust on real user queries, which may be outliers in the training data. our experiments suggest that, while simplequestions is the largest, the examples are too simple and the success on it does not indicate progress in factoid question answering in general. for example, we show that, under the same training data size, the system's accuracy on simplequestions gets about 10 points higher than that on webquestions. although the simplicity of simplequestions is pointed out in past work @xcite , our work provides an empirical evidence that this is indeed the case with a careful comparison and manual analysis using the standardized datasets. given our analysis, we suggest two possible future directions. one is to invent a clever novel data creation method that would be scalable while avoiding bias as much as possible. in this respect, we point out that a recent attempt by freebaseqa @xcite is not successful, and that significant bias still exists. another is to exploit useful information from the large dataset of simplequestions in a better way. in the last analysis, we demonstrate that a simple approach of training on a union of the datasets @xcite is not satisfactory toward this end, calling for a more sophisticated method of exploiting useful features across datasets effectively. ## datasets we use four qa datasets over a knowledge base (kb) as our target datasets. these datasets were selected because they share a common kb (freebase), and a large portion of each dataset comprises of factoid questions, which are the main focus of this paper. a factoid question asks a single fact, or a triple (subject, predicate, object) on a kb, where the object corresponds to the answer. for example, "which country is albert bolender from?" corresponds to a fact (albert bolender, people.person.nationality, ?), where the placeholder found in the kb is the target object, which is united states. free917 @xcite this is the first dataset for machine learning-based semantic parsing over freebase. it contains 917 questions on a subset of freebase, called freebase commons, covering 81 domains. @xcite find that each question tends to contain words that are directly related to the target freebase predicate. an example is "what genre of music is b12?", for which the gold predicate is music.artist.genre. we use an annotated free917 dataset by @xcite . 1 webqsp @xcite this is an extension of webquestions @xcite with gold sparql query on each question, which is missing in the original dataset. aiming at creating more natural questions than free917, each question is derived from the google suggest api, followed by filtering by crowd workers. consequently, the authors observe a larger divergence between the question words and predicates, such as "what music did beethoven compose?", for the aforementioned predicate music.artist.genre. this dataset contains 4,737 questions. simplequestions @xcite this is the largest dataset in our experiments, containing over 100,000 questions answerable by a single fact. contrary to webquestions, each question in this dataset is created from a sampled fact in freebase, which is then verbalized and paraphrased by a crowd worker. possibly due to this procedure starting from a kb fact, we find that, as in free917, this dataset also tends to verbalize a predicate with directly related terms, such as "what type of music . . .?" for music.artist.genre. foot_2 this approach eases the collection of a lot of data and is popular in data creation for semantic parsing @xcite @xcite . however, we will see in section 4.3 that it also tends to introduce certain biases, which affect models' generalization. the authors also define a subset of freebase called fb2m that covers 2m entities and 5k predicates, including all entities appearing in webquestions, and create all questions from this subset. freebaseqa @xcite this is the latest dataset aiming at more difficult factoid questions than simplequestions while maintaining the scale of data size. specifically, the questions in this dataset are first sampled from triviaqa @xcite and then filtered by heuristics to collect factoid questions answerable on freebase. although the authors argue that their procedure reliably eliminates non-factoid questions, we find several problems in this dataset, which we describe in section 4.2. ## systems now we describe four systems that we compare across the datasets. since we only deal with single fact questions in this paper, all questions can be answered by correctly predicting a subject entity e and a relation r on the kb. to search for the best pair, all systems in this paper employ a pipeline, which is comprised of three different submodules below: the systems differ in each submodule. there are also some minor variations in pipeline constructions. we select the four target systems considering their high accuracies on simplequestions, as well as the availability of the code. 4 buboqa (mohammed et al., 2018) 5 in this system, both entity linking and relation prediction are modeled with simple classifiers. despite its simplicity, this approach outperforms several more complex architectures @xcite . specifically, for entity linking, a trained lstm first detects the entity spans, which are then heuristically mapped to the candidate kb entities and scored with the levenshtein distance to the canonical entity label. relation prediction is performed independently by another classifier on top of a different lstm. finally, the best combination of (ê, r) is found according to a weighted sum of these two module scores. 6 this is an extension of an even simpler baseline of @xcite , and a similar approach is employed in @xcite . note that this system treats relation prediction as classification among the predicates appearing in the training data. this means that it cannot solve zero-shot relation prediction, which occurs to some extent especially in the dataset transfer experiment (section 4.3). on the other hand, the other three systems theoretically can handle them, as described in the following. hierarchical residual bilstm (hr-bilstm) @xcite on this system (and the next, kbqa-adapter), relation prediction is performed differently, not by classification on a fixed set of relations, but by mapping on a shared embedding space for kb relations and texts. this model simply encodes both question tokens and relation tokens (e.g., "music artist genre" for music.artist.genre) by different encoders. relation candidates are then ranked by cosine similarity between the outputs of two encoders. this method allows us to calculate the score of an unseen relation. for this system, since @xcite do not release their code, we use the implementation by @xcite . unfortunately, this implementation only includes the relation prediction module rather than the full pipeline. we thus try to reproduce the pipeline described in @xcite , using the entity linking module of buboqa. see appendix a for details. we use the same pipeline for kbqa-adapter. kbqa-adapter (wu et al., 2019) 7 this is an improvement to hr-bilstm with an additional adversarial adapter coupled with the relation encoder. the motivation of this adapter is to improve the zero-shot relation prediction performance. to this end, the adapter receives a relation embedding for r provided by kg embeddings, which is jointnre @xcite , transforming it to an embedding space where unseen relations can be handled properly. we employ the same pipeline with the buboqa entity linker for this system. knowledge embedding-based qa (keqa) (huang et al., 2019) 8 this system also builds on an external knowledge graph embedding, transe @xcite , which is used as the more direct and central part in the system. given a knowledge graph embedding, which is fixed, this model tries to map each question into an entity embedding ê and relation embedding r, using separate lstms. we expect ê to be close to the gold node embedding in the graph and r to the gold relation embedding. also, we would expect that the transition defined by the embedding model (e.g., addition for transe), f (ê, r), will get close to the answer node embedding. the query generation step of the system selects the (ê, r, ô) triple based on this intuition, by minimizing the summed distances from embeddings corresponding to ê, r, and ô to the obtained encoded embeddings. settings and notes foot_9 for all models, we employ the best architectures and hyperparameters reported in the paper or a related document. for the first three systems, we set the number of entity linking outputs as 50, and that of relation prediction as 5, which are the default settings for buboqa. note also that these three systems share the same entity linking module of buboqa. for evaluation, following the standard practice of simplequestions, we evaluate the accuracy of predicted (subject, predicate) pairs. ## conclusion through several experiments, we have shown that although the system performance on simplequestions dataset is getting better and close to the upper bound, that does not indicate a more general success of simple factoid question answering overall. the main cause of this mismatch is that, as we have seen, there is often an inverse relationship between the ease of data collection and naturalness of collected questions. we found that although the data creation of simplequestions, starting from a kb fact and verbalizing by a crowd worker, is advantageous in terms of scalability, the resulting dataset is too simple, as we demonstrated that the systems can achieve high accuracies even with a limited amount of training data. webquestions, on the other hand, is a collection of real user queries with several challenges including ambiguous entity mentions, but such questions are more difficult to collect, in particular in terms of the coverage of entities and relations. it is ideal that systems trained on a simpler and scalable dataset become robust on the questions outside of the distribution of the training data, but our experiment on dataset transfer suggests that the current approaches do not achieve this. we suppose there are two possible directions toward general simple question answering, or question answering over a knowledge base in general. the first is to improve the dataset quality. we need to create a dataset, that is real and challenging, while still being scalable. freebaseqa can be seen as an attempt toward this goal, but we found that this dataset has several issues. another direction is to invent a model or a learning mechanism that can generalize robustly from biased datasets. our data union can be seen as a simple approach toward this end, but we found that current models do not exploit useful information beyond each target dataset. more sophisticated approaches, such as distributionally robust optimization @xcite , may help. another promising way is relying on strong pretrained language models, including bert @xcite . we have not included bertbased models in this paper, because its application on simplequestion has not outperformed a simpler baseline so far @xcite , and it is also nontrivial to integrate bert with knowledge graph embeddings, which is necessary for keqa-based approach and is currently actively studied @xcite . the integration of such approaches, along with robustness evaluation as done in this paper, will be of practical importance toward robust question answering not specific to a single dataset.