Q . Can Knowledge Graphs be used to Answer B oolean Questions? A . It’s complicated! (2020)

Abstract: In this paper we explore the problem of machine reading comprehension, focusing on the BoolQ dataset of Yes/No questions. We carry out an error analysis of a BERT-based machine reading comprehension model on this dataset, revealing issues such as unstable model behaviour and some noise within the dataset itself. We then experiment with two approaches for integrating information from knowledge graphs: (i) concatenating knowledge graph triples to text passages and (ii) encoding knowledge with a Graph Neural Network. Neither of these approaches show a clear improvement and we hypothesize that this may be due to a combination of inaccuracies in the knowledge graph, imprecision in entity linking, and the models’ inability to capture additional information from knowledge graphs.

## modeling knowledge graph data our manual inspection of the results of one baseline system run reveals that approximately 20% of erroneous cases are questions involving some property of an entity or concept, or some hierarchical relationship between entities. an example of the former is (3) and the latter is (4). we hypothesize that adding knowledge graph data could help in answering such questions, as well as examples such as ( 1 ) and ( 5 ) below where the entity in the question is referred to using a different name in the passage. we use the cloudapi foot_3 to annotate text with tokens, part of speech tags, named entities with freebase foot_4 kg identifiers (mids), numbers, dates and verbnet foot_5 roles which can be used for establishing relations between entities. ## analysis conceptnet even after the filtering described in section 3.1.1, we observe that often the relations from conceptnet are too general and do not add new information, e.g. "cookie jar is a type of jar". such relations are already part of the language model. @xcite show that bert contains relational knowledge and has a strong ability to recall factual knowledge without fine-tuning. furthermore, some entities are missing, e.g. there is a "tom hanks" entity but no "meg ryan" entity, or the entity "dragon ball" contains only non-english connections, confirming the general coverage issue of kgs. 12 sensitivity we observe that the gnn is sensitive to the learning rate and hyper-parameters. better tuning may compensate for the difference in performance wrt to the bert baseline. ## conclusion in this work, we take a closer look at a bert baseline system on the boolq dataset, which reveals some inconsistencies in the data and some instability in the model. we try two approaches to integrating knowledge graph information, one based on augmenting the passage text and another using a graph neural network. neither are successful. one culprit is the lack of coverage of con-ceptnet and another is related to accuracy of the entity recognition. we also suggest that the number of questions where suitable kg data is needed and could be found might just not be enough for the models to learn from.