Database reasoning over text (2021)

Abstract: Neural models have shown impressive performance gains in answering queries from natural language text. However, existing works are unable to support database queries, such as “List/Count all female athletes who were born in 20th century”, which require reasoning over sets of relevant facts with operations such as join, filtering and aggregation. We show that while state-of-the-art transformer models perform very well for small databases, they exhibit limitations in processing noisy data, numerical operations, and queries that aggregate facts. We propose a modular architecture to answer these database-style queries over multiple spans from text and aggregating these at scale. We evaluate the architecture using WikiNLDB, a novel dataset for exploring such queries. Our architecture scales to databases containing thousands of facts whereas contemporary models are limited by how many facts can be encoded. In direct comparison on small databases, our approach increases overall answer accuracy from 85% to 90%. On larger databases, our approach retains its accuracy whereas transformer baselines could not encode the context.

## introduction question answering (qa) over text has made significant strides in recent years owing to the availability of new datasets and models. machines have surpassed human performance on the well-known squad task @xcite where models extract answer spans from a short passage of text. the subsequent body of work has further considered incorporating retrieval from large corpora such as wikipedia @xcite @xcite to identify relevant information, conditioning answer generation (chen 1 https://github.com/facebookresearch/ neuraldb facts: (8 of 500 shown) ## architecture for querying nldbs to address the aforementioned challenges, we propose an instance of a neural database architecture @xcite that operates over textual facts with parallelizable non-blocking operators before aggregating the results. the three core components of the architecture, shown in figure 2 , are a support set generator (ssg) which retrieves small sets of relevant facts called support sets, a parallelizable non-blocking select-project-join (spj) operator which generates intermediate answers that can be unioned to produce the final answer, and an optional aggregation stage which uses conventional computation to perform numerical reasoning. the key insight underlying our architecture is to leverage neural models for what they excel at, namely, reasoning over a small set of facts. neural spj operator given a single support set and a query, the spj (select-project-join) operator outputs a machine readable intermediate representation of the answer that can be generated from the support set. for example, given the query "who was born in montevideo?" and the support set {"mario sagario was born in montevideo, uruguay, ..."}, the neural spj would output the entity literal mario sagario. examples of outputs are provided in figure 3 . the spj operator is performing three functions: because the spj operator is run in parallel, it can scale independently of the limitations on the size of the input of a single transformer. in contrast, the use of self-attention when encoding all facts as one input precludes parallelization, has high latency, and is limited by the memory required to compute the self-attention. by using the spj operator to perform query-dependent information extraction, aggregations can be performed over the generated outputs using conventional computation, which trivially scales to thousands of operands. furthermore, this allows large result sets to be generated by the model, whereas accurately decoding long sequences using an encoder-decoder architecture remains an open challenge @xcite . ## the wikinldb dataset in this section we introduce wikinldb, a novel dataset for training nldbs which is generated by transforming structured data from wikidata @xcite into natural language facts and queries. wikidata stores triples of the form (s,r,o), where r is a relationship between the subject s and the object o, e.g., (tim cook, employedby, apple). the scale and breadth of wikidata enables us to generate databases of many sizes and variety. facts to automate generation of questions and answers, sentences must be grounded in wikidata identifiers. one approach to generate facts would be to use templates or collect them through grounded information extraction datasets such as t-rex @xcite . however, to ensure wider linguistic variety as well as accuracy of the mapping, we use verbalizations of knowledge graph triples that are synthesized through a sequence to sequence model. concretely, we use generated sentences from kelm @xcite , which are not grounded with wikidata ids, and generate a post-hoc mapping back to wikidata.for example, given the sentence: "the slice of life manga series the film lives on was written by osamu tezuka." we map it to the wikidata triple (q11332517,p50,q193300). our mapping is a two-step process: firstly, we look up entity names from wikipedia, returning multiple matches for osamu tezuka, and secondly filter these based on which have an author relations to the slice of life in the wikidata graph. while out of scope for this paper, this technique could be applied to generate training datasets for novel domains. wikinldb uses both atomic facts in kelm (about a single relation of an entity) or composite facts (about multiple relations). queries following previous work on large-scale question answering @xcite , queries are generated using templates. for each relation and operator, multiple templates were written by the authors where placeholders can be replaced with the subject and objects for each relation. while multiple templates are used to ensure variety, these are limited in diversity in comparison to the facts. templates were generated for the first 25 relations on wikidata with mapped data in kelm. to generate queries that require joins we apply the same technique, combining to combine two or more connected relations, chaining the entities. we further select the 15 most popular relations and generate additional templates which chain the two relations. for example, we chain (y,locatedin,z) and (x,employedby,y) to create a template for the query "does @xmath0z?". data quality we manually inspect randomly selected queries and facts and score them using the categories introduced in this section. for queries, we sample 70 instances, 10 for each query type. we score each query for fluency and intelligibility. out of 70 queries, only one question was marked as non-fluent due to a typo which was corrected for the final dataset. all 70 queries were intelligible. we observed that the clarity of some queries depended on the facts in the database to provide context (e.g. "who is male?"), but otherwise met the task requirements. to assess the quality of mapped facts from kelm, a sample of 50 was evaluated based on 6 categories: intelligibility, fluency, inclusivity (conveying information from all the mapped relations), faithfulness to these relations, and whether extraneous information (not in the mapped relations) is present. 49/50 facts were intelligible and 45/50 facts were fluent. the remaining 5 had redundant information or missing conjunctions. 50/50 facts contained all mapped relations and 48/50 were faithful to these relations. 8/50 facts had extraneous information for relations that could not be mapped. the relations that could not be mapped are not used for query generation and did not affect how answers were automatically generated. ## implementation we use the huggingface @xcite of epochs were selected through maximizing the exact-match (em) accuracy on a held-out validation set for the tasks. for each experiment, we train 3 separate models with different seeds and report mean accuracy. the spj models are only trained on the small database of 25 facts and applied to larger databases at test time. for most queries, we measure correctness using exact match (em), which is 1 if the answer string generated by the model is exactly equal to the reference answer and 0 otherwise. this metric is used to score outputs where either a boolean, null answer, string or numeric answer is expected. when a set of results is returned, we compute the f 1 score considering exact matches of set elements. when comparing models and reporting results, we report macro-averages over all instances in the test set. we collectively refer to this as answer accuracy. ## experiments & results we first consider the suitability of transformer models over small databases of 25 facts comparing two information retrieval settings: perfectir, which is representative of other question answering approaches that combine an information retrieval system to select only the facts needed to answer a query, and wholedb, where the entire database is encoded by the model, assessing resilience to unrelated information and noise. the overall scores, in ## related work database queries require reasoning over a large set of relevant and non-redundant facts and performing aggregation. while in-roads have been made to perform discrete reasoning and computation over passages @xcite , with explicit computation @xcite or differentiable modules 3099 25 50 100 250 500 1000 number of facts in db 0.2 0.4 0.6 0.8 answer accuracy spj perfectir ssg+spj t5 + tf-idf t5 + dpr ( @xcite , these use only a single passage rather than requiring aggregation over large numbers of facts from different texts. multi-hop question answering requires finding supporting evidence in multiple documents (see @xcite @xcite for datasets facilitating this research). in answering multi-hop questions, the works decompose the question into simpler sub questions @xcite , or condition each hop on the previously retrieved documents @xcite . while tasks such as complexwebquestions (talmor and berant, 2018) and break @xcite focus on complex queries that can be broken down into simpler ones, our focus is on setbased and aggregation queries where the complexity comes from the need to retrieve and process a large number of non-redundant relevant facts. in contrast to the set and count tasks in babi @xcite , where each query is based on a small context (less than 20 facts), our dataset scales from databases of 25 facts to 1000. bridging the gap between unstructured natural language data and database-style querying has been a long-standing theme in database research @xcite . the work on information extraction has developed techniques for translating segments of natural language text into triples that can be further processed by a database system. there has been significant work on translating queries posed in natural language into sql queries on a database whose schema is known @xcite @xcite , with extensions to semi-structured data and knowledge bases @xcite . more recently, systems such as break @xcite and sharc @xcite have trained models to translate a natural language query into a sequence of relational operators (or variants thereof). ## conclusions database systems are the workhorse of data analysis but they require a pre-defined schema. part of their power stems from the fact that a data analyst can explore the data by easily posing a wide variety of queries. given the rise in the amount of data that is becoming available in text, images and other modalities, we would like to build systems that enable the flexibility of posing complex queries against such data, but without the need for a pre-defined schema. this paper proposed an architecture for neural databases and the associated wikinldb dataset, as first steps towards realizing a system for querying multi-modal data. our architecture is capable of overcoming the limitations of transformer models because it runs multiple transformers in parallel, each taking a small set of facts. consequently, nldbs can scale to large databases. additional research is required in order to scale nldbs to larger datasets, more complex queries, and to multi-modal data. in particular, one of the key components of the architecture is the ssg module that retrieves the relevant facts to feed to each instance of the neural spj. we believe that in practice, the semantics of the application will provide a strong hint on which facts may be relevant. for example, when querying a large corpus of socialmedia posts, each post is a candidate support set as long as the query does not require joining data from multiple posts. in addition, we assumed that our databases describe a snapshot of the world. in practice, we may have facts that override previous ones (e.g., 'samantha works for apple', followed by 'samantha works for twitter') and we would need to reason about which facts should be ignored.