SPARQL ing Database Queries from Intermediate Question Decompositions (2021)

Abstract: To translate natural language questions into executable database queries, most approaches rely on a fully annotated training set. Annotating a large dataset with queries is difficult as it requires query-language expertise. We reduce this burden using grounded in databases intermediate question representations. These representations are simpler to collect and were originally crowdsourced within the Break dataset (Wolfson et al., 2020). Our pipeline consists of two parts: a neural semantic parser that converts natural language questions into the intermediate representations and a non-trainable transpiler to the SPARQL query language (a standard language for accessing knowledge graphs and semantic web). We chose SPARQL because its queries are structurally closer to our intermediate representations (compared to SQL). We observe that the execution accuracy of queries constructed by our model on the challenging Spider dataset is comparable with the state-of-the-art text-to-SQL methods trained with annotated SQL queries. Our code and data are publicly available ( https://github.com/yandex-research/sparqling-queries ).

## introduction the difficulty of collecting and annotating datasets for the task of translating a natural language question to an executable database query is a significant obstacle to the progress of the technology. the most popular multi-database text-to-sql dataset, spider @xcite , has 10k questions, which is smaller compared to question answering datasets of other types: the drop dataset with text paragraphs has 97k questions @xcite and the gqa dataset with images has 22m questions @xcite . the spider dataset was created by 11 yale students proficient in sql, and it is difficult to scale such a process up. recently, @xcite proposed the question decomposition meaning representation, qdmr, which is a way to decompose a question into a list of "atomic" steps representing an algorithm for answering the question. importantly, they developed a crowdsourcing pipeline to annotate qdmrs and showed that it can be used at scale: they collected 83k qdmrs for questions (all in english) coming from different datasets (including spider) and released them in the break dataset. qdmrs resemble database queries but are not connected to any execution engine and cannot be run directly. moreover, qdmrs were collected when looking only at questions and thus have no information about the database structure. entities mentioned in qdmr steps usually have counterparts in the corresponding database but do not have links to them (grounding). in this paper, we build a system for translating a natural language question first into qdmr and then into an executable query. we use modified qdmrs, where the entities described with text are replaced with their database groundings. our system consists of two translators: a neural network for text-to-qdmr and a non-trainable qdmr-to-sparql transpiler. see figure 1 , for an illustration of our system. in the text-to-qdmr part, we use an encoderdecoder model. our encoder is inspired by rattransformer @xcite and uses bert @xcite or grappa @xcite . our decoder is a syntax-guided network @xcite designed for our version of the qdmr grammar. we trained this model with full supervision, for which we automatically grounded qdmrs for a subset of spider questions. in the second part of the system, our goal was to translate grounded qdmrs into one of the existing query languages to benefit from the efficiency of database software. the most natural choice would be to use sql, but designing such a translator 1 figure 2 : database from figure 1 converted to the rdf format (the rdf graph). the red nodes correspond to the values from teacher table, the green ones -to the values from school table. arcs correspond to the relations between primary key and other values of the same row (arc:tbl:col) and along the foreign keys (arc:t_src:c_src:t_tgt:c_tgt). we evaluated our system with the execution accuracy metric on the spider dataset (splits by @xcite ) and compared it with two strong baselines: text-to-sql systems bridge @xcite and smbop @xcite from the top of the spider leaderboard. on the cleaned-up validation set, our system outperforms both baselines. on the test set with original annotation, our system is in-between the baselines. additionally, we experimented with training our models on extra data: items from break without schema but with qdmrs. this teaser experiment showed potential for further improvements. this paper is organized as follows. sections 2 and 3 present two main parts of our system. section 4 contains the experimental setup, section 5our results. we review related works in section 6 and conclude in section 7. ## text-to-qdmr parser in this section, we describe our approach to generating a grounded qdmr lf from a given question and a database schema. our encoder consists of bert-like pretrained embeddings @xcite and a relation-aware transformer @xcite . our decoder is an lstm model that generates an abstract syntax tree in the depth-first traversal order @xcite . ## conclusion in this paper, we proposed a way to use the recent qdmr format @xcite as annotation for generating executable queries to databases given a question in a natural language. using qdmrs is beneficial because they can be collected through crowdsourcing potentially easier than correct database queries. our system consists of two main parts. first, we have a learned text-to-qdmr translator that we built on top of the recent rat-sql system @xcite ) and trained on an annotated with qdmrs part of the spider dataset. second, we have a non-trainable qdmr-to-sparql translator, which generates queries executable on databases in the rdf format. we evaluated our system on the spider dataset and showed it to perform on par with the modern textto-sql methods (bridge and smbop) trained with full supervision in the form of sql queries. we also showed that additional qdmr annotations for questions not aligned with any databases could further improve the performance. the improvement shows great potential for future work.