Tackling Temporal Questions in Natural Language Interface to Databases (2022)

Abstract: Temporal aspect is one of the most challenging areas in Natural Language Interface to Databases (NLIDB). This paper addresses and examines how temporal questions being studied and supported by the research community at both levels: popular annotated dataset (e.g. Spider) and recent advanced models. We present a new dataset with accompanied databases supporting temporal questions in NLIDB. We experiment with two SOTA models (Picard and ValueNet) to investigate how our new dataset helps these models learn and improve performance in temporal aspect.

## introduction natural language interface to databases (nlidb) is a task to bridge the gap between storing complex structured data in databases (dbs) and retrieving structured data from databases using natural language questions. an nlidb system allows users to access information stored in a db by using questions in natural language (e.g. english). in reality, temporal aspect is a common dimension existing in any dbs since it is realistic and practical to store data with associated date/time in dbs. however, it is challenging to retrieve information from dbs with temporal aspect due to difficulties and limitations such as 1) complex and limited association between certain entities/attributes and temporal info in db that reflects different states of data in different time frames, and 2) complexity of natural language expressions to decipher the actual value of temporal expressions in given questions to correctly derive data from dbs. temporal aspect is a common yet distinct and complex dimension that needs to be handled carefully in nlidb. given its difficulties and complexity, temporal aspect in nlidb is currently under-attended by the research community. first of all, it is difficult to design and create database that supports temporal dimension. next, it is more challenging to create pairs of questions and associated sql queries that support the learning of temporal aspect in nlidb (discussion in section 4). our contribution is threefold: i) we investigated how temporal aspect was defined and annotated in the well-known spider dataset, ii) we created the new dataset tempq4nlidb supporting the learning and understanding temporal questions in nlidb, and iii) we experimented with two sota models -valuenet @xcite and picard @xcite -to understand how they learn and handle temporal questions. to the best of our knowledge, this work is one of the first attempts trying to explore this research area to support temporal questions in nlidb. ## background and related works nlidb or text-to-sql is a long standing nlp task with many references on its complexity and achievements obtained recently @xcite @xcite . although there is no official definition of temporal questions in nlidb, the study @xcite mentioned about temporal questions with respect to temporal databases @xcite . in contrast, the study of temporal questions is a strong interest in question answering (qa) community. in the scope of this paper, we adopt a definition of temporal questions in qa for our work that is "a temporal question is any question, which contains a temporal expression, a temporal signal, or whose answer is of temporal nature." @xcite . in qa, temporal questions can be answered by temporal information embedded in semantic relations and timeline between events in corpus or taxonomy, whereas db records associated with temporal dimension are answers in nlidb. the spider dataset @xcite and spider challenge was introduced in 2018 where participating teams can have their systems evaluated on an unseen test set which is not available to public. there are two spider sub-challenges, the first one for sql inference without values and the second for systems that handle the values in sql queries. many attempts with different approaches, based on neural networks, especially on encoder-decoder architecture, have continuously improved the state of the art (sota). first, the sequence-to-sequence approach was introduced @xcite @xcite in which a neural network with very good proven qualities in translation tasks translates an english query into an sql query. these systems infer the sql formula directly as a standard translating task from one language to another. instead of translating into sql, another approach translates questions into a representation that captures semantics of a question, namely intermediate representation (ir) @xcite @xcite . the network learns a more structured and compact form of the query itself. from ir to sql is a deterministic process: a context free grammar is used to convert one into another. the authors build on the work of @xcite that used abstract syntax tree (ast). the decoder infers the ir as an ast representation of the query. among others, valuenet @xcite ) not only uses ir approach but also extends the context free grammar to include values and became one of the best systems in the spider challenge in 2020. recently, picard @xcite , a stateof-the-art algorithm for constrained decoding was introduced and achieved top rank in spider challenge. it relies on the structure and the content of the database as well as the knowledge encapsulated in the t5 language model @xcite . ## discovering temporal aspect in spider spider @xcite is a popular large-scale complex and cross-domain dataset supporting text-to-sql task consisting of 10,181 questions and 5,693 unique complex sql queries on 200 databases with multiple tables covering 138 different domains. since there is no description of temporal aspect in spider, we investigate temporal questions in this dataset. ## tempq4nlidb dataset for nlidb since temporal dimension is practical and crucial for dbs in any real-world applications, and the lack of dataset for temporal questions in nlidb, we created tempq4nlidb -a dataset of temporal condition questions for studying and experimenting with the recent advanced nlidb models. ## error analysis we present four error types in predictions made by picard and valuenet for temporal questions. type 1. models cannot detect temporal values in question, thus, no corresponding filter created.