B lend SQL : A Scalable Dialect for Unifying Hybrid Question Answering in Relational Algebra (2024)

Abstract: Many existing end-to-end systems for hybrid question answering tasks can often be boiled down to a “prompt-and-pray” paradigm, where the user has limited control and insight into the intermediate reasoning steps used to achieve the final result. Additionally, due to the context size limitation of many transformer-based LLMs, it is often not reasonable to expect that the full structured and unstructured context will fit into a given prompt in a zero-shot setting, let alone a few-shot setting. We introduce BlendSQL, a superset of SQLite to act as a unified dialect for orchestrating reasoning across both unstructured and structured data. For hybrid question answering tasks involving multi-hop reasoning, we encode the full decomposed reasoning roadmap into a single interpretable BlendSQL query. Notably, we show that BlendSQL can scale to massive datasets and improve the performance of end-to-end systems while using 35% fewer tokens. Our code is available and installable as a package at https://github.com/parkervg/blendsql.

## introduction problem decomposition in settings involving both fine-tuning and in-context learning, it has been demonstrated that generating explicit intermediate steps for complex problems can enhance accuracy. in their aptly titled "scratchpad" framework, @xcite revealed that predicting the complete execution trace led to better results when fine-tuning language models to generate execution output. more recent work has shown that prompting llms to decompose complicated problems into smaller reasoning steps before generating the final answer can improve results @xcite @xcite . however, at their core, these methods use natural language as their intermediate reasoning representation. with blendsql, we argue that natural language alone is inherently a lossy and ambiguous intermediate reasoning representation. instead, by decomposing a problem into a sql-like syntax, we allow for more deterministic reasoning capabilities and better interpretability of intermediate steps by leveraging the compositional nature of relational algebra @xcite . agents previous works have explored agents: an experts-based framework where the main task is broken down into parts that each agent or expert foot_0 carries out @xcite @xcite @xcite . furthermore, these works are based on the presence of a 'routers' or 'planners' that understand the given task and break it down into smaller tasks. although rule-based routers have been widely used, the ability of llms to generalize to complex texts has enabled their use as planners. llms, as planners, process the input and as output generate text containing the list of subtasks @xcite , the agent(s) to use from a given set, what are the inputs to the agent(s) @xcite @xcite , what is expected as response from the agent(s) @xcite . with the scope of the tasks carried out by the agents being very narrow, recent works have primarily focused on the planner component of the framework. for the same, harnessing the in-context learning capabilities of the llm @xcite or finetuning the planner llm @xcite @xcite @xcite have been the two main methods. additionally, works have used llms' in-context learning capability for dataset creation @xcite . the parser in blendsql acts as a planner and generates the apis or ingredients to use in answering the input question. however, it differs from the previous works as it goes beyond generating a single string representation of complicated problem decomposition and simplifies the agent framework using robust and complicated abstract syntax tree (ast) structures to be encoded into a single string. it is also important to note that blendsql does not just rewrite the input statement using formal constructs @xcite , it also adds information using the reference database, input prompt, and internal knowledge of the language models. text-to-sql on the widely used text-to-sql dataset spider @xcite , many works demonstrate impressive performance in the fewshot or zero-shot setting @xcite @xcite . even in settings where text-to-sql is not the focus, existing work details the effectiveness of a common and well-understood intermediate representation like sql for other reasoning tasks. @xcite frames dialogue state tracking (dst) as a text-to-sql task by encoding domains and slots from @xcite into a serialized database format and using sql as an intermediate representation, showing significant improvement over the "traditional" key-value style prediction format. hybrid question answering unlike text-to-sql, hybrid question answering involves an im-plicit decision to access tabular data, unstructured data, or both @xcite @xcite . @xcite make this more explicit by routing between end-to-end models and a parser that generates an intermediate sql representation. their implementation frames the two as isolated, specialized systems which are unable to pass information to one another. for the scope of this work, we define hybrid question answering as answering questions using a corpus of tabular and unstructured multi-modal content. while this multi-modal content can encompass text, images, audio, and video, the primary focus of this paper is on unstructured text in conjunction with tabular data. the potential to adapt this process to other formats, such as images, is also discussed in appendix c. one prior work, unik-qa @xcite , proposes converting structured tabular data into unstructured text data. our work does the opposite, and instead converts all unstructured text data into tabular data. empirically, unik-qa improves on several knowledge-base (kb) qa benchmarks using a kb-to-text preprocessing model. for example, for the kb-to-text model mentioned in the unik-qa paper, the relationship between different entities is assumed to have a simple triplet representation: <subj, pred, @xmath0. however, relationships @xmath1.g., audio, documents, or column names of the table), which would then demand more efforts in developing dedicated kb-to-text models. neuro-symbolic frameworks most similar to our work is the binder method presented in @xcite , which integrates llm reasoning into symbolic languages like sql and python. while we take inspiration from their neurosymbolic framework for reasoning, blendsql differs in the following ways. (1) we create an api specialized for sql, enabling reasoning over multi-table databases containing unstructured content, and are capable of executing and optimizing queries containing join statements, aliases, conditional table expressions, and most other sqlite operations (2) we enable constrained decoding according to the database schema (3) we design our api such that users are able to easily create any number of custom functions themselves to use within a blendsql script. ## blendsql: overview we implement blendsql as a superset of the sqlite relational database management system (rdbms) foot_1 . an executed blendsql script returns a smoothie object as output, containing the final result and intermediate reasoning steps taken. to make a smoothie, we need some ingredients and a blender. we describe these components below. ## datasets due to the joint usage of sql-like syntax and llm-based ingredients, we expect our blend-sql to perform well on hybrid question answering datasets. to validate our hypothesis, we experimented on several challenging benchmark datasets requiring reasoning over both structured and unstructured knowledge. ## results we direct the curious reader to ## error analysis to better understand the advantages and limitations of blendsql, we annotate randomly sampled 50 blendsql question-answer pairs from hy-bridqa mistakes. our chosen denotation accuracy metric judges each datapoint as either a 1 or 0. out enough", making the results not directly comparable. * to normalize different output formats (such as "seven" and "7"), we employ the denotation accuracy metric used by @xcite , from the script here. it is unclear if the previous methods use the same evaluation script. of the 3,466 datapoints in the hybridqa dev set, we see 1,464 with an accuracy score of 0. although the number 1,464 seems intimidating, it is important to remember that we achieved a literature-comparable score of 57.76 and to realize that there are many false negatives from those errors. to distinguish those false negative errors, we describe annotation categories as the parent categories containing false negative errors and the blendsql error categories as the child categories containing only true negative errors. ## open model evaluations closed models like gpt-4-0613, while powerful, lack a level of transparency and interpretability that is valuable to the research community. to this end, we also evaluate two open-source language models built for coding tasks: deepseek-coder-6.7b-instruct 11 @xcite and starcoder2-15b 12 @xcite . both models are autoregressive decoders with a context window of 16k tokens. we run experiments on the hybridqa dataset, and use each respective model as both the parser and blender to best reproduce the core experiments with gpt-4-0613. ## future work we hope to study the ability of llms not only to use functions from a previously unseen sql dialect but also to create functions that generalize certain logical patterns, as described in @xcite . additionally, while instruction fine-tuned llms show impressive abilities in executing blendsql scripts, we hope to experiment with more specialized and inexpensive models for modular tasks such as fact verification and dpr retrieval. finally, as many errors came from blendsql scripts that were unable to execute properly, we hope to explore hybrid qa tasks as a form of interactive semantic parsing @xcite . ## conclusion we introduce blendsql, a scalable dialect for problem decomposition and hybrid questionanswering. results show competitive performance on popular benchmarks while using only ∼10 fewshot examples. additionally, on the hybridqa dataset, we improve the performance of a naive end-to-end system by 8.63%, while using 45% fewer prompt tokens. we open-source all code and present an installable python package for future researchers to further explore blendsql as an intermediate representation for hybrid questionanswering.