H idden T ables and P y QT ax: A Cooperative Game and Dataset For T able QA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies (2023)

Abstract: A myriad of different Large Language Models (LLMs) face a common challenge in contextually analyzing table question-answering tasks. These challenges are engendered from (1) finite context windows for large tables, (2) multi-faceted discrepancies amongst tokenization patterns against cell boundaries, and (3) various limitations stemming from data confidentiality in the process of using external models such as gpt-35-turbo. We propose a cooperative game dubbed “HiddenTables” as a potential resolution to this challenge. In essence, “HiddenTables” is played between the code-generating LLM “Solver” and the “Oracle” which evaluates the ability of the LLM agents to solve TableQA tasks. This game is based on natural language schemas and importantly, ensures the security of the underlying data. We provide evidential experiments on a diverse set of tables that demonstrate an LLM’s collective inability to generalize and perform on complex queries, handle compositional dependencies, and align natural language to programmatic commands when concrete table schemas are provided. Unlike encoder-based models, we have pushed the boundaries of “HiddenTables” to not be limited by the number of rows - therefore we exhibit improved efficiency in prompt and completion tokens. Our infrastructure has spawned a new dataset “PyQTax” that spans across 116,671 question-table-answer triplets and provides additional fine-grained breakdowns and labels for varying question taxonomies. Therefore, in tandem with our academic contributions regarding LLMs’ deficiency in TableQA tasks, “HiddenTables” is a tactile manifestation of how LLMs can interact with massive datasets while ensuring data security and minimizing generation costs.

## introduction encoder-based approaches in contextually analyzing table question-answering tasks for language models typically prioritize and highlight the methods' achievement in accuracy @xcite . however, in many cases, a prerequisite for these approaches to achieve such accuracy is the exposition of tabular content in its entirety and the indulgent ingestion of tokens @xcite @xcite . such liberal dispositions towards privacy and efficiency can be deemed as impractical in the tangible deployment process of language models within institutions. moreover, such necessities to expose the underlying data begs the question of whether the model actually understands the question to provide an accurate answer. in essence, our endeavor is also an intellectual pursuit to answer the "chinese room argument" with regards to language models @xcite . therefore, we propose an alternative approach for table questionanswering tasks -a cooperative game dubbed "hid-dentables". hiddentables is comprised of two agents: an "oracle" and a "solver", in which the latter generates code to answer user queries relying solely on the oracle's instructions and relaying of schema. in other words, the game is played without the solver knowing the tabular content. the solver's code is then evaluated by the secure oracle that relays the answer to the user or asks followup questions to the solver. figure 1 summarizes the environmental set-up that our method enables between a user and gpt-3.5-turbo. therefore this paper sets forth a general system architecture that can be employed across a myriad of taxonomies and tabular formats. we find that the accuracy of gpt-3.5-turbo has decreased with our cooperative game albeit with lesser tokens and tightened privacy. in summary, hiddentables and its pertinent experiments have brought forth the following contributions to the academic community: figure 1: overview of our system apparatus to encourage hiddentables. the setup requires two agents, an oracle and the solver, which may or may not be on the same device. for our purposes, the solver is a gpt-3.5-turbo llm agent that handles generation off-site, and therefore potentially offers risk of adversarial attacks. we outline the conversation between our agents, which is a message-passing channel that transfers solution code along with follow-up questions, without exposing any information from the datalake. finally, the oracle will provide the answer to the user. ## related work since the advent of transformer-based attention models, pre-trained language models have shown remarkable success in learning and encoding the semantics of tabular content @xcite . methods employing encoder-based architectures rely on masked language modeling (mlm) to learn semantics and dense representations of tabular content. yet they are pre-trained on natural language text tokenized by byte-pair-encoding or wordpiece @xcite which can misalign with tabular structure. tapas @xcite ) employed an encoder that is pre-trained with whole word masking, tabert (yin et al., 2020) leveraged masked column prediction and cell value recovery to learn structure, and grappa (yu et al., 2021) augmented pre-training with synthetic sql to inject structural properties into the model. in contrast to these encoder-based approaches, tapex @xcite relies on a bart encoder-decoder backbone @xcite to encode tables and generate answers in an autoregressive fashion. however, hiddentables relies solely on the generative power of autoregressive decoders @xcite and instruction-aligned models trained with reinforcement learning from human feedback role instructions schema query " you are an ai assistant that can answer questions from tables by writing python pandas code. " + " you will receive a question and will have to write code to best answer it. " + " you will only be provided the columns to the table and their type. " " follow these instructions:" + " 1) you must write python code to operate on a pandas dataframe named df" + " 2) use reset_index() after any groupby operation involving aggregation, and sort" + …. + " 8) if you think you cannot answer the question, look for columns such as note, comments, etc. that may contain the answer and return the correct row item" " the table df has the following columns: (rlhf) @xcite , to generate solutions based on prompts @xcite rather than fine-tuning. furthermore, prior work shows that language models can more effectively solve problems when decomposing them into steps or a chain of thought @xcite . hiddentables is inspired by using chain of thought through code, as demonstrated by @xcite for robotic programs, action plan generation in robotics @xcite , web browsing @xcite , tool apis @xcite , or the generation of valid programs for arithmetic computation @xcite . @xcite explores how llms can improve their chain of thought reasoning via intermediate actions and interactions with external sources. furthermore, binder @xcite demonstrated a neural-symbolic approach to mapping questions to a program, building upon the work in @xcite for semantic parsing and code generation. also, previous literature has explored how llms can interact with themselves through intermediate followups @xcite , chained llm prompts @xcite , or cascades @xcite . @xcite has proposed how llms can be encouraged to generate their own prompts for solving tasks. finally, memprompt @xcite demonstrated that memories of errors and user feedback can be incorporated as part of the conversation to help prevent repetitive mistakes. ## methodology our proposed framework is inspired from the "chinese room argument" -to what extent could lan-guage models truly comprehend natural language and align language to the correct solution when only given the table schema? in hiddentables, two agents exist: the oracle and the solver. the clear delineation between these two agents' respective roles not only allows the user to test the model's holistic ability to comprehend tabular content but also enables the preservation of privacy with regards to the underlying data on-premise. in this context, our proposed apparatus allows the two agents to engage in a conversation, in which the oracle may ask questions and the solver will generate code that could solve the oracle's question. next, the oracle will evaluate and follow-up which enables the solver to correct any mistakes or misunderstandings. this game is played for a maximum of seven rounds to prevent infinite cycles between the agents. throughout this process, no data entries are exposed to the solver -the solver must produce executable code relying solely on the schema and the set of instructions. ## conclusion in this work, we introduced a novel approach to evaluating the generalizability of llms across 3 table question-answering datasets. by creating a cooperative game that withholds the underlying data from the the model, hiddentables challenges the solver to make educated guesses via programmatic commands and operators to be in a state of successful retrieval. we have shown that this construct enables a computationally efficient large-scale testing of llms on massive datasets in tandem with ensuring the security of the tabular data. also, our study provides insights that this task is considerably more difficult than traditional holistic models -yet lends itself to potentially large-scale industrial applications. we have also quantified this efficiency by showcasing the number of generated tokens in contrast with those of conventional models. we also contribute pyqtax, a dataset aligning generated python code to table questions and various taxonomies for 116,703 samples. overall, our work provides a promising direction for future research in the field of table question-answering and has devised a novel construct in the deployment process of language models.