c TBLS : Augmenting Large Language Models with Conversational Tables (2023)

Abstract: Optimizing accuracy and performance while eliminating hallucinations of open-domain conversational large language models (LLMs) is an open research challenge. A particularly promising direction is to augment and ground LLMs with information from structured sources. This paper introduces Conversational Tables cTBLS, a three-step architecture to retrieve and generate dialogue responses grounded on retrieved tabular information. cTBLS uses Transformer encoder embeddings for Dense Table Retrieval and obtains up to 125% relative improvement over the retriever in the previous state-of-the-art system on the HyrbiDialogue dataset. cTBLS then uses a shared process between encoder and decoder models to perform a coarse+fine tabular knowledge (e.g., cell) ranking combined with a GPT-3.5 LLM response generator to yield a 2x relative improvement in ROUGE scores. Finally, human evaluators prefer cTBLs +80% of the time (coherency, fluency) and judge informativeness to be 4x better than the previous state-of-the-art.

## introduction equipping conversational ai with multimodal capabilities broadens the range of dialogues that humans have with such systems. a persisting challenge in multimodal conversational ai is the development of systems that produce conversationally coherent responses grounded in textual and nontextual modalities @xcite . it is well-established that large language models (llms) possess real-world knowledge stored within their parameters, as demonstrated by recent research @xcite . nevertheless, the incorporation of conversation-specific extrinsic knowledge into these models to yield precise responses remains an active area of investigation. while humans can easily retrieve contextual information from tables by examining rows and columns, llms often struggle to identify relevant information amidst conversational distractions. hybridialogue @xcite , a dataset of conversations grounded on structured and unstructured knowledge from tables and text, introduces the task of responding to messages by utilizing information from external knowledge and prior dialogue turns. the authors also present an approach and experimental results on hybridia-logue that represents the current state-of-the-art (sota). this paper proposes an extension to the sota approach of hybridialogue in the form of conversational tables (ctbls) foot_0 , a novel threestep encoder-decoder architecture designed to augment llms with tabular data in conversational settings. in the first step, ctbls uses a dual-encoder transformer-based @xcite dense table @xcite to retrieve the correct table from the entire corpus based on the user's query. the second step employs a fine-tuned dual-encoder transformer to track system state and rank cells in the retrieved table according to their relevance to the conversation. finally, ctbls utilizes gpt-3.5 to generate a natural language response by prompting it with the ranked cells. while previous research separated knowledge retrieval and response generation between encoder and decoder models, this paper demonstrates that llm decoders can perform these tasks jointly when prompted with knowledge sources ranked by language model encoders. furthermore, by pre-training the dense table retriever to perform retrieval over a corpus of tables, ctbls can be extended to new knowledge sources without retraining, by appending additional knowledge to the corpus. compared to the previous sota, experiments table retrieval identifies the table most relevant to the initial query. the retrieved table is provided to the state tracker for follow-up queries. state tracking ranks cells in the table based on their ability to answer a follow-up query. response generation utilizes a llm decoder provided with the ranked cell information and the follow-up query to convert tabular data into a natural language response and continue the conversation. details on individual components are provided in section 3. on ctbls show up to 125% relative improvement in table retrieval and a 2x relative improvement in rouge scores. in addition, human evaluators prefer ctbls +80% of the time (coherency, fluency) and judge informativeness to be 4x better than the previous sota. our contributions are as follows: 1. the introduction of conversational tables (ctbls), a novel three-step encoder-decoder architecture designed to augment llms with tabular data in conversational settings. 2. experimental results demonstrating that dense table retrieval, which utilizes neural models fine-tuned with a summary of tabular information, outperforms sparse techniques based on keyword matching for table retrieval. 3. the presentation of evidence that augmenting state-of-the-art llm decoders using knowledge sources ranked by encoder language models leads to better results on automatic (rouge-precision) and human (coherence, fluency, and informativeness) evaluation for knowledge-grounded response generation while limiting the number of api calls to these models. this paper presents the ctbls system and demonstrates its application to the hybridia-logue dataset. in section 2, we review the existing literature in the fields of table question answering and knowledge grounded response generation. section 3 describes the various components of ctbls as presented in figure 1 . in section 4, we evaluate the performance of ctbls against previous methods for conversations over tables and report experimental results from automatic and human evaluations. finally, section 5 concludes the paper and outlines potential directions for future research. ## method the challenge of developing conversational systems grounded in tabular information consists of three tasks, namely table retrieval, system state tracking, and response generation. table retrieval requires identifying the most relevant table in the dataset based on a given natural language query. system state tracking is responsible for ranking the cells in the table, enabling the system to provide responses to follow-up queries about the table. finally, response generation involves converting the ranked cells into a natural language response. ## conclusion in this paper, we introduce conversational tables (ctbls), a system designed to address multiturn dialogues that are grounded in tabular data. ctbls separates tabular dialogue into three distinct tasks, specifically table retrieval, system state tracking, and response generation. the dense table retrieval system of ctbls yields an enhancement of up to 125% relative to keyword-matching based techniques on the hybridialogue dataset, with regard to top-1 accuracy and mean reciprocal rank @ 10. furthermore, ctbls conducts system state tracking utilizing a two-step process shared between encoder and decoder models. this methodology results in natural language responses exhibiting a 2x relative improvement in rouge scores. human evaluators favor ctbls +80% of the time (coherency and fluency) and judge informativeness to be 4x better than the previous stateof-the-art.