Exploring the Impact of Table-to-Text Methods on Augmenting LLM -based Question Answering with Domain Hybrid Data (2024)

Abstract: Augmenting Large Language Models (LLMs) for Question Answering (QA) with domain specific data has attracted wide attention. However, domain data often exists in a hybrid format, including text and semi-structured tables, posing challenges for the seamless integration of information. Table-to-Text Generation is a promising solution by facilitating the transformation of hybrid data into a uniformly text-formatted corpus. Although this technique has been widely studied by the NLP community, there is currently no comparative analysis on how corpora generated by different table-to-text methods affect the performance of QA systems.In this paper, we address this research gap in two steps. First, we innovatively integrate table-to-text generation into the framework of enhancing LLM-based QA systems with domain hybrid data. Then, we utilize this framework in real-world industrial data to conduct extensive experiments on two types of QA systems (DSFT and RAG frameworks) with four representative methods: Markdown format, Template serialization, TPLM-based method, and LLM-based method. Based on the experimental results, we draw some empirical findings and explore the underlying reasons behind the success of some methods. We hope the findings of this work will provide a valuable reference for the academic and industrial communities in developing robust QA systems.

## introduction enhancing the performance of large language models (llms) in domain-specific question answering (qa) has been a focus of research, predominantly employing two key approaches @xcite : domain-specific fine-tuning (dsft) which involves training llms on the domain-specific corpus @xcite , and retrieval-augmented generation (rag) which utilizes a domain-specific corpus as an external knowledge base @xcite . these approaches, leveraging the inherent text processing strengths of llms, have been widely adopted in text-only scenarios, yielding significant improvements @xcite . however, real-world data in many domains typically exists in a hybrid format, comprising not only text but also substantial volumes of semi-structured tables, as observed in e.g., scientific literature and medical reports @xcite . these tables frequently appear alongside text within the same document, providing semantically supplementary or complementary information crucial for a comprehensive understanding of the content @xcite . in exploring the potential of leveraging hybrid data to enhance the performance of llms, it is crucial to effectively integrate these data, ensuring the coexistence of text and tables. the current methods for handling the heterogeneity of text and tables have significant drawbacks: 1) directly flattening tables by concatenating cells row by row not only results in the loss of structural information embedded in the original table but also severs the informational links between cells @xcite . 2) mapping text and tables to different vector spaces separately and then integrating them, not only increases complexity but also disrupts the semantic connection between the two types of data @xcite . one promising solution is table-to-text generation @xcite , which aims to generate natural language statements that faithfully describe the information in the provided table. through this, we can transform hybrid data into a unified natural language representation that is more suitable for use by llms, while also preserving the important information from the tables and the semantic connections between the data. al-though table-to-text generation has been widely studied by the nlp community, there is currently no comparative analysis on how corpora generated by different table-to-text methods affect the performance of domain-specific qa systems. in this work, we address this research gap by two steps. first, we innovatively integrate tableto-text generation into the framework of enhancing llm-based qa systems with domain hybrid data. then, we utilize this framework to conduct extensive experiments on two types of qa systems (dsft and rag paradigms) with four representative table-to-text methods. we choose the following four strategies: 1) markdown format; 2) template serialization; 3) tplm-based method; 4) llmbased method. these strategies differ in complexity and underlying technology. the markdown and template serialization offer simplicity, while the tplm-based and llm-based methods leverage the capabilities of advanced language models to generate more nuanced text. in terms of implementation, we collect a realworld hybrid dataset called ict-data, by extracting text and tables from numerous documents about information and communication technology (ict) products. it is important to note that the text contained in tables accounts for approximately 18% of the total content in ict-data (based on word count statistics). we employ different table-to-text methods to process the tables in ict-data, obtaining different ict corpora. these corpora are then utilized to build qa systems. moreover, we create a benchmark dataset named ictqa, which consists of qa pairs based on the knowledge of ict-data. this dataset is particularly suitable for evaluating enhanced llms, as it includes some industry-specific knowledge not covered in the general llms training stage. to our knowledge, our research is the first to comprehensively compare different table-to-text strategies on llm-based qa systems enhanced by domain hybrid data. our main findings are as follows: table -to-text generation @xcite @xcite aims to create natural language descriptions from semi-structured tabular data, such as web tables. as shown in figure 1 , we apply four representative we next provide a detailed introduction of these four methods. method resource speed diversity markdown cpu fast low template cpu fast moderate tplm-based gpu moderate high llm-based gpu or api low very high and bart @xcite , on specific table-to-text generation task datasets @xcite . in this paper, we utilize the mvp model @xcite , which initially pre-trains the bart model on numerous natural language generation datasets, followed by fine-tuning on various cross-domain table -to-text datasets. it allows customized adjustment of the output through fine-tuning, offering higher flexibility and domain adaptability, while requiring more computational resources. ## experimental setup qa systems of the dsft paradigm. within the dsft paradigm, we utilize meta's opt (1.3b to 13b) @xcite and llama2-base (7b, 13b) @xcite as foundation models. the opt models offer variable sizes to enhance robustness. to mitigate training costs, we employ the qlora @xcite strategy for pretraining and instruction fine-tuning. the instruction template can be found in appendix a.3. qa systems of the rag paradigm. we use the llama2-chat models (7b, 13b, and 70b) and gpt-3.5-turbo for inference. we divide the corpus into smaller chunks, ensuring the integrity of sentences and keeping their lengths below 3000 characters. subsequently, text chunks are vectorized using the bge embedding model @xcite . we utilize the faiss library @xcite to retrieve the vectors of the top-3 relevant text chunks based on similarity. these chunks are input to the llm with the corresponding questions for answering through the rag-chain from langchain @xcite . fair comparison. to maintain consistency and control variables, all models are trained or used under the same settings on four different corpora. detailed training parameters and gpu costs are available in appendix c. ## results in the following subsections, we will discuss three research questions regarding our study. since dsft and rag systems utilize domain corpora in different ways, we will discuss them separately in this section. for the dsft paradigm. inspired by the findings for the rag paradigm. under the same llm reader setup, retrieval accuracy in this semantic space crucially impacts rag performance @xcite . the retrieval process involves selecting the vectorized chunks with the highest similarity scores to the query vector. to investigate the impact of different methods on retrieval effectiveness, we use t-sne (van der @xcite to visualize the clustering of a query and related chunks in the semantic space at figure 5 . it could be clearly seen that chunks generated by the llm-based and markdown methods, which perform well in through the analysis of rq1 and rq2, we know that the llm-based strategy with chatgpt is outstanding and reliable in both frameworks. in case markdown chunks llm-basd chunks template chunks tplm-based chunks query figure 5: a t-sne visualization of chunk clusters in the embedding space of the rag system. 'x chunks' represents chunks related to the query (red star) from the corpus generated by x table-to-text method. ## conclusion this paper studies the impact of different tableto-text methods on llm-based qa systems enhanced by domain hybrid data. specifically, we meticulously compared four representative methods: markdown formatting, template serialization, tplm-based, and llm-based approaches. through experiments, we show the superiority of the llm-based and tplm-based methods in the dsft framework, and the excellence of the llmbased and markdown methods in the rag framework. a key discovery is the varying frequency of domain-specific terms and verbs produced by these methods, alongside the differing quality of semantic representations in the generated text chunks, which appear to be pivotal factors influencing performance disparities across the two systems. these insights not only shed light on the nuances of tableto-text generation methods but also have profound implications for the enhancement of llms. furthermore, they offer practical guidance for tailoring domain-specific qa systems to meet particular needs. hejie cui, zhang xuchao, tianjiao @xcite . domain specialization as the key to make large language models disruptive: a comprehensive survey. arxiv preprint arxiv:2305.18703. yilun zhao, haowei zhang, shengyun si, linyong nan, xiangru tang, and arman cohan. 2023b. investigating table-to-text generation capabilities of large language models in real-world information seeking scenarios. in proceedings of the 2023 conference on empirical methods in natural language processing: industry track, pages 160-175, singapore. association for computational linguistics. wanjun zhong, junjie huang, qian liu, ming zhou, jiahai wang, jian yin, and nan duan. 2022. reasoning over hybrid chain for table-and-text open domain question answering. in international joint conference on artificial intelligence (ijcai), pages 4531-4537. fengbin zhu, wenqiang lei, youcheng huang, chao wang, shuo zhang, jiancheng lv, fuli feng, and tat-seng chua. 2021. tat-qa: a question answering benchmark on a hybrid of tabular and textual content in finance. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 3277-3287.