Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction (2024)

Abstract: The task of condensing large chunks of textual information into concise and structured tables has gained attention recently due to the emergence of Large Language Models (LLMs) and their potential benefit for downstream tasks, such as text summarization and text mining. Previous approaches often generate tables that directly replicate information from the text, limiting their applicability in broader contexts, as text-to-table generation in real-life scenarios necessitates information extraction, reasoning, and integration. However, there is a lack of both datasets and methodologies towards this task. In this paper, we introduce LiveSum, a new benchmark dataset created for generating summary tables of competitions based on real-time commentary texts. We evaluate the performances of state-of-the-art LLMs on this task in both fine-tuning and zero-shot settings, and additionally propose a novel pipeline called T^3 (Text-Tuple-Table) to improve their performances. Extensive experimental results demonstrate that LLMs still struggle with this task even after fine-tuning, while our approach can offer substantial performance gains without explicit training. Further analyses demonstrate that our method exhibits strong generalization abilities, surpassing previous approaches on several other text-to-table datasets. Our codeand data can be found at https://github.com/HKUST-KnowComp/LiveSum.

## introduction reading extensive texts is demanding and timeconsuming for humans, further compounded by the challenge of effectively capturing the key elements. consequently, recent works have shifted to explore the structured summarization of text @xcite , with tables being one highly prevalent form @xcite @xcite . these approaches improve text comprehension by extracting inherent yet valuable structural player5 scores with a shot from close range to the bottom left corner, assisted by player12. player2 from the home team misses a header to the left from the center of the box. ## livesum dataset we consider the problem of generating match statistic information tables from textual live commentary. inspired by rotowire @xcite , a data-to-document dataset in the sports domain that aims to generate textual summaries by incorporating statistical data from basketball games, we instead focus on live commentary in football, which is available on bbc sports 1 . we crawl the data for the english premier league from 2014 to 2023 and obtain complete commentary for 3,771 matches. figure 2 shows the pipeline we use to construct the dataset. section 3.1 describes the generation 1 https://www.bbc.com/sport/football shot foul goal fr ee ki ck re d ca rd c o rn e r k ic k o ff s id e yell ow car d process of the live commentary, and section 3.2 describes the generation process for the summary table. more details are provided in appendix a. ## t 3 (text-tuple-table) pipeline our proposed t 3 (text-tuple-table ) pipeline is designed to mimic the intuitive steps followed by humans when performing this task. when individuals aim to summarize a table from text, they typically extract pertinent or valuable tuples from the content, guided by any provided instructions, and then organize these tuples into one or more tables. based on this concept, we divide this transformation into three stages: text-to-tuple, integration, and tuple-to-table, each of which is discussed in the following subsections. taking figure 1 as an example, we first extract key events mentioned in the text, then aggregate this information into consolidated tuples, and ultimately compile them into a table. ## experimental setup baseline models in this study, we conduct finetuning on the livesum dataset using three representative open-source llms: mistral-7b-instruct-v0.2 @xcite , llama-2 chat 7b and llama-2 chat 13b @xcite . we fine-tune these models following the current state-of-the-art fine-tuning methodologies @xcite . therefore, the outcomes represent the best results achievable with the present fine-tuning methods. we also evaluate eight state-of-the-art llms in zero-shot settings: llama-2 chat 13b, llama-2 chat 70b @xcite , mistral large (mistralai, 2024), claude 2.1 @xcite , claude 3 opus @xcite , chat-gpt (openai, 2022), and gpt-4 (openai, 2024a). because of the limitation of the input lengths, we evaluate two llms in few-shot settings: chatgpt-16k and @xcite . for each model, we conduct tests using two types of prompts. the first type directly describes the task by providing an instruction text y and accompanying it with the text x. the second type uses the chain-of-thought (cot) prompting @xcite , incorporating the phrase "let's think step by step" into the instruction text. see more details in appendix c. figure 4 : the performance of various llms under fine-tune and zero-shot settings, as well as after the application of the t 3 method on the test set of livesum dataset. the average rmse and error rate for each model are displayed, along with the error rate for each of the three difficulty sections. more results are in evaluation metric as the generated cell content in this task consists of numerical values, we utilize commonly employed metrics in regression tasks, namely the root mean square error (rmse). grouping by event difficulty furthermore, we categorize the eight types of events into three groups based on assessed difficulty: goals, due to direct descriptions of scores in the original text, and red cards, due to their rare occurrence are categorized into the easy section. shots and fouls, due to their varied expressions and descriptions, are classified into the hard section. the remaining four event types are classified as medium section. we report the rmse and er for each model across different difficulty categories to provide a more comprehensive analysis. ## experiments and analyses in this section, we will benchmark the performance of current state-of-the-art llms on the livesum dataset, and further evaluate the effectiveness and generalization of our proposed approach. we aim to answer the following research questions: rq1 (benchmarking) how do the current stateof-the-art llms perform on this dataset in finetuning and zero-shot settings? ## related work text-to-table generation many studies have been proposed to perform text-to-table generation, converting it into sequence-to-sequence problems @xcite , or framing them as question-answering problems @xcite . with the rise of llms, some research has also explored evaluating llms under finetuning or zero-shot settings, and it shows that finetuning yields highly effective results @xcite . however, these methods employ datasets that only require the model to extract relevant information from text and populate tables, which significantly limits the scope of this task. therefore, we introduce a new challenging dataset and propose a universal solution that greatly enhances the performance of llms under zero-shot setting.