O pen T 2 T : An Open-Source Toolkit for Table-to-Text Generation (2024)

Abstract: Table data is pervasive in various industries, and its comprehension and manipulation demand significant time and effort for users seeking to extract relevant information. Consequently, an increasing number of studies have been directed towards table-to-text generation tasks. However, most existing methods are benchmarked solely on a limited number of datasets with varying configurations, leading to a lack of unified, standardized, fair, and comprehensive comparison between methods. This paper presents OpenT2T, the first open-source toolkit for table-to-text generation, designed to reproduce existing large language models (LLMs) for performance comparison and expedite the development of new models.We have implemented and compared a wide range of LLMs under zero- and few-shot settings on 9 table-to-text generation datasets, covering data insight generation, table summarization, and free-form table question answering. Additionally, we maintain a public leaderboard to provide insights for future work into how to choose appropriate table-to-text generation systems for real-world scenarios.

## introduction in an era where users interact with vast amounts of structured data every day for decision-making and information-seeking purposes, the need for intuitive, user-friendly interpretations has become paramount @xcite @xcite . given this emerging necessity, table-to-text generation techniques, which transform complex tabular data into comprehensible narratives tailored to users' information needs, have drawn considerable attention @xcite @xcite . these techniques can be incorporated into a broad range of applications, including but not limited to game strategy development, financial analysis, and human resources management. while large language models (llms) have achieved remarkable progress in the areas of controllable text generation and data interpretation @xcite @xcite @xcite , the exploration of these models in table-to-text generation has been limited. additionally, existing table-to-text generation systems @xcite @xcite @xcite are benchmarked on various datasets and configurations. this has led to a lack of standardization, making comprehensive evaluation between different methods challenging. moreover, since these models are developed or evaluated within individual systems, they suffer from compatibility issues. therefore, ## opent2t tasks opent2t covers three kinds of table-to-text generation tasks: data insight generation, table summarization, and free-form table question answering (as shown in ## opent2t framework as shown in figure 1 , opent2t consists of four main modules: configuration, data, modeling, and evaluation. the users are able to test the existing table -to-text models on the included dataset. they are also allowed to add their own models or datasets into opent2t by extending corresponding modules with their proposed ones. ## opent2t leaderboard we maintain a public leaderboard at huggingface space for users to track, rank, and evaluate existing table-to-text generation systems. the detailed results of model performance can be found at https://huggingface.co/spaces/ yale-nlp/opent2t_leaderboard . users can also submit model output for automated evaluation and leaderboard updates. we believe that such a leaderboard can provide future researchers and developers with valuable insights into how to choose and develop appropriate table-to-text generation systems for real-world applications. ## related work text generation from semi-structured knowledge sources, such as web tables, has been studied extensively in recent years @xcite @xcite . however, existing table-to-text methods @xcite @xcite @xcite have been evaluated on different datasets with varying configurations and developed as individual systems, resulting in difficulties in reproducing them for performance comparison in future studies. moreover, existing works typically regard table -to-text generation as a subtask of table reasoning @xcite @xcite @xcite , which focuses primarily on numerical and logical reasoning capabilities. the ## conclusion this work presents opent2t, the first opensource framework for table-to-text generation, aimed at enabling researchers and developers to reproduce and benchmark existing table-to-text generation systems in a standardized and fair manner. opent2t serves as a comprehensive platform that allows users to compare different models on a unified ground, facilitating more transparent and reproducible research in this area. the framework is developed with highly reusable and modular components, making it flexible and extensible for a wide range of use cases. additionally, opent2t provides a suite of pre-built functionalities, including data preprocessing pipelines and evaluation metrics, which streamline the process of testing and evaluating new models. we welcome researchers and engineers to join us in developing, maintaining, and improving opent2t, in order to foster innovation and enable the rapid development of novel table-to-text generation techniques.