CATS : A Pragmatic C hinese Answer-to-Sequence Dataset with Large Scale and High Quality (2023)

Abstract: There are three problems existing in the popular data-to-text datasets. First, the large-scale datasets either contain noise or lack real application scenarios. Second, the datasets close to real applications are relatively small in size. Last, current datasets bias in the English language while leaving other languages underexplored.To alleviate these limitations, in this paper, we present CATS, a pragmatic Chinese answer-to-sequence dataset with large scale and high quality. The dataset aims to generate textual descriptions for the answer in the practical TableQA system. Further, to bridge the structural gap between the input SQL and table and establish better semantic alignments, we propose a Unified Graph Transformation approach to establish a joint encoding space for the two hybrid knowledge resources and convert this task to a graph-to-text problem. The experiment results demonstrate the effectiveness of our proposed method. Further analysis on CATS attests to both the high quality and challenges of the dataset

## introduction data-to-text (d2t) generation @xcite aims to generate a natural language description conditioned on structured or semi-structured data, such as graphs @xcite or tables @xcite . it helps people get the key points of the input data and makes the stored information accessible to a broader range of endusers. a large number of datasets have been proposed as the testbed for neural d2t models and are driving the domain. however, as shown in select , , from order by asc limit 3 select name, field, the maximum age of the winner from nobel prize order by the first award time asc limit 3 the names and fields of the first three nobel prizes awarded at the earliest time and the maximum age of their winners are: 1. nobel prize in chemistry, the field of chemistry, the maximum age of the winner is 77; 2. nobel prize in physics, the field of physics, the maximum age of the winner is 87; 3. nobel prize in physiology or medicine, the field of physiology and medicine, the maximum age of the winner is 77. 3 1 77 2 87 3 77 name field the maximum age of the winner nobel prize in chemistry chemistry 77 nobel prize in physics physics 87 nobel prize in physiology or medicine physiology and medicine 77 the names, fields and max the winners of the three no were awarded the earliest f are 1. chemistry, where th to receive the nobel prize 77; 2. physics, where the m receive the nobel prize in physiology and medicine, maximum age to receive th in physiology or medicine weathergov @xcite ) or lack practical application scenarios, e.g., totto @xcite . the shortcoming leads to a separation between research and application. second, the datasets close to practical scenarios are relatively small in size. for example, rotowire @xcite only contains 4.9k training examples, and cosql @xcite is consist of 7.8k training pairs. the small training size can easily lead to overfitting and is not conducive to training a reliable neural network model. lastly, most of the existing datasets are built for english, which leads to advanced work on d2t generation primarily focusing on english and leaving other languages underexplored. these limitations hinder the progress of d2t generation. we therefore need to investigate possible remedies. the crucial step to improving the above limitations is digging out a data-to-text task with a practical scenario. recently, cosql @xcite has proposed a practical controlled d2t task: answer-to-sequence. as shown in figure 1 , the task takes a sql query generated by a semantic parsing module, i.e., text-to-sql @xcite , and its corresponding execution result (in the form of a table) as the model input and aims to produce a natural language description as the response to users in a real-world tableqa system. the sql gives explicit signals for models on what to generate. the generated description could provide a concise and easy-to-understand summary of the result table and help users verify whether the queried result is consistent with the original question @xcite . moreover, the task also contributes to a more user-friendly humancomputer interaction. nevertheless, cosql contains only 7.8k answer-to-sequence examples for training. additionally, it is a dataset with sqlgrounded dialogue state tracking as the core, and the generation annotations are very rough. the scale and quality of cosql limit further exploring the answer-to-sequence task. in this paper, to bridge the gap between research and application of data-to-text datasets and enrich their language diversity, we comply with the cosql setting and present cats, a large-scale and high-quality chinese answer-to-sequence dataset. we manually annotate all collected sql-table pairs to obtain their descriptions. we make two efforts to improve the quality and scale of the collected sql-table pairs and guarantee they are close to practical scenarios. first, we annotate the sql-table pairs from dusql @xcite , a large-scale chinese text-to-sql dataset with a sql query distribution close to real applications. data collected in this way are named cats-d. second, we adopt an automatic data construction pipeline to collect a large number of sql-table pairs for annotation. the basic idea is automatically crawling a mount of tables from the internet to build multi-table databases and then automatically generating sql queries based on the sql grammar and constrained by the given database. data collected with this method are referred to as cats-s. compared to cats-d, cats-s expands the data scale while reducing the share of easy sqls to make the dataset more challenging. in total, cats is made up of both cats-d and cats-s, and contains 43,369 answer-to-sequence examples, which is an order of magnitude larger than cosql. the input sql and table in answer-to-sequence are heterogeneous, and there is a structural gap between them. to bridge the gap and establish better semantic alignments, we propose a unified graph transformation approach (ugt), which first converts the two sources to two undirected graphs, then builds the connection between the nodes in different graphs to obtain a unified graph. in this way, we convert this task to a graph-to-text problem @xcite . previous graph-to-text work @xcite transforms the input graph into a new token graph to apply pretrained language models, such as t5 @xcite . we consider that this transformation breaks the original input graph structure and may bring in extra noises into graph encoding. hence, we further introduce a node segment embedding (nse) to preserve original structure information. our contributions are three-fold as follows: ## dataset construction considering the limitations of existing d2t datasets, we present cats, a massive and pragmatic chinese answer-to-sequence dataset. cats is constructed by two phases: sql-table pairs collection and manual data annotation. to balance the data quality and scale and bring it closer to the practical scenario, we collect the sql-table pairs in two ways. first, we derive sql-table pairs from dusql @xcite , a text-to-sql dataset that generates the sql queries by referring to the sql query distribution in real-life applications. the dataset obtained by annotating these pairs is referred to as cats-d. besides, we implement an automatic data construction pipeline to collect massive high-quality sql-table pairs. data collected with this method are referred to as cats-s, which increases the proportion of complicated sql queries to make the dataset more challenging. ultimately, both cats-d and cats-s make up cats. we first describe how to obtain sql-table pairs for subsequent annotation and then introduce the annotation details. database building to mimic the practical tableqa system, we first follow @xcite to build a multi-table database d d by collecting all databases in dusql. in addition, we also build another multi-table database d s for expanding the size and domain of our dataset through a table collection pipeline. specifically, 100,000 high-frequency words are first summarized from the clue @xcite corpus. then, we query these words in google and download all the queried spreadsheet files. subsequently, the available tables in these spreadsheets are extracted by a table parser that can identify the potential table in a work-sheet. to protect personal privacy, we use predefined unique words to replace sensitive information in these tables, such as passwords, id numbers, credit card numbers, etc. finally, these tables are used to construct the database d s . please refer to appendix a.1 for more details. ## structure-aware approach given an input sql s and a table t, the model aims to generate a response ỹ. to bridge the gap between the two sources of information, we first propose a unified graph transformation approach (ugt), which explicitly connects the input sql and table in a unified structure. in this way, we can obtain a joint graph representation of the two sources and convert the answer-to-sequence task to a graphto-text problem. and then, we utilize a varietal transformer architecture ( ribeiro et al., 2020b) ## conclusion we present cats, a large-scale and high-quality chinese answer-to-sequence dataset, along with a series of baselines. it helps alleviate the problem of current d2t datasets' bias towards the english language. we propose a unified graph transformation method to bridge the structural gap between the sql and table. in this way, we convert the task to a graph-to-text problem. furthermore, we introduce the node segment embedding to solve the problem that transforming the input graph to a new token graph breaks the original graph's structure. experiments on cats show that our proposed model outperforms existing baseline models. we conduct further analysis on cats, which attests to both the high quality and challenges of the dataset.