Tab- CQA : A Tabular Conversational Question Answering Dataset on Financial Reports (2023)

Abstract: Existing conversational question answering (CQA) datasets have been usually constructed from unstructured texts in English. In this paper, we propose Tab-CQA, a tabular CQA dataset created from Chinese financial reports that are extracted from listed companies in a wide range of different sectors in the past 30 years. From these reports, we select 2,463 tables, and manually generate 2,463 conversations with 35,494 QA pairs. Additionally, we select 4,578 tables, from which 4,578 conversations with 73,595 QA pairs are automatically created via a template-based method. With the manually- and automatically-generated conversations, Tab-CQA contains answerable and unanswerable questions. For the answerable questions, we further diversify them to cover a wide range of skills, e.g., table retrieval, fact checking, numerical reasoning, so as to accommodate real-world scenarios. We further propose two different tabular CQA models, a text-based model and an operation-based model, and evaluate them on Tab-CQA. Experiment results show that Tab-CQA is a very challenging dataset, where a huge performance gap exists between human and neural models. We will publicly release Tab-CQA as a benchmark testbed to promote further research on Chinese tabular CQA.

## introduction conversational question answering (cqa) extends traditional question answering to a conversational scenario where questions and answers are usually related to conversation history. recent years have witnessed an upsurge of interest in both dataset building for cqa and models/approaches to cqa. previous cqa datasets have been constructed either for text span extracting tasks @xcite @xcite or sql-style table search tasks @xcite . all these datasets are in english. public cqa datasets in other languages are either rare or not available at all, which is of course not desirable for an inclusive cqa study across different classes of languages @xcite . in addition to the language dimension in diversifying cqa tasks and datasets, data source is yet another important factor. existing cqa datasets are usually constructed from unstructured texts. structured or semi-structured data, like tables, are also important sources for information gathering. furthermore, structured tables exhibit reasoning skills (e.g., more numeric reasoning) with a different distribution from those on unstructured texts. inspired by the aforementioned diversity in both languages and data sources for cqa, in this paper, we propose tab-cqa, a large-scale tabular conversational question answering dataset built from chinese financial reports. it contains 2,463 manuallygenerated conversations and 4,578 automaticallygenerated conversations, with a total of 109,089 question-answer pairs. each dialogue consists of multiple rounds of questions and answers, which are either manually created by crowdsourced workers playing roles of students and teachers or automatically created by a template-based method. for manually-created dialogues, the student reads a partially masked table and asks a series of questions. we train students to ask questions as naturally as possible, preserving the characteristics of natural conversations, such as ellipsis and coreference in dialogue. the teacher provides answers to the questions from the student by carefully checking the given completely visible table. in addition to question-answer pair collection in a conversational way, we also provide annotations to manually created dialogues. the teacher annotates an answer type for each provided answer. based on pre-annotation and analysis on financial tables, we divide answer types into three categories: table retrieval, fact checking and computation. in addition to the answer type annotation, the teacher also needs to provide conversation flow tags to control the flow of conversation, i.e., good, ok, unallowable. a tabular conversation question answering example from tab-cqa is in the appendix a. all questions in tab-cqa require reasoning across the given table and dialogue history. even for table retrieval questions, they are created in the way that is more difficult than just span extraction. the dataset also contains unanswerable questions. in addition to manually created qa pairs, we automatically generate 73,595 qa pairs based on predefined templates over tables. unlike manually labeled qa pairs, automatically-generated pairs are tagged with special labels. we then propose two different cqa models: a text-based model and an operation-based model. the text-based model is to convert the table into a passage by a multi-type network for different types of answers in tab-cqa. the operation-based model converts the table into triplets to facilitate numeric reasoning. the contributions of the work are as follows. ## related work tab-cqa is related to datasets for text-based conversational question answering and tabular question answering. it is also partially related to datasets on numerical reasoning and machine reading comprehension in chinese. the comparison of tab-cqa to other related datasets is shown in text-based conversational question answering datasets. reddy et al. (2019) propose a cqa dataset coqa. it contains 8k conversations with 127k question-answer pairs. the dataset selects passages from several domains, such as children's stories, news, and science. answers of coqa are mostly a short fragment or an entity. passages are visible to both questioners and responders in coqa. choi et al. (2018) present a cqa dataset quac that focuses mainly on information seeking. it contains 14k conversations on passages selected from wikipedia. unlike coqa, only the responder can see complete passages in quac while questioner can only see the titles of passages. due to this setting, quac may contain unanswerable questions. hence, annotators not only annotate the answerability of questions, but also provide dialogue actions to control the flow of dialogue. paritially inspired by this, we present only header rows/columns of extracted tables to the student in building our dataset. campos et al. (2020) build a domain-specific cqa dataset doqa. the dataset contains 2.4k conversations, with 10.9k question-answer pairs. doqa also includes question-answer pairs in the information retrieval scenarios. all these cqa datasets are different from tab-cqa in that they create qa-style conversations on unstructured texts written in english. tabular and database-based question answering datasets. database queries are often relatively complex. hence, @xcite propose a sql-style cqa dataset sqa to decompose complex queries into several simple questions, so that there is a contextual relationship between them. their dataset uses wikipedia tables to create 6k sequences of questions, where complex questions are decomposed into several simple questions. yu et al. (2019a) create a cross-domain corpus based on a conversational query system cosql. it collects 3k+ conversations from 200 databases covering 138 domains, containing 30k+ rounds of conversations and 10k+ annotated sql queries. cosql is significantly different from ours in that it collects sql-style queries rather than natural language questions. chen et al. (2020c) build a hybrid text-and tablebased qa dataset hybridqa. each question is aligned to a structured wikipedia table and entities in the table are linked to free texts. the dataset contains 70k question-answer pairs and 13k tables, each of which is associated with an average of 44 paragraphs. chen et al. (2020a) present an open domain qa dataset ott-qa built on the base of hybridqa. they re-annotate 45k questions, which require multi-step reasoning, aggregating information from tables and texts. @xcite and @xcite propose hybrid qa datasets, also focusing on answering questions over financial data. the two datasets contain 16,552 and 8,281 question-answer pairs, respectively. despite the similarity to tab-cqa in financial qa, the two datasets are in english and not in a conversational format. yet another dataset related to tab-cqa is tab-fact @xcite , which is not a qa dataset. the dataset focuses on table-based fact detection. inferences are regarded positive if they match corresponding table descriptions. all the above datasets are in english and questions/queries in these datasets are either sql-style or uncontextually linked. datasets on numerical reasoning. @xcite propose a qa dataset drop with numerical inference-type questions. as numbers provide important supporting information for financial statements, we create qa pairs involving numerical reasoning. ## dataset creation this section elaborates how tab-cqa is created, including details on table extraction, conversation collection and annotation. ## models for tabular cqa we propose two different models, namely textbased and operation-based model, as shown in figure 1. for the text-based model, we convert each table into a piece of text, so the task is converted into a reading comprehension form. however, in practice this approach is difficult to effectively model numerical reasoning that is pervasive in our dataset. therefore, we further propose an operation-based neural model that represent a table as a series of triplets. ## conclusions in this paper, we have presented tab-cqa, a tabular conversational question answering dataset built from tables randomly extracted from annual financial reports of chinese listed companies over the past three decades in 18 industry sectors. the dataset contains 7,041 tables, of which 2,463 tables are equipped with a manually collected conversation generated by crowdsourced workers playing the roles of students and teachers, another 4,578 tables are automatically generated according to templates. we have collected 109,089 qa pairs , covering table retrieval, fact checking and computation, 47.7% of which are associated with coreference or ellipsis. we propose two models for tab-cqa, and the experimental results indicate that the operationbased model is better than the text-based model.