Sattiy at S em E val-2021 Task 9: An Ensemble Solution for Statement Verification and Evidence Finding with Tables (2021)

Abstract: Question answering from semi-structured tables can be seen as a semantic parsing task and is significant and practical for pushing the boundary of natural language understanding. Existing research mainly focuses on understanding contents from unstructured evidence, e.g., news, natural language sentences and documents. The task of verification from structured evidence, such as tables, charts, and databases, is still less-explored. This paper describes sattiy team’s system in SemEval-2021 task 9: Statement Verification and Evidence Finding with Tables (SEM-TAB-FACT)(CITATION). This competition aims to verify statements and to find evidence from tables for scientific articles and to promote proper interpretation of the surrounding article. In this paper we exploited ensemble models of pre-trained language models over tables, TaPas and TaBERT, for Task A and adjust the result based on some rules extracted for Task B. Finally, in the leadboard, we attain the F1 scores of 0.8496 and 0.7732 in Task A for the 2-way and 3-way evaluation, respectively, and the F1 score of 0.4856 in Task B.

## introduction semantic parsing is one of the most important tasks in natural language processing. it not only needs to understand the meaning of natural language statements, but also needs to map them to meaningful executable queries, such as logical forms, sql queries, and python code @xcite @xcite . question answering from semi-structured tables is usually seen as a semantic parsing task @xcite , where questions are translated into logical forms that can be executed against the table to retrieve the correct denotation @xcite . practically, it is significant in natural language understanding to verify whether a textual hypoth-esis is entailed or refuted by evidence @xcite . the verification problem has been extensively studied in different natural language tasks, such as natural language inference (nli) @xcite , claim verification @xcite , recognizing of textual entailment (rte) @xcite , and multi-model language reasoning (nlvr/nlvr2) @xcite . however, existing research mainly focuses on verifying hypothesis from unstructured evidence, e.g., news, natural language sentences and documents. research of verification under structured evidence, such as tables, charts, and databases, is still in the exploratory stage. this year, semeval-2021 task 9: statement verification and evidence finding with tables (sem-tab-fact), aims to verify statements and find evidence from tables in scientific articles @xcite . it is an important task targeting at promoting proper interpretation of the surrounding article. the competition tries to explore table understanding from two tasks: -task a -table statement support: the task aims to determine whether a statement is fully supported, refuted, or unknown to a given table. -task b -relevant cell selection: given a statement and a table, the task is to determine which cells in the table provide evidence for supporting or refuting the statement. the competition contains the following challenges: -in task a, there is no training data for the "unknown" category and the number of tables is small in the training set. -the lexical expression of the table may be different from that in the statement. -the table structure is complex and diverse. a table may contain missing values while the the supporting evidence may be resided in cells across several rows or columns. -it is difficult to understand the statements. for example, some statements express totally different semantics meaning with only one different word. this difficulty makes it even harder to find the evidence cells from tables. to overcome these challenges, we incorporate several key technologies in our implementation: -developing a systematic way to generate data from the "unknown" category; -including additional data corpus to enrich the training data; -exploiting existing state-of-the-art pre-trained language models over tables, tabert @xcite and tapas @xcite , and ensembling them into a powerful one; -aligning contents in tables and statements while constructing manual rules for tackling task b. the test shows that our implementation can increase the performance according and finally, in the leadboard, we attain the f1 scores of 0.8496 and 0.7732 in task a for the 2-way and 3-way evaluation, respectively, and the f1 score of 0.4856 in task b. the rest of this paper is organized as follows: in sec. 2, we briefly depict related work to our implementation. in sec. 3, we detail our proposed system. in sec. 4, we present the experimental setup and analyze the results. finally, we conclude our work in sec. 5. ## related work recently, pre-trained language models (plms), e.g., bert @xcite , xlnet @xcite , and roberta @xcite , have witnessed the burgeoning of promoting various downstream nlp tasks, such as reading comprehension, named entity recognition and text classification @xcite . however, the current pretrained language models are basically trained on the general text. they are not fit for some tasks, e.g., text-to-sql, table -to-text, which need to encode the structured data, because the data in the structured table also needs to be encoded at the same time. directly applying the existing plms may face the problem of inconsistency between the encoded text from the table and the pretrained text. tabert @xcite is a newly proposed pretrained model built on top of bert and jointly learns contextual representations for utterances and the structured schema of database (db) tables. this model views the verification task completely as an nli problem by linearizing a table as a premise sentence and applies plms to encode both the table and statements into distributed representation for classification. this model excels at linguistic reasoning like paraphrasing and inference but lacks symbolic reasoning skills. intuitively, encoding more table contents, e.g., type information and content snapshots, relevant to the input utterance could potentially help answer questions that involve reasoning over information across multiple rows in the table because they can provide more hints about the meaning of a column. tapas @xcite is another newly proposed pretrained question answering model over tables implemented on bert to avoid generating the logical forms. the model can fine-tune on semantic parsing datasets, only using weak supervision, with an end-to-end differentiable recipe. another stream of work on evidence finding with table is the rule-based approaches. most evidence cells can be extracted by rules. for example, if a row head or column head appears in the statement, we infer this row or col support this statement. although rule-based approaches suffer from the low recall issue, they exhibit high precision and can be applied to adjust the result for ensemble. ## system overview we elaborate the task and present our system in the following. ## experiments in the following, we present the strong baselines and the results with analysis. we have tried different combinations of tabert and tapas pre-trained models and choose the following 6 best baselines: 1) tabert 1: the pre-trained tabert with @xmath0; 2) tabert 3: the pre-trained tabert with @xmath1; 3) tapas tfimlr: the pre-trained large tapas downloaded from tapas tabfact inter masklm large reset.zip; 4) tapas wsimlr: the pre-trained large tapas downloaded from tapas wikisql sqa inter masklm large reset.zip; 5) tapas imlr: the pre-trained large tapas downloaded from tapas inter masklm large reset.zip; 6) tapas wsmlr: the pre-trained large tapas downloaded from tapas wikisql sqa masklm large reset.zip. our proposed system is funetuned on the above models for the original training data, the original data with the tabfact data, and the augmentation data. we also tune the hyper parameters to fit a better result in the local test dataset. in sum, results in ## conclusion in this paper, we present the implementation of our ensemble system to solve the problem of semeval 2021 task 9. to include more training data and resolve the issue of lacking data from the "unknown" category in the training set, we include external corpus, the tabfact dataset, and specially construct the augmented data for the "unknown" category. content snapshot is also applied to reduce the encoding effort. six pre-trained language models over tables are funetuned on the tabfact dataset and the augmented data with content snapshot tables to evaluate the corresponding performance. an ensemble mechanism is applied to get the final result. moreover, data alignment and manual rule determination are applied to solve task b. finally, our system attains the f1 score of 0.8496 and 0.7732 in task a for 2-way and 3-way evaluation, respectively, while getting the f1 score of 0.4856 in task b.