TH i F ly_ Q ueens at S em E val-2021 Task 9: Two-stage Statement Verification with Adaptive Ensembling and Slot-based Operation (2021)

Abstract: This paper describes our system for verifying statements with tables at SemEval-2021 Task 9. We developed a two-stage verifying system based on the latest table-based pre-trained model GraPPa. Multiple networks are devised to verify different types of statements in the competition dataset and an adaptive model ensembling technique is applied to ensemble models in both stages. A statement-slot-based symbolic operation module is also used in our system to further improve the performance and stability of the system. Our model achieves second place in the 3-way classification and fourth place in the 2-way classification evaluation. Several ablation experiments show the effectiveness of different modules proposed in this paper.

## introduction verifying whether a statement is entailed, refuted, or unknown with the given table is a challenging task, which requires the system to understand the statement and table jointly and reasons from the two information resources. the studies on this task could benefit several downstream applications, e.g. fake news detection. recently, with the release of a large-scale dataset for table-based fact verification named tabfact @xcite , this task received several studies. verifying statements based on tables is a challenging task since the researches on understanding tables are not enough compared with works on free-texts, and the methods to train models understanding free-text and table jointly need further studies as well. while tabfact contains a huge number of statements and tables, the tables in the tabfact dataset are relatively simple since they do not have hierarchical column heads as tables in scientific papers do and the contents in the tables are easier to understand compared to the tables in scientific papers as well. in the semeval task 9 @xcite , the goal is to develop a system that can verify statements (subtask a) and find evidence (subtask b) based on the tables extracted from scientific tables. our team is more interested in the verifying task and only participated in subtask a. different from data in tabfact, in subtask a, we are also required to classify a new type of statement that cannot be entailed or refuted based on the given table, named "unknown" type. since no statements of this type are given in the training data, the classification of this type of statement becomes a core difficulty of the subtask. this paper describes our two-stage table-based verifying system based on the latest table-based pretrained language model grappa @xcite . the system leverages the model ensembling technique to ensemble different verifying models which are designed to solve different types of statements in the dataset in both two stages of the system. the statement-slot technique is also used to capture and solve a small part of the data by symbol calculation, which helps to increase the performance and stability of our system. our system achieves a two-way score of 84.55 and a three-way score of 83.76 in subtask a respectively. ## system description this section mainly describes the details of the two-stage table-based verifying system. the first stage classifies unknown type statements from the other two types of statements, while the second stage further classifies the entailed type statements from refuted type statements. we find that the proposed two-stage system works better than a direct three-way classification system, because 1) no unknown type statements are provided in the train set, which brings difficulty to the training process of the three-way classification system; 2) the two-stage system is closer to the processing procedure of human since you have to decide whether the table knowledge is enough to entail or refute the given statement before the further reasoning. before the two-stage deep-learning-based system, a symbolic-calculation module is added to capture and process some statements with simple sentence structures. the symbolic-calculation module has the features of low recall and high precision and is used to process some numerical type statements (the verifying process involves numerical operations) since we find that the deep-learning-based system is unstable when processing such type of statements. the flowchart of the whole system is displayed in figure 1 . ## evaluation to evaluate our proposed verifying system, we perform two ablation studies regarding the adaptive ensembling method and the statement-slot-based symbolic-calculation module used in our system on both the development set and test set. besides the ablation experiment on ensembling, we also perform an ablation experiment on the statement-slot-based symbolic-calculation module. the result of the experiment shows that the pro-posed symbolic-calculation module increases the performance on both the development set and test set by around 5 percent, which shows the benefit of the symbolic-calculation module on verifying numerical type statements. ## conclusion this paper describes our two-stage verifying system developed for semeval-2021 task 9, which leverages the latest table-based pre-trained model grappa. the two-stage verifying structure allows us to develop more targeted models on both stages of the system. multiple reasoning networks are applied behind the grappa model, and an adaptive model ensembling technique is used in both stages of the system. a statement-slot-based symboliccalculation module is also added at the top of the whole system to further improves the performance and stability of the system. ablation experiments show the effectiveness of the methods proposed in the paper.