S em E val-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents ( SEM - TAB - FACTS ) (2021)

Abstract: Understanding tables is an important and relevant task that involves understanding table structure as well as being able to compare and contrast information within cells. In this paper, we address this challenge by presenting a new dataset and tasks that addresses this goal in a shared task in SemEval 2020 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS). Our dataset contains 981 manually-generated tables and an auto-generated dataset of 1980 tables providing over 180K statement and over 16M evidence annotations. SEM-TAB-FACTS featured two sub-tasks. In sub-task A, the goal was to determine if a statement is supported, refuted or unknown in relation to a table. In sub-task B, the focus was on identifying the specific cells of a table that provide evidence for the statement. 69 teams signed up to participate in the task with 19 successful submissions to subtask A and 12 successful submissions to subtask B. We present our results and main findings from the competition.

## introduction tables are ubiquitous in documents and presentations for conveying important information in a concise manner. this is true in many domains, stretching from scientific to government documents. in fact, surrounding text in these articles are often statements summarizing or highlighting some information derived from the primary source of data in tables. a relevant example is shown in figure 1 from a business insider article analyzing the impact of @xcite . describing all the information provided in this table in a readable manner would be lengthy and considerably more difficult to understand. despite their importance, popular question answering (e.g. squad and natural question @xcite figure 1 : surrounding text often highlights some information from the table but does not capture all data. alternately, the linked text may be subjective or even misleading without the original table to check the claims. 2016; @xcite ) and truth verification tasks (e.g. semeval-2019 fact checking task @xcite ) have not focused on tables, being composed solely of written text. this is likely due to their complexity to parse and understand, despite their rich amount of information. further, the structure of tables allows much more information to be presented in an efficient manner as humans can interpret meaning in the spatial relationship between cells. however, due to their challenging nature, recent algorithms have been less successful at extracting @xcite and understanding header and data structure in tables @xcite . in addition, any hierarchical and nested headers (common in printed documents) increases the difficulty in interpreting data cells, as shown in figure 2 . in this paper, we propose to bridge this gap with statement verification and evidence finding using tables from scientific articles. this important task promotes proper interpretation of the surrounding article. in fact, the misunderstanding of tables can lead to the reporting of fake news that we see as being all too prevalent today. figure 2 : a complex table sourced from @xcite with hierarchical column and row structure. additional difficulty follows from row hierarchy not being delineated by separate columns. we present the first semeval challenge to address table understanding. we introduce a brand new dataset of 1980 tables from scientific articles that addresses two challenging tasks important to table understanding: a: statement fact verification given a statement, determine whether it is supported, refuted or unknown according to the table. ## dataset details our dataset consists of two forms of generation: (1) a crowdsourced dataset, and (2) an auto-generated dataset. ## experiments we present our baseline experimental setup for each task below. task a we employ state-of-the-art table-bert implementation 7 as proposed by wenhu chen and wang (2020). we utilize table-bert's best performing configuration (table-bert-horizontal-t+f-template) as (1) using entity-linking to find the relevant columns for a statement, (2) flattening the table by scanning horizontally to form natural statements from the relevant columns and their cell values and (3) classifying the flattened table and the statement using the sentence pair classification input template evidence example statements col i, col j 'the' + col i head + 'is' + col i val + ', when the' + col j head + 'is' + col j val col i head, col j head, col i val, col j val the code is ag3 when the locality is los aguanances3. col col val + 'is in' + col head col val, col head for entailed; col for refuted ag3 is in code. col unique or same values col for entailed; none for refuted sup./inf. has the same values. col[#] 'the maximum of' + col head +'is'+val col[#] for entailed; none for refuted the maximum of length(mm) is 2.22. col[#] 'the minimum of' + col head +'is'+val col[#] for entailed; none for refuted the minimum of length(mm) is 1.54. col[#] 'the mean of' + col head + 'is' + val col[#] the mean of length(mm) is 1.83. col[#] 'the median of' + col head + 'is' + val col[#] the median of length(mm) is 1.73. col[#] 'the mode of' + col head + 'is' + val col[#] the mode of length(mm) is 1.54, 1.73, 2.22. ## competition results we present two leaderboards for each task foot_7 . the official leaderboard is from participants who have given us detailed descriptions on their system and affirmed that they did not incorporate any information from the test set that changed their final model. this is a more accurate representation of system quality. the unverified leaderboard is composed of participants who either did not give enough detail or have affirmed that they incorporated some test data information in their final model. the participants did not have access to labels for test data but some teams altered their models upon examining the input data in the test set. although we discouraged this approach, we present the results in hopes it can give some interesting information about how much improvement might be possible with having access to input test data. 19 teams participated in task a. of the 14 teams on the official leaderboard, king001 obtained the highest score for task a for both the 2-way (88.74) and 3-way (84.48) f-scores. however, the top three participants have comparable scores. all teams except for the last two beat our best baseline in task b is a much harder task and fewer teams participated in this challenge. of the 12 teams that participated, 8 are in the official leaderboard. the best score is 65.17 by breaking-bert@iitk(65.17) which is noticeably lower than the f-scores in task a. similarly to task a the results in the unverified leaderboard are considerably higher. the full leaderboard results for all participants are in we summarize the system details for all participating teams in tables 7 (task a) and 8 (task b). in general, deep learning was the most popular approach used by the participants e.g. bil- stm with attention, bert @xcite etc. most of the participants used transformer-based models to train their systems with flavors ranging from general-domain bert @xcite to table-understanding specific versions like tapas @xcite , tabert @xcite and @xcite . one third of the participants employed some form of ensembling technique in their submission. most of the participants have used the manually generated ground-truth in the development of their systems, with only one team not finding it useful. further, a large percentage of participants have used the auto-generated ground truth in their systems with three teams not finding it helpful in their evaluation. in terms of external resources, a majority of the participants used external table understanding resources in their systems. further, most of the participants employed pre-processing techniques like acronym completion, removing special characters, etc... a substantial percentage of participants used techniques like incorporating word embeddings, entity resolution etc. finally, a large number of participants used tabfact (wenhu chen and wang, 2020) as an external dataset. we also conducted additional analyses on participant submissions on the official leaderboard. we show through the average confusion matrix for task a in breakingbert@iitk @xcite ensemble models with tapas and tablebert transformers in a hierarchical two-step method for 3-way classification (unknown vs not unknown first) beary-group used tapas model with tabfact task, and added unique features. employed prepossessing tricks like k-fold validation and replacing the characters and did hyperparameter tuning. boun (köksal et al., 2021)* used text augmentation techniques such as back translation and synonym swapping on the tapas model. domain adaptation and joint learning using semtabfacts and tabfact datasets. endworld data cleaning. ensemble combining 80 instances of trained tapas-large and label smoothing. fishtoucher motivated by tapas, used bert and enriched the embedding layer with two new token type embeddings: row and column ids* (*the team mistakenly submitted an old model version, see paper for more accurate scores) kaushik acharya (acharya, 2021) parsed statements into candidate logical form; mapped result to handwritten rules, to then execute over relevant cells (identified using string matching and universal dependency parsing) king001 trained 20 instances of tapas, sat and table-bert for an ensemble of 60 models. used preprocessing like acronym completion, rules to align the table content with the question content, label smoothing. magicpai multi-model training using models such as tabert, tapas wikisql, tapas tabfact, tapas masklm. finally rule amendments and aligning the distribution of training and test data ok-team tapas pretrained on tabfact with preprocessing of data (like transforming english numerals to arabic numerals, removing special characters etc.) paima fine-tuned tapas optimized to perform window scanning on statement-related table data. preprocessing to reduce abbreviations for table headers, and identifying operation expressions. ryanstark multi-model tabert pretrained model fusion. pre-processing such as case and abbreviations. sattiy (ruan et al., 2021) ensemble of 6 fine-tuned pre-trained models on the augmented data with content snap-shot input. augmented the data provided by expanding the labels. used fast gradient method and added disturbance to the embedding layer to obtain a more stable word representation and a more general model. skywalker deep learning, lpa rules, tapas dataset sunlp bert for sequence classification, transfer learning tapas (müller et al., 2021) ensemble of tapas (bert-large-like) models: trained with a mask-lm task on wikipedia tables, intermediate pre-training data and tabfact data. hierarchical two-step method for 3-way classification. added neutral statements during training: random and by removing one of the evidence columns. thifly queen @xcite ensemble models in a hierarchical two-step method. 8-model to identify unknown statements and 9-model ensemble to classify entailed/refuted. incorporated different ensemble weights for various statement types (count, superlative, unique). volta @xcite finetuned tapas that was pretrained on tabfact. pre-processing to standardize multiple header rows to a single header. yaoxu added numeric and enumerate features to tapas and also statistic information (such as count) as a new row/column to the table . team description breakingbert @iitk an ensemble of an individual cell-based nli approach and a similarity approach with the cells and statement fishtoucher bert cls tokens for statement and table cells are used to determine cell relationships to each other, and the statement (for relevant cells) kaushik acharya relevant cells are output as part of task a ryanstark bow approach with rules applied based on word matches in header and data cells. volta finetuned tapas for cell selection. different models for entailed and refuted statements. used transfer learning and header standardization. out of the entailed and refuted statements, ones that require numerical reasoning, like range, count or comparisons seemed to be most challenging. the statements with the highest accuracy (> 95%) generally had most words or numbers exactly overlapping with those in the table. in task b, out of the statements with less than 30% evidence f-score, 86% were ones with a refuted relationship. conversely, the statements with greater than 70% fscore, 74% were ones with an entailed relationship. this shows that it is more difficult to find the most direct evidence to prove that a statement is refuted by a table than it is to show the positive evidence that a particular statement is supported by it. we believe this is an interesting line of research for future studies. ## conclusion and future works in this paper, we presented the data and competition results for sem-tab-facts, shared task 9 of semeval 2021. we created a large dataset via automated and crowdsourced fact verification as well as evidence finding for tables. our 19 teams had a variety of techniques to tackle this unique but very relevant problem. the evidence finding scores are still quite low and have a large improvement potential. additionally, the test set may be expanded in future versions of this task with a combination of manually generated, natural, and automated statements.